2024-09-13T22:12:28.751 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T22:12:28.767 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T22:12:28.869 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289 branch: squid description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7904289' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 workunit: branch: squid sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 3819 sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 sleep_before_teardown: 0 subset: 43/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 targets: smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJCVbF4vkJPzt2Tr8FNqwfTN7rAC4mX8MpIqLTYXUmbsr6K8yfICctWh6yXynK7Swat0SSTp6ylFs0kc02sk+E= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDIL7QteOgmhgb+9Sq2GH67PYjPXYs/WFt1a4rxsFeGyHKfAbXwpiZXJ/kyQggcZX2Xq4S9TBmNOw7ybRI8joN4= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T22:12:28.869 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa; will attempt to use it 2024-09-13T22:12:28.870 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ecab7be58963a5bd03910cd73eef66405d7fdd50/qa/tasks 2024-09-13T22:12:28.871 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T22:12:28.873 INFO:teuthology.task.internal:Checking packages... 2024-09-13T22:12:28.894 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ecab7be58963a5bd03910cd73eef66405d7fdd50' 2024-09-13T22:12:28.894 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T22:12:28.894 INFO:teuthology.packaging:ref: None 2024-09-13T22:12:28.894 INFO:teuthology.packaging:tag: None 2024-09-13T22:12:28.894 INFO:teuthology.packaging:branch: squid 2024-09-13T22:12:28.894 INFO:teuthology.packaging:sha1: ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:12:28.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-13T22:12:29.125 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-324-gecab7be5-1jammy 2024-09-13T22:12:29.126 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T22:12:29.132 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T22:12:29.132 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T22:12:29.138 INFO:teuthology.task.internal:Saving configuration 2024-09-13T22:12:29.151 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T22:12:29.157 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T22:12:29.190 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 22:06:29.728900', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMpZzh73HlMcsQtIJXvVlbRMCAR7XWv05UaBuhKtmbjy'} 2024-09-13T22:12:29.223 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 22:06:29.726916', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDkA4bMh9BFq/w10eD06lU8oeWKm2RoOF7jo2DibxehI'} 2024-09-13T22:12:29.223 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T22:12:29.230 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-13T22:12:29.230 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-13T22:12:29.230 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T22:12:29.302 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f39a48cf1c0>, signals=[15]) 2024-09-13T22:12:29.302 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T22:12:29.308 INFO:teuthology.task.internal:Opening connections... 2024-09-13T22:12:29.308 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-09-13T22:12:29.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:12:29.397 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-09-13T22:12:29.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:12:29.484 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T22:12:29.490 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-09-13T22:12:29.496 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-09-13T22:12:29.496 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="Ubuntu" 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="22.04" 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_CODENAME=jammy 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:ID=ubuntu 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE=debian 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T22:12:29.544 INFO:teuthology.orchestra.run.smithi178.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T22:12:29.545 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T22:12:29.545 INFO:teuthology.orchestra.run.smithi178.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T22:12:29.545 INFO:teuthology.orchestra.run.smithi178.stdout:UBUNTU_CODENAME=jammy 2024-09-13T22:12:29.545 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-09-13T22:12:29.578 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-09-13T22:12:29.583 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-09-13T22:12:29.583 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-09-13T22:12:29.631 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Ubuntu" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="22.04" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_CODENAME=jammy 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:ID=ubuntu 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE=debian 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T22:12:29.632 INFO:teuthology.orchestra.run.smithi192.stdout:UBUNTU_CODENAME=jammy 2024-09-13T22:12:29.633 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-09-13T22:12:29.661 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T22:12:29.669 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T22:12:29.674 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T22:12:29.674 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-09-13T22:12:29.676 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-09-13T22:12:29.681 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T22:12:29.686 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T22:12:29.686 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-09-13T22:12:29.721 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-09-13T22:12:29.737 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T22:12:29.820 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T22:12:29.834 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T22:12:29.834 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T22:12:29.834 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T22:12:29.835 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T22:12:29.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:12:29.839 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-09-13T22:12:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:12:29.840 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-09-13T22:12:29.884 INFO:teuthology.orchestra.run.smithi178.stdout:5.15.0-56-generic 2024-09-13T22:12:29.884 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.15.0-56-generic 2024-09-13T22:12:29.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2024-09-13T22:12:29.889 INFO:teuthology.orchestra.run.smithi192.stdout:5.15.0-56-generic 2024-09-13T22:12:29.889 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.15.0-56-generic 2024-09-13T22:12:29.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2024-09-13T22:12:30.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-09-13T22:12:30.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2024-09-13T22:12:30.328 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T22:12:30.330 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T22:12:30.347 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T22:12:30.347 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T22:12:30.395 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T22:12:30.408 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T22:12:30.750 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T22:12:30.765 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T22:12:30.776 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T22:12:30.781 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T22:12:30.890 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T22:12:30.904 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T22:12:31.081 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T22:12:31.082 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T22:12:31.109 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T22:12:31.201 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T22:12:31.217 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T22:12:31.226 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T22:12:31.227 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T22:12:31.242 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T22:12:31.254 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T22:12:31.256 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T22:12:31.257 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T22:12:31.258 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T22:12:31.258 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T22:12:31.267 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T22:12:31.268 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T22:12:31.272 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T22:12:31.273 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T22:12:31.323 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T22:12:31.324 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T22:12:31.327 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T22:12:31.335 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T22:12:31.343 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T22:12:31.343 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T22:12:31.360 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T22:12:31.360 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T22:12:31.361 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T22:12:31.369 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T22:12:31.369 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T22:12:31.376 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T22:12:31.381 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T22:12:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T22:12:31.392 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T22:12:31.393 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T22:12:31.393 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T22:12:31.399 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T22:12:31.399 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T22:12:31.400 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T22:12:31.407 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T22:12:31.408 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T22:12:31.411 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T22:12:31.412 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T22:12:31.417 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T22:12:31.417 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T22:12:31.417 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T22:12:31.418 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T22:12:31.418 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T22:12:31.419 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T22:12:31.423 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T22:12:31.424 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T22:12:31.434 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T22:12:31.437 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T22:12:31.437 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T22:12:31.438 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T22:12:31.473 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T22:12:31.474 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T22:12:31.488 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T22:12:31.488 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T22:12:31.489 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T22:12:31.490 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T22:12:31.496 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T22:12:31.497 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T22:12:31.497 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T22:12:31.498 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T22:12:31.499 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T22:12:31.500 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T22:12:31.500 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T22:12:31.504 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T22:12:31.505 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T22:12:31.505 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T22:12:31.570 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T22:12:31.570 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T22:12:31.571 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T22:12:31.571 INFO:teuthology.orchestra.run.smithi178.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T22:12:31.572 INFO:teuthology.orchestra.run.smithi178.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T22:12:31.586 INFO:teuthology.orchestra.run.smithi192.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T22:12:31.586 INFO:teuthology.orchestra.run.smithi192.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T22:12:37.759 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 16.0 MB in 3s (5,839 kB/s) 2024-09-13T22:12:37.762 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 16.0 MB in 3s (5,784 kB/s) 2024-09-13T22:12:39.170 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-09-13T22:12:39.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T22:12:39.252 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-09-13T22:12:39.283 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-09-13T22:12:39.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T22:12:39.406 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-09-13T22:12:39.459 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-09-13T22:12:39.460 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-09-13T22:12:39.620 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2024-09-13T22:12:39.620 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2024-09-13T22:12:39.650 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T22:12:39.650 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T22:12:39.650 INFO:teuthology.orchestra.run.smithi178.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T22:12:39.651 INFO:teuthology.orchestra.run.smithi178.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T22:12:39.651 INFO:teuthology.orchestra.run.smithi178.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T22:12:39.651 INFO:teuthology.orchestra.run.smithi178.stdout: python2.7-minimal 2024-09-13T22:12:39.651 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T22:12:39.653 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2024-09-13T22:12:39.653 INFO:teuthology.orchestra.run.smithi178.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T22:12:39.653 INFO:teuthology.orchestra.run.smithi178.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T22:12:39.653 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T22:12:39.654 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2024-09-13T22:12:39.655 INFO:teuthology.orchestra.run.smithi178.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T22:12:39.655 INFO:teuthology.orchestra.run.smithi178.stdout:Recommended packages: 2024-09-13T22:12:39.655 INFO:teuthology.orchestra.run.smithi178.stdout: thermald 2024-09-13T22:12:39.706 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2024-09-13T22:12:39.706 INFO:teuthology.orchestra.run.smithi178.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T22:12:39.706 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T22:12:39.707 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T22:12:39.708 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages will be upgraded: 2024-09-13T22:12:39.708 INFO:teuthology.orchestra.run.smithi178.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T22:12:39.800 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T22:12:39.800 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T22:12:39.800 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T22:12:39.800 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T22:12:39.801 INFO:teuthology.orchestra.run.smithi192.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T22:12:39.801 INFO:teuthology.orchestra.run.smithi192.stdout: python2.7-minimal 2024-09-13T22:12:39.801 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T22:12:39.802 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2024-09-13T22:12:39.802 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T22:12:39.802 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T22:12:39.802 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T22:12:39.803 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2024-09-13T22:12:39.803 INFO:teuthology.orchestra.run.smithi192.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T22:12:39.804 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2024-09-13T22:12:39.804 INFO:teuthology.orchestra.run.smithi192.stdout: thermald 2024-09-13T22:12:39.848 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2024-09-13T22:12:39.848 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T22:12:39.849 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T22:12:39.849 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T22:12:39.850 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2024-09-13T22:12:39.851 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T22:12:39.889 INFO:teuthology.orchestra.run.smithi178.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T22:12:39.890 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 113 MB of archives. 2024-09-13T22:12:39.890 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T22:12:39.890 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T22:12:39.901 INFO:teuthology.orchestra.run.smithi192.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T22:12:39.901 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 113 MB of archives. 2024-09-13T22:12:39.901 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T22:12:39.901 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T22:12:40.312 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T22:12:40.400 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T22:12:41.200 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T22:12:41.200 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T22:12:41.201 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T22:12:41.327 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T22:12:41.347 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T22:12:41.388 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T22:12:41.739 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T22:12:41.806 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 113 MB in 2s (72.6 MB/s) 2024-09-13T22:12:42.001 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T22:12:43.980 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T22:12:43.980 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T22:12:43.980 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T22:12:44.026 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T22:12:44.032 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:12:44.147 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:12:44.401 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T22:12:44.457 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T22:12:44.842 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 113 MB in 5s (23.9 MB/s) 2024-09-13T22:12:45.026 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T22:12:46.323 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T22:12:46.345 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:12:46.430 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:12:46.951 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T22:12:46.972 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:12:47.009 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:12:47.107 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T22:12:47.111 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:12:47.264 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:12:49.382 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T22:12:49.403 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:12:49.493 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:12:49.935 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T22:12:49.956 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:12:49.993 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:12:54.090 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T22:12:54.217 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T22:12:54.584 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T22:12:54.713 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T22:12:55.007 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T22:12:55.018 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T22:12:55.048 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T22:12:56.775 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T22:12:56.910 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T22:12:57.209 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T22:12:57.330 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T22:12:57.650 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T22:12:57.672 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T22:12:57.717 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T22:13:01.360 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T22:13:01.396 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:13:01.443 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:03.652 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T22:13:03.689 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T22:13:03.727 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:04.247 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T22:13:04.372 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T22:13:04.756 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T22:13:04.861 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:04.979 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T22:13:05.106 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:06.406 INFO:teuthology.orchestra.run.smithi192.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T22:13:06.406 INFO:teuthology.orchestra.run.smithi192.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T22:13:06.406 INFO:teuthology.orchestra.run.smithi192.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T22:13:06.406 INFO:teuthology.orchestra.run.smithi192.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:06.585 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:07.366 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T22:13:07.492 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T22:13:07.842 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T22:13:07.961 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:07.964 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:08.089 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T22:13:08.226 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:09.338 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T22:13:09.447 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T22:13:09.527 INFO:teuthology.orchestra.run.smithi178.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T22:13:09.527 INFO:teuthology.orchestra.run.smithi178.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T22:13:09.528 INFO:teuthology.orchestra.run.smithi178.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T22:13:09.528 INFO:teuthology.orchestra.run.smithi178.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:09.565 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:09.656 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T22:13:09.656 INFO:teuthology.orchestra.run.smithi192.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:09.663 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:11.072 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:12.891 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T22:13:13.017 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T22:13:13.118 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T22:13:13.175 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T22:13:13.175 INFO:teuthology.orchestra.run.smithi178.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:23.696 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T22:13:23.696 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub' 2024-09-13T22:13:23.722 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T22:13:23.747 INFO:teuthology.orchestra.run.smithi192.stdout:Generating grub configuration file ... 2024-09-13T22:13:24.032 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T22:13:24.055 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:24.399 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T22:13:24.405 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T22:13:24.463 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T22:13:24.469 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T22:13:24.636 INFO:teuthology.orchestra.run.smithi192.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T22:13:24.636 INFO:teuthology.orchestra.run.smithi192.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T22:13:24.636 INFO:teuthology.orchestra.run.smithi192.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T22:13:24.676 INFO:teuthology.orchestra.run.smithi192.stdout:done 2024-09-13T22:13:25.215 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 21 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2024-09-13T22:13:25.242 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.15.0.121.121 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2024-09-13T22:13:25.243 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2024-09-13T22:13:25.243 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T22:13:25.243 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T22:13:25.244 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T22:13:25.244 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T22:13:25.244 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T22:13:25.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2024-09-13T22:13:25.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2024-09-13T22:13:25.675 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T22:13:25.676 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T22:13:25.746 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T22:13:26.127 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T22:13:26.311 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 128 kB in 1s (176 kB/s) 2024-09-13T22:13:27.185 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T22:13:27.185 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub' 2024-09-13T22:13:27.195 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T22:13:27.223 INFO:teuthology.orchestra.run.smithi178.stdout:Generating grub configuration file ... 2024-09-13T22:13:27.339 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-09-13T22:13:27.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T22:13:27.421 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-09-13T22:13:27.495 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T22:13:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:27.644 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2024-09-13T22:13:27.644 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2024-09-13T22:13:27.819 INFO:teuthology.orchestra.run.smithi192.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T22:13:27.820 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T22:13:27.820 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T22:13:27.820 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T22:13:27.820 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T22:13:27.821 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T22:13:27.821 INFO:teuthology.orchestra.run.smithi192.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T22:13:27.821 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T22:13:27.833 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T22:13:27.836 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T22:13:27.862 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T22:13:27.864 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 21 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2024-09-13T22:13:27.880 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.15.0.121.121 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2024-09-13T22:13:27.881 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2024-09-13T22:13:27.881 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-09-13T22:13:27.903 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T22:13:27.909 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T22:13:27.927 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.vcImDTw3Ja 2024-09-13T22:13:27.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vcImDTw3Ja 2024-09-13T22:13:27.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.vcImDTw3Ja 2024-09-13T22:13:28.084 INFO:teuthology.orchestra.run.smithi178.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T22:13:28.084 INFO:teuthology.orchestra.run.smithi178.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T22:13:28.085 INFO:teuthology.orchestra.run.smithi178.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T22:13:28.099 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.vcImDTw3Ja is 10KB 2024-09-13T22:13:28.111 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.vcImDTw3Ja 2024-09-13T22:13:28.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T22:13:28.121 INFO:teuthology.orchestra.run.smithi178.stdout:done 2024-09-13T22:13:28.171 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:13:28.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T22:13:28.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T22:13:28.265 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T22:13:28.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo update-grub 2024-09-13T22:13:28.625 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 21 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.15.0.121.121 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T22:13:28.649 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2024-09-13T22:13:28.650 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2024-09-13T22:13:28.650 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2024-09-13T22:13:28.650 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2024-09-13T22:13:28.650 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T22:13:28.650 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T22:13:28.650 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T22:13:28.650 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T22:13:28.650 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T22:13:28.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2024-09-13T22:13:28.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-09-13T22:13:28.896 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T22:13:29.018 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T22:13:29.100 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T22:13:29.181 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T22:13:29.615 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub' 2024-09-13T22:13:29.616 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T22:13:29.619 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-09-13T22:13:29.866 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T22:13:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:30.125 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T22:13:30.130 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T22:13:30.171 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T22:13:30.176 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T22:13:30.296 INFO:teuthology.orchestra.run.smithi192.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T22:13:30.296 INFO:teuthology.orchestra.run.smithi192.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T22:13:30.296 INFO:teuthology.orchestra.run.smithi192.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T22:13:30.306 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-09-13T22:13:30.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-09-13T22:13:30.338 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-09-13T22:13:30.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T22:13:30.421 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-09-13T22:13:30.624 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-09-13T22:13:30.625 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-09-13T22:13:30.791 INFO:teuthology.orchestra.run.smithi178.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T22:13:30.792 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T22:13:30.832 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T22:13:30.834 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic 2024-09-13T22:13:30.849 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic 2024-09-13T22:13:30.849 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2024-09-13T22:13:30.849 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2024-09-13T22:13:30.849 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2024-09-13T22:13:30.849 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 21 2024-09-13T22:13:30.849 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.15.0.121.121 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2024-09-13T22:13:30.850 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2024-09-13T22:13:30.851 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-09-13T22:13:30.899 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.WLG8fHwWta 2024-09-13T22:13:30.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WLG8fHwWta 2024-09-13T22:13:30.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.WLG8fHwWta 2024-09-13T22:13:31.081 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.WLG8fHwWta is 10KB 2024-09-13T22:13:31.094 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.WLG8fHwWta 2024-09-13T22:13:31.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T22:13:31.155 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:13:31.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T22:13:31.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T22:13:31.219 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T22:13:31.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo update-grub 2024-09-13T22:13:32.469 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub' 2024-09-13T22:13:32.470 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T22:13:32.472 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-09-13T22:13:32.641 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T22:13:32.647 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T22:13:32.886 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T22:13:32.892 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T22:13:32.933 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T22:13:32.938 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T22:13:33.057 INFO:teuthology.orchestra.run.smithi178.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T22:13:33.057 INFO:teuthology.orchestra.run.smithi178.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T22:13:33.057 INFO:teuthology.orchestra.run.smithi178.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T22:13:33.084 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-09-13T22:13:33.093 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-09-13T22:14:00.338 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T22:14:00.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:14:00.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:14:03.097 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T22:14:03.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:14:03.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:14:18.685 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-09-13T22:14:21.505 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-09-13T22:14:27.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:14:27.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:14:30.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:14:30.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:14:30.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-09-13T22:14:33.601 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-09-13T22:14:42.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:14:42.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:14:45.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:14:45.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:14:48.701 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-09-13T22:15:03.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:15:03.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:15:14.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-09-13T22:15:29.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:15:29.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:15:29.481 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-09-13T22:15:30.027 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:15:30.027 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-13T22:15:30.028 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-09-13T22:15:30.073 INFO:teuthology.orchestra.run.smithi192.stdout:5.15.0-121-generic 2024-09-13T22:15:30.073 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T22:15:30.073 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T22:15:30.074 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T22:15:31.075 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T22:15:31.075 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T22:15:31.437 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-09-13T22:15:31.482 DEBUG:teuthology.parallel:result is None 2024-09-13T22:15:35.469 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-09-13T22:15:36.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:15:36.048 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-13T22:15:36.049 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-09-13T22:15:36.097 INFO:teuthology.orchestra.run.smithi178.stdout:5.15.0-121-generic 2024-09-13T22:15:36.097 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T22:15:36.097 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T22:15:36.097 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T22:15:37.098 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T22:15:37.098 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T22:15:37.444 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-09-13T22:15:37.475 DEBUG:teuthology.parallel:result is None 2024-09-13T22:15:37.475 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T22:15:37.481 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T22:15:37.481 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T22:15:37.484 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T22:15:37.490 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T22:15:37.495 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T22:15:37.505 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T22:15:37.506 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T22:15:37.530 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T22:15:37.559 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T22:15:37.566 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T22:15:37.566 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T22:15:37.590 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T22:15:37.611 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T22:15:37.611 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T22:15:37.619 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T22:15:37.624 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T22:15:37.625 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T22:15:37.633 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T22:15:37.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T22:15:37.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T22:15:37.679 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T22:15:37.688 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T22:15:37.688 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T22:15:37.718 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T22:15:37.724 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T22:15:37.769 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T22:15:37.821 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:15:37.821 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T22:15:37.903 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T22:15:37.909 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T22:15:37.957 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:15:37.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T22:15:38.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-09-13T22:15:38.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-09-13T22:15:38.103 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T22:15:38.110 INFO:teuthology.task.internal:Starting timer... 2024-09-13T22:15:38.110 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T22:15:38.123 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T22:15:38.130 DEBUG:teuthology.task.selinux:Excluding smithi178: OS 'ubuntu' does not support SELinux 2024-09-13T22:15:38.130 DEBUG:teuthology.task.selinux:Excluding smithi192: OS 'ubuntu' does not support SELinux 2024-09-13T22:15:38.131 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T22:15:38.131 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T22:15:38.131 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T22:15:38.131 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T22:15:38.140 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T22:15:38.150 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T22:15:38.151 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi178.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T22:23:06.888 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi178.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-09-13T22:23:06.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:23:06.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:23:07.004 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-09-13T22:23:07.085 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-13T22:23:07.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:23:07.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T22:23:07.177 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-09-13T22:23:07.261 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-13T22:23:07.261 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T22:23:07.272 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T22:23:07.272 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T22:23:07.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T22:23:07.275 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T22:23:07.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Command line: ntpd -gq 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: ---------------------------------------------------- 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: corporation. Support and training for ntp-4 are 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: available at https://www.nwtime.org/support 2024-09-13T22:23:07.303 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: ---------------------------------------------------- 2024-09-13T22:23:07.304 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: proto: precision = 0.045 usec (-24) 2024-09-13T22:23:07.304 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: basedate set to 2022-02-04 2024-09-13T22:23:07.304 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: gps base set to 2022-02-06 (week 2196) 2024-09-13T22:23:07.305 INFO:teuthology.orchestra.run.smithi178.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T22:23:07.305 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T22:23:07.305 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T22:23:07.305 INFO:teuthology.orchestra.run.smithi178.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T22:23:07.306 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T22:23:07.306 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T22:23:07.306 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T22:23:07.306 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listen normally on 3 enp3s0f1 172.21.15.178:123 2024-09-13T22:23:07.306 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listen normally on 4 lo [::1]:123 2024-09-13T22:23:07.306 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ef9%5]:123 2024-09-13T22:23:07.307 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:07 ntpd[18529]: Listening on routing socket on fd #22 for interface updates 2024-09-13T22:23:07.335 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T22:23:07.335 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Command line: ntpd -gq 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: ---------------------------------------------------- 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: corporation. Support and training for ntp-4 are 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: available at https://www.nwtime.org/support 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: ---------------------------------------------------- 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: proto: precision = 0.048 usec (-24) 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: basedate set to 2022-02-04 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: gps base set to 2022-02-06 (week 2196) 2024-09-13T22:23:07.336 INFO:teuthology.orchestra.run.smithi192.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T22:23:07.337 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T22:23:07.337 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T22:23:07.337 INFO:teuthology.orchestra.run.smithi192.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listen normally on 3 enp3s0f1 172.21.15.192:123 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listen normally on 4 lo [::1]:123 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8049%5]:123 2024-09-13T22:23:07.338 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:07 ntpd[18490]: Listening on routing socket on fd #22 for interface updates 2024-09-13T22:23:08.303 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:08 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:08.303 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:08 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:08.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:08 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:08.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:08 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:08.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:08 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:08.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:08 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:08.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:08 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:08.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:08 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:08.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:08 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:08.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:08 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:08.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:08 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:08.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:08 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:09.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:09.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:09 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:09.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:09.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:09.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:09.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:09 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:10.303 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:10 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:10.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:10 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:10.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:10 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:10.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:10 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:10.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:10 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:10.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:10 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:10.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:10 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:10.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:10 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:10.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:10 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:10.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:10 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:10.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:10 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:10.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:10 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:11.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:11.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:11 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:11.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:11.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:11.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:11.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:11.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:11 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:12.303 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:12 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:12.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:12 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:12.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:12 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:12.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:12 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:12.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:12 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:12.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:12 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:12.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:12 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:12.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:12 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:12.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:12 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:12.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:12 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:12.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:12 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:12.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:12 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:13 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:13.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:13.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:13.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:13.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:13 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:14.303 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stdout:13 Sep 22:23:14 ntpd[18529]: ntpd: time slew -0.000851 s 2024-09-13T22:23:14.304 INFO:teuthology.orchestra.run.smithi178.stdout:ntpd: time slew -0.000851s 2024-09-13T22:23:14.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T22:23:14.305 INFO:teuthology.orchestra.run.smithi178.stderr:13 Sep 22:23:14 ntpd[18529]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T22:23:14.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:14.335 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:14.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T22:23:14.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T22:23:14.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T22:23:14.336 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T22:23:14.336 INFO:teuthology.orchestra.run.smithi192.stdout:13 Sep 22:23:14 ntpd[18490]: ntpd: time slew -0.000707 s 2024-09-13T22:23:14.336 INFO:teuthology.orchestra.run.smithi192.stdout:ntpd: time slew -0.000707s 2024-09-13T22:23:14.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T22:23:14.337 INFO:teuthology.orchestra.run.smithi192.stderr:13 Sep 22:23:14 ntpd[18490]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T22:23:14.356 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T22:23:14.356 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2024-09-13T22:23:14.356 INFO:teuthology.orchestra.run.smithi178.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.356 INFO:teuthology.orchestra.run.smithi178.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.356 INFO:teuthology.orchestra.run.smithi178.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.356 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.383 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T22:23:14.383 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2024-09-13T22:23:14.383 INFO:teuthology.orchestra.run.smithi192.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.383 INFO:teuthology.orchestra.run.smithi192.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.384 INFO:teuthology.orchestra.run.smithi192.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.384 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T22:23:14.385 INFO:teuthology.run_tasks:Running task install... 2024-09-13T22:23:14.397 DEBUG:teuthology.task.install:project ceph 2024-09-13T22:23:14.397 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}} 2024-09-13T22:23:14.397 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'} 2024-09-13T22:23:14.398 INFO:teuthology.task.install:Using flavor: default 2024-09-13T22:23:14.406 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T22:23:14.406 INFO:teuthology.task.install:extra packages: [] 2024-09-13T22:23:14.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-key list | grep Ceph 2024-09-13T22:23:14.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-key list | grep Ceph 2024-09-13T22:23:14.464 INFO:teuthology.orchestra.run.smithi178.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T22:23:14.485 INFO:teuthology.orchestra.run.smithi192.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T22:23:14.490 INFO:teuthology.orchestra.run.smithi178.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T22:23:14.490 INFO:teuthology.orchestra.run.smithi178.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T22:23:14.491 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T22:23:14.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:23:14.513 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T22:23:14.513 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T22:23:14.514 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T22:23:14.514 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:23:14.646 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default/ 2024-09-13T22:23:14.647 INFO:teuthology.task.install.deb:Package version is 19.1.1-324-gecab7be5-1jammy 2024-09-13T22:23:14.661 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default/ 2024-09-13T22:23:14.661 INFO:teuthology.task.install.deb:Package version is 19.1.1-324-gecab7be5-1jammy 2024-09-13T22:23:14.768 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:23:14.768 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T22:23:14.778 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:23:14.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T22:23:14.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-09-13T22:23:14.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2024-09-13T22:23:14.949 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T22:23:14.949 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T22:23:14.960 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T22:23:14.960 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T22:23:14.960 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T22:23:14.971 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T22:23:14.977 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T22:23:14.989 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T22:23:15.008 INFO:teuthology.orchestra.run.smithi178.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T22:23:15.017 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T22:23:15.029 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-13T22:23:15.038 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-13T22:23:15.051 INFO:teuthology.orchestra.run.smithi178.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T22:23:15.060 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T22:23:15.073 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-09-13T22:23:15.081 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-13T22:23:15.102 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-13T22:23:15.108 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-09-13T22:23:15.294 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 29.0 kB in 0s (90.0 kB/s) 2024-09-13T22:23:15.301 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 29.0 kB in 0s (85.4 kB/s) 2024-09-13T22:23:16.255 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-09-13T22:23:16.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-324-gecab7be5-1jammy cephadm=19.1.1-324-gecab7be5-1jammy ceph-mds=19.1.1-324-gecab7be5-1jammy ceph-mgr=19.1.1-324-gecab7be5-1jammy ceph-common=19.1.1-324-gecab7be5-1jammy ceph-fuse=19.1.1-324-gecab7be5-1jammy ceph-test=19.1.1-324-gecab7be5-1jammy ceph-volume=19.1.1-324-gecab7be5-1jammy radosgw=19.1.1-324-gecab7be5-1jammy python3-rados=19.1.1-324-gecab7be5-1jammy python3-rgw=19.1.1-324-gecab7be5-1jammy python3-cephfs=19.1.1-324-gecab7be5-1jammy python3-rbd=19.1.1-324-gecab7be5-1jammy libcephfs2=19.1.1-324-gecab7be5-1jammy libcephfs-dev=19.1.1-324-gecab7be5-1jammy librados2=19.1.1-324-gecab7be5-1jammy librbd1=19.1.1-324-gecab7be5-1jammy rbd-fuse=19.1.1-324-gecab7be5-1jammy 2024-09-13T22:23:16.278 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-09-13T22:23:16.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-324-gecab7be5-1jammy cephadm=19.1.1-324-gecab7be5-1jammy ceph-mds=19.1.1-324-gecab7be5-1jammy ceph-mgr=19.1.1-324-gecab7be5-1jammy ceph-common=19.1.1-324-gecab7be5-1jammy ceph-fuse=19.1.1-324-gecab7be5-1jammy ceph-test=19.1.1-324-gecab7be5-1jammy ceph-volume=19.1.1-324-gecab7be5-1jammy radosgw=19.1.1-324-gecab7be5-1jammy python3-rados=19.1.1-324-gecab7be5-1jammy python3-rgw=19.1.1-324-gecab7be5-1jammy python3-cephfs=19.1.1-324-gecab7be5-1jammy python3-rbd=19.1.1-324-gecab7be5-1jammy libcephfs2=19.1.1-324-gecab7be5-1jammy libcephfs-dev=19.1.1-324-gecab7be5-1jammy librados2=19.1.1-324-gecab7be5-1jammy librbd1=19.1.1-324-gecab7be5-1jammy rbd-fuse=19.1.1-324-gecab7be5-1jammy 2024-09-13T22:23:16.334 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-09-13T22:23:16.359 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-09-13T22:23:16.543 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2024-09-13T22:23:16.543 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2024-09-13T22:23:16.567 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-09-13T22:23:16.568 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-09-13T22:23:16.737 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T22:23:16.737 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T22:23:16.737 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T22:23:16.737 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T22:23:16.738 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T22:23:16.738 INFO:teuthology.orchestra.run.smithi192.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T22:23:16.738 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T22:23:16.739 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2024-09-13T22:23:16.739 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T22:23:16.740 INFO:teuthology.orchestra.run.smithi192.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet zip 2024-09-13T22:23:16.741 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2024-09-13T22:23:16.742 INFO:teuthology.orchestra.run.smithi192.stdout: btrfs-tools 2024-09-13T22:23:16.775 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T22:23:16.775 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T22:23:16.775 INFO:teuthology.orchestra.run.smithi178.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T22:23:16.775 INFO:teuthology.orchestra.run.smithi178.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T22:23:16.776 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T22:23:16.776 INFO:teuthology.orchestra.run.smithi178.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T22:23:16.776 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T22:23:16.777 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2024-09-13T22:23:16.777 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T22:23:16.777 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T22:23:16.777 INFO:teuthology.orchestra.run.smithi178.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T22:23:16.777 INFO:teuthology.orchestra.run.smithi178.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T22:23:16.778 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T22:23:16.778 INFO:teuthology.orchestra.run.smithi178.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T22:23:16.778 INFO:teuthology.orchestra.run.smithi178.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T22:23:16.779 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet zip 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T22:23:16.780 INFO:teuthology.orchestra.run.smithi178.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T22:23:16.781 INFO:teuthology.orchestra.run.smithi178.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T22:23:16.781 INFO:teuthology.orchestra.run.smithi178.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T22:23:16.781 INFO:teuthology.orchestra.run.smithi178.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T22:23:16.781 INFO:teuthology.orchestra.run.smithi178.stdout:Recommended packages: 2024-09-13T22:23:16.781 INFO:teuthology.orchestra.run.smithi178.stdout: btrfs-tools 2024-09-13T22:23:16.848 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2024-09-13T22:23:16.848 INFO:teuthology.orchestra.run.smithi192.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T22:23:16.848 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T22:23:16.848 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T22:23:16.849 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T22:23:16.849 INFO:teuthology.orchestra.run.smithi192.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T22:23:16.849 INFO:teuthology.orchestra.run.smithi192.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T22:23:16.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T22:23:16.851 INFO:teuthology.orchestra.run.smithi192.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T22:23:16.884 INFO:teuthology.orchestra.run.smithi178.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T22:23:16.885 INFO:teuthology.orchestra.run.smithi178.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T22:23:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T22:23:16.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T22:23:16.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T22:23:16.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T22:23:16.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T22:23:16.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T22:23:16.887 INFO:teuthology.orchestra.run.smithi178.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T22:23:16.949 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T22:23:16.949 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 181 MB of archives. 2024-09-13T22:23:16.949 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-13T22:23:16.950 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T22:23:16.994 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T22:23:16.994 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 181 MB of archives. 2024-09-13T22:23:16.995 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-13T22:23:16.995 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-324-gecab7be5-1jammy [3,591 kB] 2024-09-13T22:23:17.028 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-324-gecab7be5-1jammy [3,591 kB] 2024-09-13T22:23:17.039 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T22:23:17.039 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T22:23:17.042 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T22:23:17.059 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T22:23:17.119 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T22:23:17.123 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T22:23:17.136 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T22:23:17.141 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T22:23:17.141 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T22:23:17.141 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T22:23:17.150 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T22:23:17.152 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T22:23:17.155 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T22:23:17.156 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T22:23:17.157 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T22:23:17.160 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T22:23:17.174 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T22:23:17.175 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T22:23:17.176 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T22:23:17.176 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T22:23:17.177 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T22:23:17.178 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T22:23:17.178 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T22:23:17.179 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T22:23:17.182 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-324-gecab7be5-1jammy [3,241 kB] 2024-09-13T22:23:17.192 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T22:23:17.192 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T22:23:17.209 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T22:23:17.211 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T22:23:17.212 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T22:23:17.212 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T22:23:17.213 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T22:23:17.213 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T22:23:17.214 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T22:23:17.214 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T22:23:17.226 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T22:23:17.227 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T22:23:17.242 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-324-gecab7be5-1jammy [3,241 kB] 2024-09-13T22:23:17.243 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-324-gecab7be5-1jammy [718 kB] 2024-09-13T22:23:17.243 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T22:23:17.249 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T22:23:17.249 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T22:23:17.249 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T22:23:17.250 INFO:teuthology.orchestra.run.smithi178.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T22:23:17.251 INFO:teuthology.orchestra.run.smithi178.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T22:23:17.253 INFO:teuthology.orchestra.run.smithi178.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T22:23:17.254 INFO:teuthology.orchestra.run.smithi178.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T22:23:17.260 INFO:teuthology.orchestra.run.smithi178.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T22:23:17.261 INFO:teuthology.orchestra.run.smithi178.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T22:23:17.278 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-324-gecab7be5-1jammy [356 kB] 2024-09-13T22:23:17.278 INFO:teuthology.orchestra.run.smithi178.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T22:23:17.278 INFO:teuthology.orchestra.run.smithi178.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T22:23:17.279 INFO:teuthology.orchestra.run.smithi178.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T22:23:17.281 INFO:teuthology.orchestra.run.smithi178.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T22:23:17.285 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-324-gecab7be5-1jammy [32.8 kB] 2024-09-13T22:23:17.286 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-324-gecab7be5-1jammy [180 kB] 2024-09-13T22:23:17.290 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-324-gecab7be5-1jammy [67.8 kB] 2024-09-13T22:23:17.291 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-324-gecab7be5-1jammy [333 kB] 2024-09-13T22:23:17.297 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-324-gecab7be5-1jammy [6,875 kB] 2024-09-13T22:23:17.333 INFO:teuthology.orchestra.run.smithi178.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T22:23:17.334 INFO:teuthology.orchestra.run.smithi178.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T22:23:17.335 INFO:teuthology.orchestra.run.smithi178.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T22:23:17.335 INFO:teuthology.orchestra.run.smithi178.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T22:23:17.336 INFO:teuthology.orchestra.run.smithi178.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T22:23:17.354 INFO:teuthology.orchestra.run.smithi178.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T22:23:17.355 INFO:teuthology.orchestra.run.smithi178.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T22:23:17.355 INFO:teuthology.orchestra.run.smithi178.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T22:23:17.365 INFO:teuthology.orchestra.run.smithi178.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-324-gecab7be5-1jammy [718 kB] 2024-09-13T22:23:17.370 INFO:teuthology.orchestra.run.smithi178.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T22:23:17.370 INFO:teuthology.orchestra.run.smithi178.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T22:23:17.371 INFO:teuthology.orchestra.run.smithi178.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T22:23:17.372 INFO:teuthology.orchestra.run.smithi178.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T22:23:17.372 INFO:teuthology.orchestra.run.smithi178.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T22:23:17.372 INFO:teuthology.orchestra.run.smithi178.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T22:23:17.378 INFO:teuthology.orchestra.run.smithi178.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-324-gecab7be5-1jammy [356 kB] 2024-09-13T22:23:17.387 INFO:teuthology.orchestra.run.smithi178.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-324-gecab7be5-1jammy [32.8 kB] 2024-09-13T22:23:17.387 INFO:teuthology.orchestra.run.smithi178.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-324-gecab7be5-1jammy [180 kB] 2024-09-13T22:23:17.387 INFO:teuthology.orchestra.run.smithi178.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T22:23:17.388 INFO:teuthology.orchestra.run.smithi178.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T22:23:17.391 INFO:teuthology.orchestra.run.smithi178.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-324-gecab7be5-1jammy [67.8 kB] 2024-09-13T22:23:17.391 INFO:teuthology.orchestra.run.smithi178.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T22:23:17.392 INFO:teuthology.orchestra.run.smithi178.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-324-gecab7be5-1jammy [333 kB] 2024-09-13T22:23:17.405 INFO:teuthology.orchestra.run.smithi178.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T22:23:17.406 INFO:teuthology.orchestra.run.smithi178.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T22:23:17.406 INFO:teuthology.orchestra.run.smithi178.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T22:23:17.409 INFO:teuthology.orchestra.run.smithi178.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T22:23:17.412 INFO:teuthology.orchestra.run.smithi178.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T22:23:17.413 INFO:teuthology.orchestra.run.smithi178.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T22:23:17.413 INFO:teuthology.orchestra.run.smithi178.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T22:23:17.422 INFO:teuthology.orchestra.run.smithi178.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T22:23:17.422 INFO:teuthology.orchestra.run.smithi178.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T22:23:17.425 INFO:teuthology.orchestra.run.smithi178.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T22:23:17.426 INFO:teuthology.orchestra.run.smithi178.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-324-gecab7be5-1jammy [6,875 kB] 2024-09-13T22:23:17.439 INFO:teuthology.orchestra.run.smithi178.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T22:23:17.439 INFO:teuthology.orchestra.run.smithi178.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T22:23:17.440 INFO:teuthology.orchestra.run.smithi178.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T22:23:17.441 INFO:teuthology.orchestra.run.smithi178.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T22:23:17.442 INFO:teuthology.orchestra.run.smithi178.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T22:23:17.448 INFO:teuthology.orchestra.run.smithi178.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T22:23:17.448 INFO:teuthology.orchestra.run.smithi178.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T22:23:17.456 INFO:teuthology.orchestra.run.smithi178.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T22:23:17.458 INFO:teuthology.orchestra.run.smithi178.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T22:23:17.458 INFO:teuthology.orchestra.run.smithi178.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T22:23:17.466 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T22:23:17.478 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T22:23:17.495 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-324-gecab7be5-1jammy [112 kB] 2024-09-13T22:23:17.497 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-324-gecab7be5-1jammy [472 kB] 2024-09-13T22:23:17.530 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-324-gecab7be5-1jammy [25.9 MB] 2024-09-13T22:23:17.559 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T22:23:17.668 INFO:teuthology.orchestra.run.smithi178.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-324-gecab7be5-1jammy [112 kB] 2024-09-13T22:23:17.669 INFO:teuthology.orchestra.run.smithi178.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-324-gecab7be5-1jammy [472 kB] 2024-09-13T22:23:17.679 INFO:teuthology.orchestra.run.smithi178.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-324-gecab7be5-1jammy [25.9 MB] 2024-09-13T22:23:17.788 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T22:23:17.799 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T22:23:17.829 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T22:23:17.837 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T22:23:17.839 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T22:23:17.840 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T22:23:17.858 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T22:23:17.862 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T22:23:17.866 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T22:23:17.872 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T22:23:17.951 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T22:23:17.956 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T22:23:17.959 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T22:23:17.963 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T22:23:17.964 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T22:23:17.965 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T22:23:17.968 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T22:23:17.968 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T22:23:17.969 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T22:23:18.035 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T22:23:18.052 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T22:23:18.053 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T22:23:18.054 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T22:23:18.059 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T22:23:18.060 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T22:23:18.117 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T22:23:18.118 INFO:teuthology.orchestra.run.smithi192.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T22:23:18.120 INFO:teuthology.orchestra.run.smithi192.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T22:23:18.121 INFO:teuthology.orchestra.run.smithi192.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T22:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T22:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T22:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T22:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T22:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T22:23:18.279 INFO:teuthology.orchestra.run.smithi192.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T22:23:18.280 INFO:teuthology.orchestra.run.smithi192.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T22:23:18.282 INFO:teuthology.orchestra.run.smithi192.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T22:23:18.284 INFO:teuthology.orchestra.run.smithi192.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T22:23:18.288 INFO:teuthology.orchestra.run.smithi192.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T22:23:18.361 INFO:teuthology.orchestra.run.smithi192.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T22:23:18.363 INFO:teuthology.orchestra.run.smithi192.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T22:23:18.365 INFO:teuthology.orchestra.run.smithi192.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T22:23:18.370 INFO:teuthology.orchestra.run.smithi192.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T22:23:18.371 INFO:teuthology.orchestra.run.smithi192.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T22:23:18.443 INFO:teuthology.orchestra.run.smithi192.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T22:23:18.452 INFO:teuthology.orchestra.run.smithi192.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T22:23:18.516 INFO:teuthology.orchestra.run.smithi192.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-324-gecab7be5-1jammy [5,145 kB] 2024-09-13T22:23:18.611 INFO:teuthology.orchestra.run.smithi192.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T22:23:18.613 INFO:teuthology.orchestra.run.smithi192.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T22:23:18.615 INFO:teuthology.orchestra.run.smithi192.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-324-gecab7be5-1jammy [245 kB] 2024-09-13T22:23:18.616 INFO:teuthology.orchestra.run.smithi192.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T22:23:18.617 INFO:teuthology.orchestra.run.smithi192.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T22:23:18.619 INFO:teuthology.orchestra.run.smithi192.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T22:23:18.628 INFO:teuthology.orchestra.run.smithi178.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-324-gecab7be5-1jammy [5,145 kB] 2024-09-13T22:23:18.643 INFO:teuthology.orchestra.run.smithi192.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-324-gecab7be5-1jammy [124 kB] 2024-09-13T22:23:18.646 INFO:teuthology.orchestra.run.smithi192.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-324-gecab7be5-1jammy [1,555 kB] 2024-09-13T22:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T22:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T22:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T22:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T22:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T22:23:18.778 INFO:teuthology.orchestra.run.smithi192.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T22:23:18.779 INFO:teuthology.orchestra.run.smithi192.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T22:23:18.779 INFO:teuthology.orchestra.run.smithi192.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-324-gecab7be5-1jammy [6,143 kB] 2024-09-13T22:23:18.780 INFO:teuthology.orchestra.run.smithi192.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T22:23:18.781 INFO:teuthology.orchestra.run.smithi192.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T22:23:18.782 INFO:teuthology.orchestra.run.smithi192.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T22:23:18.783 INFO:teuthology.orchestra.run.smithi192.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T22:23:18.790 INFO:teuthology.orchestra.run.smithi192.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T22:23:18.809 INFO:teuthology.orchestra.run.smithi178.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-324-gecab7be5-1jammy [245 kB] 2024-09-13T22:23:18.816 INFO:teuthology.orchestra.run.smithi178.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-324-gecab7be5-1jammy [124 kB] 2024-09-13T22:23:18.818 INFO:teuthology.orchestra.run.smithi178.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-324-gecab7be5-1jammy [1,555 kB] 2024-09-13T22:23:18.860 INFO:teuthology.orchestra.run.smithi192.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T22:23:18.864 INFO:teuthology.orchestra.run.smithi192.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T22:23:18.866 INFO:teuthology.orchestra.run.smithi192.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T22:23:18.883 INFO:teuthology.orchestra.run.smithi178.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-324-gecab7be5-1jammy [6,143 kB] 2024-09-13T22:23:18.941 INFO:teuthology.orchestra.run.smithi192.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T22:23:18.948 INFO:teuthology.orchestra.run.smithi192.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T22:23:18.951 INFO:teuthology.orchestra.run.smithi192.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T22:23:18.952 INFO:teuthology.orchestra.run.smithi192.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T22:23:18.964 INFO:teuthology.orchestra.run.smithi192.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T22:23:18.965 INFO:teuthology.orchestra.run.smithi192.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T22:23:18.973 INFO:teuthology.orchestra.run.smithi192.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T22:23:19.022 INFO:teuthology.orchestra.run.smithi192.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T22:23:19.024 INFO:teuthology.orchestra.run.smithi192.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T22:23:19.025 INFO:teuthology.orchestra.run.smithi192.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T22:23:19.026 INFO:teuthology.orchestra.run.smithi192.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-324-gecab7be5-1jammy [22.8 MB] 2024-09-13T22:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T22:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T22:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T22:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T22:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T22:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T22:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T22:23:19.151 INFO:teuthology.orchestra.run.smithi178.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-324-gecab7be5-1jammy [22.8 MB] 2024-09-13T22:23:20.186 INFO:teuthology.orchestra.run.smithi192.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-324-gecab7be5-1jammy [14.1 kB] 2024-09-13T22:23:20.187 INFO:teuthology.orchestra.run.smithi192.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-324-gecab7be5-1jammy [877 kB] 2024-09-13T22:23:20.204 INFO:teuthology.orchestra.run.smithi192.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-324-gecab7be5-1jammy [2,462 kB] 2024-09-13T22:23:20.246 INFO:teuthology.orchestra.run.smithi178.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-324-gecab7be5-1jammy [14.1 kB] 2024-09-13T22:23:20.247 INFO:teuthology.orchestra.run.smithi178.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-324-gecab7be5-1jammy [877 kB] 2024-09-13T22:23:20.254 INFO:teuthology.orchestra.run.smithi192.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-324-gecab7be5-1jammy [794 kB] 2024-09-13T22:23:20.264 INFO:teuthology.orchestra.run.smithi178.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-324-gecab7be5-1jammy [2,462 kB] 2024-09-13T22:23:20.290 INFO:teuthology.orchestra.run.smithi192.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-324-gecab7be5-1jammy [152 kB] 2024-09-13T22:23:20.293 INFO:teuthology.orchestra.run.smithi192.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-324-gecab7be5-1jammy [2,275 kB] 2024-09-13T22:23:20.332 INFO:teuthology.orchestra.run.smithi178.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-324-gecab7be5-1jammy [794 kB] 2024-09-13T22:23:20.362 INFO:teuthology.orchestra.run.smithi178.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-324-gecab7be5-1jammy [152 kB] 2024-09-13T22:23:20.365 INFO:teuthology.orchestra.run.smithi178.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-324-gecab7be5-1jammy [2,275 kB] 2024-09-13T22:23:20.411 INFO:teuthology.orchestra.run.smithi192.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-324-gecab7be5-1jammy [8,625 kB] 2024-09-13T22:23:20.466 INFO:teuthology.orchestra.run.smithi178.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-324-gecab7be5-1jammy [8,625 kB] 2024-09-13T22:23:20.845 INFO:teuthology.orchestra.run.smithi192.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-324-gecab7be5-1jammy [14.2 kB] 2024-09-13T22:23:20.845 INFO:teuthology.orchestra.run.smithi192.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-324-gecab7be5-1jammy [51.4 MB] 2024-09-13T22:23:20.867 INFO:teuthology.orchestra.run.smithi178.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-324-gecab7be5-1jammy [14.2 kB] 2024-09-13T22:23:20.868 INFO:teuthology.orchestra.run.smithi178.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-324-gecab7be5-1jammy [51.4 MB] 2024-09-13T22:23:23.092 INFO:teuthology.orchestra.run.smithi178.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-324-gecab7be5-1jammy [130 kB] 2024-09-13T22:23:23.095 INFO:teuthology.orchestra.run.smithi178.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-324-gecab7be5-1jammy [39.0 kB] 2024-09-13T22:23:23.096 INFO:teuthology.orchestra.run.smithi178.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-324-gecab7be5-1jammy [13.6 MB] 2024-09-13T22:23:23.179 INFO:teuthology.orchestra.run.smithi192.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-324-gecab7be5-1jammy [130 kB] 2024-09-13T22:23:23.182 INFO:teuthology.orchestra.run.smithi192.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-324-gecab7be5-1jammy [39.0 kB] 2024-09-13T22:23:23.183 INFO:teuthology.orchestra.run.smithi192.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-324-gecab7be5-1jammy [13.6 MB] 2024-09-13T22:23:23.471 INFO:teuthology.orchestra.run.smithi178.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-324-gecab7be5-1jammy [91.7 kB] 2024-09-13T22:23:23.597 INFO:teuthology.orchestra.run.smithi192.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-324-gecab7be5-1jammy [91.7 kB] 2024-09-13T22:23:24.046 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 181 MB in 7s (27.5 MB/s) 2024-09-13T22:23:24.165 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 181 MB in 7s (26.8 MB/s) 2024-09-13T22:23:24.194 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T22:23:24.293 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T22:23:24.299 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T22:23:24.323 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T22:23:24.368 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T22:23:24.423 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T22:23:24.428 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T22:23:24.513 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T22:23:24.638 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T22:23:24.653 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T22:23:24.696 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T22:23:24.933 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T22:23:24.959 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T22:23:25.000 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T22:23:25.024 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T22:23:25.050 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T22:23:25.091 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T22:23:25.361 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T22:23:25.387 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T22:23:25.427 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T22:23:25.527 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T22:23:25.554 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T22:23:25.603 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:23:25.881 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T22:23:25.907 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T22:23:25.956 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:23:26.014 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T22:23:26.041 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T22:23:26.080 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:23:26.334 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T22:23:26.349 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T22:23:26.384 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:23:26.425 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T22:23:26.451 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T22:23:26.492 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:23:26.746 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T22:23:26.772 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T22:23:26.794 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T22:23:26.810 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T22:23:26.812 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:23:26.845 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T22:23:27.174 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T22:23:27.181 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librados2. 2024-09-13T22:23:27.200 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T22:23:27.208 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../007-librados2_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:27.232 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T22:23:27.250 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librados2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:27.551 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librados2. 2024-09-13T22:23:27.577 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../007-librados2_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librados2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:27.700 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librbd1. 2024-09-13T22:23:27.727 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../008-librbd1_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:27.767 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librbd1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:28.079 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librbd1. 2024-09-13T22:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../008-librbd1_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librbd1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:28.236 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T22:23:28.263 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:28.303 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libcephfs2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:28.555 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rados. 2024-09-13T22:23:28.571 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../010-python3-rados_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T22:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:28.613 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rados (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:28.916 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T22:23:28.943 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:28.946 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rados. 2024-09-13T22:23:28.973 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../010-python3-rados_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:28.983 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-ceph-argparse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:29.176 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rados (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:29.218 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T22:23:29.234 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:29.277 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cephfs (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:29.512 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T22:23:29.538 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:29.555 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T22:23:29.579 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-argparse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:29.581 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:29.613 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:29.831 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T22:23:29.857 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:29.874 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T22:23:29.898 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cephfs (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:29.901 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T22:23:29.941 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T22:23:30.242 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T22:23:30.252 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T22:23:30.265 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:30.279 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T22:23:30.300 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:30.318 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T22:23:30.620 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T22:23:30.635 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T22:23:30.670 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T22:23:30.854 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T22:23:30.870 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:30.913 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rbd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:30.998 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T22:23:31.024 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T22:23:31.064 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T22:23:31.224 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T22:23:31.250 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T22:23:31.282 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T22:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T22:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:31.375 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rbd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:31.526 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T22:23:31.542 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T22:23:31.576 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T22:23:31.695 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T22:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T22:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T22:23:31.854 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T22:23:31.870 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T22:23:31.912 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T22:23:31.997 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T22:23:32.024 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T22:23:32.064 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T22:23:32.241 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua5.1. 2024-09-13T22:23:32.267 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T22:23:32.307 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T22:23:32.367 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T22:23:32.382 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T22:23:32.425 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T22:23:32.593 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua-any. 2024-09-13T22:23:32.620 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T22:23:32.652 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T22:23:32.787 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package lua5.1. 2024-09-13T22:23:32.813 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T22:23:32.856 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T22:23:32.888 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package zip. 2024-09-13T22:23:32.903 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T22:23:32.946 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T22:23:33.140 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package lua-any. 2024-09-13T22:23:33.155 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T22:23:33.190 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T22:23:33.258 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package unzip. 2024-09-13T22:23:33.284 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T22:23:33.324 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T22:23:33.426 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package zip. 2024-09-13T22:23:33.441 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T22:23:33.484 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T22:23:33.719 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package luarocks. 2024-09-13T22:23:33.746 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T22:23:33.786 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T22:23:33.787 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package unzip. 2024-09-13T22:23:33.813 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T22:23:33.862 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T22:23:34.169 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librgw2. 2024-09-13T22:23:34.196 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../025-librgw2_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:34.235 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librgw2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:34.508 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package luarocks. 2024-09-13T22:23:34.534 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T22:23:34.574 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T22:23:34.763 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T22:23:34.778 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:34.821 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:34.957 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librgw2. 2024-09-13T22:23:34.983 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../025-librgw2_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:35.023 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librgw2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:35.107 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T22:23:35.133 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T22:23:35.182 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T22:23:35.477 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T22:23:35.503 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:35.551 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libradosstriper1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:35.567 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T22:23:35.582 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:35.617 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:35.962 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T22:23:35.988 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T22:23:36.022 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-common. 2024-09-13T22:23:36.028 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T22:23:36.048 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../029-ceph-common_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:36.088 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:36.349 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T22:23:36.374 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:36.415 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libradosstriper1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-common. 2024-09-13T22:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../029-ceph-common_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:37.469 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-base. 2024-09-13T22:23:37.495 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../030-ceph-base_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:37.544 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-base (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:38.073 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T22:23:38.124 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T22:23:38.139 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T22:23:38.323 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-base. 2024-09-13T22:23:38.349 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../030-ceph-base_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:38.399 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-base (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:38.442 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T22:23:38.469 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T22:23:38.509 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T22:23:38.820 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T22:23:38.847 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T22:23:38.886 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T22:23:38.952 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T22:23:38.979 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T22:23:39.019 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T22:23:39.158 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T22:23:39.174 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T22:23:39.208 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T22:23:39.347 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T22:23:39.373 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T22:23:39.503 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T22:23:39.529 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T22:23:39.569 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T22:23:39.588 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T22:23:39.906 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T22:23:39.916 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T22:23:39.933 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T22:23:39.943 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T22:23:39.973 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T22:23:39.983 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T22:23:40.251 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-portend. 2024-09-13T22:23:40.277 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T22:23:40.278 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T22:23:40.303 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T22:23:40.309 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T22:23:40.344 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T22:23:40.595 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T22:23:40.622 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T22:23:40.639 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T22:23:40.662 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T22:23:40.665 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T22:23:40.705 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T22:23:41.008 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T22:23:41.034 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T22:23:41.074 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T22:23:41.157 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T22:23:41.183 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T22:23:41.223 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T22:23:41.378 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-portend. 2024-09-13T22:23:41.404 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T22:23:41.444 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T22:23:41.560 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T22:23:41.587 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T22:23:41.627 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T22:23:41.688 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T22:23:41.704 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T22:23:41.738 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T22:23:41.938 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T22:23:41.965 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T22:23:42.005 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T22:23:42.024 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T22:23:42.051 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T22:23:42.091 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T22:23:42.308 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T22:23:42.335 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T22:23:42.375 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T22:23:42.419 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T22:23:42.446 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T22:23:42.486 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T22:23:42.636 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-mako. 2024-09-13T22:23:42.663 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T22:23:42.703 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T22:23:42.800 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T22:23:42.826 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T22:23:42.866 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T22:23:43.031 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T22:23:43.058 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T22:23:43.098 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T22:23:43.169 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T22:23:43.196 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T22:23:43.236 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T22:23:43.401 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T22:23:43.427 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T22:23:43.467 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T22:23:43.522 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-mako. 2024-09-13T22:23:43.549 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T22:23:43.580 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T22:23:43.778 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webob. 2024-09-13T22:23:43.794 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T22:23:43.828 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T22:23:43.850 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T22:23:43.878 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T22:23:43.917 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T22:23:44.132 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T22:23:44.158 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T22:23:44.202 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T22:23:44.220 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T22:23:44.247 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T22:23:44.303 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T22:23:44.485 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T22:23:44.511 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T22:23:44.551 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T22:23:44.623 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webob. 2024-09-13T22:23:44.649 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T22:23:44.689 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T22:23:44.854 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-paste. 2024-09-13T22:23:44.881 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T22:23:44.921 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T22:23:45.051 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T22:23:45.078 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T22:23:45.121 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T22:23:45.258 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T22:23:45.284 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T22:23:45.324 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T22:23:45.387 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T22:23:45.414 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T22:23:45.445 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T22:23:45.627 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T22:23:45.654 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T22:23:45.694 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T22:23:45.757 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-paste. 2024-09-13T22:23:45.784 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T22:23:45.824 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T22:23:45.972 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T22:23:45.998 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T22:23:46.038 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T22:23:46.118 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T22:23:46.142 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T22:23:46.177 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T22:23:46.334 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T22:23:46.361 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T22:23:46.400 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T22:23:46.471 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T22:23:46.498 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T22:23:46.538 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T22:23:46.739 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T22:23:46.765 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T22:23:46.805 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T22:23:46.849 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T22:23:46.876 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T22:23:46.916 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T22:23:47.134 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-idna. 2024-09-13T22:23:47.161 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T22:23:47.200 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T22:23:47.244 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T22:23:47.270 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T22:23:47.310 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T22:23:47.511 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T22:23:47.538 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T22:23:47.578 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T22:23:47.666 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T22:23:47.692 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T22:23:47.723 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T22:23:47.890 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-requests. 2024-09-13T22:23:47.917 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T22:23:47.948 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T22:23:48.027 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-idna. 2024-09-13T22:23:48.054 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T22:23:48.118 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T22:23:48.218 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T22:23:48.244 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T22:23:48.284 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T22:23:48.463 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T22:23:48.478 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T22:23:48.521 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T22:23:48.537 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T22:23:48.553 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:48.587 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-modules-core (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:48.850 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests. 2024-09-13T22:23:48.876 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T22:23:48.916 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T22:23:48.976 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T22:23:49.003 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:49.042 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:49.211 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T22:23:49.237 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T22:23:49.278 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T22:23:49.287 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T22:23:49.302 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T22:23:49.337 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T22:23:49.539 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T22:23:49.554 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:49.597 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-modules-core (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:49.632 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T22:23:49.659 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T22:23:49.698 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T22:23:50.043 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T22:23:50.070 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:50.110 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:50.253 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T22:23:50.280 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:50.677 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:51.347 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T22:23:51.356 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T22:23:51.362 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:51.383 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T22:23:51.396 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mon (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:51.422 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T22:23:51.734 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T22:23:51.760 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T22:23:51.800 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T22:23:52.084 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T22:23:52.111 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:52.150 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-osd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:52.170 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T22:23:52.196 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:52.236 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:52.640 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T22:23:52.667 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:52.706 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mon (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:53.204 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph. 2024-09-13T22:23:53.219 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../066-ceph_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:53.252 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T22:23:53.262 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:53.278 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:53.318 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-osd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:53.582 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T22:23:53.609 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:53.640 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:53.993 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T22:23:54.020 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:54.060 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mds (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:54.372 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph. 2024-09-13T22:23:54.387 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../066-ceph_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:54.430 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:54.454 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package cephadm. 2024-09-13T22:23:54.482 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../069-cephadm_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:54.521 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:54.733 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T22:23:54.760 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:54.800 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:54.815 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T22:23:54.842 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T22:23:54.874 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T22:23:55.128 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T22:23:55.155 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:55.186 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mds (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:55.210 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T22:23:55.237 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T22:23:55.276 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T22:23:55.589 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephadm. 2024-09-13T22:23:55.616 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../069-cephadm_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:23:55.655 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:56.008 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T22:23:56.035 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T22:23:56.075 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T22:23:56.274 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-babel. 2024-09-13T22:23:56.301 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T22:23:56.340 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T22:23:56.444 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T22:23:56.460 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T22:23:56.494 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T22:23:56.660 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T22:23:56.687 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T22:23:56.726 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T22:23:57.080 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T22:23:57.107 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:57.146 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:57.492 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T22:23:57.519 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T22:23:57.542 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-babel. 2024-09-13T22:23:57.558 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T22:23:57.569 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T22:23:57.608 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T22:23:57.844 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-routes. 2024-09-13T22:23:57.872 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T22:23:57.911 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T22:23:57.936 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T22:23:57.963 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T22:23:58.002 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T22:23:58.239 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T22:23:58.266 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:58.305 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-dashboard (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:58.347 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T22:23:58.374 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:58.413 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:23:58.759 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T22:23:58.786 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T22:23:58.825 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T22:23:59.111 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-routes. 2024-09-13T22:23:59.139 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T22:23:59.178 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T22:23:59.519 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T22:23:59.523 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T22:23:59.535 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T22:23:59.550 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:23:59.569 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T22:23:59.589 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-dashboard (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:00.065 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T22:24:00.082 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T22:24:00.131 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T22:24:00.485 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T22:24:00.512 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T22:24:00.552 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T22:24:00.854 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T22:24:00.881 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T22:24:00.920 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T22:24:00.920 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T22:24:00.935 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T22:24:00.970 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T22:24:01.490 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T22:24:01.506 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T22:24:01.513 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T22:24:01.541 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:24:01.548 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T22:24:01.666 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:01.911 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T22:24:01.938 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T22:24:01.977 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T22:24:02.288 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T22:24:02.315 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T22:24:02.354 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T22:24:02.561 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T22:24:02.589 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T22:24:02.628 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T22:24:02.956 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T22:24:02.983 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T22:24:03.022 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T22:24:03.080 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T22:24:03.108 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:24:03.232 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:03.351 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T22:24:03.379 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T22:24:03.418 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T22:24:03.704 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T22:24:03.732 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T22:24:03.770 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T22:24:04.100 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T22:24:04.127 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T22:24:04.166 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T22:24:04.211 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T22:24:04.239 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T22:24:04.277 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T22:24:04.470 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T22:24:04.497 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T22:24:04.528 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T22:24:04.556 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T22:24:04.583 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T22:24:04.614 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T22:24:04.814 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T22:24:04.842 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T22:24:04.881 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T22:24:04.942 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T22:24:04.970 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T22:24:05.009 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T22:24:05.193 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T22:24:05.220 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T22:24:05.259 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T22:24:05.529 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T22:24:05.556 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T22:24:05.595 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T22:24:05.848 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T22:24:05.865 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:24:05.874 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T22:24:05.901 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T22:24:05.915 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:05.932 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T22:24:06.243 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T22:24:06.270 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T22:24:06.302 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T22:24:06.309 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T22:24:06.329 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T22:24:06.377 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T22:24:06.663 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T22:24:06.690 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T22:24:06.729 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T22:24:06.747 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T22:24:06.774 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T22:24:06.822 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T22:24:07.159 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T22:24:07.186 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T22:24:07.225 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T22:24:07.232 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package jq. 2024-09-13T22:24:07.260 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T22:24:07.299 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T22:24:07.526 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package socat. 2024-09-13T22:24:07.542 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T22:24:07.576 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T22:24:07.830 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T22:24:07.846 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T22:24:07.880 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T22:24:07.973 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T22:24:08.001 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:24:08.039 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:08.201 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-test. 2024-09-13T22:24:08.217 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../097-ceph-test_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:08.251 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-test (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:08.326 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T22:24:08.353 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T22:24:08.384 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T22:24:08.729 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T22:24:08.756 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T22:24:08.795 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T22:24:09.064 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package jq. 2024-09-13T22:24:09.081 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T22:24:09.123 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T22:24:09.342 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package socat. 2024-09-13T22:24:09.358 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T22:24:09.392 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T22:24:09.696 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T22:24:09.723 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T22:24:09.762 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T22:24:10.125 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-test. 2024-09-13T22:24:10.141 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../097-ceph-test_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:10.174 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-test (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:10.589 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T22:24:10.616 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:24:10.672 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-volume (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:10.976 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T22:24:10.992 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:11.026 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libcephfs-dev (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:11.287 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T22:24:11.315 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T22:24:11.345 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T22:24:11.755 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T22:24:11.783 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T22:24:11.850 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T22:24:12.221 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T22:24:12.249 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T22:24:12.288 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T22:24:12.480 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T22:24:12.507 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-324-gecab7be5-1jammy_all.deb ... 2024-09-13T22:24:12.554 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-volume (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:12.677 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package pkg-config. 2024-09-13T22:24:12.705 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T22:24:12.735 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T22:24:12.892 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T22:24:12.919 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:12.967 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs-dev (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:12.979 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T22:24:12.995 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T22:24:13.038 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T22:24:13.353 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T22:24:13.380 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T22:24:13.424 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T22:24:13.427 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T22:24:13.440 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T22:24:13.483 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T22:24:13.766 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T22:24:13.786 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T22:24:13.793 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T22:24:13.813 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T22:24:13.832 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T22:24:13.852 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T22:24:14.172 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T22:24:14.200 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T22:24:14.255 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T22:24:14.286 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T22:24:14.314 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T22:24:14.361 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T22:24:14.567 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T22:24:14.595 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T22:24:14.634 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T22:24:14.742 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package pkg-config. 2024-09-13T22:24:14.770 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T22:24:14.808 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T22:24:14.953 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-py. 2024-09-13T22:24:14.981 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T22:24:15.020 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T22:24:15.078 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T22:24:15.106 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T22:24:15.145 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T22:24:15.340 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T22:24:15.368 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T22:24:15.415 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T22:24:15.540 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T22:24:15.567 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T22:24:15.598 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T22:24:15.845 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T22:24:15.872 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T22:24:15.901 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T22:24:15.911 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T22:24:15.928 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T22:24:15.967 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T22:24:16.197 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-toml. 2024-09-13T22:24:16.225 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T22:24:16.263 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T22:24:16.270 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T22:24:16.298 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T22:24:16.337 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T22:24:16.552 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T22:24:16.580 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T22:24:16.618 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T22:24:16.648 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T22:24:16.676 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T22:24:16.716 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T22:24:16.963 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T22:24:16.991 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T22:24:17.001 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-py. 2024-09-13T22:24:17.029 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T22:24:17.030 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T22:24:17.060 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T22:24:17.299 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T22:24:17.316 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T22:24:17.330 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T22:24:17.346 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T22:24:17.357 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T22:24:17.379 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T22:24:17.809 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T22:24:17.837 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T22:24:17.875 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T22:24:18.095 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package radosgw. 2024-09-13T22:24:18.123 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../116-radosgw_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:18.161 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking radosgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:18.195 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-toml. 2024-09-13T22:24:18.223 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T22:24:18.262 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T22:24:18.548 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T22:24:18.576 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T22:24:18.614 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T22:24:18.930 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T22:24:18.946 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:18.980 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking rbd-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:18.993 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T22:24:19.021 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T22:24:19.060 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T22:24:19.256 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T22:24:19.329 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T22:24:19.345 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T22:24:19.387 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T22:24:19.535 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T22:24:19.645 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T22:24:19.908 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T22:24:20.125 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package radosgw. 2024-09-13T22:24:20.153 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../116-radosgw_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:20.191 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking radosgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:20.227 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T22:24:20.658 INFO:teuthology.orchestra.run.smithi178.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T22:24:20.713 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:21.224 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user cephadm....done 2024-09-13T22:24:21.228 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T22:24:21.244 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-324-gecab7be5-1jammy_amd64.deb ... 2024-09-13T22:24:21.277 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking rbd-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:21.377 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T22:24:21.573 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T22:24:21.679 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T22:24:21.816 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T22:24:21.917 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T22:24:21.938 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T22:24:22.064 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T22:24:22.170 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T22:24:22.332 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T22:24:22.491 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T22:24:22.724 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T22:24:22.851 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T22:24:22.930 INFO:teuthology.orchestra.run.smithi192.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T22:24:23.011 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:23.160 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T22:24:23.521 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T22:24:23.595 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user cephadm....done 2024-09-13T22:24:23.729 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T22:24:23.780 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T22:24:24.039 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T22:24:24.324 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T22:24:24.407 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T22:24:24.584 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T22:24:24.656 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T22:24:24.783 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T22:24:24.851 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T22:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T22:24:25.102 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-ceph-argparse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:25.327 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T22:24:25.362 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T22:24:25.436 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T22:24:25.488 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T22:24:25.605 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T22:24:25.706 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T22:24:25.721 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T22:24:25.974 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T22:24:26.081 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T22:24:26.332 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T22:24:26.387 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T22:24:26.414 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T22:24:26.477 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T22:24:26.592 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T22:24:26.603 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T22:24:26.835 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T22:24:26.979 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T22:24:27.078 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T22:24:27.105 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T22:24:27.365 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T22:24:27.379 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T22:24:27.624 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T22:24:27.629 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-argparse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:27.889 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T22:24:27.934 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T22:24:28.015 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T22:24:28.060 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T22:24:28.141 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T22:24:28.267 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T22:24:28.526 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T22:24:28.736 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T22:24:28.880 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T22:24:28.915 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T22:24:28.980 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T22:24:28.996 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T22:24:29.105 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T22:24:29.114 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T22:24:29.214 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T22:24:29.465 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T22:24:29.507 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T22:24:29.565 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T22:24:29.825 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T22:24:29.926 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T22:24:30.084 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T22:24:30.160 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T22:24:30.402 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T22:24:30.503 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T22:24:30.528 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T22:24:30.846 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T22:24:31.139 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T22:24:31.181 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T22:24:31.432 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T22:24:31.558 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:31.685 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T22:24:31.824 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T22:24:31.851 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T22:24:31.950 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T22:24:32.152 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T22:24:32.251 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T22:24:32.652 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T22:24:32.886 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T22:24:33.263 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T22:24:33.341 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T22:24:33.547 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T22:24:33.609 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:24:33.727 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T22:24:33.874 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T22:24:33.994 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T22:24:34.108 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T22:24:34.137 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T22:24:34.247 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T22:24:34.259 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:34.531 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T22:24:34.553 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T22:24:34.782 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T22:24:34.853 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T22:24:35.033 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:24:35.142 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T22:24:35.452 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T22:24:35.703 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T22:24:35.963 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T22:24:35.984 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T22:24:36.214 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T22:24:36.226 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:24:36.336 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T22:24:36.456 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T22:24:36.579 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T22:24:36.705 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T22:24:36.724 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T22:24:36.822 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T22:24:36.859 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T22:24:37.107 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T22:24:37.135 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T22:24:37.294 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T22:24:37.367 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T22:24:37.587 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T22:24:37.617 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:24:37.743 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T22:24:37.880 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T22:24:38.061 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T22:24:38.139 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T22:24:38.265 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T22:24:38.321 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T22:24:38.400 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T22:24:38.488 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T22:24:38.580 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T22:24:38.710 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T22:24:38.836 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T22:24:38.841 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T22:24:39.082 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T22:24:39.229 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T22:24:39.341 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T22:24:39.480 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T22:24:39.484 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T22:24:39.611 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T22:24:39.769 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T22:24:39.870 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T22:24:39.874 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:24:39.991 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T22:24:40.162 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T22:24:40.293 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T22:24:40.455 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T22:24:40.594 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T22:24:40.706 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T22:24:40.807 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T22:24:40.861 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T22:24:40.908 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T22:24:40.987 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T22:24:40.997 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T22:24:41.243 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T22:24:41.255 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T22:24:41.344 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T22:24:41.515 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T22:24:41.729 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T22:24:41.860 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T22:24:41.989 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T22:24:42.106 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T22:24:42.236 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T22:24:42.348 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T22:24:42.474 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T22:24:42.521 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T22:24:42.784 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T22:24:42.848 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T22:24:42.974 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librados2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:43.085 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T22:24:43.109 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libsqlite3-mod-ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:43.235 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T22:24:43.361 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T22:24:43.487 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T22:24:43.563 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libcephfs2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:43.672 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libradosstriper1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:43.763 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T22:24:43.789 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T22:24:44.047 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T22:24:44.075 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librbd1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:44.184 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-modules-core (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:44.293 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:44.384 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T22:24:44.489 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T22:24:44.489 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T22:24:44.794 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T22:24:44.978 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libcephfs-dev (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:45.070 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T22:24:45.104 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rados (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:45.221 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librgw2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:45.322 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T22:24:45.423 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T22:24:45.549 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librados2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:45.675 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libsqlite3-mod-ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:45.817 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T22:24:46.120 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.221 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libradosstriper1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.347 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T22:24:46.618 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rbd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.624 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librbd1 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.719 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up rbd-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.750 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-modules-core (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.861 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.876 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:46.987 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cephfs (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:47.090 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T22:24:47.090 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T22:24:47.121 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:47.583 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs-dev (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:47.664 INFO:teuthology.orchestra.run.smithi178.stdout:Adding group ceph....done 2024-09-13T22:24:47.686 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rados (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:47.796 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librgw2 (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:47.922 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T22:24:48.041 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user ceph....done 2024-09-13T22:24:48.209 INFO:teuthology.orchestra.run.smithi178.stdout:Setting system user ceph properties....done 2024-09-13T22:24:48.224 INFO:teuthology.orchestra.run.smithi178.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T22:24:48.363 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T22:24:48.729 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T22:24:49.117 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rbd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.181 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-test (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.243 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up rbd-fuse (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.307 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up radosgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.377 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.478 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cephfs (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.612 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-common (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:49.765 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T22:24:49.765 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T22:24:50.155 INFO:teuthology.orchestra.run.smithi192.stdout:Adding group ceph....done 2024-09-13T22:24:50.234 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-base (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:50.544 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T22:24:50.556 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user ceph....done 2024-09-13T22:24:50.700 INFO:teuthology.orchestra.run.smithi192.stdout:Setting system user ceph properties....done 2024-09-13T22:24:50.714 INFO:teuthology.orchestra.run.smithi192.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T22:24:50.849 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T22:24:50.994 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mds (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:51.148 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T22:24:51.226 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T22:24:51.226 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T22:24:51.621 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-test (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:51.679 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:51.747 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up radosgw (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:51.895 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T22:24:51.895 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T22:24:52.237 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T22:24:52.237 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T22:24:52.363 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-osd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:52.646 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T22:24:52.646 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T22:24:52.775 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-base (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:53.115 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-k8sevents (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:53.220 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T22:24:53.249 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:53.501 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mon (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:53.727 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mds (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:53.936 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T22:24:53.936 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T22:24:53.999 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T22:24:53.999 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T22:24:54.394 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:54.479 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:54.545 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:54.671 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-dashboard (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:54.682 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T22:24:54.683 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T22:24:54.822 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-volume (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:55.138 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-osd (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:55.208 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T22:24:55.309 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T22:24:55.411 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T22:24:55.411 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T22:24:55.453 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T22:24:55.906 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-k8sevents (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:56.032 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:56.184 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mon (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:56.397 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T22:24:56.398 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T22:24:56.411 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T22:24:56.893 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-cephadm (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:57.027 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:57.089 INFO:teuthology.orchestra.run.smithi178.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T22:24:57.093 DEBUG:teuthology.parallel:result is None 2024-09-13T22:24:57.128 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-dashboard (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:57.305 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-volume (19.1.1-324-gecab7be5-1jammy) ... 2024-09-13T22:24:57.675 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T22:24:57.859 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T22:24:58.003 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T22:24:58.727 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T22:24:59.404 INFO:teuthology.orchestra.run.smithi192.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T22:24:59.408 DEBUG:teuthology.parallel:result is None 2024-09-13T22:24:59.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:24:59.558 DEBUG:teuthology.orchestra.run.smithi178:> dpkg-query -W -f '${Version}' ceph 2024-09-13T22:24:59.578 INFO:teuthology.orchestra.run.smithi178.stdout:19.1.1-324-gecab7be5-1jammy 2024-09-13T22:24:59.578 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-324-gecab7be5-1jammy 2024-09-13T22:24:59.578 INFO:teuthology.task.install:The correct ceph version 19.1.1-324-gecab7be5-1jammy is installed. 2024-09-13T22:24:59.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:24:59.727 DEBUG:teuthology.orchestra.run.smithi192:> dpkg-query -W -f '${Version}' ceph 2024-09-13T22:24:59.746 INFO:teuthology.orchestra.run.smithi192.stdout:19.1.1-324-gecab7be5-1jammy 2024-09-13T22:24:59.746 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-324-gecab7be5-1jammy 2024-09-13T22:24:59.746 INFO:teuthology.task.install:The correct ceph version 19.1.1-324-gecab7be5-1jammy is installed. 2024-09-13T22:24:59.748 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T22:24:59.748 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:24:59.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T22:24:59.763 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:24:59.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T22:24:59.801 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T22:24:59.802 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:24:59.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T22:24:59.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T22:24:59.873 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:24:59.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T22:24:59.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T22:24:59.941 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T22:24:59.941 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:24:59.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T22:24:59.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T22:25:00.009 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:25:00.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T22:25:00.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T22:25:00.077 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T22:25:00.077 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:25:00.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T22:25:00.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T22:25:00.145 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:25:00.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T22:25:00.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T22:25:00.212 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T22:25:00.307 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'cephadm_mode': 'root'} 2024-09-13T22:25:00.307 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:25:00.308 INFO:tasks.cephadm:Cluster fsid is 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:25:00.308 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T22:25:00.308 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.178:6789]', 'mon.c': '[v1:172.21.15.178:6790]', 'mon.b': '[v1:172.21.15.192:6789]'} 2024-09-13T22:25:00.308 INFO:tasks.cephadm:First mon is mon.a on smithi178 2024-09-13T22:25:00.308 INFO:tasks.cephadm:First mgr is y 2024-09-13T22:25:00.308 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T22:25:00.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2024-09-13T22:25:00.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-09-13T22:25:00.338 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-13T22:25:00.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:25:00.488 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/56074/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-324-gecab7be5', 'node_name': '172.21.2.17+braggi17', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-324-gecab7be5-1jammy'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-09-13 19:22:31.141342', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-09-13T22:25:00.632 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 ecab7be58963a5bd03910cd73eef66405d7fdd50 from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:25:00.633 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-09-13T22:25:00.634 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-09-13T22:25:00.634 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T22:25:01.091 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r-- 1 ubuntu ubuntu 791838 Sep 13 22:25 /home/ubuntu/cephtest/cephadm 2024-09-13T22:25:01.092 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/ecab7be58963a5bd03910cd73eef66405d7fdd50/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T22:25:01.553 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-rw-r-- 1 ubuntu ubuntu 791838 Sep 13 22:25 /home/ubuntu/cephtest/cephadm 2024-09-13T22:25:01.553 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T22:25:01.561 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T22:25:01.580 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 on all hosts... 2024-09-13T22:25:01.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 pull 2024-09-13T22:25:01.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 pull 2024-09-13T22:25:01.784 INFO:teuthology.orchestra.run.smithi178.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50... 2024-09-13T22:25:01.796 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50... 2024-09-13T22:27:16.718 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:27:16.718 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable)", 2024-09-13T22:27:16.718 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "0652ff2b5d78add567c2474cd2436ae21ad32722283bcbf25450e33edcd34e34", 2024-09-13T22:27:16.718 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-09-13T22:27:16.719 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b893710f6933dd3672a2a633ce5ac92081563259207d21cd52290129179ca808" 2024-09-13T22:27:16.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:27:16.719 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:27:17.984 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:27:17.985 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_version": "ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable)", 2024-09-13T22:27:17.985 INFO:teuthology.orchestra.run.smithi178.stdout: "image_id": "0652ff2b5d78add567c2474cd2436ae21ad32722283bcbf25450e33edcd34e34", 2024-09-13T22:27:17.985 INFO:teuthology.orchestra.run.smithi178.stdout: "repo_digests": [ 2024-09-13T22:27:17.985 INFO:teuthology.orchestra.run.smithi178.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b893710f6933dd3672a2a633ce5ac92081563259207d21cd52290129179ca808" 2024-09-13T22:27:17.985 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:27:17.985 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:27:18.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2024-09-13T22:27:18.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-09-13T22:27:18.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2024-09-13T22:27:18.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-09-13T22:27:18.120 INFO:tasks.cephadm:Writing seed config... 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-13T22:27:18.122 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-13T22:27:18.123 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-13T22:27:18.124 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T22:27:18.124 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-13T22:27:18.124 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:27:18.125 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T22:27:18.148 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 03ae43e2-721f-11ef-bceb-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-13T22:27:18.148 DEBUG:teuthology.orchestra.run.smithi178:mon.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a.service 2024-09-13T22:27:18.192 DEBUG:teuthology.orchestra.run.smithi178:mgr.y> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y.service 2024-09-13T22:27:18.235 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T22:27:18.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 -v bootstrap --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.178:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T22:27:18.458 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-13T22:27:18.458 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50', '-v', 'bootstrap', '--fsid', '03ae43e2-721f-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.178:6789]', '--skip-admin-label'] 2024-09-13T22:27:18.459 INFO:teuthology.orchestra.run.smithi178.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T22:27:18.459 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying podman|docker is present... 2024-09-13T22:27:18.460 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying lvm2 is present... 2024-09-13T22:27:18.460 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying time synchronization is in place... 2024-09-13T22:27:18.464 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T22:27:18.464 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T22:27:18.467 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T22:27:18.467 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.471 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-09-13T22:27:18.471 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-09-13T22:27:18.477 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-09-13T22:27:18.478 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.478 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-09-13T22:27:18.478 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout masked 2024-09-13T22:27:18.481 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-09-13T22:27:18.481 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.485 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-09-13T22:27:18.486 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-09-13T22:27:18.491 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-09-13T22:27:18.491 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.495 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout enabled 2024-09-13T22:27:18.498 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout active 2024-09-13T22:27:18.498 INFO:teuthology.orchestra.run.smithi178.stdout:Unit ntp.service is enabled and running 2024-09-13T22:27:18.498 INFO:teuthology.orchestra.run.smithi178.stdout:Repeating the final host check... 2024-09-13T22:27:18.499 INFO:teuthology.orchestra.run.smithi178.stdout:docker (/usr/bin/docker) is present 2024-09-13T22:27:18.499 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl is present 2024-09-13T22:27:18.499 INFO:teuthology.orchestra.run.smithi178.stdout:lvcreate is present 2024-09-13T22:27:18.502 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T22:27:18.502 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T22:27:18.505 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T22:27:18.505 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.508 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-09-13T22:27:18.509 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-09-13T22:27:18.513 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-09-13T22:27:18.513 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.518 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-09-13T22:27:18.518 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout masked 2024-09-13T22:27:18.521 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-09-13T22:27:18.521 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.524 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-09-13T22:27:18.525 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-09-13T22:27:18.529 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-09-13T22:27:18.529 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout inactive 2024-09-13T22:27:18.534 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout enabled 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stdout active 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Unit ntp.service is enabled and running 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Host looks OK 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Cluster fsid: 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Acquiring lock 140087134419792 on /run/cephadm/03ae43e2-721f-11ef-bceb-c7b262605968.lock 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Lock 140087134419792 acquired on /run/cephadm/03ae43e2-721f-11ef-bceb-c7b262605968.lock 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying IP 172.21.15.178 port 6789 ... 2024-09-13T22:27:18.538 INFO:teuthology.orchestra.run.smithi178.stdout:Base mon IP(s) is [172.21.15.178:6789], mon addrv is [v1:172.21.15.178:6789] 2024-09-13T22:27:18.541 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 2024-09-13T22:27:18.541 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.178 metric 100 2024-09-13T22:27:18.541 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 158.69.67.47 via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.178 metric 100 2024-09-13T22:27:18.541 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-09-13T22:27:18.541 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.178 metric 100 2024-09-13T22:27:18.542 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 172.21.0.1 dev enp3s0f1 proto dhcp scope link src 172.21.15.178 metric 100 2024-09-13T22:27:18.542 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 172.21.0.2 dev enp3s0f1 proto dhcp scope link src 172.21.15.178 metric 100 2024-09-13T22:27:18.542 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.178 metric 100 2024-09-13T22:27:18.543 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T22:27:18.544 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-09-13T22:27:18.544 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1784sec pref medium 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7ef9/64 scope link 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:Mon IP `172.21.15.178` is in CIDR network `172.21.0.0/20` 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:Mon IP `172.21.15.178` is in CIDR network `172.21.0.1/32` 2024-09-13T22:27:18.546 INFO:teuthology.orchestra.run.smithi178.stdout:Mon IP `172.21.15.178` is in CIDR network `172.21.0.2/32` 2024-09-13T22:27:18.547 INFO:teuthology.orchestra.run.smithi178.stdout:Mon IP `172.21.15.178` is in CIDR network `172.21.15.254/32` 2024-09-13T22:27:18.547 INFO:teuthology.orchestra.run.smithi178.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.1/32', '172.21.0.2/32', '172.21.15.254/32'] 2024-09-13T22:27:18.547 INFO:teuthology.orchestra.run.smithi178.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T22:27:18.547 INFO:teuthology.orchestra.run.smithi178.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50... 2024-09-13T22:27:18.794 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/docker: stdout ecab7be58963a5bd03910cd73eef66405d7fdd50: Pulling from ceph-ci/ceph 2024-09-13T22:27:18.794 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/docker: stdout Digest: sha256:b893710f6933dd3672a2a633ce5ac92081563259207d21cd52290129179ca808 2024-09-13T22:27:18.794 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:27:18.795 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 2024-09-13T22:27:20.398 INFO:teuthology.orchestra.run.smithi178.stdout:ceph: stdout ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable) 2024-09-13T22:27:20.398 INFO:teuthology.orchestra.run.smithi178.stdout:Ceph version: ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable) 2024-09-13T22:27:20.398 INFO:teuthology.orchestra.run.smithi178.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T22:27:21.827 INFO:teuthology.orchestra.run.smithi178.stdout:stat: stdout 167 167 2024-09-13T22:27:21.828 INFO:teuthology.orchestra.run.smithi178.stdout:Creating initial keys... 2024-09-13T22:27:23.210 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-authtool: stdout AQBKvORmmDH5KRAALVoolztYgm12JnodzjpVkA== 2024-09-13T22:27:24.597 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-authtool: stdout AQBMvORmm8QLBBAAl6oGNGEOwW4yKD/gTadusg== 2024-09-13T22:27:25.954 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-authtool: stdout AQBNvORm3HtdFxAA/MoT7uQeynWGydfwN5+xwA== 2024-09-13T22:27:25.955 INFO:teuthology.orchestra.run.smithi178.stdout:Creating initial monmap... 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool for a [v1:172.21.15.178:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:setting min_mon_release = quincy 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/monmaptool: set fsid to 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:27.410 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T22:27:27.411 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:27:27.411 INFO:teuthology.orchestra.run.smithi178.stdout:Creating mon... 2024-09-13T22:27:28.835 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.270+0000 7f37ddf0cc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.270+0000 7f37ddf0cc80 1 imported monmap: 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-09-13T22:27:26.887333+0000 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.270+0000 7f37ddf0cc80 0 /usr/bin/ceph-mon: set fsid to 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:28.836 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: RocksDB version: 7.9.2 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Git sha 0 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Compile date 2024-09-13 16:59:10 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: DB SUMMARY 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: DB Session ID: LOFUGD419AH8OXL8NGA3 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.error_if_exists: 0 2024-09-13T22:27:28.837 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.create_if_missing: 1 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.paranoid_checks: 1 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.env: 0x5609a0ee1940 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.info_log: 0x5609a1dd6bc0 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.statistics: (nil) 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.use_fsync: 0 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_log_file_size: 0 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T22:27:28.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.allow_fallocate: 1 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.use_direct_reads: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.db_log_dir: 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.wal_dir: 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T22:27:28.839 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.write_buffer_manager: 0x5609a1ddd4a0 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T22:27:28.840 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.unordered_write: 0 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.row_cache: None 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.wal_filter: None 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.two_write_queues: 0 2024-09-13T22:27:28.841 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.wal_compression: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.atomic_flush: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.log_readahead_size: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-13T22:27:28.842 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_background_jobs: 2 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_background_compactions: -1 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_subcompactions: 1 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_open_files: -1 2024-09-13T22:27:28.843 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_background_flushes: -1 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Compression algorithms supported: 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kZSTD supported: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kXpressCompression supported: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kBZip2Compression supported: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kLZ4Compression supported: 1 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kZlibCompression supported: 1 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-13T22:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: kSnappyCompression supported: 1 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.845 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.merge_operator: 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_filter: None 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5609a1dd6800) 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-09-13T22:27:28.846 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x5609a1dc11f0 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-09-13T22:27:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-09-13T22:27:28.848 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression: NoCompression 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.num_levels: 7 2024-09-13T22:27:28.849 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-13T22:27:28.850 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T22:27:28.851 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T22:27:28.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.inplace_update_support: 0 2024-09-13T22:27:28.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.bloom_locality: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.max_successive_merges: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.ttl: 2592000 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T22:27:28.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.enable_blob_files: false 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.min_blob_size: 0 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-13T22:27:28.855 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cd009c98-c068-4ad0-a42a-d66be345defd 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.274+0000 7f37ddf0cc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37ddf0cc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5609a1deee00 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37ddf0cc80 4 rocksdb: DB pointer 0x5609a1ec8000 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37d5697640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37d5697640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:27:28.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T22:27:28.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x5609a1dc11f0#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-09-13T22:27:28.858 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-09-13T22:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr 2024-09-13T22:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37ddf0cc80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-09-13T22:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37ddf0cc80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-09-13T22:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-13T22:27:28.278+0000 7f37ddf0cc80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-09-13T22:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout:create mon.a on 2024-09-13T22:27:29.078 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-09-13T22:27:29.282 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-09-13T22:27:29.504 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-03ae43e2-721f-11ef-bceb-c7b262605968.target -> /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968.target. 2024-09-13T22:27:29.504 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-03ae43e2-721f-11ef-bceb-c7b262605968.target -> /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968.target. 2024-09-13T22:27:29.806 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a 2024-09-13T22:27:29.806 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to reset failed state of unit ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-13T22:27:29.992 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968.target.wants/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a.service -> /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service. 2024-09-13T22:27:30.000 INFO:teuthology.orchestra.run.smithi178.stdout:firewalld does not appear to be present 2024-09-13T22:27:30.001 INFO:teuthology.orchestra.run.smithi178.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T22:27:30.001 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for mon to start... 2024-09-13T22:27:30.001 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for mon... 2024-09-13T22:27:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:31 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.479398+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T22:27:32.238 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T22:27:32.238 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout id: 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:32.238 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T22:27:32.238 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout services: 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.176552s) 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout data: 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:mon is available 2024-09-13T22:27:32.239 INFO:teuthology.orchestra.run.smithi178.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T22:27:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499322+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499508+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499519+0000 mon.a (mon.0) 4 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499526+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-09-13T22:27:26.887333+0000 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499532+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499538+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499544+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.499551+0000 mon.a (mon.0) 9 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.501899+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.504140+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: cluster 2024-09-13T22:27:31.504568+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-09-13T22:27:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:32 smithi178 bash[22355]: audit 2024-09-13T22:27:31.675830+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.178:0/2423162236' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-13T22:27:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:33 smithi178 bash[22355]: audit 2024-09-13T22:27:33.351792+0000 mon.a (mon.0) 14 : audit [INF] from='client.? v1:172.21.15.178:0/96584854' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T22:27:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:33 smithi178 bash[22355]: audit 2024-09-13T22:27:33.356712+0000 mon.a (mon.0) 15 : audit [INF] from='client.? v1:172.21.15.178:0/96584854' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T22:27:33.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:33.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T22:27:33.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout fsid = 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:33.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.178:6789] 2024-09-13T22:27:33.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T22:27:33.852 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T22:27:33.853 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T22:27:33.854 INFO:teuthology.orchestra.run.smithi178.stdout:Generating new minimal ceph.conf... 2024-09-13T22:27:35.408 INFO:teuthology.orchestra.run.smithi178.stdout:Restarting the monitor... 2024-09-13T22:27:35.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:34 smithi178 bash[22355]: audit 2024-09-13T22:27:34.929859+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? v1:172.21.15.178:0/1852010625' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:27:35.668 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:35 smithi178 systemd[1]: Stopping Ceph mon.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T22:27:35.668 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:35 smithi178 bash[22355]: debug 2024-09-13T22:27:35.538+0000 7f5aaa8d5640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-13T22:27:35.668 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:35 smithi178 bash[22355]: debug 2024-09-13T22:27:35.538+0000 7f5aaa8d5640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T22:27:36.198 INFO:teuthology.orchestra.run.smithi178.stdout:Setting public_network to 172.21.0.0/20,172.21.15.254/32,172.21.0.1/32,172.21.0.2/32 in mon config section 2024-09-13T22:27:36.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:36 smithi178 bash[22740]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-mon-a 2024-09-13T22:27:36.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:36 smithi178 bash[22800]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-mon-a 2024-09-13T22:27:36.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:36 smithi178 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T22:27:36.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:36 smithi178 systemd[1]: Stopped Ceph mon.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:27:36.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:36 smithi178 systemd[1]: Started Ceph mon.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:27:37.774 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.550+0000 7f8b9528ec80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-13T22:27:37.774 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.550+0000 7f8b9528ec80 0 ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable), process ceph-mon, pid 7 2024-09-13T22:27:37.774 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.550+0000 7f8b9528ec80 0 pidfile_write: ignore empty --pid-file 2024-09-13T22:27:37.774 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.554+0000 7f8b9528ec80 0 load: jerasure load: lrc 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: RocksDB version: 7.9.2 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Git sha 0 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Compile date 2024-09-13 16:59:10 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: DB SUMMARY 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: DB Session ID: Y9P3L0S7UI8JUYCTIUCR 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: CURRENT file: CURRENT 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: IDENTITY file: IDENTITY 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92865 ; 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.error_if_exists: 0 2024-09-13T22:27:37.775 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.create_if_missing: 0 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.paranoid_checks: 1 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.env: 0x559f88b82940 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.info_log: 0x559f89801680 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.statistics: (nil) 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.use_fsync: 0 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_log_file_size: 0 2024-09-13T22:27:37.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.allow_fallocate: 1 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.use_direct_reads: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.db_log_dir: 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.wal_dir: 2024-09-13T22:27:37.777 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.write_buffer_manager: 0x559f89810dc0 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-13T22:27:37.778 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.unordered_write: 0 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.row_cache: None 2024-09-13T22:27:37.779 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.wal_filter: None 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.two_write_queues: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.wal_compression: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.atomic_flush: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.log_readahead_size: 0 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T22:27:37.780 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_background_jobs: 2 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_background_compactions: -1 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_subcompactions: 1 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T22:27:37.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_open_files: -1 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_background_flushes: -1 2024-09-13T22:27:37.782 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Compression algorithms supported: 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kZSTD supported: 0 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kXpressCompression supported: 0 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kBZip2Compression supported: 0 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kLZ4Compression supported: 1 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kZlibCompression supported: 1 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: kSnappyCompression supported: 1 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T22:27:37.783 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-13T22:27:37.784 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T22:27:37.784 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.merge_operator: 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_filter: None 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559f898012c0) 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cache_index_and_filter_blocks: 1 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: pin_top_level_index_and_filter: 1 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: index_type: 0 2024-09-13T22:27:37.785 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: data_block_index_type: 0 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: index_shortening: 1 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: data_block_hash_table_util_ratio: 0.750000 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: checksum: 4 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: no_block_cache: 0 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_cache: 0x559f897f71f0 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_cache_name: BinnedLRUCache 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_cache_options: 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: capacity : 536870912 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: num_shard_bits : 4 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: strict_capacity_limit : 0 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: high_pri_pool_ratio: 0.000 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_cache_compressed: (nil) 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: persistent_cache: (nil) 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_size: 4096 2024-09-13T22:27:37.786 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_size_deviation: 10 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_restart_interval: 16 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: index_block_restart_interval: 1 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: metadata_block_size: 4096 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: partition_filters: 0 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: use_delta_encoding: 1 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: filter_policy: bloomfilter 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: whole_key_filtering: 1 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: verify_compression: 0 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: read_amp_bytes_per_bit: 0 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: format_version: 5 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: enable_index_compression: 1 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: block_align: 0 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: max_auto_readahead_size: 262144 2024-09-13T22:27:37.787 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: prepopulate_block_cache: 0 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: initial_auto_readahead_size: 8192 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: num_file_reads_for_auto_readahead: 2 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression: NoCompression 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.num_levels: 7 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T22:27:37.788 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T22:27:37.789 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T22:27:37.790 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T22:27:37.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.inplace_update_support: 0 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.bloom_locality: 0 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.max_successive_merges: 0 2024-09-13T22:27:37.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.ttl: 2592000 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.enable_blob_files: false 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.min_blob_size: 0 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T22:27:37.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cd009c98-c068-4ad0-a42a-d66be345defd 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266457560295, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266457561311, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88777, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 286, "table_properties": {"data_size": 86839, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11865, "raw_average_key_size": 48, "raw_value_size": 80318, "raw_average_value_size": 327, "num_data_blocks": 11, "num_entries": 245, "num_filter_entries": 245, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726266457, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cd009c98-c068-4ad0-a42a-d66be345defd", "db_session_id": "Y9P3L0S7UI8JUYCTIUCR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266457561439, "job": 1, "event": "recovery_finished"} 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T22:27:37.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559f89822e00 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.558+0000 7f8b9528ec80 4 rocksdb: DB pointer 0x559f89922000 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b8b057640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b8b057640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: ** DB Stats ** 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: ** Compaction Stats [default] ** 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:27:37.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: L0 2/0 88.52 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Sum 2/0 88.52 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: ** Compaction Stats [default] ** 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T22:27:37.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: AddFile(Keys): cumulative 0, interval 0 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Cumulative compaction: 0.00 GB write, 24.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Interval compaction: 0.00 GB write, 24.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Block cache BinnedLRUCache@0x559f897f71f0#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: ** File Read Latency Histogram By Level [default] ** 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 starting mon.a rank 0 at public addrs v1:172.21.15.178:6789/0 at bind addrs v1:172.21.15.178:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 1 mon.a@-1(???) e1 preinit fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 mon.a@-1(???).mds e1 new map 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 mon.a@-1(???).mds e1 print_map 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: e1 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: btime 2024-09-13T22:27:31:499941+0000 2024-09-13T22:27:37.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: legacy client fscid: -1 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: No filesystems configured 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: debug 2024-09-13T22:27:37.562+0000 7f8b9528ec80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591378+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591456+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591464+0000 mon.a (mon.0) 3 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:37.798 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591475+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-09-13T22:27:26.887333+0000 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591491+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591501+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591511+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591521+0000 mon.a (mon.0) 8 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.591984+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.592013+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-13T22:27:37.799 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:37 smithi178 bash[22840]: cluster 2024-09-13T22:27:37.592404+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-09-13T22:27:38.354 INFO:teuthology.orchestra.run.smithi178.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T22:27:38.380 INFO:teuthology.orchestra.run.smithi178.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T22:27:38.380 INFO:teuthology.orchestra.run.smithi178.stdout:Creating mgr... 2024-09-13T22:27:38.380 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T22:27:38.381 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T22:27:38.686 INFO:teuthology.orchestra.run.smithi178.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y 2024-09-13T22:27:38.686 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Failed to reset failed state of unit ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-13T22:27:38.762 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:38 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:27:38.900 INFO:teuthology.orchestra.run.smithi178.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968.target.wants/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y.service -> /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service. 2024-09-13T22:27:38.907 INFO:teuthology.orchestra.run.smithi178.stdout:firewalld does not appear to be present 2024-09-13T22:27:38.907 INFO:teuthology.orchestra.run.smithi178.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T22:27:38.907 INFO:teuthology.orchestra.run.smithi178.stdout:firewalld does not appear to be present 2024-09-13T22:27:38.907 INFO:teuthology.orchestra.run.smithi178.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-13T22:27:38.907 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for mgr to start... 2024-09-13T22:27:38.907 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for mgr... 2024-09-13T22:27:39.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:38 smithi178 bash[22840]: audit 2024-09-13T22:27:37.762957+0000 mon.a (mon.0) 12 : audit [INF] from='client.? v1:172.21.15.178:0/3076641377' entity='client.admin' 2024-09-13T22:27:39.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:38 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:27:40.681 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:40 smithi178 bash[23095]: debug 2024-09-13T22:27:40.621+0000 7f5b25cb4040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T22:27:41.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:40 smithi178 bash[22840]: audit 2024-09-13T22:27:40.686459+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.178:0/1668648730' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T22:27:41.245 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 0 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-13T22:27:41.246 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T22:27:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "btime": "2024-09-13T22:27:31:499941+0000", 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T22:27:41.248 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:41.249 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T22:27:31.500775+0000", 2024-09-13T22:27:41.250 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T22:27:41.250 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:41.250 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T22:27:41.250 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:27:41.250 INFO:teuthology.orchestra.run.smithi178.stdout:mgr not available, waiting (1/15)... 2024-09-13T22:27:42.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:41 smithi178 bash[23095]: debug 2024-09-13T22:27:41.813+0000 7f5b25cb4040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T22:27:42.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:41 smithi178 bash[23095]: debug 2024-09-13T22:27:41.877+0000 7f5b25cb4040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T22:27:42.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:42 smithi178 bash[23095]: debug 2024-09-13T22:27:42.025+0000 7f5b25cb4040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T22:27:42.931 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:42 smithi178 bash[23095]: debug 2024-09-13T22:27:42.669+0000 7f5b25cb4040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T22:27:42.931 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:42 smithi178 bash[23095]: debug 2024-09-13T22:27:42.737+0000 7f5b25cb4040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T22:27:42.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:42 smithi178 bash[23095]: debug 2024-09-13T22:27:42.805+0000 7f5b25cb4040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T22:27:43.339 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:42 smithi178 bash[23095]: debug 2024-09-13T22:27:42.989+0000 7f5b25cb4040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T22:27:43.340 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.065+0000 7f5b25cb4040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T22:27:43.340 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.129+0000 7f5b25cb4040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T22:27:43.650 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.385+0000 7f5b25cb4040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T22:27:43.651 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.453+0000 7f5b25cb4040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T22:27:43.931 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.645+0000 7f5b25cb4040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T22:27:43.931 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.725+0000 7f5b25cb4040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T22:27:44.278 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:43 smithi178 bash[23095]: debug 2024-09-13T22:27:43.973+0000 7f5b25cb4040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T22:27:44.278 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:44 smithi178 bash[23095]: debug 2024-09-13T22:27:44.149+0000 7f5b25cb4040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:27:44.555 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:44 smithi178 bash[23095]: debug 2024-09-13T22:27:44.321+0000 7f5b25cb4040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T22:27:44.555 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:44 smithi178 bash[23095]: debug 2024-09-13T22:27:44.401+0000 7f5b25cb4040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T22:27:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:44 smithi178 bash[22840]: audit 2024-09-13T22:27:44.544957+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? v1:172.21.15.178:0/3684718271' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T22:27:44.931 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:44 smithi178 bash[23095]: debug 2024-09-13T22:27:44.585+0000 7f5b25cb4040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:27:45.046 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:45.046 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:27:45.046 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 0 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-13T22:27:45.047 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T22:27:45.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T22:27:45.052 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "btime": "2024-09-13T22:27:31:499941+0000", 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.053 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T22:27:31.500775+0000", 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:45.054 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T22:27:45.055 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:27:45.055 INFO:teuthology.orchestra.run.smithi178.stdout:mgr not available, waiting (2/15)... 2024-09-13T22:27:45.681 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:45 smithi178 bash[23095]: debug 2024-09-13T22:27:45.393+0000 7f5b25cb4040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T22:27:45.682 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:45 smithi178 bash[23095]: debug 2024-09-13T22:27:45.465+0000 7f5b25cb4040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T22:27:46.149 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:45 smithi178 bash[23095]: debug 2024-09-13T22:27:45.689+0000 7f5b25cb4040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T22:27:46.149 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:45 smithi178 bash[23095]: debug 2024-09-13T22:27:45.761+0000 7f5b25cb4040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T22:27:46.431 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[23095]: debug 2024-09-13T22:27:46.145+0000 7f5b25cb4040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T22:27:46.431 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[23095]: debug 2024-09-13T22:27:46.225+0000 7f5b25cb4040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T22:27:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: cluster 2024-09-13T22:27:46.248780+0000 mon.a (mon.0) 15 : cluster [INF] Activating manager daemon y 2024-09-13T22:27:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: cluster 2024-09-13T22:27:46.255893+0000 mon.a (mon.0) 16 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00729031s) 2024-09-13T22:27:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.257099+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:27:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.257286+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.257443+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.257595+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.257763+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.257912+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.258070+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.258424+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T22:27:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: cluster 2024-09-13T22:27:46.285614+0000 mon.a (mon.0) 25 : cluster [INF] Manager daemon y is now available 2024-09-13T22:27:46.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.297132+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:27:46.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:46 smithi178 bash[22840]: audit 2024-09-13T22:27:46.299152+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:27:47.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:47 smithi178 bash[22840]: audit 2024-09-13T22:27:46.301572+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' 2024-09-13T22:27:47.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:47 smithi178 bash[22840]: audit 2024-09-13T22:27:46.305334+0000 mon.a (mon.0) 29 : audit [INF] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' 2024-09-13T22:27:47.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:47 smithi178 bash[22840]: audit 2024-09-13T22:27:46.307049+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.14100 v1:172.21.15.178:0/4119380179' entity='mgr.y' 2024-09-13T22:27:47.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:47 smithi178 bash[22840]: cluster 2024-09-13T22:27:47.262726+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e3: y(active, since 1.0141s) 2024-09-13T22:27:49.196 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:49.196 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:27:49.196 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:27:49.196 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T22:27:49.196 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T22:27:49.196 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 0 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T22:27:49.198 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:49.199 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-09-13T22:27:49.199 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T22:27:49.199 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T22:27:49.200 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T22:27:49.201 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T22:27:49.201 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T22:27:49.201 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T22:27:49.203 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "btime": "2024-09-13T22:27:31:499941+0000", 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T22:27:49.204 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T22:27:49.205 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T22:27:49.205 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ], 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T22:27:31.500775+0000", 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout }, 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T22:27:49.206 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:27:49.207 INFO:teuthology.orchestra.run.smithi178.stdout:mgr is available 2024-09-13T22:27:49.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:49 smithi178 bash[22840]: cluster 2024-09-13T22:27:48.310821+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-09-13T22:27:49.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:49 smithi178 bash[22840]: audit 2024-09-13T22:27:48.670998+0000 mon.a (mon.0) 33 : audit [DBG] from='client.? v1:172.21.15.178:0/2116205778' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T22:27:50.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:50 smithi178 bash[22840]: audit 2024-09-13T22:27:50.529639+0000 mon.a (mon.0) 34 : audit [INF] from='client.? v1:172.21.15.178:0/990855216' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T22:27:51.096 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:51.096 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T22:27:51.096 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout fsid = 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:27:51.096 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.178:6789] 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 2024-09-13T22:27:51.097 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T22:27:51.098 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T22:27:51.098 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T22:27:51.098 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling cephadm module... 2024-09-13T22:27:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:52 smithi178 bash[22840]: audit 2024-09-13T22:27:52.552876+0000 mon.a (mon.0) 35 : audit [INF] from='client.? v1:172.21.15.178:0/2880938597' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-13T22:27:52.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:52 smithi178 bash[23095]: ignoring --setuser ceph since I am not root 2024-09-13T22:27:52.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:52 smithi178 bash[23095]: ignoring --setgroup ceph since I am not root 2024-09-13T22:27:52.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:52 smithi178 bash[23095]: debug 2024-09-13T22:27:52.761+0000 7f5995bd0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T22:27:53.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:53 smithi178 bash[22840]: audit 2024-09-13T22:27:52.604318+0000 mon.a (mon.0) 36 : audit [INF] from='client.? v1:172.21.15.178:0/2880938597' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T22:27:53.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:53 smithi178 bash[22840]: cluster 2024-09-13T22:27:52.609586+0000 mon.a (mon.0) 37 : cluster [DBG] mgrmap e5: y(active, since 6s) 2024-09-13T22:27:54.202 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:53 smithi178 bash[23095]: debug 2024-09-13T22:27:53.977+0000 7f5995bd0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T22:27:54.202 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:54 smithi178 bash[23095]: debug 2024-09-13T22:27:54.045+0000 7f5995bd0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T22:27:54.532 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:54 smithi178 bash[23095]: debug 2024-09-13T22:27:54.197+0000 7f5995bd0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T22:27:54.860 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:54 smithi178 bash[22840]: audit 2024-09-13T22:27:54.519920+0000 mon.a (mon.0) 38 : audit [DBG] from='client.? v1:172.21.15.178:0/504369533' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T22:27:55.067 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for the mgr to restart... 2024-09-13T22:27:55.068 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for mgr epoch 5... 2024-09-13T22:27:55.158 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:54 smithi178 bash[23095]: debug 2024-09-13T22:27:54.857+0000 7f5995bd0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T22:27:55.158 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:54 smithi178 bash[23095]: debug 2024-09-13T22:27:54.925+0000 7f5995bd0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T22:27:55.158 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:54 smithi178 bash[23095]: debug 2024-09-13T22:27:54.993+0000 7f5995bd0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T22:27:55.431 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.185+0000 7f5995bd0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T22:27:55.432 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.261+0000 7f5995bd0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T22:27:55.432 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.329+0000 7f5995bd0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T22:27:55.852 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.585+0000 7f5995bd0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T22:27:55.852 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.653+0000 7f5995bd0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T22:27:55.852 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.849+0000 7f5995bd0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T22:27:56.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:55 smithi178 bash[23095]: debug 2024-09-13T22:27:55.925+0000 7f5995bd0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T22:27:56.527 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:56 smithi178 bash[23095]: debug 2024-09-13T22:27:56.185+0000 7f5995bd0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T22:27:56.527 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:56 smithi178 bash[23095]: debug 2024-09-13T22:27:56.361+0000 7f5995bd0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:27:56.783 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:56 smithi178 bash[23095]: debug 2024-09-13T22:27:56.521+0000 7f5995bd0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T22:27:56.783 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:56 smithi178 bash[23095]: debug 2024-09-13T22:27:56.601+0000 7f5995bd0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T22:27:57.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:56 smithi178 bash[23095]: debug 2024-09-13T22:27:56.781+0000 7f5995bd0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:27:57.880 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:57 smithi178 bash[23095]: debug 2024-09-13T22:27:57.581+0000 7f5995bd0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T22:27:57.881 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:57 smithi178 bash[23095]: debug 2024-09-13T22:27:57.653+0000 7f5995bd0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T22:27:58.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:57 smithi178 bash[23095]: debug 2024-09-13T22:27:57.877+0000 7f5995bd0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T22:27:58.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:57 smithi178 bash[23095]: debug 2024-09-13T22:27:57.949+0000 7f5995bd0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[23095]: debug 2024-09-13T22:27:58.337+0000 7f5995bd0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[23095]: debug 2024-09-13T22:27:58.413+0000 7f5995bd0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: cluster 2024-09-13T22:27:58.436349+0000 mon.a (mon.0) 39 : cluster [INF] Active manager daemon y restarted 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: cluster 2024-09-13T22:27:58.436858+0000 mon.a (mon.0) 40 : cluster [INF] Activating manager daemon y 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: cluster 2024-09-13T22:27:58.441087+0000 mon.a (mon.0) 41 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: cluster 2024-09-13T22:27:58.441216+0000 mon.a (mon.0) 42 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00453411s) 2024-09-13T22:27:58.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: audit 2024-09-13T22:27:58.444091+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:27:58.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: audit 2024-09-13T22:27:58.445640+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T22:27:58.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: audit 2024-09-13T22:27:58.446437+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:27:58.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: audit 2024-09-13T22:27:58.446807+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:27:58.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: audit 2024-09-13T22:27:58.447056+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:27:58.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: cluster 2024-09-13T22:27:58.472733+0000 mon.a (mon.0) 48 : cluster [INF] Manager daemon y is now available 2024-09-13T22:27:58.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:58 smithi178 bash[22840]: audit 2024-09-13T22:27:58.487057+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:27:59.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: cephadm 2024-09-13T22:27:58.484554+0000 mgr.y (mgr.14118) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-09-13T22:27:59.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.492165+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:27:59.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.495336+0000 mon.a (mon.0) 51 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:27:59.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.498318+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:27:59.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.507171+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:27:59.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.509728+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:27:59.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.923661+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:27:59.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: audit 2024-09-13T22:27:58.928424+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:27:59.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:27:59 smithi178 bash[22840]: cluster 2024-09-13T22:27:59.447010+0000 mon.a (mon.0) 57 : cluster [DBG] mgrmap e7: y(active, since 1.01032s) 2024-09-13T22:28:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:28:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-13T22:28:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T22:28:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:28:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:mgr epoch 5 is available 2024-09-13T22:28:00.048 INFO:teuthology.orchestra.run.smithi178.stdout:Setting orchestrator backend to cephadm... 2024-09-13T22:28:00.432 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: debug 2024-09-13T22:28:00.029+0000 7f58fcf0f640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:22:28:00] ENGINE Error in HTTPServer.serve 2024-09-13T22:28:00.432 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: Traceback (most recent call last): 2024-09-13T22:28:00.432 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: self._connections.run(self.expiration_interval) 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: self._run(expiration_interval) 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:28:00.433 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: s = self.context.wrap_socket( 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: return self.sslsocket_class._create( 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: self.do_handshake() 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: self._sslobj.do_handshake() 2024-09-13T22:28:00.434 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[23095]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:28:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[22840]: audit 2024-09-13T22:27:59.445265+0000 mgr.y (mgr.14118) 2 : audit [DBG] from='client.14122 v1:172.21.15.178:0/2605437445' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T22:28:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[22840]: audit 2024-09-13T22:27:59.459605+0000 mgr.y (mgr.14118) 3 : audit [DBG] from='client.14122 v1:172.21.15.178:0/2605437445' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T22:28:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:00 smithi178 bash[22840]: audit 2024-09-13T22:28:00.030311+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: cephadm 2024-09-13T22:27:59.806821+0000 mgr.y (mgr.14118) 4 : cephadm [INF] [13/Sep/2024:22:27:59] ENGINE Bus STARTING 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: cephadm 2024-09-13T22:27:59.908420+0000 mgr.y (mgr.14118) 5 : cephadm [INF] [13/Sep/2024:22:27:59] ENGINE Serving on http://172.21.15.178:8765 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: cephadm 2024-09-13T22:28:00.029124+0000 mgr.y (mgr.14118) 6 : cephadm [INF] [13/Sep/2024:22:28:00] ENGINE Serving on https://172.21.15.178:7150 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: cephadm 2024-09-13T22:28:00.029337+0000 mgr.y (mgr.14118) 7 : cephadm [INF] [13/Sep/2024:22:28:00] ENGINE Bus STARTED 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: cephadm 2024-09-13T22:28:00.033170+0000 mgr.y (mgr.14118) 8 : cephadm [ERR] [13/Sep/2024:22:28:00] ENGINE Error in HTTPServer.serve 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: Traceback (most recent call last): 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: self._connections.run(self.expiration_interval) 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:28:01.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: self._run(expiration_interval) 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: s = self.context.wrap_socket( 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: return self.sslsocket_class._create( 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: self.do_handshake() 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: self._sslobj.do_handshake() 2024-09-13T22:28:01.796 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:28:01.797 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:01 smithi178 bash[22840]: cluster 2024-09-13T22:28:00.503942+0000 mon.a (mon.0) 59 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-09-13T22:28:02.917 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:02 smithi178 bash[22840]: audit 2024-09-13T22:28:01.592309+0000 mgr.y (mgr.14118) 9 : audit [DBG] from='client.14130 v1:172.21.15.178:0/1814230857' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:02.917 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:02 smithi178 bash[22840]: audit 2024-09-13T22:28:01.600363+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:02.917 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:02 smithi178 bash[22840]: audit 2024-09-13T22:28:01.619595+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:28:04.022 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T22:28:04.022 INFO:teuthology.orchestra.run.smithi178.stdout:Generating ssh key... 2024-09-13T22:28:04.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:03 smithi178 bash[22840]: audit 2024-09-13T22:28:03.431569+0000 mgr.y (mgr.14118) 10 : audit [DBG] from='client.14132 v1:172.21.15.178:0/4124886717' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:05.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: Generating public/private rsa key pair. 2024-09-13T22:28:05.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: Your identification has been saved in /tmp/tmpfr3be37w/key 2024-09-13T22:28:05.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: Your public key has been saved in /tmp/tmpfr3be37w/key.pub 2024-09-13T22:28:05.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: The key fingerprint is: 2024-09-13T22:28:05.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: SHA256:kcy/XcyS/5+js/lZH84rcGv+5eJEeUHQMp83FYS0Q60 ceph-03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: The key's randomart image is: 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: +---[RSA 3072]----+ 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | .+*+.| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | o . .+oo.| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | = o=.o| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | o E.=o| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | S . o * +| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | + * . | 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | . + +.o| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | *=+B| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: | o*OXO| 2024-09-13T22:28:05.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:05 smithi178 bash[23095]: +----[SHA256]-----+ 2024-09-13T22:28:06.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:06 smithi178 bash[22840]: audit 2024-09-13T22:28:05.300358+0000 mgr.y (mgr.14118) 11 : audit [DBG] from='client.14134 v1:172.21.15.178:0/1481733566' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:06.791 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:06 smithi178 bash[22840]: cephadm 2024-09-13T22:28:05.300657+0000 mgr.y (mgr.14118) 12 : cephadm [INF] Generating ssh key... 2024-09-13T22:28:06.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:06 smithi178 bash[22840]: audit 2024-09-13T22:28:05.499504+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:06.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:06 smithi178 bash[22840]: audit 2024-09-13T22:28:05.502411+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:07.838 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeyMILlcc8AAHLI8q0489XmVkEHTSNkt3r2Bj3CrKhTygYeTPL2c9blPo9AJS9QJ7gCp/SAlggO1TprGPoDCKnPtoPePnOJT9qeDPJI0IAnlrvZGk+QPni6rWpWi/t5w2GO9JqSjedAxQVbI7A2jKkmN7IdNSLM3NyZQHpUanbND8jnlRgq7f3G327FmH1TW1LJfFd1tRMOwzpHkJokQ51qeI9/OtreVn1ANzYTvgzbHS+0VtiSYxwlDrY5DhKrv+dAkaqScVM+wKjcfecJg9U/z3RTtsNFqGHSmesEYAFcBOmmTQPKnz4Xb7tuU4TtGIbcCVdQyrMcdikJxlKDE7wdZD6yF2mBU7EwR1998qJYSZomC6M0bGEu5vybI1yuvkrY2CRGJ0NMfmFG+bgYo6yxzkQFjBdlj3sBZ4GK1mNtUrbaXxTHWJkQfBe9iXqQWXCILA1p0T7HwcojDtQLNu3Lhg9raW8JSwUMid6UOqAZjSjykP5BKCru5mxenr9cvc= ceph-03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:28:07.838 INFO:teuthology.orchestra.run.smithi178.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T22:28:07.838 INFO:teuthology.orchestra.run.smithi178.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T22:28:07.838 INFO:teuthology.orchestra.run.smithi178.stdout:Adding host smithi178... 2024-09-13T22:28:08.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:07 smithi178 bash[22840]: audit 2024-09-13T22:28:07.296567+0000 mgr.y (mgr.14118) 13 : audit [DBG] from='client.14136 v1:172.21.15.178:0/4263437060' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:10.054 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:09 smithi178 bash[22840]: audit 2024-09-13T22:28:09.206612+0000 mgr.y (mgr.14118) 14 : audit [DBG] from='client.14138 v1:172.21.15.178:0/2939057200' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "addr": "172.21.15.178", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:10.844 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:10 smithi178 bash[22840]: cephadm 2024-09-13T22:28:10.192803+0000 mgr.y (mgr.14118) 15 : cephadm [INF] Deploying cephadm binary to smithi178 2024-09-13T22:28:13.993 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout Added host 'smithi178' with addr '172.21.15.178' 2024-09-13T22:28:13.994 INFO:teuthology.orchestra.run.smithi178.stdout:Deploying unmanaged mon service... 2024-09-13T22:28:14.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:14 smithi178 bash[22840]: audit 2024-09-13T22:28:13.303467+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:14.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:14 smithi178 bash[22840]: cephadm 2024-09-13T22:28:13.303888+0000 mgr.y (mgr.14118) 16 : cephadm [INF] Added host smithi178 2024-09-13T22:28:14.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:14 smithi178 bash[22840]: audit 2024-09-13T22:28:13.304589+0000 mon.a (mon.0) 65 : audit [DBG] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:28:16.951 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T22:28:16.951 INFO:teuthology.orchestra.run.smithi178.stdout:Deploying unmanaged mgr service... 2024-09-13T22:28:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:17 smithi178 bash[22840]: audit 2024-09-13T22:28:16.122779+0000 mgr.y (mgr.14118) 17 : audit [DBG] from='client.14140 v1:172.21.15.178:0/2602169634' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:17 smithi178 bash[22840]: cephadm 2024-09-13T22:28:16.124185+0000 mgr.y (mgr.14118) 18 : cephadm [INF] Saving service mon spec with placement count:5 2024-09-13T22:28:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:17 smithi178 bash[22840]: audit 2024-09-13T22:28:16.126677+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:17 smithi178 bash[22840]: audit 2024-09-13T22:28:16.979392+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:18 smithi178 bash[22840]: audit 2024-09-13T22:28:17.494410+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:18 smithi178 bash[22840]: audit 2024-09-13T22:28:18.472989+0000 mon.a (mon.0) 69 : audit [INF] from='mgr.14118 v1:172.21.15.178:0/498629585' entity='mgr.y' 2024-09-13T22:28:19.042 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T22:28:19.858 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:19 smithi178 bash[22840]: cluster 2024-09-13T22:28:18.448392+0000 mgr.y (mgr.14118) 19 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:19.858 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:19 smithi178 bash[22840]: audit 2024-09-13T22:28:18.466723+0000 mgr.y (mgr.14118) 20 : audit [DBG] from='client.14142 v1:172.21.15.178:0/2919213410' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:19.858 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:19 smithi178 bash[22840]: cephadm 2024-09-13T22:28:18.468729+0000 mgr.y (mgr.14118) 21 : cephadm [INF] Saving service mgr spec with placement count:2 2024-09-13T22:28:21.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:21 smithi178 bash[22840]: audit 2024-09-13T22:28:20.344999+0000 mon.a (mon.0) 70 : audit [INF] from='client.? v1:172.21.15.178:0/3810131680' entity='client.admin' 2024-09-13T22:28:21.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:21 smithi178 bash[22840]: cluster 2024-09-13T22:28:20.448775+0000 mgr.y (mgr.14118) 22 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:22.765 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling the dashboard module... 2024-09-13T22:28:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:23 smithi178 bash[22840]: audit 2024-09-13T22:28:22.254284+0000 mon.a (mon.0) 71 : audit [INF] from='client.? v1:172.21.15.178:0/2210017107' entity='client.admin' 2024-09-13T22:28:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:23 smithi178 bash[22840]: cluster 2024-09-13T22:28:22.449096+0000 mgr.y (mgr.14118) 23 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:24 smithi178 bash[22840]: audit 2024-09-13T22:28:24.292985+0000 mon.a (mon.0) 72 : audit [INF] from='client.? v1:172.21.15.178:0/4180290250' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-13T22:28:24.682 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:24 smithi178 bash[23095]: ignoring --setuser ceph since I am not root 2024-09-13T22:28:24.682 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:24 smithi178 bash[23095]: ignoring --setgroup ceph since I am not root 2024-09-13T22:28:24.682 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:24 smithi178 bash[23095]: debug 2024-09-13T22:28:24.504+0000 7f67939cc040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T22:28:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:25 smithi178 bash[22840]: audit 2024-09-13T22:28:24.344258+0000 mon.a (mon.0) 73 : audit [INF] from='client.? v1:172.21.15.178:0/4180290250' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T22:28:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:25 smithi178 bash[22840]: cluster 2024-09-13T22:28:24.347832+0000 mon.a (mon.0) 74 : cluster [DBG] mgrmap e9: y(active, since 25s) 2024-09-13T22:28:26.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:25 smithi178 bash[23095]: debug 2024-09-13T22:28:25.684+0000 7f67939cc040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T22:28:26.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:25 smithi178 bash[23095]: debug 2024-09-13T22:28:25.748+0000 7f67939cc040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T22:28:26.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:25 smithi178 bash[23095]: debug 2024-09-13T22:28:25.904+0000 7f67939cc040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T22:28:26.911 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:26 smithi178 bash[22840]: audit 2024-09-13T22:28:26.485846+0000 mon.a (mon.0) 75 : audit [DBG] from='client.? v1:172.21.15.178:0/769392493' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T22:28:26.911 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:26 smithi178 bash[23095]: debug 2024-09-13T22:28:26.580+0000 7f67939cc040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T22:28:26.911 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:26 smithi178 bash[23095]: debug 2024-09-13T22:28:26.652+0000 7f67939cc040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T22:28:26.912 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:26 smithi178 bash[23095]: debug 2024-09-13T22:28:26.720+0000 7f67939cc040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T22:28:27.030 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:28:27.030 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T22:28:27.030 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T22:28:27.030 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T22:28:27.031 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T22:28:27.031 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:28:27.031 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for the mgr to restart... 2024-09-13T22:28:27.031 INFO:teuthology.orchestra.run.smithi178.stdout:Waiting for mgr epoch 9... 2024-09-13T22:28:27.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:26 smithi178 bash[23095]: debug 2024-09-13T22:28:26.904+0000 7f67939cc040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T22:28:27.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:26 smithi178 bash[23095]: debug 2024-09-13T22:28:26.984+0000 7f67939cc040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T22:28:27.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:27 smithi178 bash[23095]: debug 2024-09-13T22:28:27.056+0000 7f67939cc040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T22:28:27.580 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:27 smithi178 bash[23095]: debug 2024-09-13T22:28:27.312+0000 7f67939cc040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T22:28:27.580 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:27 smithi178 bash[23095]: debug 2024-09-13T22:28:27.380+0000 7f67939cc040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T22:28:27.907 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:27 smithi178 bash[23095]: debug 2024-09-13T22:28:27.576+0000 7f67939cc040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T22:28:27.908 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:27 smithi178 bash[23095]: debug 2024-09-13T22:28:27.652+0000 7f67939cc040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T22:28:28.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:27 smithi178 bash[23095]: debug 2024-09-13T22:28:27.904+0000 7f67939cc040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T22:28:28.182 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:28 smithi178 bash[23095]: debug 2024-09-13T22:28:28.088+0000 7f67939cc040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:28:28.514 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:28 smithi178 bash[23095]: debug 2024-09-13T22:28:28.252+0000 7f67939cc040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T22:28:28.514 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:28 smithi178 bash[23095]: debug 2024-09-13T22:28:28.332+0000 7f67939cc040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T22:28:29.614 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:28 smithi178 bash[23095]: debug 2024-09-13T22:28:28.508+0000 7f67939cc040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:28:29.614 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:29 smithi178 bash[23095]: debug 2024-09-13T22:28:29.312+0000 7f67939cc040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T22:28:29.614 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:29 smithi178 bash[23095]: debug 2024-09-13T22:28:29.388+0000 7f67939cc040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T22:28:29.614 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:29 smithi178 bash[23095]: debug 2024-09-13T22:28:29.608+0000 7f67939cc040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T22:28:29.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:29 smithi178 bash[23095]: debug 2024-09-13T22:28:29.680+0000 7f67939cc040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[23095]: debug 2024-09-13T22:28:30.068+0000 7f67939cc040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[23095]: debug 2024-09-13T22:28:30.144+0000 7f67939cc040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: cluster 2024-09-13T22:28:30.168657+0000 mon.a (mon.0) 76 : cluster [INF] Active manager daemon y restarted 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: cluster 2024-09-13T22:28:30.169096+0000 mon.a (mon.0) 77 : cluster [INF] Activating manager daemon y 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: cluster 2024-09-13T22:28:30.172556+0000 mon.a (mon.0) 78 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: cluster 2024-09-13T22:28:30.172685+0000 mon.a (mon.0) 79 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0037884s) 2024-09-13T22:28:30.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: audit 2024-09-13T22:28:30.175208+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:28:30.361 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: audit 2024-09-13T22:28:30.176302+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T22:28:30.361 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: audit 2024-09-13T22:28:30.177303+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:28:30.361 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: audit 2024-09-13T22:28:30.177404+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:28:30.361 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: audit 2024-09-13T22:28:30.177546+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:28:30.361 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:30 smithi178 bash[22840]: cluster 2024-09-13T22:28:30.204860+0000 mon.a (mon.0) 85 : cluster [INF] Manager daemon y is now available 2024-09-13T22:28:31.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[22840]: audit 2024-09-13T22:28:30.223959+0000 mon.a (mon.0) 86 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:28:31.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[22840]: audit 2024-09-13T22:28:30.240583+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:28:31.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[22840]: audit 2024-09-13T22:28:30.242915+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:28:31.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[22840]: cluster 2024-09-13T22:28:31.177444+0000 mon.a (mon.0) 89 : cluster [DBG] mgrmap e11: y(active, since 1.00854s) 2024-09-13T22:28:31.856 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout { 2024-09-13T22:28:31.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T22:28:31.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T22:28:31.857 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout } 2024-09-13T22:28:31.857 INFO:teuthology.orchestra.run.smithi178.stdout:mgr epoch 9 is available 2024-09-13T22:28:31.857 INFO:teuthology.orchestra.run.smithi178.stdout:Generating a dashboard self-signed certificate... 2024-09-13T22:28:31.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: debug 2024-09-13T22:28:31.572+0000 7f66f9488640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:22:28:31] ENGINE Error in HTTPServer.serve 2024-09-13T22:28:31.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: Traceback (most recent call last): 2024-09-13T22:28:31.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:28:31.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: self._connections.run(self.expiration_interval) 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: self._run(expiration_interval) 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: s = self.context.wrap_socket( 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: return self.sslsocket_class._create( 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: self.do_handshake() 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:28:31.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: self._sslobj.do_handshake() 2024-09-13T22:28:31.934 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:28:31 smithi178 bash[23095]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:28:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:32 smithi178 bash[22840]: cephadm 2024-09-13T22:28:31.352897+0000 mgr.y (mgr.14150) 3 : cephadm [INF] [13/Sep/2024:22:28:31] ENGINE Bus STARTING 2024-09-13T22:28:33.502 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: cephadm 2024-09-13T22:28:31.454677+0000 mgr.y (mgr.14150) 4 : cephadm [INF] [13/Sep/2024:22:28:31] ENGINE Serving on http://172.21.15.178:8765 2024-09-13T22:28:33.502 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: cephadm 2024-09-13T22:28:31.575706+0000 mgr.y (mgr.14150) 5 : cephadm [INF] [13/Sep/2024:22:28:31] ENGINE Serving on https://172.21.15.178:7150 2024-09-13T22:28:33.502 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: cephadm 2024-09-13T22:28:31.575880+0000 mgr.y (mgr.14150) 6 : cephadm [INF] [13/Sep/2024:22:28:31] ENGINE Bus STARTED 2024-09-13T22:28:33.502 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: cephadm 2024-09-13T22:28:31.577502+0000 mgr.y (mgr.14150) 7 : cephadm [ERR] [13/Sep/2024:22:28:31] ENGINE Error in HTTPServer.serve 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: Traceback (most recent call last): 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: self._connections.run(self.expiration_interval) 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: self._run(expiration_interval) 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: s = self.context.wrap_socket( 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:28:33.503 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: return self.sslsocket_class._create( 2024-09-13T22:28:33.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:28:33.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: self.do_handshake() 2024-09-13T22:28:33.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:28:33.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: self._sslobj.do_handshake() 2024-09-13T22:28:33.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:28:33.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:33 smithi178 bash[22840]: cluster 2024-09-13T22:28:32.228478+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-09-13T22:28:34.055 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-13T22:28:34.055 INFO:teuthology.orchestra.run.smithi178.stdout:Creating initial admin user... 2024-09-13T22:28:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:34 smithi178 bash[22840]: audit 2024-09-13T22:28:33.376282+0000 mgr.y (mgr.14150) 8 : audit [DBG] from='client.14162 v1:172.21.15.178:0/2981340109' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:34 smithi178 bash[22840]: audit 2024-09-13T22:28:33.487678+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:34 smithi178 bash[22840]: audit 2024-09-13T22:28:33.489726+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:36.275 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nWyjYks6hvtHHv4eI8Zs7O8jNpWN3V5t./dQ8Lz49ctm/1/aPrl0a", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726266515, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T22:28:36.276 INFO:teuthology.orchestra.run.smithi178.stdout:Fetching dashboard port number... 2024-09-13T22:28:37.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:36 smithi178 bash[22840]: audit 2024-09-13T22:28:35.488759+0000 mgr.y (mgr.14150) 9 : audit [DBG] from='client.14164 v1:172.21.15.178:0/2771616573' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:37.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:36 smithi178 bash[22840]: audit 2024-09-13T22:28:35.721198+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:37.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:36 smithi178 bash[22840]: audit 2024-09-13T22:28:35.768704+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:38.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:37 smithi178 bash[22840]: cluster 2024-09-13T22:28:36.774122+0000 mon.a (mon.0) 95 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-09-13T22:28:38.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:37 smithi178 bash[22840]: audit 2024-09-13T22:28:36.865014+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:38.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:37 smithi178 bash[22840]: audit 2024-09-13T22:28:37.675288+0000 mon.a (mon.0) 97 : audit [DBG] from='client.? v1:172.21.15.178:0/4120024732' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-13T22:28:38.255 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T22:28:38.255 INFO:teuthology.orchestra.run.smithi178.stdout:firewalld does not appear to be present 2024-09-13T22:28:38.255 INFO:teuthology.orchestra.run.smithi178.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T22:28:38.258 INFO:teuthology.orchestra.run.smithi178.stdout:Ceph Dashboard is now available at: 2024-09-13T22:28:38.258 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:38.258 INFO:teuthology.orchestra.run.smithi178.stdout: URL: https://smithi178.front.sepia.ceph.com:8443/ 2024-09-13T22:28:38.258 INFO:teuthology.orchestra.run.smithi178.stdout: User: admin 2024-09-13T22:28:38.259 INFO:teuthology.orchestra.run.smithi178.stdout: Password: crl7h24bsy 2024-09-13T22:28:38.259 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:38.259 INFO:teuthology.orchestra.run.smithi178.stdout:Saving cluster configuration to /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config directory 2024-09-13T22:28:40.345 INFO:teuthology.orchestra.run.smithi178.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T22:28:40.345 INFO:teuthology.orchestra.run.smithi178.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T22:28:40.345 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: ceph telemetry on 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout:For more information see: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-13T22:28:40.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:28:40.347 INFO:teuthology.orchestra.run.smithi178.stdout:Bootstrap complete. 2024-09-13T22:28:40.400 INFO:tasks.cephadm:Fetching config... 2024-09-13T22:28:40.401 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:28:40.401 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T22:28:40.407 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T22:28:40.408 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:28:40.408 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T22:28:40.455 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T22:28:40.455 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:28:40.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-13T22:28:40.512 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T22:28:40.512 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:28:40.512 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T22:28:40.559 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T22:28:40.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeyMILlcc8AAHLI8q0489XmVkEHTSNkt3r2Bj3CrKhTygYeTPL2c9blPo9AJS9QJ7gCp/SAlggO1TprGPoDCKnPtoPePnOJT9qeDPJI0IAnlrvZGk+QPni6rWpWi/t5w2GO9JqSjedAxQVbI7A2jKkmN7IdNSLM3NyZQHpUanbND8jnlRgq7f3G327FmH1TW1LJfFd1tRMOwzpHkJokQ51qeI9/OtreVn1ANzYTvgzbHS+0VtiSYxwlDrY5DhKrv+dAkaqScVM+wKjcfecJg9U/z3RTtsNFqGHSmesEYAFcBOmmTQPKnz4Xb7tuU4TtGIbcCVdQyrMcdikJxlKDE7wdZD6yF2mBU7EwR1998qJYSZomC6M0bGEu5vybI1yuvkrY2CRGJ0NMfmFG+bgYo6yxzkQFjBdlj3sBZ4GK1mNtUrbaXxTHWJkQfBe9iXqQWXCILA1p0T7HwcojDtQLNu3Lhg9raW8JSwUMid6UOqAZjSjykP5BKCru5mxenr9cvc= ceph-03ae43e2-721f-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T22:28:40.622 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeyMILlcc8AAHLI8q0489XmVkEHTSNkt3r2Bj3CrKhTygYeTPL2c9blPo9AJS9QJ7gCp/SAlggO1TprGPoDCKnPtoPePnOJT9qeDPJI0IAnlrvZGk+QPni6rWpWi/t5w2GO9JqSjedAxQVbI7A2jKkmN7IdNSLM3NyZQHpUanbND8jnlRgq7f3G327FmH1TW1LJfFd1tRMOwzpHkJokQ51qeI9/OtreVn1ANzYTvgzbHS+0VtiSYxwlDrY5DhKrv+dAkaqScVM+wKjcfecJg9U/z3RTtsNFqGHSmesEYAFcBOmmTQPKnz4Xb7tuU4TtGIbcCVdQyrMcdikJxlKDE7wdZD6yF2mBU7EwR1998qJYSZomC6M0bGEu5vybI1yuvkrY2CRGJ0NMfmFG+bgYo6yxzkQFjBdlj3sBZ4GK1mNtUrbaXxTHWJkQfBe9iXqQWXCILA1p0T7HwcojDtQLNu3Lhg9raW8JSwUMid6UOqAZjSjykP5BKCru5mxenr9cvc= ceph-03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:28:40.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeyMILlcc8AAHLI8q0489XmVkEHTSNkt3r2Bj3CrKhTygYeTPL2c9blPo9AJS9QJ7gCp/SAlggO1TprGPoDCKnPtoPePnOJT9qeDPJI0IAnlrvZGk+QPni6rWpWi/t5w2GO9JqSjedAxQVbI7A2jKkmN7IdNSLM3NyZQHpUanbND8jnlRgq7f3G327FmH1TW1LJfFd1tRMOwzpHkJokQ51qeI9/OtreVn1ANzYTvgzbHS+0VtiSYxwlDrY5DhKrv+dAkaqScVM+wKjcfecJg9U/z3RTtsNFqGHSmesEYAFcBOmmTQPKnz4Xb7tuU4TtGIbcCVdQyrMcdikJxlKDE7wdZD6yF2mBU7EwR1998qJYSZomC6M0bGEu5vybI1yuvkrY2CRGJ0NMfmFG+bgYo6yxzkQFjBdlj3sBZ4GK1mNtUrbaXxTHWJkQfBe9iXqQWXCILA1p0T7HwcojDtQLNu3Lhg9raW8JSwUMid6UOqAZjSjykP5BKCru5mxenr9cvc= ceph-03ae43e2-721f-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T22:28:40.654 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeyMILlcc8AAHLI8q0489XmVkEHTSNkt3r2Bj3CrKhTygYeTPL2c9blPo9AJS9QJ7gCp/SAlggO1TprGPoDCKnPtoPePnOJT9qeDPJI0IAnlrvZGk+QPni6rWpWi/t5w2GO9JqSjedAxQVbI7A2jKkmN7IdNSLM3NyZQHpUanbND8jnlRgq7f3G327FmH1TW1LJfFd1tRMOwzpHkJokQ51qeI9/OtreVn1ANzYTvgzbHS+0VtiSYxwlDrY5DhKrv+dAkaqScVM+wKjcfecJg9U/z3RTtsNFqGHSmesEYAFcBOmmTQPKnz4Xb7tuU4TtGIbcCVdQyrMcdikJxlKDE7wdZD6yF2mBU7EwR1998qJYSZomC6M0bGEu5vybI1yuvkrY2CRGJ0NMfmFG+bgYo6yxzkQFjBdlj3sBZ4GK1mNtUrbaXxTHWJkQfBe9iXqQWXCILA1p0T7HwcojDtQLNu3Lhg9raW8JSwUMid6UOqAZjSjykP5BKCru5mxenr9cvc= ceph-03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:28:40.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T22:28:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:40 smithi178 bash[22840]: audit 2024-09-13T22:28:39.785517+0000 mon.a (mon.0) 98 : audit [INF] from='client.? v1:172.21.15.178:0/4211160254' entity='client.admin' 2024-09-13T22:28:45.509 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:28:47.767 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T22:28:47.767 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T22:28:48.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:47 smithi178 bash[22840]: audit 2024-09-13T22:28:46.968874+0000 mon.a (mon.0) 99 : audit [INF] from='client.? v1:172.21.15.178:0/254950001' entity='client.admin' 2024-09-13T22:28:49.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:49 smithi178 bash[22840]: audit 2024-09-13T22:28:48.063311+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:49.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:49 smithi178 bash[22840]: audit 2024-09-13T22:28:48.066014+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:49.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:49 smithi178 bash[22840]: audit 2024-09-13T22:28:48.067067+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:28:49.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:49 smithi178 bash[22840]: audit 2024-09-13T22:28:48.070628+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:49.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:49 smithi178 bash[22840]: audit 2024-09-13T22:28:48.089828+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:28:49.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:49 smithi178 bash[22840]: audit 2024-09-13T22:28:48.097692+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:51 smithi178 bash[22840]: cluster 2024-09-13T22:28:50.179240+0000 mgr.y (mgr.14150) 10 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:52.489 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:28:53.499 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:53 smithi178 bash[22840]: cluster 2024-09-13T22:28:52.179651+0000 mgr.y (mgr.14150) 11 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:54.470 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-09-13T22:28:54.470 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:28:54.470 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-09-13T22:28:54.477 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:28:54.477 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:28:54.522 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-09-13T22:28:54.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch host add smithi192 2024-09-13T22:28:55.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:53.815755+0000 mgr.y (mgr.14150) 12 : audit [DBG] from='client.14172 v1:172.21.15.178:0/1177560791' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:28:55.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:53.820754+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:55.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:53.821843+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:28:55.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:53.824000+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:28:55.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:53.824759+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: cephadm 2024-09-13T22:28:53.825840+0000 mgr.y (mgr.14150) 13 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: cephadm 2024-09-13T22:28:53.915043+0000 mgr.y (mgr.14150) 14 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: cephadm 2024-09-13T22:28:53.997613+0000 mgr.y (mgr.14150) 15 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: cephadm 2024-09-13T22:28:54.080152+0000 mgr.y (mgr.14150) 16 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: cluster 2024-09-13T22:28:54.180004+0000 mgr.y (mgr.14150) 17 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:54.215368+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:54.217796+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:55.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:54 smithi178 bash[22840]: audit 2024-09-13T22:28:54.220228+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:28:57.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:57 smithi178 bash[22840]: cluster 2024-09-13T22:28:56.180417+0000 mgr.y (mgr.14150) 18 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:28:59.239 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:28:59.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:28:59 smithi178 bash[22840]: cluster 2024-09-13T22:28:58.180844+0000 mgr.y (mgr.14150) 19 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:01 smithi178 bash[22840]: cluster 2024-09-13T22:29:00.181249+0000 mgr.y (mgr.14150) 20 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:02.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:02 smithi178 bash[22840]: audit 2024-09-13T22:29:00.554689+0000 mgr.y (mgr.14150) 21 : audit [DBG] from='client.14174 v1:172.21.15.178:0/1123505540' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:03.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:03 smithi178 bash[22840]: cephadm 2024-09-13T22:29:01.676138+0000 mgr.y (mgr.14150) 22 : cephadm [INF] Deploying cephadm binary to smithi192 2024-09-13T22:29:03.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:03 smithi178 bash[22840]: cluster 2024-09-13T22:29:02.181673+0000 mgr.y (mgr.14150) 23 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:04.950 INFO:teuthology.orchestra.run.smithi178.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-09-13T22:29:05.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:05 smithi178 bash[22840]: cluster 2024-09-13T22:29:04.182059+0000 mgr.y (mgr.14150) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:05.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:05 smithi178 bash[22840]: audit 2024-09-13T22:29:04.948657+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:05.578 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:05 smithi178 bash[22840]: audit 2024-09-13T22:29:04.950369+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:05.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T22:29:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:06 smithi178 bash[22840]: cephadm 2024-09-13T22:29:04.949549+0000 mgr.y (mgr.14150) 25 : cephadm [INF] Added host smithi192 2024-09-13T22:29:06.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:06 smithi178 bash[22840]: audit 2024-09-13T22:29:05.494070+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:07 smithi178 bash[22840]: cluster 2024-09-13T22:29:06.182416+0000 mgr.y (mgr.14150) 26 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:07.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:07 smithi178 bash[22840]: audit 2024-09-13T22:29:07.023283+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:09.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:09 smithi178 bash[22840]: cluster 2024-09-13T22:29:08.182805+0000 mgr.y (mgr.14150) 27 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:09.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:09 smithi178 bash[22840]: audit 2024-09-13T22:29:08.218487+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:10.303 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:29:11.668 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:29:11.669 INFO:teuthology.orchestra.run.smithi178.stdout:[{"addr": "172.21.15.178", "hostname": "smithi178", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-09-13T22:29:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:11 smithi178 bash[22840]: cluster 2024-09-13T22:29:10.183146+0000 mgr.y (mgr.14150) 28 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:12.266 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T22:29:12.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T22:29:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:13 smithi178 bash[22840]: audit 2024-09-13T22:29:11.667720+0000 mgr.y (mgr.14150) 29 : audit [DBG] from='client.14176 v1:172.21.15.178:0/3856056139' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:29:13.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:13 smithi178 bash[22840]: cluster 2024-09-13T22:29:12.183543+0000 mgr.y (mgr.14150) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.737316+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.742224+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.747979+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.752979+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.754696+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:29:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.756644+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:13.758261+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: cephadm 2024-09-13T22:29:13.760550+0000 mgr.y (mgr.14150) 31 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: cephadm 2024-09-13T22:29:13.942582+0000 mgr.y (mgr.14150) 32 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: cephadm 2024-09-13T22:29:14.096048+0000 mgr.y (mgr.14150) 33 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: cluster 2024-09-13T22:29:14.183934+0000 mgr.y (mgr.14150) 34 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: cephadm 2024-09-13T22:29:14.238841+0000 mgr.y (mgr.14150) 35 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:14.383187+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:14.387695+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:15.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:14 smithi178 bash[22840]: audit 2024-09-13T22:29:14.392773+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:16.989 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:29:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:17 smithi178 bash[22840]: cluster 2024-09-13T22:29:16.184344+0000 mgr.y (mgr.14150) 36 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:19.238 INFO:teuthology.orchestra.run.smithi178.stderr:adjusted tunables profile to default 2024-09-13T22:29:19.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:19 smithi178 bash[22840]: cluster 2024-09-13T22:29:18.184747+0000 mgr.y (mgr.14150) 37 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:19.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:19 smithi178 bash[22840]: audit 2024-09-13T22:29:18.413387+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.178:0/3236304514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T22:29:19.871 INFO:tasks.cephadm:Adding mon.a on smithi178 2024-09-13T22:29:19.872 INFO:tasks.cephadm:Adding mon.c on smithi178 2024-09-13T22:29:19.872 INFO:tasks.cephadm:Adding mon.b on smithi192 2024-09-13T22:29:19.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b' 2024-09-13T22:29:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:20 smithi178 bash[22840]: audit 2024-09-13T22:29:19.238247+0000 mon.a (mon.0) 129 : audit [INF] from='client.? v1:172.21.15.178:0/3236304514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T22:29:20.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:20 smithi178 bash[22840]: cluster 2024-09-13T22:29:19.242781+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:21.099 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:29:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:21 smithi178 bash[22840]: cluster 2024-09-13T22:29:20.185166+0000 mgr.y (mgr.14150) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:22.493 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-09-13T22:29:22.994 DEBUG:teuthology.orchestra.run.smithi178:mon.c> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.c.service 2024-09-13T22:29:22.997 DEBUG:teuthology.orchestra.run.smithi192:mon.b> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.b.service 2024-09-13T22:29:22.999 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T22:29:22.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: cluster 2024-09-13T22:29:22.185557+0000 mgr.y (mgr.14150) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.491767+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.493349+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.496647+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.498115+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.505426+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.509067+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:23.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:23 smithi178 bash[22840]: audit 2024-09-13T22:29:22.510577+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:24.238 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:29:24.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:24 smithi178 bash[22840]: audit 2024-09-13T22:29:22.481410+0000 mgr.y (mgr.14150) 40 : audit [DBG] from='client.14180 v1:172.21.15.192:0/298312509' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:24.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:24 smithi178 bash[22840]: cephadm 2024-09-13T22:29:22.486626+0000 mgr.y (mgr.14150) 41 : cephadm [INF] Saving service mon spec with placement smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b;count:3 2024-09-13T22:29:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:24 smithi178 bash[22840]: cephadm 2024-09-13T22:29:22.512310+0000 mgr.y (mgr.14150) 42 : cephadm [INF] Deploying daemon mon.b on smithi192 2024-09-13T22:29:25.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:25 smithi178 bash[22840]: cluster 2024-09-13T22:29:24.185960+0000 mgr.y (mgr.14150) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:26.124 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-09-13T22:29:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:29:26.125 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","modified":"2024-09-13T22:27:26.887333Z","created":"2024-09-13T22:27:26.887333Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T22:29:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:26 smithi178 bash[22840]: audit 2024-09-13T22:29:26.124364+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.192:0/3835244938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:29:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:27 smithi178 bash[22840]: cluster 2024-09-13T22:29:26.186388+0000 mgr.y (mgr.14150) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:27.913 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T22:29:27.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T22:29:28.179 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:28 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:28.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:28 smithi192 systemd[1]: Started Ceph mon.b for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:29:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.189+0000 7fe9f79e9c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.189+0000 7fe9f79e9c80 0 ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable), process ceph-mon, pid 7 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.189+0000 7fe9f79e9c80 0 pidfile_write: ignore empty --pid-file 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 0 load: jerasure load: lrc 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: RocksDB version: 7.9.2 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Git sha 0 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Compile date 2024-09-13 16:59:10 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: DB SUMMARY 2024-09-13T22:29:29.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: DB Session ID: GDEZQ6E2YPES9DQGZLAO 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: CURRENT file: CURRENT 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: IDENTITY file: IDENTITY 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.error_if_exists: 0 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.create_if_missing: 0 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.paranoid_checks: 1 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T22:29:29.412 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.env: 0x55c85206d940 2024-09-13T22:29:29.413 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-13T22:29:29.413 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.info_log: 0x55c854201620 2024-09-13T22:29:29.413 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-13T22:29:29.413 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.statistics: (nil) 2024-09-13T22:29:29.413 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.use_fsync: 0 2024-09-13T22:29:29.413 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_log_file_size: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.allow_fallocate: 1 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.use_direct_reads: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-13T22:29:29.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.db_log_dir: 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.wal_dir: 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.write_buffer_manager: 0x55c854210dc0 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T22:29:29.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.unordered_write: 0 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.row_cache: None 2024-09-13T22:29:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.wal_filter: None 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.two_write_queues: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.wal_compression: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.atomic_flush: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.log_readahead_size: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-13T22:29:29.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_background_jobs: 2 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_background_compactions: -1 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_subcompactions: 1 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T22:29:29.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_open_files: -1 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T22:29:29.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_background_flushes: -1 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Compression algorithms supported: 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kZSTD supported: 0 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kXpressCompression supported: 0 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kBZip2Compression supported: 0 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kLZ4Compression supported: 1 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kZlibCompression supported: 1 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: kSnappyCompression supported: 1 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T22:29:29.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-13T22:29:29.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T22:29:29.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T22:29:29.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.merge_operator: 2024-09-13T22:29:29.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_filter: None 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8542012a0) 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cache_index_and_filter_blocks: 1 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T22:29:29.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T22:29:29.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: pin_top_level_index_and_filter: 1 2024-09-13T22:29:29.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: index_type: 0 2024-09-13T22:29:29.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: data_block_index_type: 0 2024-09-13T22:29:29.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: index_shortening: 1 2024-09-13T22:29:29.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: data_block_hash_table_util_ratio: 0.750000 2024-09-13T22:29:29.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: checksum: 4 2024-09-13T22:29:29.425 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: no_block_cache: 0 2024-09-13T22:29:29.425 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_cache: 0x55c8541f71f0 2024-09-13T22:29:29.425 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_cache_name: BinnedLRUCache 2024-09-13T22:29:29.425 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_cache_options: 2024-09-13T22:29:29.425 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: capacity : 536870912 2024-09-13T22:29:29.425 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: num_shard_bits : 4 2024-09-13T22:29:29.426 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: strict_capacity_limit : 0 2024-09-13T22:29:29.426 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: high_pri_pool_ratio: 0.000 2024-09-13T22:29:29.426 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_cache_compressed: (nil) 2024-09-13T22:29:29.426 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: persistent_cache: (nil) 2024-09-13T22:29:29.426 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_size: 4096 2024-09-13T22:29:29.426 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_size_deviation: 10 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_restart_interval: 16 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: index_block_restart_interval: 1 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: metadata_block_size: 4096 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: partition_filters: 0 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: use_delta_encoding: 1 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: filter_policy: bloomfilter 2024-09-13T22:29:29.427 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: whole_key_filtering: 1 2024-09-13T22:29:29.428 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: verify_compression: 0 2024-09-13T22:29:29.428 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: read_amp_bytes_per_bit: 0 2024-09-13T22:29:29.428 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: format_version: 5 2024-09-13T22:29:29.428 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: enable_index_compression: 1 2024-09-13T22:29:29.428 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: block_align: 0 2024-09-13T22:29:29.428 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: max_auto_readahead_size: 262144 2024-09-13T22:29:29.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: prepopulate_block_cache: 0 2024-09-13T22:29:29.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: initial_auto_readahead_size: 8192 2024-09-13T22:29:29.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: num_file_reads_for_auto_readahead: 2 2024-09-13T22:29:29.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-13T22:29:29.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-13T22:29:29.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression: NoCompression 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.num_levels: 7 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T22:29:29.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T22:29:29.431 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T22:29:29.431 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T22:29:29.431 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T22:29:29.431 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T22:29:29.431 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:29:29.431 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T22:29:29.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T22:29:29.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:29:29.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:29:29.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T22:29:29.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-13T22:29:29.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-13T22:29:29.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T22:29:29.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:29:29.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:29:29.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T22:29:29.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-13T22:29:29.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:29:29.434 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T22:29:29.434 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T22:29:29.434 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T22:29:29.434 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-13T22:29:29.434 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T22:29:29.434 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T22:29:29.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T22:29:29.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T22:29:29.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T22:29:29.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T22:29:29.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T22:29:29.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T22:29:29.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-13T22:29:29.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T22:29:29.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T22:29:29.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-13T22:29:29.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T22:29:29.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T22:29:29.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T22:29:29.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T22:29:29.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T22:29:29.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T22:29:29.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T22:29:29.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T22:29:29.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T22:29:29.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T22:29:29.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T22:29:29.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.inplace_update_support: 0 2024-09-13T22:29:29.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T22:29:29.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T22:29:29.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T22:29:29.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T22:29:29.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.bloom_locality: 0 2024-09-13T22:29:29.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.max_successive_merges: 0 2024-09-13T22:29:29.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T22:29:29.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-13T22:29:29.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-13T22:29:29.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-13T22:29:29.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.ttl: 2592000 2024-09-13T22:29:29.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T22:29:29.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T22:29:29.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T22:29:29.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.enable_blob_files: false 2024-09-13T22:29:29.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.min_blob_size: 0 2024-09-13T22:29:29.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-13T22:29:29.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T22:29:29.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T22:29:29.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T22:29:29.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T22:29:29.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T22:29:29.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-13T22:29:29.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T22:29:29.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T22:29:29.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-13T22:29:29.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d711d6cc-4451-48dd-9ec3-891290617b93 2024-09-13T22:29:29.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266569196565, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-13T22:29:29.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-13T22:29:29.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266569196980, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726266569, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d711d6cc-4451-48dd-9ec3-891290617b93", "db_session_id": "GDEZQ6E2YPES9DQGZLAO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-13T22:29:29.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266569197098, "job": 1, "event": "recovery_finished"} 2024-09-13T22:29:29.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.193+0000 7fe9f79e9c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-13T22:29:29.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T22:29:29.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c854222e00 2024-09-13T22:29:29.446 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 4 rocksdb: DB pointer 0x55c854322000 2024-09-13T22:29:29.446 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9ed7b2640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T22:29:29.446 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9ed7b2640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T22:29:29.446 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: ** DB Stats ** 2024-09-13T22:29:29.446 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:29:29.446 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T22:29:29.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:29:29.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:29:29.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T22:29:29.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:29:29.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:29:29.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: ** Compaction Stats [default] ** 2024-09-13T22:29:29.448 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:29:29.448 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T22:29:29.448 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:29.448 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:29.448 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:29.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: ** Compaction Stats [default] ** 2024-09-13T22:29:29.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:29:29.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T22:29:29.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:29.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T22:29:29.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:29:29.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T22:29:29.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T22:29:29.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T22:29:29.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T22:29:29.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: AddFile(Keys): cumulative 0, interval 0 2024-09-13T22:29:29.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:29:29.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:29:29.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T22:29:29.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Block cache BinnedLRUCache@0x55c8541f71f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-09-13T22:29:29.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-13T22:29:29.451 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: ** File Read Latency Histogram By Level [default] ** 2024-09-13T22:29:29.452 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-13T22:29:29.452 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 0 using public_addrv v1:172.21.15.192:6789/0 2024-09-13T22:29:29.452 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 0 starting mon.b rank -1 at public addrs v1:172.21.15.192:6789/0 at bind addrs v1:172.21.15.192:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:29.452 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.197+0000 7fe9f79e9c80 1 mon.b@-1(???) e0 preinit fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:29.452 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.217+0000 7fe9f07b8640 0 mon.b@-1(synchronizing).mds e1 new map 2024-09-13T22:29:29.452 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.217+0000 7fe9f07b8640 0 mon.b@-1(synchronizing).mds e1 print_map 2024-09-13T22:29:29.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: e1 2024-09-13T22:29:29.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: btime 2024-09-13T22:27:31:499941+0000 2024-09-13T22:29:29.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T22:29:29.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-13T22:29:29.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: legacy client fscid: -1 2024-09-13T22:29:29.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: 2024-09-13T22:29:29.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: No filesystems configured 2024-09-13T22:29:29.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.217+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-13T22:29:29.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-13T22:29:29.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.217+0000 7fe9ebfaf640 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-09-13T22:29:29.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-13T22:29:29.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-13T22:29:29.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-13T22:29:29.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f4fc1640 4 rocksdb: (Original Log Time 2024/09/13-22:29:29.221966) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-09-13T22:29:29.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f4fc1640 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-09-13T22:29:29.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-13T22:29:29.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f4fc1640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266569222041, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1056, "num_deletes": 13, "total_data_size": 2236319, "memory_usage": 2260616, "flush_reason": "Manual Compaction"} 2024-09-13T22:29:29.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f4fc1640 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-09-13T22:29:29.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T22:29:29.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:29:29.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:29:29.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:29:29.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:00.554689+0000 mgr.y (mgr.14150) 21 : audit [DBG] from='client.14174 v1:172.21.15.178:0/1123505540' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:29.457 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:01.676138+0000 mgr.y (mgr.14150) 22 : cephadm [INF] Deploying cephadm binary to smithi192 2024-09-13T22:29:29.457 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:02.181673+0000 mgr.y (mgr.14150) 23 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.457 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:04.182059+0000 mgr.y (mgr.14150) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.457 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:04.948657+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.457 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:04.950369+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:04.949549+0000 mgr.y (mgr.14150) 25 : cephadm [INF] Added host smithi192 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:05.494070+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:06.182416+0000 mgr.y (mgr.14150) 26 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:07.023283+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:08.182805+0000 mgr.y (mgr.14150) 27 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:08.218487+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:10.183146+0000 mgr.y (mgr.14150) 28 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:11.667720+0000 mgr.y (mgr.14150) 29 : audit [DBG] from='client.14176 v1:172.21.15.178:0/3856056139' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:29:29.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:12.183543+0000 mgr.y (mgr.14150) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.737316+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.742224+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.747979+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.752979+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.754696+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:29:29.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.756644+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:29.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:13.758261+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:29.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:13.760550+0000 mgr.y (mgr.14150) 31 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:29:29.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:13.942582+0000 mgr.y (mgr.14150) 32 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:29:29.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:14.096048+0000 mgr.y (mgr.14150) 33 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:29:29.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:14.183934+0000 mgr.y (mgr.14150) 34 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:14.238841+0000 mgr.y (mgr.14150) 35 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:29:29.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:14.383187+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:14.387695+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.461 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:14.392773+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:16.184344+0000 mgr.y (mgr.14150) 36 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:18.184747+0000 mgr.y (mgr.14150) 37 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:18.413387+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.178:0/3236304514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:19.238247+0000 mon.a (mon.0) 129 : audit [INF] from='client.? v1:172.21.15.178:0/3236304514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:19.242781+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:20.185166+0000 mgr.y (mgr.14150) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.462 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:22.185557+0000 mgr.y (mgr.14150) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.491767+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.493349+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.496647+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.498115+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.505426+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.509067+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:29.463 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.510577+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:22.481410+0000 mgr.y (mgr.14150) 40 : audit [DBG] from='client.14180 v1:172.21.15.192:0/298312509' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:22.486626+0000 mgr.y (mgr.14150) 41 : cephadm [INF] Saving service mon spec with placement smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b;count:3 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cephadm 2024-09-13T22:29:22.512310+0000 mgr.y (mgr.14150) 42 : cephadm [INF] Deploying daemon mon.b on smithi192 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:24.185960+0000 mgr.y (mgr.14150) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: audit 2024-09-13T22:29:26.124364+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.192:0/3835244938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: cluster 2024-09-13T22:29:26.186388+0000 mgr.y (mgr.14150) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.464 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:29 smithi192 bash[24608]: debug 2024-09-13T22:29:29.221+0000 7fe9f07b8640 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T22:29:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: cluster 2024-09-13T22:29:28.186826+0000 mgr.y (mgr.14150) 45 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:29.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: audit 2024-09-13T22:29:28.224881+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: audit 2024-09-13T22:29:28.230158+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: audit 2024-09-13T22:29:28.236023+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:29.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: audit 2024-09-13T22:29:28.237316+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:29.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: audit 2024-09-13T22:29:28.238994+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:29.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:29 smithi178 bash[22840]: cephadm 2024-09-13T22:29:28.240903+0000 mgr.y (mgr.14150) 46 : cephadm [INF] Deploying daemon mon.c on smithi178 2024-09-13T22:29:32.329 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:32 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:32.329 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:29:32 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:32 smithi178 systemd[1]: Started Ceph mon.c for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:29:32.669 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:32 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:32.669 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:29:32 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:33.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.583+0000 7f58ffbb9c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-13T22:29:33.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.583+0000 7f58ffbb9c80 0 ceph version 19.1.1-324-gecab7be5 (ecab7be58963a5bd03910cd73eef66405d7fdd50) squid (stable), process ceph-mon, pid 7 2024-09-13T22:29:33.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.583+0000 7f58ffbb9c80 0 pidfile_write: ignore empty --pid-file 2024-09-13T22:29:33.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 0 load: jerasure load: lrc 2024-09-13T22:29:33.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: RocksDB version: 7.9.2 2024-09-13T22:29:33.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Git sha 0 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Compile date 2024-09-13 16:59:10 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: DB SUMMARY 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: DB Session ID: KJOT6O7SWHBEN6ADIGCH 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: CURRENT file: CURRENT 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: IDENTITY file: IDENTITY 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.error_if_exists: 0 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.create_if_missing: 0 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.paranoid_checks: 1 2024-09-13T22:29:33.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.env: 0x55b198cf0940 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.fs: PosixFileSystem 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.info_log: 0x55b199d27620 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.statistics: (nil) 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.use_fsync: 0 2024-09-13T22:29:33.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_log_file_size: 0 2024-09-13T22:29:33.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T22:29:33.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T22:29:33.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-13T22:29:33.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-13T22:29:33.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.allow_fallocate: 1 2024-09-13T22:29:33.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.use_direct_reads: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.create_missing_column_families: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.db_log_dir: 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.wal_dir: 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T22:29:33.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.advise_random_on_open: 1 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.write_buffer_manager: 0x55b199d36dc0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.rate_limiter: (nil) 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.unordered_write: 0 2024-09-13T22:29:33.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.row_cache: None 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.wal_filter: None 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.two_write_queues: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.manual_wal_flush: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.wal_compression: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.atomic_flush: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T22:29:33.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.log_readahead_size: 0 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_background_jobs: 2 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_background_compactions: -1 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_subcompactions: 1 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T22:29:33.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_total_wal_size: 0 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_open_files: -1 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bytes_per_sync: 0 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_background_flushes: -1 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Compression algorithms supported: 2024-09-13T22:29:33.940 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kZSTD supported: 0 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kXpressCompression supported: 0 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kBZip2Compression supported: 0 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kLZ4Compression supported: 1 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kZlibCompression supported: 1 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: kSnappyCompression supported: 1 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T22:29:33.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.merge_operator: 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_filter: None 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_filter_factory: None 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b199d272a0) 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cache_index_and_filter_blocks: 1 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: pin_top_level_index_and_filter: 1 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: index_type: 0 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: data_block_index_type: 0 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: index_shortening: 1 2024-09-13T22:29:33.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: data_block_hash_table_util_ratio: 0.750000 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: checksum: 4 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: no_block_cache: 0 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_cache: 0x55b199d1d1f0 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_cache_name: BinnedLRUCache 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_cache_options: 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: capacity : 536870912 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: num_shard_bits : 4 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: strict_capacity_limit : 0 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: high_pri_pool_ratio: 0.000 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_cache_compressed: (nil) 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: persistent_cache: (nil) 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_size: 4096 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_size_deviation: 10 2024-09-13T22:29:33.943 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_restart_interval: 16 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: index_block_restart_interval: 1 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: metadata_block_size: 4096 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: partition_filters: 0 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: use_delta_encoding: 1 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: filter_policy: bloomfilter 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: whole_key_filtering: 1 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: verify_compression: 0 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: read_amp_bytes_per_bit: 0 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: format_version: 5 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: enable_index_compression: 1 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: block_align: 0 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: max_auto_readahead_size: 262144 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: prepopulate_block_cache: 0 2024-09-13T22:29:33.944 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: initial_auto_readahead_size: 8192 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: num_file_reads_for_auto_readahead: 2 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression: NoCompression 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.num_levels: 7 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T22:29:33.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.level: 32767 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T22:29:33.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T22:29:33.947 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.enabled: false 2024-09-13T22:29:33.947 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T22:29:33.948 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T22:29:33.948 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T22:29:33.948 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T22:29:33.948 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-13T22:29:33.948 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T22:29:33.949 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.arena_block_size: 1048576 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T22:29:33.950 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.inplace_update_support: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.bloom_locality: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.max_successive_merges: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.force_consistency_checks: 1 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-13T22:29:33.951 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.ttl: 2592000 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.enable_blob_files: false 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.min_blob_size: 0 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.blob_file_size: 268435456 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-09-13T22:29:33.952 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4964717a-5f95-4972-9a8e-905c99bea02b 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266573593923, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266573594375, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726266573, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4964717a-5f95-4972-9a8e-905c99bea02b", "db_session_id": "KJOT6O7SWHBEN6ADIGCH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266573594504, "job": 1, "event": "recovery_finished"} 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.587+0000 7f58ffbb9c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b199d48e00 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 4 rocksdb: DB pointer 0x55b199e48000 2024-09-13T22:29:33.953 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58f5982640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58f5982640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: ** DB Stats ** 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: ** Compaction Stats [default] ** 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:33.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: ** Compaction Stats [default] ** 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: AddFile(Keys): cumulative 0, interval 0 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:29:33.955 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Block cache BinnedLRUCache@0x55b199d1d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: ** File Read Latency Histogram By Level [default] ** 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 0 using public_addrv v1:172.21.15.178:6790/0 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 0 starting mon.c rank -1 at public addrs v1:172.21.15.178:6790/0 at bind addrs v1:172.21.15.178:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.591+0000 7f58ffbb9c80 1 mon.c@-1(???) e0 preinit fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 0 mon.c@-1(synchronizing).mds e1 new map 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 0 mon.c@-1(synchronizing).mds e1 print_map 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: e1 2024-09-13T22:29:33.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: btime 2024-09-13T22:27:31:499941+0000 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: legacy client fscid: -1 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: No filesystems configured 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f417f640 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58fd191640 4 rocksdb: (Original Log Time 2024/09/13-22:29:33.612957) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58fd191640 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58fd191640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266573613162, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1070, "num_deletes": 13, "total_data_size": 2242486, "memory_usage": 2267296, "flush_reason": "Manual Compaction"} 2024-09-13T22:29:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58fd191640 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:00.554689+0000 mgr.y (mgr.14150) 21 : audit [DBG] from='client.14174 v1:172.21.15.178:0/1123505540' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:01.676138+0000 mgr.y (mgr.14150) 22 : cephadm [INF] Deploying cephadm binary to smithi192 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:02.181673+0000 mgr.y (mgr.14150) 23 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:04.182059+0000 mgr.y (mgr.14150) 24 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:04.948657+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:04.950369+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:04.949549+0000 mgr.y (mgr.14150) 25 : cephadm [INF] Added host smithi192 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:05.494070+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:06.182416+0000 mgr.y (mgr.14150) 26 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:07.023283+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:08.182805+0000 mgr.y (mgr.14150) 27 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:08.218487+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:10.183146+0000 mgr.y (mgr.14150) 28 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:11.667720+0000 mgr.y (mgr.14150) 29 : audit [DBG] from='client.14176 v1:172.21.15.178:0/3856056139' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:12.183543+0000 mgr.y (mgr.14150) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.737316+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.959 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.742224+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.747979+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.752979+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.754696+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.756644+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:13.758261+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:13.760550+0000 mgr.y (mgr.14150) 31 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:13.942582+0000 mgr.y (mgr.14150) 32 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:14.096048+0000 mgr.y (mgr.14150) 33 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:14.183934+0000 mgr.y (mgr.14150) 34 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:14.238841+0000 mgr.y (mgr.14150) 35 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:14.383187+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:14.387695+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.960 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:14.392773+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:16.184344+0000 mgr.y (mgr.14150) 36 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:18.184747+0000 mgr.y (mgr.14150) 37 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:18.413387+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.178:0/3236304514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:19.238247+0000 mon.a (mon.0) 129 : audit [INF] from='client.? v1:172.21.15.178:0/3236304514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:19.242781+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:20.185166+0000 mgr.y (mgr.14150) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:22.185557+0000 mgr.y (mgr.14150) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.491767+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.493349+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.496647+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.498115+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.505426+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.509067+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:33.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.510577+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:22.481410+0000 mgr.y (mgr.14150) 40 : audit [DBG] from='client.14180 v1:172.21.15.192:0/298312509' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:22.486626+0000 mgr.y (mgr.14150) 41 : cephadm [INF] Saving service mon spec with placement smithi178:[v1:172.21.15.178:6789]=a;smithi178:[v1:172.21.15.178:6790]=c;smithi192:[v1:172.21.15.192:6789]=b;count:3 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:22.512310+0000 mgr.y (mgr.14150) 42 : cephadm [INF] Deploying daemon mon.b on smithi192 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:24.185960+0000 mgr.y (mgr.14150) 43 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:26.124364+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.192:0/3835244938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:26.186388+0000 mgr.y (mgr.14150) 44 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cluster 2024-09-13T22:29:28.186826+0000 mgr.y (mgr.14150) 45 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:28.224881+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:28.230158+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:28.236023+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:28.237316+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: audit 2024-09-13T22:29:28.238994+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:33.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: cephadm 2024-09-13T22:29:28.240903+0000 mgr.y (mgr.14150) 46 : cephadm [INF] Deploying daemon mon.c on smithi178 2024-09-13T22:29:33.963 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.607+0000 7f58f8988640 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T22:29:33.963 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.611+0000 7f58fd191640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726266573617785, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1989892, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1075, "table_properties": {"data_size": 1984385, "index_size": 2391, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2053, "raw_key_size": 24718, "raw_average_key_size": 30, "raw_value_size": 1967462, "raw_average_value_size": 2444, "num_data_blocks": 113, "num_entries": 805, "num_filter_entries": 805, "num_deletions": 13, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726266573, "oldest_key_time": 1726266573, "file_creation_time": 1726266573, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4964717a-5f95-4972-9a8e-905c99bea02b", "db_session_id": "KJOT6O7SWHBEN6ADIGCH", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-09-13T22:29:33.963 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:33 smithi178 bash[29889]: debug 2024-09-13T22:29:33.611+0000 7f58fd191640 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 4856 microseconds, and 3847 cpu microseconds. 2024-09-13T22:29:34.578 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-09-13T22:29:34.578 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:29:34.578 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","modified":"2024-09-13T22:29:29.228250Z","created":"2024-09-13T22:27:26.887333Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T22:29:34.589 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:28.186826+0000 mgr.y (mgr.14150) 45 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:28.224881+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:28.230158+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:28.236023+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:28.237316+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:28.238994+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cephadm 2024-09-13T22:29:28.240903+0000 mgr.y (mgr.14150) 46 : cephadm [INF] Deploying daemon mon.c on smithi178 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:29.232541+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:29.232941+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:29.252237+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:30.187173+0000 mgr.y (mgr.14150) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:30.227134+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:31.227699+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:31.229933+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:32.187571+0000 mgr.y (mgr.14150) 48 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:32.227148+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:33.227732+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:33.620347+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:34.228116+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.271818+0000 mon.a (mon.0) 154 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277724+0000 mon.a (mon.0) 155 : cluster [DBG] monmap epoch 2 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277752+0000 mon.a (mon.0) 156 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277768+0000 mon.a (mon.0) 157 : cluster [DBG] last_changed 2024-09-13T22:29:29.228250+0000 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277781+0000 mon.a (mon.0) 158 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277799+0000 mon.a (mon.0) 159 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277816+0000 mon.a (mon.0) 160 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:34.591 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277834+0000 mon.a (mon.0) 161 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.277850+0000 mon.a (mon.0) 162 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.295783+0000 mon.a (mon.0) 163 : cluster [DBG] fsmap 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.295833+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.296095+0000 mon.a (mon.0) 165 : cluster [DBG] mgrmap e13: y(active, since 64s) 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.296321+0000 mon.a (mon.0) 166 : cluster [INF] overall HEALTH_OK 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:34.303393+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:34.310564+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:34.321247+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:34 smithi192 bash[24608]: audit 2024-09-13T22:29:34.340729+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:29.232541+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:29.232941+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:29.252237+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:30.187173+0000 mgr.y (mgr.14150) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:30.227134+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:31.227699+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:31.229933+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:32.187571+0000 mgr.y (mgr.14150) 48 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:32.227148+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:33.227732+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:33.620347+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:34.228116+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.271818+0000 mon.a (mon.0) 154 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277724+0000 mon.a (mon.0) 155 : cluster [DBG] monmap epoch 2 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277752+0000 mon.a (mon.0) 156 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277768+0000 mon.a (mon.0) 157 : cluster [DBG] last_changed 2024-09-13T22:29:29.228250+0000 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277781+0000 mon.a (mon.0) 158 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277799+0000 mon.a (mon.0) 159 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277816+0000 mon.a (mon.0) 160 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277834+0000 mon.a (mon.0) 161 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.277850+0000 mon.a (mon.0) 162 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:34.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.295783+0000 mon.a (mon.0) 163 : cluster [DBG] fsmap 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.295833+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.296095+0000 mon.a (mon.0) 165 : cluster [DBG] mgrmap e13: y(active, since 64s) 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.296321+0000 mon.a (mon.0) 166 : cluster [INF] overall HEALTH_OK 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:34.303393+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:34.310564+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:34.321247+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:34.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:34 smithi178 bash[22840]: audit 2024-09-13T22:29:34.340729+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:35 smithi192 bash[24608]: cluster 2024-09-13T22:29:34.187978+0000 mgr.y (mgr.14150) 49 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:35 smithi192 bash[24608]: audit 2024-09-13T22:29:34.386351+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:35 smithi192 bash[24608]: audit 2024-09-13T22:29:34.577733+0000 mon.a (mon.0) 172 : audit [DBG] from='client.? v1:172.21.15.192:0/2995696991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:29:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:35 smithi192 bash[24608]: audit 2024-09-13T22:29:34.620024+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:35 smithi192 bash[24608]: audit 2024-09-13T22:29:35.227935+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:35 smithi192 bash[24608]: audit 2024-09-13T22:29:35.243752+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:35 smithi178 bash[22840]: cluster 2024-09-13T22:29:34.187978+0000 mgr.y (mgr.14150) 49 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:35 smithi178 bash[22840]: audit 2024-09-13T22:29:34.386351+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:35 smithi178 bash[22840]: audit 2024-09-13T22:29:34.577733+0000 mon.a (mon.0) 172 : audit [DBG] from='client.? v1:172.21.15.192:0/2995696991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:29:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:35 smithi178 bash[22840]: audit 2024-09-13T22:29:34.620024+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:35 smithi178 bash[22840]: audit 2024-09-13T22:29:35.227935+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:35 smithi178 bash[22840]: audit 2024-09-13T22:29:35.243752+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:36.174 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T22:29:36.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T22:29:40.382 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:29:41.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:35.628046+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:29:41.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:35.628427+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:35.628635+0000 mon.a (mon.0) 178 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:35.648445+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:35.648951+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:36.188361+0000 mgr.y (mgr.14150) 50 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:36.620701+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:37.620912+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:38.188751+0000 mgr.y (mgr.14150) 51 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:38.620951+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:39.620371+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.189140+0000 mgr.y (mgr.14150) 52 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:40.621217+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.648373+0000 mon.a (mon.0) 186 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654141+0000 mon.a (mon.0) 187 : cluster [DBG] monmap epoch 3 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654172+0000 mon.a (mon.0) 188 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654189+0000 mon.a (mon.0) 189 : cluster [DBG] last_changed 2024-09-13T22:29:35.622057+0000 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654205+0000 mon.a (mon.0) 190 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654222+0000 mon.a (mon.0) 191 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654239+0000 mon.a (mon.0) 192 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654256+0000 mon.a (mon.0) 193 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654275+0000 mon.a (mon.0) 194 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.654292+0000 mon.a (mon.0) 195 : cluster [DBG] 2: v1:172.21.15.178:6790/0 mon.c 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.671840+0000 mon.a (mon.0) 196 : cluster [DBG] fsmap 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.671892+0000 mon.a (mon.0) 197 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.672157+0000 mon.a (mon.0) 198 : cluster [DBG] mgrmap e13: y(active, since 70s) 2024-09-13T22:29:41.162 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.672443+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.679524+0000 mon.a (mon.0) 200 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.679570+0000 mon.a (mon.0) 201 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: cluster 2024-09-13T22:29:40.679596+0000 mon.a (mon.0) 202 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.178:6790/0 is down (out of quorum) 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:40.684373+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:40.689871+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:40.695532+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.163 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:40 smithi192 bash[24608]: audit 2024-09-13T22:29:40.701137+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:35.628046+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:35.628427+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:35.628635+0000 mon.a (mon.0) 178 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:35.648445+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:35.648951+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:36.188361+0000 mgr.y (mgr.14150) 50 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:36.620701+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:37.620912+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:38.188751+0000 mgr.y (mgr.14150) 51 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:38.620951+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:39.620371+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.189140+0000 mgr.y (mgr.14150) 52 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:40.621217+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.648373+0000 mon.a (mon.0) 186 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654141+0000 mon.a (mon.0) 187 : cluster [DBG] monmap epoch 3 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654172+0000 mon.a (mon.0) 188 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654189+0000 mon.a (mon.0) 189 : cluster [DBG] last_changed 2024-09-13T22:29:35.622057+0000 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654205+0000 mon.a (mon.0) 190 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654222+0000 mon.a (mon.0) 191 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654239+0000 mon.a (mon.0) 192 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654256+0000 mon.a (mon.0) 193 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654275+0000 mon.a (mon.0) 194 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.654292+0000 mon.a (mon.0) 195 : cluster [DBG] 2: v1:172.21.15.178:6790/0 mon.c 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.671840+0000 mon.a (mon.0) 196 : cluster [DBG] fsmap 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.671892+0000 mon.a (mon.0) 197 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.672157+0000 mon.a (mon.0) 198 : cluster [DBG] mgrmap e13: y(active, since 70s) 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.672443+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.679524+0000 mon.a (mon.0) 200 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.679570+0000 mon.a (mon.0) 201 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: cluster 2024-09-13T22:29:40.679596+0000 mon.a (mon.0) 202 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.178:6790/0 is down (out of quorum) 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:40.684373+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:40.689871+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.185 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:40.695532+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.185 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:40 smithi178 bash[22840]: audit 2024-09-13T22:29:40.701137+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:41.869 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:29:41.870 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","modified":"2024-09-13T22:29:35.622057Z","created":"2024-09-13T22:27:26.887333Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6790","nonce":0}]},"addr":"172.21.15.178:6790/0","public_addr":"172.21.15.178:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T22:29:41.870 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-09-13T22:29:42.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.351014+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:42.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.352519+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:42.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: cephadm 2024-09-13T22:29:41.354370+0000 mgr.y (mgr.14150) 53 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: cephadm 2024-09-13T22:29:41.354775+0000 mgr.y (mgr.14150) 54 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.620836+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.623879+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.627778+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.635464+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.641302+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.647327+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.683152+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.687649+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.693428+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.698800+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.699801+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:42.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.700821+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:42.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:41 smithi192 bash[24608]: audit 2024-09-13T22:29:41.701749+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:42.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.351014+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:42.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.352519+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:42.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: cephadm 2024-09-13T22:29:41.354370+0000 mgr.y (mgr.14150) 53 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: cephadm 2024-09-13T22:29:41.354775+0000 mgr.y (mgr.14150) 54 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.620836+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.623879+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.627778+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.635464+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.641302+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.647327+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.683152+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.687649+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.693428+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.698800+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.699801+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:42.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.700821+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:42.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:41 smithi178 bash[22840]: audit 2024-09-13T22:29:41.701749+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:42.488 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T22:29:42.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T22:29:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:29.232541+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:29:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:29.232941+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:29.252237+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:30.187173+0000 mgr.y (mgr.14150) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:30.227134+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:31.227699+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:31.229933+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:32.187571+0000 mgr.y (mgr.14150) 48 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:32.227148+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:33.227732+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:33.620347+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.228116+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.271818+0000 mon.a (mon.0) 154 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277724+0000 mon.a (mon.0) 155 : cluster [DBG] monmap epoch 2 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277752+0000 mon.a (mon.0) 156 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:42.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277768+0000 mon.a (mon.0) 157 : cluster [DBG] last_changed 2024-09-13T22:29:29.228250+0000 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277781+0000 mon.a (mon.0) 158 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277799+0000 mon.a (mon.0) 159 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277816+0000 mon.a (mon.0) 160 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277834+0000 mon.a (mon.0) 161 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.277850+0000 mon.a (mon.0) 162 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.295783+0000 mon.a (mon.0) 163 : cluster [DBG] fsmap 2024-09-13T22:29:42.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.295833+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.296095+0000 mon.a (mon.0) 165 : cluster [DBG] mgrmap e13: y(active, since 64s) 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.296321+0000 mon.a (mon.0) 166 : cluster [INF] overall HEALTH_OK 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.303393+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.310564+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.321247+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.340729+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:34.187978+0000 mgr.y (mgr.14150) 49 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.386351+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.577733+0000 mon.a (mon.0) 172 : audit [DBG] from='client.? v1:172.21.15.192:0/2995696991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:34.620024+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:35.227935+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:35.243752+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:35.628046+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:35.628427+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:35.628635+0000 mon.a (mon.0) 178 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:35.648445+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:35.648951+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:36.188361+0000 mgr.y (mgr.14150) 50 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:36.620701+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:37.620912+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:38.188751+0000 mgr.y (mgr.14150) 51 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:38.620951+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:39.620371+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.936 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.189140+0000 mgr.y (mgr.14150) 52 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:40.621217+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.648373+0000 mon.a (mon.0) 186 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654141+0000 mon.a (mon.0) 187 : cluster [DBG] monmap epoch 3 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654172+0000 mon.a (mon.0) 188 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654189+0000 mon.a (mon.0) 189 : cluster [DBG] last_changed 2024-09-13T22:29:35.622057+0000 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654205+0000 mon.a (mon.0) 190 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654222+0000 mon.a (mon.0) 191 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654239+0000 mon.a (mon.0) 192 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654256+0000 mon.a (mon.0) 193 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654275+0000 mon.a (mon.0) 194 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.654292+0000 mon.a (mon.0) 195 : cluster [DBG] 2: v1:172.21.15.178:6790/0 mon.c 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.671840+0000 mon.a (mon.0) 196 : cluster [DBG] fsmap 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.671892+0000 mon.a (mon.0) 197 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:42.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.672157+0000 mon.a (mon.0) 198 : cluster [DBG] mgrmap e13: y(active, since 70s) 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.672443+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.679524+0000 mon.a (mon.0) 200 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.679570+0000 mon.a (mon.0) 201 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cluster 2024-09-13T22:29:40.679596+0000 mon.a (mon.0) 202 : cluster [WRN] mon.c (rank 2) addr v1:172.21.15.178:6790/0 is down (out of quorum) 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:40.684373+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:40.689871+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:40.695532+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:40.701137+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.351014+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.352519+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cephadm 2024-09-13T22:29:41.354370+0000 mgr.y (mgr.14150) 53 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: cephadm 2024-09-13T22:29:41.354775+0000 mgr.y (mgr.14150) 54 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:29:42.938 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.620836+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.623879+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.627778+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.635464+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.641302+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.647327+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.683152+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.687649+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.693428+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.698800+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.699801+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.700821+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:42.939 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:42 smithi178 bash[29889]: audit 2024-09-13T22:29:41.701749+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:37.623699+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.660447+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.660877+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.661013+0000 mon.a (mon.0) 223 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.684407+0000 mon.a (mon.0) 224 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688228+0000 mon.a (mon.0) 225 : cluster [DBG] monmap epoch 3 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688246+0000 mon.a (mon.0) 226 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688255+0000 mon.a (mon.0) 227 : cluster [DBG] last_changed 2024-09-13T22:29:35.622057+0000 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688265+0000 mon.a (mon.0) 228 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688279+0000 mon.a (mon.0) 229 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688290+0000 mon.a (mon.0) 230 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:43.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688300+0000 mon.a (mon.0) 231 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688310+0000 mon.a (mon.0) 232 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688319+0000 mon.a (mon.0) 233 : cluster [DBG] 2: v1:172.21.15.178:6790/0 mon.c 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688762+0000 mon.a (mon.0) 234 : cluster [DBG] fsmap 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688795+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.688927+0000 mon.a (mon.0) 236 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.689063+0000 mon.a (mon.0) 237 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.689080+0000 mon.a (mon.0) 238 : cluster [INF] Cluster is now healthy 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[22840]: cluster 2024-09-13T22:29:42.705626+0000 mon.a (mon.0) 239 : cluster [INF] overall HEALTH_OK 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:37.623699+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-13T22:29:43.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.660447+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.660877+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.661013+0000 mon.a (mon.0) 223 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.684407+0000 mon.a (mon.0) 224 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688228+0000 mon.a (mon.0) 225 : cluster [DBG] monmap epoch 3 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688246+0000 mon.a (mon.0) 226 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688255+0000 mon.a (mon.0) 227 : cluster [DBG] last_changed 2024-09-13T22:29:35.622057+0000 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688265+0000 mon.a (mon.0) 228 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688279+0000 mon.a (mon.0) 229 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688290+0000 mon.a (mon.0) 230 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688300+0000 mon.a (mon.0) 231 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688310+0000 mon.a (mon.0) 232 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688319+0000 mon.a (mon.0) 233 : cluster [DBG] 2: v1:172.21.15.178:6790/0 mon.c 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688762+0000 mon.a (mon.0) 234 : cluster [DBG] fsmap 2024-09-13T22:29:43.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688795+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:43.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.688927+0000 mon.a (mon.0) 236 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-09-13T22:29:43.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.689063+0000 mon.a (mon.0) 237 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-13T22:29:43.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.689080+0000 mon.a (mon.0) 238 : cluster [INF] Cluster is now healthy 2024-09-13T22:29:43.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:43 smithi178 bash[29889]: cluster 2024-09-13T22:29:42.705626+0000 mon.a (mon.0) 239 : cluster [INF] overall HEALTH_OK 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:37.623699+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.660447+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.660877+0000 mon.b (mon.1) 4 : cluster [INF] mon.b calling monitor election 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.661013+0000 mon.a (mon.0) 223 : cluster [INF] mon.a calling monitor election 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.684407+0000 mon.a (mon.0) 224 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688228+0000 mon.a (mon.0) 225 : cluster [DBG] monmap epoch 3 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688246+0000 mon.a (mon.0) 226 : cluster [DBG] fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688255+0000 mon.a (mon.0) 227 : cluster [DBG] last_changed 2024-09-13T22:29:35.622057+0000 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688265+0000 mon.a (mon.0) 228 : cluster [DBG] created 2024-09-13T22:27:26.887333+0000 2024-09-13T22:29:43.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688279+0000 mon.a (mon.0) 229 : cluster [DBG] min_mon_release 19 (squid) 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688290+0000 mon.a (mon.0) 230 : cluster [DBG] election_strategy: 1 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688300+0000 mon.a (mon.0) 231 : cluster [DBG] 0: v1:172.21.15.178:6789/0 mon.a 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688310+0000 mon.a (mon.0) 232 : cluster [DBG] 1: v1:172.21.15.192:6789/0 mon.b 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688319+0000 mon.a (mon.0) 233 : cluster [DBG] 2: v1:172.21.15.178:6790/0 mon.c 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688762+0000 mon.a (mon.0) 234 : cluster [DBG] fsmap 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688795+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.688927+0000 mon.a (mon.0) 236 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.689063+0000 mon.a (mon.0) 237 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.689080+0000 mon.a (mon.0) 238 : cluster [INF] Cluster is now healthy 2024-09-13T22:29:43.911 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:43 smithi192 bash[24608]: cluster 2024-09-13T22:29:42.705626+0000 mon.a (mon.0) 239 : cluster [INF] overall HEALTH_OK 2024-09-13T22:29:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: audit 2024-09-13T22:29:43.621115+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: audit 2024-09-13T22:29:44.119069+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: audit 2024-09-13T22:29:44.124174+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: cephadm 2024-09-13T22:29:44.125358+0000 mgr.y (mgr.14150) 60 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: audit 2024-09-13T22:29:44.125779+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: audit 2024-09-13T22:29:44.126698+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: audit 2024-09-13T22:29:44.127624+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: cephadm 2024-09-13T22:29:44.128968+0000 mgr.y (mgr.14150) 61 : cephadm [INF] Reconfiguring daemon mon.c on smithi178 2024-09-13T22:29:44.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:44 smithi192 bash[24608]: cluster 2024-09-13T22:29:44.190032+0000 mgr.y (mgr.14150) 62 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: audit 2024-09-13T22:29:43.621115+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: audit 2024-09-13T22:29:44.119069+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: audit 2024-09-13T22:29:44.124174+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: cephadm 2024-09-13T22:29:44.125358+0000 mgr.y (mgr.14150) 60 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: audit 2024-09-13T22:29:44.125779+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: audit 2024-09-13T22:29:44.126698+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: audit 2024-09-13T22:29:44.127624+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: cephadm 2024-09-13T22:29:44.128968+0000 mgr.y (mgr.14150) 61 : cephadm [INF] Reconfiguring daemon mon.c on smithi178 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[22840]: cluster 2024-09-13T22:29:44.190032+0000 mgr.y (mgr.14150) 62 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:44.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: audit 2024-09-13T22:29:43.621115+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: audit 2024-09-13T22:29:44.119069+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: audit 2024-09-13T22:29:44.124174+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: cephadm 2024-09-13T22:29:44.125358+0000 mgr.y (mgr.14150) 60 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: audit 2024-09-13T22:29:44.125779+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: audit 2024-09-13T22:29:44.126698+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: audit 2024-09-13T22:29:44.127624+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: cephadm 2024-09-13T22:29:44.128968+0000 mgr.y (mgr.14150) 61 : cephadm [INF] Reconfiguring daemon mon.c on smithi178 2024-09-13T22:29:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:44 smithi178 bash[29889]: cluster 2024-09-13T22:29:44.190032+0000 mgr.y (mgr.14150) 62 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:47.223 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:29:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: cluster 2024-09-13T22:29:46.190473+0000 mgr.y (mgr.14150) 63 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: audit 2024-09-13T22:29:46.312835+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:47.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: audit 2024-09-13T22:29:46.318670+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:47.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: cephadm 2024-09-13T22:29:46.320066+0000 mgr.y (mgr.14150) 64 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-13T22:29:47.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: audit 2024-09-13T22:29:46.320673+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:47.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: audit 2024-09-13T22:29:46.322083+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:47.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: audit 2024-09-13T22:29:46.323449+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:47.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:47 smithi192 bash[24608]: cephadm 2024-09-13T22:29:46.324876+0000 mgr.y (mgr.14150) 65 : cephadm [INF] Reconfiguring daemon mon.b on smithi192 2024-09-13T22:29:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: cluster 2024-09-13T22:29:46.190473+0000 mgr.y (mgr.14150) 63 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: audit 2024-09-13T22:29:46.312835+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: audit 2024-09-13T22:29:46.318670+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: cephadm 2024-09-13T22:29:46.320066+0000 mgr.y (mgr.14150) 64 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: audit 2024-09-13T22:29:46.320673+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: audit 2024-09-13T22:29:46.322083+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: audit 2024-09-13T22:29:46.323449+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[22840]: cephadm 2024-09-13T22:29:46.324876+0000 mgr.y (mgr.14150) 65 : cephadm [INF] Reconfiguring daemon mon.b on smithi192 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: cluster 2024-09-13T22:29:46.190473+0000 mgr.y (mgr.14150) 63 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: audit 2024-09-13T22:29:46.312835+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: audit 2024-09-13T22:29:46.318670+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: cephadm 2024-09-13T22:29:46.320066+0000 mgr.y (mgr.14150) 64 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: audit 2024-09-13T22:29:46.320673+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T22:29:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: audit 2024-09-13T22:29:46.322083+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T22:29:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: audit 2024-09-13T22:29:46.323449+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:47 smithi178 bash[29889]: cephadm 2024-09-13T22:29:46.324876+0000 mgr.y (mgr.14150) 65 : cephadm [INF] Reconfiguring daemon mon.b on smithi192 2024-09-13T22:29:48.546 INFO:teuthology.orchestra.run.smithi178.stdout:# minimal ceph.conf for 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:48.546 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-09-13T22:29:48.547 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:29:48.547 INFO:teuthology.orchestra.run.smithi178.stdout: mon_host = 172.21.15.178:6789/0 172.21.15.192:6789/0 v1:172.21.15.178:6790/0 2024-09-13T22:29:49.150 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T22:29:49.151 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:29:49.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T22:29:49.164 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:29:49.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:29:49.218 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:29:49.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T22:29:49.232 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:29:49.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:29:49.286 INFO:tasks.cephadm:Adding mgr.y on smithi178 2024-09-13T22:29:49.286 INFO:tasks.cephadm:Adding mgr.x on smithi192 2024-09-13T22:29:49.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi178=y;smithi192=x' 2024-09-13T22:29:49.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: cluster 2024-09-13T22:29:48.190924+0000 mgr.y (mgr.14150) 66 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:49.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.497116+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.503617+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.505770+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:49.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.508717+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:49.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.510155+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:49.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.518215+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[22840]: audit 2024-09-13T22:29:48.546837+0000 mon.a (mon.0) 257 : audit [DBG] from='client.? v1:172.21.15.178:0/1729382929' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:49.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: cluster 2024-09-13T22:29:48.190924+0000 mgr.y (mgr.14150) 66 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:49.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.497116+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.503617+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.505770+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:49.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.508717+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:49.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.510155+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:49.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.518215+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:49 smithi178 bash[29889]: audit 2024-09-13T22:29:48.546837+0000 mon.a (mon.0) 257 : audit [DBG] from='client.? v1:172.21.15.178:0/1729382929' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:49.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: cluster 2024-09-13T22:29:48.190924+0000 mgr.y (mgr.14150) 66 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.497116+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.503617+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.505770+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.508717+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.510155+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.518215+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:49.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:49 smithi192 bash[24608]: audit 2024-09-13T22:29:48.546837+0000 mon.a (mon.0) 257 : audit [DBG] from='client.? v1:172.21.15.178:0/1729382929' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:51.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:51 smithi192 bash[24608]: cluster 2024-09-13T22:29:50.191350+0000 mgr.y (mgr.14150) 67 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:51 smithi178 bash[22840]: cluster 2024-09-13T22:29:50.191350+0000 mgr.y (mgr.14150) 67 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:51 smithi178 bash[29889]: cluster 2024-09-13T22:29:50.191350+0000 mgr.y (mgr.14150) 67 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:53.048 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:29:53.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:53 smithi192 bash[24608]: cluster 2024-09-13T22:29:52.191782+0000 mgr.y (mgr.14150) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:53 smithi178 bash[22840]: cluster 2024-09-13T22:29:52.191782+0000 mgr.y (mgr.14150) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:53 smithi178 bash[29889]: cluster 2024-09-13T22:29:52.191782+0000 mgr.y (mgr.14150) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:54.432 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-09-13T22:29:54.980 DEBUG:teuthology.orchestra.run.smithi192:mgr.x> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.x.service 2024-09-13T22:29:54.982 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T22:29:54.982 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:29:54.982 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T22:29:54.989 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T22:29:54.989 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-09-13T22:29:55.039 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T22:29:55.039 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.039 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-09-13T22:29:55.040 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.040 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-09-13 22:19:37.406472464 +0000 2024-09-13T22:29:55.040 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-09-13 22:19:37.174474536 +0000 2024-09-13T22:29:55.040 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-09-13 22:19:37.174474536 +0000 2024-09-13T22:29:55.040 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-09-13T22:29:55.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T22:29:55.093 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-09-13T22:29:55.093 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-09-13T22:29:55.094 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000319457 s, 1.6 MB/s 2024-09-13T22:29:55.095 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T22:29:55.144 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-09-13T22:29:55.190 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-09-13 22:19:37.866468354 +0000 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-09-13 22:19:37.618470570 +0000 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-09-13 22:19:37.618470570 +0000 2024-09-13T22:29:55.191 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-09-13T22:29:55.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T22:29:55.245 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-09-13T22:29:55.245 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-09-13T22:29:55.245 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000308977 s, 1.7 MB/s 2024-09-13T22:29:55.246 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T22:29:55.297 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-09-13T22:29:55.343 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T22:29:55.343 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.344 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 914 Links: 1 2024-09-13T22:29:55.344 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.344 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-09-13 22:19:38.394463639 +0000 2024-09-13T22:29:55.344 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-09-13 22:19:38.118466104 +0000 2024-09-13T22:29:55.344 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-09-13 22:19:38.118466104 +0000 2024-09-13T22:29:55.344 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-09-13T22:29:55.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T22:29:55.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: cluster 2024-09-13T22:29:54.192178+0000 mgr.y (mgr.14150) 69 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.431277+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.432921+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.436409+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.437939+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.446284+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.449344+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.451722+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T22:29:55.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.456149+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T22:29:55.397 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[22840]: audit 2024-09-13T22:29:54.457768+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:55.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: cluster 2024-09-13T22:29:54.192178+0000 mgr.y (mgr.14150) 69 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:55.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.431277+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:55.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.432921+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:55.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.436409+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:55.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.437939+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:55.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.446284+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:55.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.449344+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T22:29:55.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.451722+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T22:29:55.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.456149+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T22:29:55.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:55 smithi178 bash[29889]: audit 2024-09-13T22:29:54.457768+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:55.399 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-09-13T22:29:55.399 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-09-13T22:29:55.399 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000210276 s, 2.4 MB/s 2024-09-13T22:29:55.399 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T22:29:55.453 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-09-13 22:19:38.618461638 +0000 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-09-13 22:19:38.614461674 +0000 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-09-13 22:19:38.614461674 +0000 2024-09-13T22:29:55.499 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-09-13T22:29:55.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T22:29:55.553 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-09-13T22:29:55.554 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-09-13T22:29:55.554 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000324454 s, 1.6 MB/s 2024-09-13T22:29:55.555 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T22:29:55.604 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:29:55.604 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T22:29:55.611 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T22:29:55.611 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-09-13T22:29:55.658 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T22:29:55.658 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.658 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 871 Links: 1 2024-09-13T22:29:55.658 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.658 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-13 22:19:37.885563812 +0000 2024-09-13T22:29:55.658 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-13 22:19:37.633566063 +0000 2024-09-13T22:29:55.659 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-13 22:19:37.633566063 +0000 2024-09-13T22:29:55.659 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-09-13T22:29:55.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: cluster 2024-09-13T22:29:54.192178+0000 mgr.y (mgr.14150) 69 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.431277+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.432921+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.436409+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.437939+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.446284+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.449344+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.451722+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.456149+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T22:29:55.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:55 smithi192 bash[24608]: audit 2024-09-13T22:29:54.457768+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:29:55.673 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-13T22:29:55.673 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-13T22:29:55.673 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000443979 s, 1.2 MB/s 2024-09-13T22:29:55.674 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T22:29:55.723 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-13 22:19:38.341559739 +0000 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-13 22:19:38.089561990 +0000 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-13 22:19:38.089561990 +0000 2024-09-13T22:29:55.774 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-09-13T22:29:55.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T22:29:55.832 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-13T22:29:55.832 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-13T22:29:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000442816 s, 1.2 MB/s 2024-09-13T22:29:55.834 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T22:29:55.883 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-09-13T22:29:55.934 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T22:29:55.934 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:55.935 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-09-13T22:29:55.935 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:55.935 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-13 22:19:38.841555273 +0000 2024-09-13T22:29:55.935 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-13 22:19:38.597557453 +0000 2024-09-13T22:29:55.935 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-13 22:19:38.597557453 +0000 2024-09-13T22:29:55.935 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-09-13T22:29:55.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T22:29:55.989 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-13T22:29:55.989 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-13T22:29:55.989 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000421502 s, 1.2 MB/s 2024-09-13T22:29:55.990 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T22:29:56.043 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-09-13T22:29:56.089 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T22:29:56.089 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T22:29:56.090 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-09-13T22:29:56.090 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T22:29:56.090 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-13 22:19:39.065553272 +0000 2024-09-13T22:29:56.090 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-13 22:19:39.061553307 +0000 2024-09-13T22:29:56.090 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-13 22:19:39.061553307 +0000 2024-09-13T22:29:56.090 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-09-13T22:29:56.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T22:29:56.146 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-13T22:29:56.146 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-13T22:29:56.146 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000268411 s, 1.9 MB/s 2024-09-13T22:29:56.147 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T22:29:56.196 INFO:tasks.cephadm:Deploying osd.0 on smithi178 with /dev/vg_nvme/lv_4... 2024-09-13T22:29:56.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T22:29:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:56 smithi192 bash[24608]: audit 2024-09-13T22:29:54.422373+0000 mgr.y (mgr.14150) 70 : audit [DBG] from='client.24115 v1:172.21.15.192:0/2708224461' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi178=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:56 smithi192 bash[24608]: cephadm 2024-09-13T22:29:54.425392+0000 mgr.y (mgr.14150) 71 : cephadm [INF] Saving service mgr spec with placement smithi178=y;smithi192=x;count:2 2024-09-13T22:29:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:56 smithi192 bash[24608]: cephadm 2024-09-13T22:29:54.459464+0000 mgr.y (mgr.14150) 72 : cephadm [INF] Deploying daemon mgr.x on smithi192 2024-09-13T22:29:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:56 smithi178 bash[22840]: audit 2024-09-13T22:29:54.422373+0000 mgr.y (mgr.14150) 70 : audit [DBG] from='client.24115 v1:172.21.15.192:0/2708224461' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi178=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:56.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:56 smithi178 bash[22840]: cephadm 2024-09-13T22:29:54.425392+0000 mgr.y (mgr.14150) 71 : cephadm [INF] Saving service mgr spec with placement smithi178=y;smithi192=x;count:2 2024-09-13T22:29:56.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:56 smithi178 bash[22840]: cephadm 2024-09-13T22:29:54.459464+0000 mgr.y (mgr.14150) 72 : cephadm [INF] Deploying daemon mgr.x on smithi192 2024-09-13T22:29:56.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:56 smithi178 bash[29889]: audit 2024-09-13T22:29:54.422373+0000 mgr.y (mgr.14150) 70 : audit [DBG] from='client.24115 v1:172.21.15.192:0/2708224461' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi178=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:29:56.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:56 smithi178 bash[29889]: cephadm 2024-09-13T22:29:54.425392+0000 mgr.y (mgr.14150) 71 : cephadm [INF] Saving service mgr spec with placement smithi178=y;smithi192=x;count:2 2024-09-13T22:29:56.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:56 smithi178 bash[29889]: cephadm 2024-09-13T22:29:54.459464+0000 mgr.y (mgr.14150) 72 : cephadm [INF] Deploying daemon mgr.x on smithi192 2024-09-13T22:29:57.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:57.314 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:57.314 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:57.314 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:57.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:57.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:57 smithi192 bash[24608]: cluster 2024-09-13T22:29:56.192585+0000 mgr.y (mgr.14150) 73 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:57.659 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:29:57.660 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:57 smithi192 systemd[1]: Started Ceph mgr.x for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:29:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:57 smithi178 bash[22840]: cluster 2024-09-13T22:29:56.192585+0000 mgr.y (mgr.14150) 73 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:57 smithi178 bash[29889]: cluster 2024-09-13T22:29:56.192585+0000 mgr.y (mgr.14150) 73 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:58.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[24608]: audit 2024-09-13T22:29:57.459772+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[24608]: audit 2024-09-13T22:29:57.466339+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[24608]: audit 2024-09-13T22:29:57.473539+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[24608]: audit 2024-09-13T22:29:57.480184+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[24608]: audit 2024-09-13T22:29:57.537617+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:58.775 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[24608]: cluster 2024-09-13T22:29:58.193003+0000 mgr.y (mgr.14150) 74 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:58.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[22840]: audit 2024-09-13T22:29:57.459772+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[22840]: audit 2024-09-13T22:29:57.466339+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[22840]: audit 2024-09-13T22:29:57.473539+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[22840]: audit 2024-09-13T22:29:57.480184+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[22840]: audit 2024-09-13T22:29:57.537617+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[22840]: cluster 2024-09-13T22:29:58.193003+0000 mgr.y (mgr.14150) 74 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:58.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[29889]: audit 2024-09-13T22:29:57.459772+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[29889]: audit 2024-09-13T22:29:57.466339+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[29889]: audit 2024-09-13T22:29:57.473539+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[29889]: audit 2024-09-13T22:29:57.480184+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:29:58.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[29889]: audit 2024-09-13T22:29:57.537617+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:29:58.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:29:58 smithi178 bash[29889]: cluster 2024-09-13T22:29:58.193003+0000 mgr.y (mgr.14150) 74 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:29:59.159 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:58 smithi192 bash[25468]: debug 2024-09-13T22:29:58.773+0000 7fd39f889040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T22:30:00.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:00 smithi192 bash[24608]: cluster 2024-09-13T22:30:00.000159+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-09-13T22:30:00.409 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:29:59 smithi192 bash[25468]: debug 2024-09-13T22:29:59.965+0000 7fd39f889040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T22:30:00.409 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:00 smithi192 bash[25468]: debug 2024-09-13T22:30:00.029+0000 7fd39f889040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T22:30:00.409 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:00 smithi192 bash[25468]: debug 2024-09-13T22:30:00.181+0000 7fd39f889040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T22:30:00.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:00 smithi178 bash[29889]: cluster 2024-09-13T22:30:00.000159+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-09-13T22:30:00.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:00 smithi178 bash[22840]: cluster 2024-09-13T22:30:00.000159+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-09-13T22:30:00.935 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:30:01.155 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:00 smithi192 bash[25468]: debug 2024-09-13T22:30:00.829+0000 7fd39f889040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T22:30:01.155 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:00 smithi192 bash[25468]: debug 2024-09-13T22:30:00.901+0000 7fd39f889040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T22:30:01.155 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:00 smithi192 bash[25468]: debug 2024-09-13T22:30:00.969+0000 7fd39f889040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T22:30:01.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[24608]: cluster 2024-09-13T22:30:00.193432+0000 mgr.y (mgr.14150) 75 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:01.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[24608]: audit 2024-09-13T22:30:00.251470+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:01.409 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.153+0000 7fd39f889040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T22:30:01.410 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.229+0000 7fd39f889040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T22:30:01.410 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.297+0000 7fd39f889040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T22:30:01.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:01 smithi178 bash[22840]: cluster 2024-09-13T22:30:00.193432+0000 mgr.y (mgr.14150) 75 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:01.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:01 smithi178 bash[22840]: audit 2024-09-13T22:30:00.251470+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:01.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:01 smithi178 bash[29889]: cluster 2024-09-13T22:30:00.193432+0000 mgr.y (mgr.14150) 75 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:01.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:01 smithi178 bash[29889]: audit 2024-09-13T22:30:00.251470+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:01.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.553+0000 7fd39f889040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T22:30:01.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.617+0000 7fd39f889040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T22:30:02.140 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.813+0000 7fd39f889040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T22:30:02.140 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:01 smithi192 bash[25468]: debug 2024-09-13T22:30:01.889+0000 7fd39f889040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T22:30:02.409 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:02 smithi192 bash[25468]: debug 2024-09-13T22:30:02.137+0000 7fd39f889040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T22:30:02.409 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:02 smithi192 bash[25468]: debug 2024-09-13T22:30:02.313+0000 7fd39f889040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:30:02.757 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:02 smithi192 bash[25468]: debug 2024-09-13T22:30:02.473+0000 7fd39f889040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T22:30:02.757 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:02 smithi192 bash[25468]: debug 2024-09-13T22:30:02.557+0000 7fd39f889040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T22:30:03.161 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:02 smithi192 bash[25468]: debug 2024-09-13T22:30:02.753+0000 7fd39f889040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:30:03.564 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: cluster 2024-09-13T22:30:02.193762+0000 mgr.y (mgr.14150) 76 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.756807+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.762807+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.763786+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.764807+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.771477+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.830325+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.831475+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[29889]: audit 2024-09-13T22:30:02.832403+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:03.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: cluster 2024-09-13T22:30:02.193762+0000 mgr.y (mgr.14150) 76 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.756807+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.762807+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.763786+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.764807+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.771477+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.830325+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.831475+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:03 smithi178 bash[22840]: audit 2024-09-13T22:30:02.832403+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:03.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: cluster 2024-09-13T22:30:02.193762+0000 mgr.y (mgr.14150) 76 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.756807+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.762807+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.763786+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.764807+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.771477+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.830325+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.831475+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T22:30:03.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[24608]: audit 2024-09-13T22:30:02.832403+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:03.860 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[25468]: debug 2024-09-13T22:30:03.561+0000 7fd39f889040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T22:30:03.860 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[25468]: debug 2024-09-13T22:30:03.637+0000 7fd39f889040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T22:30:03.860 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[25468]: debug 2024-09-13T22:30:03.857+0000 7fd39f889040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T22:30:04.159 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:03 smithi192 bash[25468]: debug 2024-09-13T22:30:03.929+0000 7fd39f889040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T22:30:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:04 smithi178 bash[29889]: cephadm 2024-09-13T22:30:02.829630+0000 mgr.y (mgr.14150) 77 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-13T22:30:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:04 smithi178 bash[29889]: cephadm 2024-09-13T22:30:02.833688+0000 mgr.y (mgr.14150) 78 : cephadm [INF] Reconfiguring daemon mgr.y on smithi178 2024-09-13T22:30:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:04 smithi178 bash[22840]: cephadm 2024-09-13T22:30:02.829630+0000 mgr.y (mgr.14150) 77 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-13T22:30:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:04 smithi178 bash[22840]: cephadm 2024-09-13T22:30:02.833688+0000 mgr.y (mgr.14150) 78 : cephadm [INF] Reconfiguring daemon mgr.y on smithi178 2024-09-13T22:30:04.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:04 smithi192 bash[24608]: cephadm 2024-09-13T22:30:02.829630+0000 mgr.y (mgr.14150) 77 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-13T22:30:04.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:04 smithi192 bash[24608]: cephadm 2024-09-13T22:30:02.833688+0000 mgr.y (mgr.14150) 78 : cephadm [INF] Reconfiguring daemon mgr.y on smithi178 2024-09-13T22:30:04.659 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:04 smithi192 bash[25468]: debug 2024-09-13T22:30:04.317+0000 7fd39f889040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T22:30:04.660 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:30:04 smithi192 bash[25468]: debug 2024-09-13T22:30:04.393+0000 7fd39f889040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T22:30:04.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:30:04.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi178:vg_nvme/lv_4 2024-09-13T22:30:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:05 smithi192 bash[24608]: cluster 2024-09-13T22:30:04.194098+0000 mgr.y (mgr.14150) 79 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:05 smithi192 bash[24608]: cluster 2024-09-13T22:30:04.421971+0000 mon.a (mon.0) 282 : cluster [DBG] Standby manager daemon x started 2024-09-13T22:30:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:05 smithi192 bash[24608]: audit 2024-09-13T22:30:04.424291+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T22:30:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:05 smithi192 bash[24608]: audit 2024-09-13T22:30:04.424809+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T22:30:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:05 smithi192 bash[24608]: audit 2024-09-13T22:30:04.425905+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T22:30:05.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:05 smithi192 bash[24608]: audit 2024-09-13T22:30:04.426388+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[29889]: cluster 2024-09-13T22:30:04.194098+0000 mgr.y (mgr.14150) 79 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[29889]: cluster 2024-09-13T22:30:04.421971+0000 mon.a (mon.0) 282 : cluster [DBG] Standby manager daemon x started 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[29889]: audit 2024-09-13T22:30:04.424291+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[29889]: audit 2024-09-13T22:30:04.424809+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[29889]: audit 2024-09-13T22:30:04.425905+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[29889]: audit 2024-09-13T22:30:04.426388+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[22840]: cluster 2024-09-13T22:30:04.194098+0000 mgr.y (mgr.14150) 79 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[22840]: cluster 2024-09-13T22:30:04.421971+0000 mon.a (mon.0) 282 : cluster [DBG] Standby manager daemon x started 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[22840]: audit 2024-09-13T22:30:04.424291+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T22:30:05.670 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[22840]: audit 2024-09-13T22:30:04.424809+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T22:30:05.671 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[22840]: audit 2024-09-13T22:30:04.425905+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T22:30:05.671 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:05 smithi178 bash[22840]: audit 2024-09-13T22:30:04.426388+0000 mon.b (mon.1) 8 : audit [DBG] from='mgr.? v1:172.21.15.192:0/3938332492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T22:30:06.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:06 smithi192 bash[24608]: cluster 2024-09-13T22:30:05.273585+0000 mon.a (mon.0) 283 : cluster [DBG] mgrmap e14: y(active, since 95s), standbys: x 2024-09-13T22:30:06.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:06 smithi192 bash[24608]: audit 2024-09-13T22:30:05.274265+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T22:30:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:06 smithi178 bash[22840]: cluster 2024-09-13T22:30:05.273585+0000 mon.a (mon.0) 283 : cluster [DBG] mgrmap e14: y(active, since 95s), standbys: x 2024-09-13T22:30:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:06 smithi178 bash[22840]: audit 2024-09-13T22:30:05.274265+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T22:30:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:06 smithi178 bash[29889]: cluster 2024-09-13T22:30:05.273585+0000 mon.a (mon.0) 283 : cluster [DBG] mgrmap e14: y(active, since 95s), standbys: x 2024-09-13T22:30:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:06 smithi178 bash[29889]: audit 2024-09-13T22:30:05.274265+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T22:30:07.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: cluster 2024-09-13T22:30:06.194480+0000 mgr.y (mgr.14150) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:07.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: audit 2024-09-13T22:30:06.406188+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: audit 2024-09-13T22:30:06.412898+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: audit 2024-09-13T22:30:06.415887+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:07.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: audit 2024-09-13T22:30:06.419039+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:07.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: audit 2024-09-13T22:30:06.420481+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:07.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:07 smithi192 bash[24608]: audit 2024-09-13T22:30:06.428795+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: cluster 2024-09-13T22:30:06.194480+0000 mgr.y (mgr.14150) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: audit 2024-09-13T22:30:06.406188+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: audit 2024-09-13T22:30:06.412898+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: audit 2024-09-13T22:30:06.415887+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: audit 2024-09-13T22:30:06.419039+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: audit 2024-09-13T22:30:06.420481+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[22840]: audit 2024-09-13T22:30:06.428795+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: cluster 2024-09-13T22:30:06.194480+0000 mgr.y (mgr.14150) 80 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: audit 2024-09-13T22:30:06.406188+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: audit 2024-09-13T22:30:06.412898+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: audit 2024-09-13T22:30:06.415887+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: audit 2024-09-13T22:30:06.419039+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: audit 2024-09-13T22:30:06.420481+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:07 smithi178 bash[29889]: audit 2024-09-13T22:30:06.428795+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:09.552 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:30:09.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:09 smithi192 bash[24608]: cluster 2024-09-13T22:30:08.194924+0000 mgr.y (mgr.14150) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:09.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:09 smithi178 bash[22840]: cluster 2024-09-13T22:30:08.194924+0000 mgr.y (mgr.14150) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:09 smithi178 bash[29889]: cluster 2024-09-13T22:30:08.194924+0000 mgr.y (mgr.14150) 81 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[22840]: cluster 2024-09-13T22:30:10.195362+0000 mgr.y (mgr.14150) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[22840]: audit 2024-09-13T22:30:10.925112+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:30:11.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[22840]: audit 2024-09-13T22:30:10.927857+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:30:11.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[22840]: audit 2024-09-13T22:30:10.928683+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:11.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[29889]: cluster 2024-09-13T22:30:10.195362+0000 mgr.y (mgr.14150) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:11.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[29889]: audit 2024-09-13T22:30:10.925112+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:30:11.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[29889]: audit 2024-09-13T22:30:10.927857+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:30:11.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:11 smithi178 bash[29889]: audit 2024-09-13T22:30:10.928683+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:11.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:11 smithi192 bash[24608]: cluster 2024-09-13T22:30:10.195362+0000 mgr.y (mgr.14150) 82 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:11.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:11 smithi192 bash[24608]: audit 2024-09-13T22:30:10.925112+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:30:11.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:11 smithi192 bash[24608]: audit 2024-09-13T22:30:10.927857+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:30:11.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:11 smithi192 bash[24608]: audit 2024-09-13T22:30:10.928683+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:12.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:12 smithi178 bash[22840]: audit 2024-09-13T22:30:10.922451+0000 mgr.y (mgr.14150) 83 : audit [DBG] from='client.14208 v1:172.21.15.178:0/3313593242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:30:12.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:12 smithi178 bash[29889]: audit 2024-09-13T22:30:10.922451+0000 mgr.y (mgr.14150) 83 : audit [DBG] from='client.14208 v1:172.21.15.178:0/3313593242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:30:12.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:12 smithi192 bash[24608]: audit 2024-09-13T22:30:10.922451+0000 mgr.y (mgr.14150) 83 : audit [DBG] from='client.14208 v1:172.21.15.178:0/3313593242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:30:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:13 smithi178 bash[22840]: cluster 2024-09-13T22:30:12.195809+0000 mgr.y (mgr.14150) 84 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:13 smithi178 bash[29889]: cluster 2024-09-13T22:30:12.195809+0000 mgr.y (mgr.14150) 84 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:13.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:13 smithi192 bash[24608]: cluster 2024-09-13T22:30:12.195809+0000 mgr.y (mgr.14150) 84 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:15 smithi178 bash[22840]: cluster 2024-09-13T22:30:14.196241+0000 mgr.y (mgr.14150) 85 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:15.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:15 smithi178 bash[29889]: cluster 2024-09-13T22:30:14.196241+0000 mgr.y (mgr.14150) 85 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:15.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:15 smithi192 bash[24608]: cluster 2024-09-13T22:30:14.196241+0000 mgr.y (mgr.14150) 85 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:17 smithi178 bash[22840]: cluster 2024-09-13T22:30:16.196697+0000 mgr.y (mgr.14150) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:17 smithi178 bash[29889]: cluster 2024-09-13T22:30:16.196697+0000 mgr.y (mgr.14150) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:17.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:17 smithi192 bash[24608]: cluster 2024-09-13T22:30:16.196697+0000 mgr.y (mgr.14150) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:18 smithi178 bash[22840]: cluster 2024-09-13T22:30:18.197096+0000 mgr.y (mgr.14150) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:18.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:18 smithi178 bash[29889]: cluster 2024-09-13T22:30:18.197096+0000 mgr.y (mgr.14150) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:18 smithi192 bash[24608]: cluster 2024-09-13T22:30:18.197096+0000 mgr.y (mgr.14150) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[22840]: audit 2024-09-13T22:30:19.185503+0000 mon.a (mon.0) 294 : audit [INF] from='client.? v1:172.21.15.178:0/2290988680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac"}]: dispatch 2024-09-13T22:30:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[22840]: audit 2024-09-13T22:30:19.187199+0000 mon.a (mon.0) 295 : audit [INF] from='client.? v1:172.21.15.178:0/2290988680' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac"}]': finished 2024-09-13T22:30:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[22840]: cluster 2024-09-13T22:30:19.190222+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-13T22:30:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[22840]: audit 2024-09-13T22:30:19.190507+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:19.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[29889]: audit 2024-09-13T22:30:19.185503+0000 mon.a (mon.0) 294 : audit [INF] from='client.? v1:172.21.15.178:0/2290988680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac"}]: dispatch 2024-09-13T22:30:19.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[29889]: audit 2024-09-13T22:30:19.187199+0000 mon.a (mon.0) 295 : audit [INF] from='client.? v1:172.21.15.178:0/2290988680' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac"}]': finished 2024-09-13T22:30:19.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[29889]: cluster 2024-09-13T22:30:19.190222+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-13T22:30:19.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:19 smithi178 bash[29889]: audit 2024-09-13T22:30:19.190507+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:19.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:19 smithi192 bash[24608]: audit 2024-09-13T22:30:19.185503+0000 mon.a (mon.0) 294 : audit [INF] from='client.? v1:172.21.15.178:0/2290988680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac"}]: dispatch 2024-09-13T22:30:19.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:19 smithi192 bash[24608]: audit 2024-09-13T22:30:19.187199+0000 mon.a (mon.0) 295 : audit [INF] from='client.? v1:172.21.15.178:0/2290988680' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac"}]': finished 2024-09-13T22:30:19.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:19 smithi192 bash[24608]: cluster 2024-09-13T22:30:19.190222+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-13T22:30:19.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:19 smithi192 bash[24608]: audit 2024-09-13T22:30:19.190507+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:20.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:20 smithi192 bash[24608]: audit 2024-09-13T22:30:19.781884+0000 mon.a (mon.0) 298 : audit [DBG] from='client.? v1:172.21.15.178:0/2496189778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:30:20.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:20 smithi192 bash[24608]: cluster 2024-09-13T22:30:20.197445+0000 mgr.y (mgr.14150) 88 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:20 smithi178 bash[22840]: audit 2024-09-13T22:30:19.781884+0000 mon.a (mon.0) 298 : audit [DBG] from='client.? v1:172.21.15.178:0/2496189778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:30:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:20 smithi178 bash[22840]: cluster 2024-09-13T22:30:20.197445+0000 mgr.y (mgr.14150) 88 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:20.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:20 smithi178 bash[29889]: audit 2024-09-13T22:30:19.781884+0000 mon.a (mon.0) 298 : audit [DBG] from='client.? v1:172.21.15.178:0/2496189778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:30:20.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:20 smithi178 bash[29889]: cluster 2024-09-13T22:30:20.197445+0000 mgr.y (mgr.14150) 88 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:23.568 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:23 smithi178 bash[22840]: cluster 2024-09-13T22:30:22.197856+0000 mgr.y (mgr.14150) 89 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:23.568 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:23 smithi178 bash[29889]: cluster 2024-09-13T22:30:22.197856+0000 mgr.y (mgr.14150) 89 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:23 smithi192 bash[24608]: cluster 2024-09-13T22:30:22.197856+0000 mgr.y (mgr.14150) 89 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:25.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:25 smithi192 bash[24608]: cluster 2024-09-13T22:30:24.198230+0000 mgr.y (mgr.14150) 90 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:25.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:25 smithi178 bash[22840]: cluster 2024-09-13T22:30:24.198230+0000 mgr.y (mgr.14150) 90 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:25.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:25 smithi178 bash[29889]: cluster 2024-09-13T22:30:24.198230+0000 mgr.y (mgr.14150) 90 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:27.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:27 smithi192 bash[24608]: cluster 2024-09-13T22:30:26.198622+0000 mgr.y (mgr.14150) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:27 smithi178 bash[22840]: cluster 2024-09-13T22:30:26.198622+0000 mgr.y (mgr.14150) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:27 smithi178 bash[29889]: cluster 2024-09-13T22:30:26.198622+0000 mgr.y (mgr.14150) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:29.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:29 smithi192 bash[24608]: cluster 2024-09-13T22:30:28.199066+0000 mgr.y (mgr.14150) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:29 smithi178 bash[22840]: cluster 2024-09-13T22:30:28.199066+0000 mgr.y (mgr.14150) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:29 smithi178 bash[29889]: cluster 2024-09-13T22:30:28.199066+0000 mgr.y (mgr.14150) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:31 smithi192 bash[24608]: cluster 2024-09-13T22:30:30.199380+0000 mgr.y (mgr.14150) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:31 smithi178 bash[22840]: cluster 2024-09-13T22:30:30.199380+0000 mgr.y (mgr.14150) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:31 smithi178 bash[29889]: cluster 2024-09-13T22:30:30.199380+0000 mgr.y (mgr.14150) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:33.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:33 smithi192 bash[24608]: cluster 2024-09-13T22:30:32.199731+0000 mgr.y (mgr.14150) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:33.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:33 smithi192 bash[24608]: audit 2024-09-13T22:30:32.727017+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T22:30:33.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:33 smithi192 bash[24608]: audit 2024-09-13T22:30:32.728704+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:33 smithi178 bash[22840]: cluster 2024-09-13T22:30:32.199731+0000 mgr.y (mgr.14150) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:33 smithi178 bash[22840]: audit 2024-09-13T22:30:32.727017+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T22:30:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:33 smithi178 bash[22840]: audit 2024-09-13T22:30:32.728704+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:33 smithi178 bash[29889]: cluster 2024-09-13T22:30:32.199731+0000 mgr.y (mgr.14150) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:33 smithi178 bash[29889]: audit 2024-09-13T22:30:32.727017+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T22:30:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:33 smithi178 bash[29889]: audit 2024-09-13T22:30:32.728704+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:34 smithi178 bash[29889]: cephadm 2024-09-13T22:30:32.730083+0000 mgr.y (mgr.14150) 95 : cephadm [INF] Deploying daemon osd.0 on smithi178 2024-09-13T22:30:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:34 smithi178 bash[22840]: cephadm 2024-09-13T22:30:32.730083+0000 mgr.y (mgr.14150) 95 : cephadm [INF] Deploying daemon osd.0 on smithi178 2024-09-13T22:30:34.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:34 smithi192 bash[24608]: cephadm 2024-09-13T22:30:32.730083+0000 mgr.y (mgr.14150) 95 : cephadm [INF] Deploying daemon osd.0 on smithi178 2024-09-13T22:30:35.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:35 smithi178 bash[22840]: cluster 2024-09-13T22:30:34.200053+0000 mgr.y (mgr.14150) 96 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:35.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:35 smithi178 bash[29889]: cluster 2024-09-13T22:30:34.200053+0000 mgr.y (mgr.14150) 96 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:35 smithi192 bash[24608]: cluster 2024-09-13T22:30:34.200053+0000 mgr.y (mgr.14150) 96 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:37.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:30:37.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:30:37.181 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:30:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:30:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:37 smithi192 bash[24608]: cluster 2024-09-13T22:30:36.200437+0000 mgr.y (mgr.14150) 97 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:37.680 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:30:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:30:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:30:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:37 smithi178 bash[22840]: cluster 2024-09-13T22:30:36.200437+0000 mgr.y (mgr.14150) 97 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:30:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:37 smithi178 bash[29889]: cluster 2024-09-13T22:30:36.200437+0000 mgr.y (mgr.14150) 97 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:38.543 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:38 smithi178 bash[22840]: audit 2024-09-13T22:30:37.380513+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:38.543 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:38 smithi178 bash[22840]: audit 2024-09-13T22:30:37.384269+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:38.543 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:38 smithi178 bash[22840]: audit 2024-09-13T22:30:37.388384+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:38.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:38 smithi178 bash[29889]: audit 2024-09-13T22:30:37.380513+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:38.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:38 smithi178 bash[29889]: audit 2024-09-13T22:30:37.384269+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:38.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:38 smithi178 bash[29889]: audit 2024-09-13T22:30:37.388384+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:38 smithi192 bash[24608]: audit 2024-09-13T22:30:37.380513+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:38 smithi192 bash[24608]: audit 2024-09-13T22:30:37.384269+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:38 smithi192 bash[24608]: audit 2024-09-13T22:30:37.388384+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:39.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:39 smithi192 bash[24608]: cluster 2024-09-13T22:30:38.200902+0000 mgr.y (mgr.14150) 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:39 smithi178 bash[22840]: cluster 2024-09-13T22:30:38.200902+0000 mgr.y (mgr.14150) 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:39 smithi178 bash[29889]: cluster 2024-09-13T22:30:38.200902+0000 mgr.y (mgr.14150) 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:41.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:41 smithi192 bash[24608]: cluster 2024-09-13T22:30:40.201350+0000 mgr.y (mgr.14150) 99 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:41 smithi178 bash[22840]: cluster 2024-09-13T22:30:40.201350+0000 mgr.y (mgr.14150) 99 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:41 smithi178 bash[29889]: cluster 2024-09-13T22:30:40.201350+0000 mgr.y (mgr.14150) 99 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:43.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:43 smithi192 bash[24608]: cluster 2024-09-13T22:30:42.201720+0000 mgr.y (mgr.14150) 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:43.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:43 smithi178 bash[22840]: cluster 2024-09-13T22:30:42.201720+0000 mgr.y (mgr.14150) 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:43.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:43 smithi178 bash[29889]: cluster 2024-09-13T22:30:42.201720+0000 mgr.y (mgr.14150) 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[22840]: audit 2024-09-13T22:30:43.681678+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[22840]: audit 2024-09-13T22:30:43.688481+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[22840]: cluster 2024-09-13T22:30:44.202061+0000 mgr.y (mgr.14150) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[22840]: audit 2024-09-13T22:30:44.345532+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[22840]: audit 2024-09-13T22:30:44.346319+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:44.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[22840]: audit 2024-09-13T22:30:44.351597+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:44.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[29889]: audit 2024-09-13T22:30:43.681678+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:44.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[29889]: audit 2024-09-13T22:30:43.688481+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:44.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[29889]: cluster 2024-09-13T22:30:44.202061+0000 mgr.y (mgr.14150) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:44.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[29889]: audit 2024-09-13T22:30:44.345532+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[29889]: audit 2024-09-13T22:30:44.346319+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:44.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:44 smithi178 bash[29889]: audit 2024-09-13T22:30:44.351597+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:45.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:44 smithi192 bash[24608]: audit 2024-09-13T22:30:43.681678+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:45.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:44 smithi192 bash[24608]: audit 2024-09-13T22:30:43.688481+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:45.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:44 smithi192 bash[24608]: cluster 2024-09-13T22:30:44.202061+0000 mgr.y (mgr.14150) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:45.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:44 smithi192 bash[24608]: audit 2024-09-13T22:30:44.345532+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:30:45.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:44 smithi192 bash[24608]: audit 2024-09-13T22:30:44.346319+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:30:45.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:44 smithi192 bash[24608]: audit 2024-09-13T22:30:44.351597+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:47.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:47 smithi178 bash[22840]: cluster 2024-09-13T22:30:46.202407+0000 mgr.y (mgr.14150) 102 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:47.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:47 smithi178 bash[22840]: audit 2024-09-13T22:30:46.805868+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T22:30:47.615 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:47 smithi178 bash[29889]: cluster 2024-09-13T22:30:46.202407+0000 mgr.y (mgr.14150) 102 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:47.615 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:47 smithi178 bash[29889]: audit 2024-09-13T22:30:46.805868+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T22:30:47.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:47 smithi192 bash[24608]: cluster 2024-09-13T22:30:46.202407+0000 mgr.y (mgr.14150) 102 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:47 smithi192 bash[24608]: audit 2024-09-13T22:30:46.805868+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T22:30:48.080 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 0 on host 'smithi178' 2024-09-13T22:30:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:47.256592+0000 mon.a (mon.0) 310 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T22:30:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: cluster 2024-09-13T22:30:47.261543+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-13T22:30:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:47.262374+0000 mon.a (mon.0) 312 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:30:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:47.263007+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:48.063929+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:48.070637+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:48.077771+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:48.258289+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: cluster 2024-09-13T22:30:48.261781+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[22840]: audit 2024-09-13T22:30:48.262409+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:47.256592+0000 mon.a (mon.0) 310 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: cluster 2024-09-13T22:30:47.261543+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:47.262374+0000 mon.a (mon.0) 312 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:47.263007+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:48.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:48.063929+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:48.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:48.070637+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:48.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:48.077771+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:48.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:48.258289+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:30:48.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: cluster 2024-09-13T22:30:48.261781+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-13T22:30:48.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:48 smithi178 bash[29889]: audit 2024-09-13T22:30:48.262409+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:47.256592+0000 mon.a (mon.0) 310 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: cluster 2024-09-13T22:30:47.261543+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:47.262374+0000 mon.a (mon.0) 312 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:47.263007+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:48.063929+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:48.070637+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:48.077771+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:30:48.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:48.258289+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 v1:172.21.15.178:6801/1489719354' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:30:48.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: cluster 2024-09-13T22:30:48.261781+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-13T22:30:48.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:48 smithi192 bash[24608]: audit 2024-09-13T22:30:48.262409+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:48.678 DEBUG:teuthology.orchestra.run.smithi178:osd.0> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T22:30:48.680 INFO:tasks.cephadm:Deploying osd.1 on smithi178 with /dev/vg_nvme/lv_3... 2024-09-13T22:30:48.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T22:30:49.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:49 smithi192 bash[24608]: cluster 2024-09-13T22:30:48.202695+0000 mgr.y (mgr.14150) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:49.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:49 smithi192 bash[24608]: audit 2024-09-13T22:30:48.269152+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:49.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:49 smithi178 bash[22840]: cluster 2024-09-13T22:30:48.202695+0000 mgr.y (mgr.14150) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:49.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:49 smithi178 bash[22840]: audit 2024-09-13T22:30:48.269152+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:49.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:49 smithi178 bash[29889]: cluster 2024-09-13T22:30:48.202695+0000 mgr.y (mgr.14150) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:49.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:49 smithi178 bash[29889]: audit 2024-09-13T22:30:48.269152+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:50.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:50 smithi192 bash[24608]: cluster 2024-09-13T22:30:47.848082+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:30:50.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:50 smithi192 bash[24608]: cluster 2024-09-13T22:30:47.848218+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:30:50.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:50 smithi192 bash[24608]: audit 2024-09-13T22:30:49.266202+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:50.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:50 smithi192 bash[24608]: cluster 2024-09-13T22:30:49.271929+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 v1:172.21.15.178:6801/1489719354 boot 2024-09-13T22:30:50.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:50 smithi192 bash[24608]: cluster 2024-09-13T22:30:49.272000+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-13T22:30:50.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:50 smithi192 bash[24608]: audit 2024-09-13T22:30:49.272538+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[22840]: cluster 2024-09-13T22:30:47.848082+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[22840]: cluster 2024-09-13T22:30:47.848218+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[22840]: audit 2024-09-13T22:30:49.266202+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[22840]: cluster 2024-09-13T22:30:49.271929+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 v1:172.21.15.178:6801/1489719354 boot 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[22840]: cluster 2024-09-13T22:30:49.272000+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[22840]: audit 2024-09-13T22:30:49.272538+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[29889]: cluster 2024-09-13T22:30:47.848082+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[29889]: cluster 2024-09-13T22:30:47.848218+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:30:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[29889]: audit 2024-09-13T22:30:49.266202+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:50.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[29889]: cluster 2024-09-13T22:30:49.271929+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 v1:172.21.15.178:6801/1489719354 boot 2024-09-13T22:30:50.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[29889]: cluster 2024-09-13T22:30:49.272000+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-13T22:30:50.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:50 smithi178 bash[29889]: audit 2024-09-13T22:30:49.272538+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:30:51.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:51 smithi192 bash[24608]: cluster 2024-09-13T22:30:50.203102+0000 mgr.y (mgr.14150) 104 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:51.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:51 smithi192 bash[24608]: cluster 2024-09-13T22:30:50.291676+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-13T22:30:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:51 smithi178 bash[22840]: cluster 2024-09-13T22:30:50.203102+0000 mgr.y (mgr.14150) 104 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:51 smithi178 bash[22840]: cluster 2024-09-13T22:30:50.291676+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-13T22:30:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:51 smithi178 bash[29889]: cluster 2024-09-13T22:30:50.203102+0000 mgr.y (mgr.14150) 104 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T22:30:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:51 smithi178 bash[29889]: cluster 2024-09-13T22:30:50.291676+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-13T22:30:53.122 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:30:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:53 smithi192 bash[24608]: cluster 2024-09-13T22:30:52.203541+0000 mgr.y (mgr.14150) 105 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:53 smithi178 bash[22840]: cluster 2024-09-13T22:30:52.203541+0000 mgr.y (mgr.14150) 105 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:53 smithi178 bash[29889]: cluster 2024-09-13T22:30:52.203541+0000 mgr.y (mgr.14150) 105 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:55 smithi192 bash[24608]: cluster 2024-09-13T22:30:54.204065+0000 mgr.y (mgr.14150) 106 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:55 smithi178 bash[22840]: cluster 2024-09-13T22:30:54.204065+0000 mgr.y (mgr.14150) 106 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:55 smithi178 bash[29889]: cluster 2024-09-13T22:30:54.204065+0000 mgr.y (mgr.14150) 106 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:57.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:57 smithi192 bash[24608]: cluster 2024-09-13T22:30:56.204681+0000 mgr.y (mgr.14150) 107 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:57 smithi178 bash[22840]: cluster 2024-09-13T22:30:56.204681+0000 mgr.y (mgr.14150) 107 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:57 smithi178 bash[29889]: cluster 2024-09-13T22:30:56.204681+0000 mgr.y (mgr.14150) 107 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:57.895 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:30:57.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi178:vg_nvme/lv_3 2024-09-13T22:30:59.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:30:59 smithi192 bash[24608]: cluster 2024-09-13T22:30:58.205049+0000 mgr.y (mgr.14150) 108 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:59.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:30:59 smithi178 bash[29889]: cluster 2024-09-13T22:30:58.205049+0000 mgr.y (mgr.14150) 108 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:30:59.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:30:59 smithi178 bash[22840]: cluster 2024-09-13T22:30:58.205049+0000 mgr.y (mgr.14150) 108 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: cluster 2024-09-13T22:31:00.205466+0000 mgr.y (mgr.14150) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.770916+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.777076+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.779497+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.785564+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.788787+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.790319+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:01.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:01 smithi192 bash[24608]: audit 2024-09-13T22:31:00.799175+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: cluster 2024-09-13T22:31:00.205466+0000 mgr.y (mgr.14150) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.770916+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.777076+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.779497+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.785564+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.788787+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.790319+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[22840]: audit 2024-09-13T22:31:00.799175+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: cluster 2024-09-13T22:31:00.205466+0000 mgr.y (mgr.14150) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.770916+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.777076+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.779497+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.785564+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.788787+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.790319+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:01 smithi178 bash[29889]: audit 2024-09-13T22:31:00.799175+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:02.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:02 smithi192 bash[24608]: cephadm 2024-09-13T22:31:00.763899+0000 mgr.y (mgr.14150) 110 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:31:02.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:02 smithi192 bash[24608]: cephadm 2024-09-13T22:31:00.780712+0000 mgr.y (mgr.14150) 111 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 16241M 2024-09-13T22:31:02.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:02 smithi178 bash[22840]: cephadm 2024-09-13T22:31:00.763899+0000 mgr.y (mgr.14150) 110 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:31:02.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:02 smithi178 bash[22840]: cephadm 2024-09-13T22:31:00.780712+0000 mgr.y (mgr.14150) 111 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 16241M 2024-09-13T22:31:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:02 smithi178 bash[29889]: cephadm 2024-09-13T22:31:00.763899+0000 mgr.y (mgr.14150) 110 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:31:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:02 smithi178 bash[29889]: cephadm 2024-09-13T22:31:00.780712+0000 mgr.y (mgr.14150) 111 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 16241M 2024-09-13T22:31:03.224 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:31:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:03 smithi192 bash[24608]: cluster 2024-09-13T22:31:02.205923+0000 mgr.y (mgr.14150) 112 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:03 smithi178 bash[22840]: cluster 2024-09-13T22:31:02.205923+0000 mgr.y (mgr.14150) 112 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:03 smithi178 bash[29889]: cluster 2024-09-13T22:31:02.205923+0000 mgr.y (mgr.14150) 112 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:05.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:05 smithi192 bash[24608]: cluster 2024-09-13T22:31:04.206363+0000 mgr.y (mgr.14150) 113 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:05.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:05 smithi192 bash[24608]: audit 2024-09-13T22:31:04.565469+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:31:05.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:05 smithi192 bash[24608]: audit 2024-09-13T22:31:04.568308+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:31:05.661 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:05 smithi192 bash[24608]: audit 2024-09-13T22:31:04.569327+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[22840]: cluster 2024-09-13T22:31:04.206363+0000 mgr.y (mgr.14150) 113 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[22840]: audit 2024-09-13T22:31:04.565469+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:31:05.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[22840]: audit 2024-09-13T22:31:04.568308+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:31:05.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[22840]: audit 2024-09-13T22:31:04.569327+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:05.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[29889]: cluster 2024-09-13T22:31:04.206363+0000 mgr.y (mgr.14150) 113 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:05.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[29889]: audit 2024-09-13T22:31:04.565469+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:31:05.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[29889]: audit 2024-09-13T22:31:04.568308+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:31:05.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:05 smithi178 bash[29889]: audit 2024-09-13T22:31:04.569327+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:06.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:06 smithi192 bash[24608]: audit 2024-09-13T22:31:04.561777+0000 mgr.y (mgr.14150) 114 : audit [DBG] from='client.14235 v1:172.21.15.178:0/1022562167' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:31:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:06 smithi178 bash[22840]: audit 2024-09-13T22:31:04.561777+0000 mgr.y (mgr.14150) 114 : audit [DBG] from='client.14235 v1:172.21.15.178:0/1022562167' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:31:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:06 smithi178 bash[29889]: audit 2024-09-13T22:31:04.561777+0000 mgr.y (mgr.14150) 114 : audit [DBG] from='client.14235 v1:172.21.15.178:0/1022562167' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:31:07.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:07 smithi192 bash[24608]: cluster 2024-09-13T22:31:06.206872+0000 mgr.y (mgr.14150) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:07 smithi178 bash[22840]: cluster 2024-09-13T22:31:06.206872+0000 mgr.y (mgr.14150) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:07 smithi178 bash[29889]: cluster 2024-09-13T22:31:06.206872+0000 mgr.y (mgr.14150) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:09.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:09 smithi192 bash[24608]: cluster 2024-09-13T22:31:08.207346+0000 mgr.y (mgr.14150) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:09 smithi178 bash[22840]: cluster 2024-09-13T22:31:08.207346+0000 mgr.y (mgr.14150) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:09 smithi178 bash[29889]: cluster 2024-09-13T22:31:08.207346+0000 mgr.y (mgr.14150) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:11.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:11 smithi192 bash[24608]: cluster 2024-09-13T22:31:10.207810+0000 mgr.y (mgr.14150) 117 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:11 smithi178 bash[29889]: cluster 2024-09-13T22:31:10.207810+0000 mgr.y (mgr.14150) 117 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:11 smithi178 bash[22840]: cluster 2024-09-13T22:31:10.207810+0000 mgr.y (mgr.14150) 117 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:13.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:13 smithi192 bash[24608]: cluster 2024-09-13T22:31:12.208289+0000 mgr.y (mgr.14150) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:13.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:13 smithi192 bash[24608]: audit 2024-09-13T22:31:13.065004+0000 mon.c (mon.2) 3 : audit [INF] from='client.? v1:172.21.15.178:0/1877479194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]: dispatch 2024-09-13T22:31:13.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:13 smithi192 bash[24608]: audit 2024-09-13T22:31:13.065421+0000 mon.a (mon.0) 336 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]: dispatch 2024-09-13T22:31:13.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:13 smithi192 bash[24608]: audit 2024-09-13T22:31:13.067522+0000 mon.a (mon.0) 337 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]': finished 2024-09-13T22:31:13.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:13 smithi192 bash[24608]: cluster 2024-09-13T22:31:13.072789+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-13T22:31:13.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:13 smithi192 bash[24608]: audit 2024-09-13T22:31:13.073390+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[29889]: cluster 2024-09-13T22:31:12.208289+0000 mgr.y (mgr.14150) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[29889]: audit 2024-09-13T22:31:13.065004+0000 mon.c (mon.2) 3 : audit [INF] from='client.? v1:172.21.15.178:0/1877479194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]: dispatch 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[29889]: audit 2024-09-13T22:31:13.065421+0000 mon.a (mon.0) 336 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]: dispatch 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[29889]: audit 2024-09-13T22:31:13.067522+0000 mon.a (mon.0) 337 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]': finished 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[29889]: cluster 2024-09-13T22:31:13.072789+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[29889]: audit 2024-09-13T22:31:13.073390+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[22840]: cluster 2024-09-13T22:31:12.208289+0000 mgr.y (mgr.14150) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[22840]: audit 2024-09-13T22:31:13.065004+0000 mon.c (mon.2) 3 : audit [INF] from='client.? v1:172.21.15.178:0/1877479194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]: dispatch 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[22840]: audit 2024-09-13T22:31:13.065421+0000 mon.a (mon.0) 336 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]: dispatch 2024-09-13T22:31:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[22840]: audit 2024-09-13T22:31:13.067522+0000 mon.a (mon.0) 337 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa"}]': finished 2024-09-13T22:31:13.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[22840]: cluster 2024-09-13T22:31:13.072789+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-13T22:31:13.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:13 smithi178 bash[22840]: audit 2024-09-13T22:31:13.073390+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:14.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:14 smithi192 bash[24608]: audit 2024-09-13T22:31:13.737660+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? v1:172.21.15.178:0/829798883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:31:14.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:14 smithi178 bash[29889]: audit 2024-09-13T22:31:13.737660+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? v1:172.21.15.178:0/829798883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:31:14.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:14 smithi178 bash[22840]: audit 2024-09-13T22:31:13.737660+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? v1:172.21.15.178:0/829798883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:31:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:15 smithi192 bash[24608]: cluster 2024-09-13T22:31:14.208745+0000 mgr.y (mgr.14150) 119 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:15 smithi178 bash[22840]: cluster 2024-09-13T22:31:14.208745+0000 mgr.y (mgr.14150) 119 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:15.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:15 smithi178 bash[29889]: cluster 2024-09-13T22:31:14.208745+0000 mgr.y (mgr.14150) 119 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:17 smithi192 bash[24608]: cluster 2024-09-13T22:31:16.209243+0000 mgr.y (mgr.14150) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:17 smithi178 bash[22840]: cluster 2024-09-13T22:31:16.209243+0000 mgr.y (mgr.14150) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:17 smithi178 bash[29889]: cluster 2024-09-13T22:31:16.209243+0000 mgr.y (mgr.14150) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:19 smithi192 bash[24608]: cluster 2024-09-13T22:31:18.209676+0000 mgr.y (mgr.14150) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:19.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:19 smithi178 bash[22840]: cluster 2024-09-13T22:31:18.209676+0000 mgr.y (mgr.14150) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:19 smithi178 bash[29889]: cluster 2024-09-13T22:31:18.209676+0000 mgr.y (mgr.14150) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:21 smithi192 bash[24608]: cluster 2024-09-13T22:31:20.210160+0000 mgr.y (mgr.14150) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:21.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:21 smithi178 bash[22840]: cluster 2024-09-13T22:31:20.210160+0000 mgr.y (mgr.14150) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:21 smithi178 bash[29889]: cluster 2024-09-13T22:31:20.210160+0000 mgr.y (mgr.14150) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:23.642 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:23 smithi178 bash[22840]: cluster 2024-09-13T22:31:22.210647+0000 mgr.y (mgr.14150) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:23.642 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:23 smithi178 bash[29889]: cluster 2024-09-13T22:31:22.210647+0000 mgr.y (mgr.14150) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:23 smithi192 bash[24608]: cluster 2024-09-13T22:31:22.210647+0000 mgr.y (mgr.14150) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:25.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:25 smithi192 bash[24608]: cluster 2024-09-13T22:31:24.211126+0000 mgr.y (mgr.14150) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:25 smithi178 bash[22840]: cluster 2024-09-13T22:31:24.211126+0000 mgr.y (mgr.14150) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:25.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:25 smithi178 bash[29889]: cluster 2024-09-13T22:31:24.211126+0000 mgr.y (mgr.14150) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:26 smithi192 bash[24608]: audit 2024-09-13T22:31:26.053314+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T22:31:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:26 smithi192 bash[24608]: audit 2024-09-13T22:31:26.054796+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:26.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:26 smithi178 bash[29889]: audit 2024-09-13T22:31:26.053314+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T22:31:26.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:26 smithi178 bash[29889]: audit 2024-09-13T22:31:26.054796+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:26.670 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:26 smithi178 bash[22840]: audit 2024-09-13T22:31:26.053314+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T22:31:26.671 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:26 smithi178 bash[22840]: audit 2024-09-13T22:31:26.054796+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:27.649 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:27 smithi178 bash[22840]: cephadm 2024-09-13T22:31:26.056161+0000 mgr.y (mgr.14150) 125 : cephadm [INF] Deploying daemon osd.1 on smithi178 2024-09-13T22:31:27.649 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:27 smithi178 bash[22840]: cluster 2024-09-13T22:31:26.211455+0000 mgr.y (mgr.14150) 126 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:27.649 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:27 smithi178 bash[29889]: cephadm 2024-09-13T22:31:26.056161+0000 mgr.y (mgr.14150) 125 : cephadm [INF] Deploying daemon osd.1 on smithi178 2024-09-13T22:31:27.649 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:27 smithi178 bash[29889]: cluster 2024-09-13T22:31:26.211455+0000 mgr.y (mgr.14150) 126 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:27 smithi192 bash[24608]: cephadm 2024-09-13T22:31:26.056161+0000 mgr.y (mgr.14150) 125 : cephadm [INF] Deploying daemon osd.1 on smithi178 2024-09-13T22:31:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:27 smithi192 bash[24608]: cluster 2024-09-13T22:31:26.211455+0000 mgr.y (mgr.14150) 126 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:29.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:29 smithi178 bash[29889]: cluster 2024-09-13T22:31:28.211784+0000 mgr.y (mgr.14150) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:29.399 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:29 smithi178 bash[22840]: cluster 2024-09-13T22:31:28.211784+0000 mgr.y (mgr.14150) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:29 smithi192 bash[24608]: cluster 2024-09-13T22:31:28.211784+0000 mgr.y (mgr.14150) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:30.382 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.382 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.383 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.383 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.681 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:30.681 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:31:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:31:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:31 smithi192 bash[24608]: cluster 2024-09-13T22:31:30.212088+0000 mgr.y (mgr.14150) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:31 smithi192 bash[24608]: audit 2024-09-13T22:31:30.638591+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:31:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:31 smithi192 bash[24608]: audit 2024-09-13T22:31:30.642822+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:31 smithi192 bash[24608]: audit 2024-09-13T22:31:30.647328+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:31.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[22840]: cluster 2024-09-13T22:31:30.212088+0000 mgr.y (mgr.14150) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:31.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[22840]: audit 2024-09-13T22:31:30.638591+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:31:31.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[22840]: audit 2024-09-13T22:31:30.642822+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:31.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[22840]: audit 2024-09-13T22:31:30.647328+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[29889]: cluster 2024-09-13T22:31:30.212088+0000 mgr.y (mgr.14150) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[29889]: audit 2024-09-13T22:31:30.638591+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:31:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[29889]: audit 2024-09-13T22:31:30.642822+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:31 smithi178 bash[29889]: audit 2024-09-13T22:31:30.647328+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:33 smithi192 bash[24608]: cluster 2024-09-13T22:31:32.212397+0000 mgr.y (mgr.14150) 129 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:33 smithi178 bash[22840]: cluster 2024-09-13T22:31:32.212397+0000 mgr.y (mgr.14150) 129 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:33 smithi178 bash[29889]: cluster 2024-09-13T22:31:32.212397+0000 mgr.y (mgr.14150) 129 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:35 smithi192 bash[24608]: cluster 2024-09-13T22:31:34.212798+0000 mgr.y (mgr.14150) 130 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:35.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:35 smithi178 bash[22840]: cluster 2024-09-13T22:31:34.212798+0000 mgr.y (mgr.14150) 130 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:35 smithi178 bash[29889]: cluster 2024-09-13T22:31:34.212798+0000 mgr.y (mgr.14150) 130 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:37 smithi192 bash[24608]: cluster 2024-09-13T22:31:36.213257+0000 mgr.y (mgr.14150) 131 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:37 smithi192 bash[24608]: audit 2024-09-13T22:31:36.983118+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:37 smithi192 bash[24608]: audit 2024-09-13T22:31:36.987648+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:37 smithi192 bash[24608]: audit 2024-09-13T22:31:36.988397+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:37 smithi192 bash[24608]: audit 2024-09-13T22:31:36.989047+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:37 smithi192 bash[24608]: audit 2024-09-13T22:31:36.994581+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[29889]: cluster 2024-09-13T22:31:36.213257+0000 mgr.y (mgr.14150) 131 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[29889]: audit 2024-09-13T22:31:36.983118+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[29889]: audit 2024-09-13T22:31:36.987648+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[29889]: audit 2024-09-13T22:31:36.988397+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[29889]: audit 2024-09-13T22:31:36.989047+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[29889]: audit 2024-09-13T22:31:36.994581+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[22840]: cluster 2024-09-13T22:31:36.213257+0000 mgr.y (mgr.14150) 131 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[22840]: audit 2024-09-13T22:31:36.983118+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[22840]: audit 2024-09-13T22:31:36.987648+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[22840]: audit 2024-09-13T22:31:36.988397+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[22840]: audit 2024-09-13T22:31:36.989047+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:37 smithi178 bash[22840]: audit 2024-09-13T22:31:36.994581+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:38 smithi192 bash[24608]: cluster 2024-09-13T22:31:38.213725+0000 mgr.y (mgr.14150) 132 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:38 smithi178 bash[22840]: cluster 2024-09-13T22:31:38.213725+0000 mgr.y (mgr.14150) 132 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:38.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:38 smithi178 bash[29889]: cluster 2024-09-13T22:31:38.213725+0000 mgr.y (mgr.14150) 132 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:39.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:39 smithi192 bash[24608]: audit 2024-09-13T22:31:39.505499+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T22:31:39.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:39 smithi178 bash[22840]: audit 2024-09-13T22:31:39.505499+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T22:31:39.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:39 smithi178 bash[29889]: audit 2024-09-13T22:31:39.505499+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T22:31:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:40 smithi192 bash[24608]: audit 2024-09-13T22:31:39.631674+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T22:31:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:40 smithi192 bash[24608]: cluster 2024-09-13T22:31:39.635241+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-13T22:31:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:40 smithi192 bash[24608]: audit 2024-09-13T22:31:39.635502+0000 mon.a (mon.0) 354 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:31:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:40 smithi192 bash[24608]: audit 2024-09-13T22:31:39.635728+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:40 smithi192 bash[24608]: cluster 2024-09-13T22:31:40.214171+0000 mgr.y (mgr.14150) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:40.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[22840]: audit 2024-09-13T22:31:39.631674+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[22840]: cluster 2024-09-13T22:31:39.635241+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[22840]: audit 2024-09-13T22:31:39.635502+0000 mon.a (mon.0) 354 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[22840]: audit 2024-09-13T22:31:39.635728+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[22840]: cluster 2024-09-13T22:31:40.214171+0000 mgr.y (mgr.14150) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[29889]: audit 2024-09-13T22:31:39.631674+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[29889]: cluster 2024-09-13T22:31:39.635241+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[29889]: audit 2024-09-13T22:31:39.635502+0000 mon.a (mon.0) 354 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[29889]: audit 2024-09-13T22:31:39.635728+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:40 smithi178 bash[29889]: cluster 2024-09-13T22:31:40.214171+0000 mgr.y (mgr.14150) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:41.247 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 1 on host 'smithi178' 2024-09-13T22:31:41.871 DEBUG:teuthology.orchestra.run.smithi178:osd.1> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T22:31:41.873 INFO:tasks.cephadm:Deploying osd.2 on smithi178 with /dev/vg_nvme/lv_2... 2024-09-13T22:31:41.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T22:31:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:40.633451+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:31:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: cluster 2024-09-13T22:31:40.636695+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-13T22:31:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:40.637631+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:40.640777+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:41.223239+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:31:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:41.232394+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:41.241012+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:41 smithi192 bash[24608]: audit 2024-09-13T22:31:41.640698+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:40.633451+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: cluster 2024-09-13T22:31:40.636695+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:40.637631+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:40.640777+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:41.223239+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:41.232394+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:41.241012+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:41.925 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[29889]: audit 2024-09-13T22:31:41.640698+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:40.633451+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 v1:172.21.15.178:6805/4187084287' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: cluster 2024-09-13T22:31:40.636695+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:40.637631+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:40.640777+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:41.223239+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:41.232394+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:41.241012+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:41.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:41 smithi178 bash[22840]: audit 2024-09-13T22:31:41.640698+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[22840]: cluster 2024-09-13T22:31:40.498173+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[22840]: cluster 2024-09-13T22:31:40.498264+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[22840]: cluster 2024-09-13T22:31:41.650431+0000 mon.a (mon.0) 364 : cluster [INF] osd.1 v1:172.21.15.178:6805/4187084287 boot 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[22840]: cluster 2024-09-13T22:31:41.650504+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[22840]: audit 2024-09-13T22:31:41.651109+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[22840]: cluster 2024-09-13T22:31:42.214735+0000 mgr.y (mgr.14150) 134 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[29889]: cluster 2024-09-13T22:31:40.498173+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:31:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[29889]: cluster 2024-09-13T22:31:40.498264+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:31:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[29889]: cluster 2024-09-13T22:31:41.650431+0000 mon.a (mon.0) 364 : cluster [INF] osd.1 v1:172.21.15.178:6805/4187084287 boot 2024-09-13T22:31:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[29889]: cluster 2024-09-13T22:31:41.650504+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-13T22:31:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[29889]: audit 2024-09-13T22:31:41.651109+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:42.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:42 smithi178 bash[29889]: cluster 2024-09-13T22:31:42.214735+0000 mgr.y (mgr.14150) 134 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:42 smithi192 bash[24608]: cluster 2024-09-13T22:31:40.498173+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:31:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:42 smithi192 bash[24608]: cluster 2024-09-13T22:31:40.498264+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:31:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:42 smithi192 bash[24608]: cluster 2024-09-13T22:31:41.650431+0000 mon.a (mon.0) 364 : cluster [INF] osd.1 v1:172.21.15.178:6805/4187084287 boot 2024-09-13T22:31:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:42 smithi192 bash[24608]: cluster 2024-09-13T22:31:41.650504+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-13T22:31:43.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:42 smithi192 bash[24608]: audit 2024-09-13T22:31:41.651109+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:31:43.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:42 smithi192 bash[24608]: cluster 2024-09-13T22:31:42.214735+0000 mgr.y (mgr.14150) 134 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T22:31:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:43 smithi178 bash[22840]: cluster 2024-09-13T22:31:42.666875+0000 mon.a (mon.0) 367 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-13T22:31:43.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:43 smithi178 bash[29889]: cluster 2024-09-13T22:31:42.666875+0000 mon.a (mon.0) 367 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-13T22:31:44.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:43 smithi192 bash[24608]: cluster 2024-09-13T22:31:42.666875+0000 mon.a (mon.0) 367 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-13T22:31:44.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:44 smithi178 bash[22840]: cluster 2024-09-13T22:31:44.215221+0000 mgr.y (mgr.14150) 135 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:44 smithi178 bash[29889]: cluster 2024-09-13T22:31:44.215221+0000 mgr.y (mgr.14150) 135 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:45.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:44 smithi192 bash[24608]: cluster 2024-09-13T22:31:44.215221+0000 mgr.y (mgr.14150) 135 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:46.297 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:31:47.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:47 smithi192 bash[24608]: cluster 2024-09-13T22:31:46.215755+0000 mgr.y (mgr.14150) 136 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:47.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:47 smithi178 bash[22840]: cluster 2024-09-13T22:31:46.215755+0000 mgr.y (mgr.14150) 136 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:47 smithi178 bash[29889]: cluster 2024-09-13T22:31:46.215755+0000 mgr.y (mgr.14150) 136 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:48 smithi192 bash[24608]: cluster 2024-09-13T22:31:48.216270+0000 mgr.y (mgr.14150) 137 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:48.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:48 smithi178 bash[22840]: cluster 2024-09-13T22:31:48.216270+0000 mgr.y (mgr.14150) 137 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:48 smithi178 bash[29889]: cluster 2024-09-13T22:31:48.216270+0000 mgr.y (mgr.14150) 137 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:51 smithi192 bash[24608]: cluster 2024-09-13T22:31:50.216772+0000 mgr.y (mgr.14150) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:51 smithi178 bash[22840]: cluster 2024-09-13T22:31:50.216772+0000 mgr.y (mgr.14150) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:51 smithi178 bash[29889]: cluster 2024-09-13T22:31:50.216772+0000 mgr.y (mgr.14150) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:52.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: cephadm 2024-09-13T22:31:51.653742+0000 mgr.y (mgr.14150) 139 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.660946+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.667599+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.669720+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.670913+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: cephadm 2024-09-13T22:31:51.671852+0000 mgr.y (mgr.14150) 140 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 8120M 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.677898+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.682450+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.684003+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: audit 2024-09-13T22:31:51.693380+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[22840]: cluster 2024-09-13T22:31:52.217264+0000 mgr.y (mgr.14150) 141 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: cephadm 2024-09-13T22:31:51.653742+0000 mgr.y (mgr.14150) 139 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.660946+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.667599+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.669720+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.670913+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: cephadm 2024-09-13T22:31:51.671852+0000 mgr.y (mgr.14150) 140 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 8120M 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.677898+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.682450+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.684003+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: audit 2024-09-13T22:31:51.693380+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:52.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:52 smithi178 bash[29889]: cluster 2024-09-13T22:31:52.217264+0000 mgr.y (mgr.14150) 141 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: cephadm 2024-09-13T22:31:51.653742+0000 mgr.y (mgr.14150) 139 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:31:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.660946+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.667599+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.669720+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.670913+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: cephadm 2024-09-13T22:31:51.671852+0000 mgr.y (mgr.14150) 140 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 8120M 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.677898+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.682450+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.684003+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: audit 2024-09-13T22:31:51.693380+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:31:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:52 smithi192 bash[24608]: cluster 2024-09-13T22:31:52.217264+0000 mgr.y (mgr.14150) 141 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:53.919 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:31:53.959 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi178:vg_nvme/lv_2 2024-09-13T22:31:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:55 smithi192 bash[24608]: cluster 2024-09-13T22:31:54.217576+0000 mgr.y (mgr.14150) 142 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:55 smithi178 bash[22840]: cluster 2024-09-13T22:31:54.217576+0000 mgr.y (mgr.14150) 142 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:55.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:55 smithi178 bash[29889]: cluster 2024-09-13T22:31:54.217576+0000 mgr.y (mgr.14150) 142 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:57.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:57 smithi192 bash[24608]: cluster 2024-09-13T22:31:56.217999+0000 mgr.y (mgr.14150) 143 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:57.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:57 smithi178 bash[22840]: cluster 2024-09-13T22:31:56.217999+0000 mgr.y (mgr.14150) 143 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:57 smithi178 bash[29889]: cluster 2024-09-13T22:31:56.217999+0000 mgr.y (mgr.14150) 143 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:58.762 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:31:59.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:31:59 smithi192 bash[24608]: cluster 2024-09-13T22:31:58.218455+0000 mgr.y (mgr.14150) 144 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:59.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:31:59 smithi178 bash[22840]: cluster 2024-09-13T22:31:58.218455+0000 mgr.y (mgr.14150) 144 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:31:59.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:31:59 smithi178 bash[29889]: cluster 2024-09-13T22:31:58.218455+0000 mgr.y (mgr.14150) 144 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:00.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:00 smithi192 bash[24608]: audit 2024-09-13T22:32:00.238581+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:32:00.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:00 smithi192 bash[24608]: audit 2024-09-13T22:32:00.244048+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:32:00.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:00 smithi192 bash[24608]: audit 2024-09-13T22:32:00.244918+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:00.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:00 smithi178 bash[22840]: audit 2024-09-13T22:32:00.238581+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:32:00.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:00 smithi178 bash[22840]: audit 2024-09-13T22:32:00.244048+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:32:00.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:00 smithi178 bash[22840]: audit 2024-09-13T22:32:00.244918+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:00.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:00 smithi178 bash[29889]: audit 2024-09-13T22:32:00.238581+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:32:00.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:00 smithi178 bash[29889]: audit 2024-09-13T22:32:00.244048+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:32:00.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:00 smithi178 bash[29889]: audit 2024-09-13T22:32:00.244918+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:01.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:01 smithi192 bash[24608]: cluster 2024-09-13T22:32:00.218916+0000 mgr.y (mgr.14150) 145 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:01.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:01 smithi192 bash[24608]: audit 2024-09-13T22:32:00.232577+0000 mgr.y (mgr.14150) 146 : audit [DBG] from='client.14262 v1:172.21.15.178:0/877141034' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:32:01.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:01 smithi178 bash[22840]: cluster 2024-09-13T22:32:00.218916+0000 mgr.y (mgr.14150) 145 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:01.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:01 smithi178 bash[22840]: audit 2024-09-13T22:32:00.232577+0000 mgr.y (mgr.14150) 146 : audit [DBG] from='client.14262 v1:172.21.15.178:0/877141034' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:32:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:01 smithi178 bash[29889]: cluster 2024-09-13T22:32:00.218916+0000 mgr.y (mgr.14150) 145 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:01 smithi178 bash[29889]: audit 2024-09-13T22:32:00.232577+0000 mgr.y (mgr.14150) 146 : audit [DBG] from='client.14262 v1:172.21.15.178:0/877141034' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:32:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:03 smithi192 bash[24608]: cluster 2024-09-13T22:32:02.219352+0000 mgr.y (mgr.14150) 147 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:03 smithi178 bash[22840]: cluster 2024-09-13T22:32:02.219352+0000 mgr.y (mgr.14150) 147 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:03 smithi178 bash[29889]: cluster 2024-09-13T22:32:02.219352+0000 mgr.y (mgr.14150) 147 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:05.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:05 smithi192 bash[24608]: cluster 2024-09-13T22:32:04.219817+0000 mgr.y (mgr.14150) 148 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:05 smithi178 bash[22840]: cluster 2024-09-13T22:32:04.219817+0000 mgr.y (mgr.14150) 148 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:05.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:05 smithi178 bash[29889]: cluster 2024-09-13T22:32:04.219817+0000 mgr.y (mgr.14150) 148 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:07 smithi178 bash[22840]: cluster 2024-09-13T22:32:06.220286+0000 mgr.y (mgr.14150) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:07.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:07 smithi178 bash[29889]: cluster 2024-09-13T22:32:06.220286+0000 mgr.y (mgr.14150) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:07.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:07 smithi192 bash[24608]: cluster 2024-09-13T22:32:06.220286+0000 mgr.y (mgr.14150) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:08 smithi192 bash[24608]: cluster 2024-09-13T22:32:08.220607+0000 mgr.y (mgr.14150) 150 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:08.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:08 smithi178 bash[22840]: cluster 2024-09-13T22:32:08.220607+0000 mgr.y (mgr.14150) 150 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:08.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:08 smithi178 bash[29889]: cluster 2024-09-13T22:32:08.220607+0000 mgr.y (mgr.14150) 150 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:09 smithi192 bash[24608]: audit 2024-09-13T22:32:08.763012+0000 mon.a (mon.0) 379 : audit [INF] from='client.? v1:172.21.15.178:0/2795898046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a79075-5102-4f6c-b41a-255102064624"}]: dispatch 2024-09-13T22:32:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:09 smithi192 bash[24608]: audit 2024-09-13T22:32:08.765690+0000 mon.a (mon.0) 380 : audit [INF] from='client.? v1:172.21.15.178:0/2795898046' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a79075-5102-4f6c-b41a-255102064624"}]': finished 2024-09-13T22:32:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:09 smithi192 bash[24608]: cluster 2024-09-13T22:32:08.770463+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-13T22:32:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:09 smithi192 bash[24608]: audit 2024-09-13T22:32:08.770737+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:09 smithi192 bash[24608]: audit 2024-09-13T22:32:09.389713+0000 mon.a (mon.0) 383 : audit [DBG] from='client.? v1:172.21.15.178:0/3139526638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:32:09.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[22840]: audit 2024-09-13T22:32:08.763012+0000 mon.a (mon.0) 379 : audit [INF] from='client.? v1:172.21.15.178:0/2795898046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a79075-5102-4f6c-b41a-255102064624"}]: dispatch 2024-09-13T22:32:09.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[22840]: audit 2024-09-13T22:32:08.765690+0000 mon.a (mon.0) 380 : audit [INF] from='client.? v1:172.21.15.178:0/2795898046' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a79075-5102-4f6c-b41a-255102064624"}]': finished 2024-09-13T22:32:09.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[22840]: cluster 2024-09-13T22:32:08.770463+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-13T22:32:09.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[22840]: audit 2024-09-13T22:32:08.770737+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:09.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[22840]: audit 2024-09-13T22:32:09.389713+0000 mon.a (mon.0) 383 : audit [DBG] from='client.? v1:172.21.15.178:0/3139526638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:32:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[29889]: audit 2024-09-13T22:32:08.763012+0000 mon.a (mon.0) 379 : audit [INF] from='client.? v1:172.21.15.178:0/2795898046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a79075-5102-4f6c-b41a-255102064624"}]: dispatch 2024-09-13T22:32:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[29889]: audit 2024-09-13T22:32:08.765690+0000 mon.a (mon.0) 380 : audit [INF] from='client.? v1:172.21.15.178:0/2795898046' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a79075-5102-4f6c-b41a-255102064624"}]': finished 2024-09-13T22:32:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[29889]: cluster 2024-09-13T22:32:08.770463+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-13T22:32:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[29889]: audit 2024-09-13T22:32:08.770737+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:09 smithi178 bash[29889]: audit 2024-09-13T22:32:09.389713+0000 mon.a (mon.0) 383 : audit [DBG] from='client.? v1:172.21.15.178:0/3139526638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:32:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:10 smithi192 bash[24608]: cluster 2024-09-13T22:32:10.221105+0000 mgr.y (mgr.14150) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:10.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:10 smithi178 bash[22840]: cluster 2024-09-13T22:32:10.221105+0000 mgr.y (mgr.14150) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:10 smithi178 bash[29889]: cluster 2024-09-13T22:32:10.221105+0000 mgr.y (mgr.14150) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:11 smithi192 bash[24608]: cluster 2024-09-13T22:32:10.554973+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:32:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:11 smithi192 bash[24608]: cluster 2024-09-13T22:32:10.555071+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:32:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:11 smithi192 bash[24608]: cluster 2024-09-13T22:32:10.640327+0000 mon.a (mon.0) 384 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-13T22:32:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:11 smithi192 bash[24608]: audit 2024-09-13T22:32:10.640490+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[22840]: cluster 2024-09-13T22:32:10.554973+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:32:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[22840]: cluster 2024-09-13T22:32:10.555071+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:32:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[22840]: cluster 2024-09-13T22:32:10.640327+0000 mon.a (mon.0) 384 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-13T22:32:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[22840]: audit 2024-09-13T22:32:10.640490+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:11.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[29889]: cluster 2024-09-13T22:32:10.554973+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:32:11.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[29889]: cluster 2024-09-13T22:32:10.555071+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:32:11.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[29889]: cluster 2024-09-13T22:32:10.640327+0000 mon.a (mon.0) 384 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-13T22:32:11.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:11 smithi178 bash[29889]: audit 2024-09-13T22:32:10.640490+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:12 smithi192 bash[24608]: cluster 2024-09-13T22:32:12.221597+0000 mgr.y (mgr.14150) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:12.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:12 smithi178 bash[22840]: cluster 2024-09-13T22:32:12.221597+0000 mgr.y (mgr.14150) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:12 smithi178 bash[29889]: cluster 2024-09-13T22:32:12.221597+0000 mgr.y (mgr.14150) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:15 smithi192 bash[24608]: cluster 2024-09-13T22:32:14.222098+0000 mgr.y (mgr.14150) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:15 smithi178 bash[22840]: cluster 2024-09-13T22:32:14.222098+0000 mgr.y (mgr.14150) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:15 smithi178 bash[29889]: cluster 2024-09-13T22:32:14.222098+0000 mgr.y (mgr.14150) 153 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:17 smithi192 bash[24608]: cluster 2024-09-13T22:32:16.222579+0000 mgr.y (mgr.14150) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:17 smithi178 bash[22840]: cluster 2024-09-13T22:32:16.222579+0000 mgr.y (mgr.14150) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:17 smithi178 bash[29889]: cluster 2024-09-13T22:32:16.222579+0000 mgr.y (mgr.14150) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:19.555 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:19 smithi178 bash[29889]: cluster 2024-09-13T22:32:18.223058+0000 mgr.y (mgr.14150) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:19.555 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:19 smithi178 bash[22840]: cluster 2024-09-13T22:32:18.223058+0000 mgr.y (mgr.14150) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:19 smithi192 bash[24608]: cluster 2024-09-13T22:32:18.223058+0000 mgr.y (mgr.14150) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:21.649 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:21 smithi178 bash[22840]: cluster 2024-09-13T22:32:20.223487+0000 mgr.y (mgr.14150) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:21.649 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:21 smithi178 bash[29889]: cluster 2024-09-13T22:32:20.223487+0000 mgr.y (mgr.14150) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:21 smithi192 bash[24608]: cluster 2024-09-13T22:32:20.223487+0000 mgr.y (mgr.14150) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:23 smithi192 bash[24608]: cluster 2024-09-13T22:32:22.223937+0000 mgr.y (mgr.14150) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:23 smithi192 bash[24608]: audit 2024-09-13T22:32:22.293067+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T22:32:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:23 smithi192 bash[24608]: audit 2024-09-13T22:32:22.294696+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:23 smithi192 bash[24608]: cephadm 2024-09-13T22:32:22.296048+0000 mgr.y (mgr.14150) 158 : cephadm [INF] Deploying daemon osd.2 on smithi178 2024-09-13T22:32:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[22840]: cluster 2024-09-13T22:32:22.223937+0000 mgr.y (mgr.14150) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[22840]: audit 2024-09-13T22:32:22.293067+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T22:32:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[22840]: audit 2024-09-13T22:32:22.294696+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[22840]: cephadm 2024-09-13T22:32:22.296048+0000 mgr.y (mgr.14150) 158 : cephadm [INF] Deploying daemon osd.2 on smithi178 2024-09-13T22:32:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[29889]: cluster 2024-09-13T22:32:22.223937+0000 mgr.y (mgr.14150) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[29889]: audit 2024-09-13T22:32:22.293067+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T22:32:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[29889]: audit 2024-09-13T22:32:22.294696+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:23 smithi178 bash[29889]: cephadm 2024-09-13T22:32:22.296048+0000 mgr.y (mgr.14150) 158 : cephadm [INF] Deploying daemon osd.2 on smithi178 2024-09-13T22:32:25.651 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:25 smithi178 bash[22840]: cluster 2024-09-13T22:32:24.224384+0000 mgr.y (mgr.14150) 159 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:25.651 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:25 smithi178 bash[29889]: cluster 2024-09-13T22:32:24.224384+0000 mgr.y (mgr.14150) 159 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:25 smithi192 bash[24608]: cluster 2024-09-13T22:32:24.224384+0000 mgr.y (mgr.14150) 159 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:26.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.679 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.680 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.680 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.930 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.930 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:26.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:26 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:32:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:27 smithi192 bash[24608]: cluster 2024-09-13T22:32:26.224753+0000 mgr.y (mgr.14150) 160 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:27 smithi192 bash[24608]: audit 2024-09-13T22:32:26.951460+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:32:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:27 smithi192 bash[24608]: audit 2024-09-13T22:32:26.955322+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:27 smithi192 bash[24608]: audit 2024-09-13T22:32:26.959474+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:27.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[22840]: cluster 2024-09-13T22:32:26.224753+0000 mgr.y (mgr.14150) 160 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:27.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[22840]: audit 2024-09-13T22:32:26.951460+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:32:27.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[22840]: audit 2024-09-13T22:32:26.955322+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:27.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[22840]: audit 2024-09-13T22:32:26.959474+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[29889]: cluster 2024-09-13T22:32:26.224753+0000 mgr.y (mgr.14150) 160 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[29889]: audit 2024-09-13T22:32:26.951460+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:32:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[29889]: audit 2024-09-13T22:32:26.955322+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:27 smithi178 bash[29889]: audit 2024-09-13T22:32:26.959474+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:28 smithi192 bash[24608]: cluster 2024-09-13T22:32:28.225207+0000 mgr.y (mgr.14150) 161 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:28.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:28 smithi178 bash[29889]: cluster 2024-09-13T22:32:28.225207+0000 mgr.y (mgr.14150) 161 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:28.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:28 smithi178 bash[22840]: cluster 2024-09-13T22:32:28.225207+0000 mgr.y (mgr.14150) 161 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:31.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:31 smithi178 bash[22840]: cluster 2024-09-13T22:32:30.225651+0000 mgr.y (mgr.14150) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:31 smithi178 bash[29889]: cluster 2024-09-13T22:32:30.225651+0000 mgr.y (mgr.14150) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:31.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:31 smithi192 bash[24608]: cluster 2024-09-13T22:32:30.225651+0000 mgr.y (mgr.14150) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:33.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:33 smithi178 bash[22840]: cluster 2024-09-13T22:32:32.226102+0000 mgr.y (mgr.14150) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:33 smithi178 bash[29889]: cluster 2024-09-13T22:32:32.226102+0000 mgr.y (mgr.14150) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:33 smithi192 bash[24608]: cluster 2024-09-13T22:32:32.226102+0000 mgr.y (mgr.14150) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:34 smithi192 bash[24608]: audit 2024-09-13T22:32:33.317789+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:34 smithi192 bash[24608]: audit 2024-09-13T22:32:33.324317+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:34 smithi192 bash[24608]: audit 2024-09-13T22:32:33.950657+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:34 smithi192 bash[24608]: audit 2024-09-13T22:32:33.952093+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:32:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:34 smithi192 bash[24608]: audit 2024-09-13T22:32:33.959804+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[22840]: audit 2024-09-13T22:32:33.317789+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[22840]: audit 2024-09-13T22:32:33.324317+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[22840]: audit 2024-09-13T22:32:33.950657+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[22840]: audit 2024-09-13T22:32:33.952093+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:32:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[22840]: audit 2024-09-13T22:32:33.959804+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[29889]: audit 2024-09-13T22:32:33.317789+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[29889]: audit 2024-09-13T22:32:33.324317+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[29889]: audit 2024-09-13T22:32:33.950657+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[29889]: audit 2024-09-13T22:32:33.952093+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:32:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:34 smithi178 bash[29889]: audit 2024-09-13T22:32:33.959804+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:35 smithi192 bash[24608]: cluster 2024-09-13T22:32:34.226411+0000 mgr.y (mgr.14150) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:35.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:35 smithi178 bash[22840]: cluster 2024-09-13T22:32:34.226411+0000 mgr.y (mgr.14150) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:35 smithi178 bash[29889]: cluster 2024-09-13T22:32:34.226411+0000 mgr.y (mgr.14150) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:36.602 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:36 smithi178 bash[22840]: audit 2024-09-13T22:32:36.250643+0000 mon.a (mon.0) 396 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T22:32:36.602 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:36 smithi178 bash[29889]: audit 2024-09-13T22:32:36.250643+0000 mon.a (mon.0) 396 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T22:32:36.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:36 smithi192 bash[24608]: audit 2024-09-13T22:32:36.250643+0000 mon.a (mon.0) 396 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T22:32:37.070 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 2 on host 'smithi178' 2024-09-13T22:32:37.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: cluster 2024-09-13T22:32:36.226939+0000 mgr.y (mgr.14150) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:37.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: audit 2024-09-13T22:32:36.329123+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T22:32:37.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: cluster 2024-09-13T22:32:36.333114+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-13T22:32:37.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: audit 2024-09-13T22:32:36.333407+0000 mon.a (mon.0) 399 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:32:37.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: audit 2024-09-13T22:32:36.333730+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:37.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: audit 2024-09-13T22:32:37.053043+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: audit 2024-09-13T22:32:37.060134+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[29889]: audit 2024-09-13T22:32:37.067991+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: cluster 2024-09-13T22:32:36.226939+0000 mgr.y (mgr.14150) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: audit 2024-09-13T22:32:36.329123+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: cluster 2024-09-13T22:32:36.333114+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: audit 2024-09-13T22:32:36.333407+0000 mon.a (mon.0) 399 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: audit 2024-09-13T22:32:36.333730+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:37.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: audit 2024-09-13T22:32:37.053043+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:32:37.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: audit 2024-09-13T22:32:37.060134+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:37.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:37 smithi178 bash[22840]: audit 2024-09-13T22:32:37.067991+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: cluster 2024-09-13T22:32:36.226939+0000 mgr.y (mgr.14150) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: audit 2024-09-13T22:32:36.329123+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: cluster 2024-09-13T22:32:36.333114+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: audit 2024-09-13T22:32:36.333407+0000 mon.a (mon.0) 399 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: audit 2024-09-13T22:32:36.333730+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: audit 2024-09-13T22:32:37.053043+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: audit 2024-09-13T22:32:37.060134+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:37 smithi192 bash[24608]: audit 2024-09-13T22:32:37.067991+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:37.799 DEBUG:teuthology.orchestra.run.smithi178:osd.2> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T22:32:37.801 INFO:tasks.cephadm:Deploying osd.3 on smithi178 with /dev/vg_nvme/lv_1... 2024-09-13T22:32:37.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T22:32:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:38 smithi192 bash[24608]: audit 2024-09-13T22:32:37.330981+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:32:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:38 smithi192 bash[24608]: cluster 2024-09-13T22:32:37.335430+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-13T22:32:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:38 smithi192 bash[24608]: audit 2024-09-13T22:32:37.336918+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:38 smithi192 bash[24608]: audit 2024-09-13T22:32:37.344646+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[22840]: audit 2024-09-13T22:32:37.330981+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:32:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[22840]: cluster 2024-09-13T22:32:37.335430+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-13T22:32:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[22840]: audit 2024-09-13T22:32:37.336918+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[22840]: audit 2024-09-13T22:32:37.344646+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[29889]: audit 2024-09-13T22:32:37.330981+0000 mon.a (mon.0) 404 : audit [INF] from='osd.2 v1:172.21.15.178:6809/2303802730' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:32:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[29889]: cluster 2024-09-13T22:32:37.335430+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-13T22:32:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[29889]: audit 2024-09-13T22:32:37.336918+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:38 smithi178 bash[29889]: audit 2024-09-13T22:32:37.344646+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:39 smithi192 bash[24608]: cluster 2024-09-13T22:32:37.282765+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:32:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:39 smithi192 bash[24608]: cluster 2024-09-13T22:32:37.282916+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:32:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:39 smithi192 bash[24608]: cluster 2024-09-13T22:32:38.227488+0000 mgr.y (mgr.14150) 166 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:39 smithi192 bash[24608]: cluster 2024-09-13T22:32:38.345449+0000 mon.a (mon.0) 408 : cluster [INF] osd.2 v1:172.21.15.178:6809/2303802730 boot 2024-09-13T22:32:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:39 smithi192 bash[24608]: cluster 2024-09-13T22:32:38.345532+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-13T22:32:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:39 smithi192 bash[24608]: audit 2024-09-13T22:32:38.346352+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:39.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[22840]: cluster 2024-09-13T22:32:37.282765+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:32:39.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[22840]: cluster 2024-09-13T22:32:37.282916+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:32:39.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[22840]: cluster 2024-09-13T22:32:38.227488+0000 mgr.y (mgr.14150) 166 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:39.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[22840]: cluster 2024-09-13T22:32:38.345449+0000 mon.a (mon.0) 408 : cluster [INF] osd.2 v1:172.21.15.178:6809/2303802730 boot 2024-09-13T22:32:39.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[22840]: cluster 2024-09-13T22:32:38.345532+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[22840]: audit 2024-09-13T22:32:38.346352+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[29889]: cluster 2024-09-13T22:32:37.282765+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[29889]: cluster 2024-09-13T22:32:37.282916+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[29889]: cluster 2024-09-13T22:32:38.227488+0000 mgr.y (mgr.14150) 166 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[29889]: cluster 2024-09-13T22:32:38.345449+0000 mon.a (mon.0) 408 : cluster [INF] osd.2 v1:172.21.15.178:6809/2303802730 boot 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[29889]: cluster 2024-09-13T22:32:38.345532+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-13T22:32:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:39 smithi178 bash[29889]: audit 2024-09-13T22:32:38.346352+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:32:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:40 smithi192 bash[24608]: cluster 2024-09-13T22:32:40.227993+0000 mgr.y (mgr.14150) 167 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:40 smithi192 bash[24608]: audit 2024-09-13T22:32:40.341625+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:32:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:40 smithi192 bash[24608]: audit 2024-09-13T22:32:40.787548+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T22:32:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:40 smithi192 bash[24608]: cluster 2024-09-13T22:32:40.789449+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-13T22:32:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:40 smithi192 bash[24608]: audit 2024-09-13T22:32:40.790837+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:32:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[29889]: cluster 2024-09-13T22:32:40.227993+0000 mgr.y (mgr.14150) 167 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[29889]: audit 2024-09-13T22:32:40.341625+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:32:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[29889]: audit 2024-09-13T22:32:40.787548+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T22:32:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[29889]: cluster 2024-09-13T22:32:40.789449+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-13T22:32:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[29889]: audit 2024-09-13T22:32:40.790837+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:32:41.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[22840]: cluster 2024-09-13T22:32:40.227993+0000 mgr.y (mgr.14150) 167 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:41.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[22840]: audit 2024-09-13T22:32:40.341625+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:32:41.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[22840]: audit 2024-09-13T22:32:40.787548+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T22:32:41.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[22840]: cluster 2024-09-13T22:32:40.789449+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-13T22:32:41.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:40 smithi178 bash[22840]: audit 2024-09-13T22:32:40.790837+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:32:42.116 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:32:43.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:42 smithi192 bash[24608]: audit 2024-09-13T22:32:41.789702+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:32:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:42 smithi192 bash[24608]: cluster 2024-09-13T22:32:41.797188+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-13T22:32:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:42 smithi192 bash[24608]: cluster 2024-09-13T22:32:42.228565+0000 mgr.y (mgr.14150) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:43.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:42 smithi178 bash[29889]: audit 2024-09-13T22:32:41.789702+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:32:43.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:42 smithi178 bash[29889]: cluster 2024-09-13T22:32:41.797188+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-13T22:32:43.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:42 smithi178 bash[29889]: cluster 2024-09-13T22:32:42.228565+0000 mgr.y (mgr.14150) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:42 smithi178 bash[22840]: audit 2024-09-13T22:32:41.789702+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:32:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:42 smithi178 bash[22840]: cluster 2024-09-13T22:32:41.797188+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-13T22:32:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:42 smithi178 bash[22840]: cluster 2024-09-13T22:32:42.228565+0000 mgr.y (mgr.14150) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:44.060 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:43 smithi178 bash[22840]: cluster 2024-09-13T22:32:42.805030+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-13T22:32:44.060 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:43 smithi178 bash[22840]: audit 2024-09-13T22:32:43.453903+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:44.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:43 smithi178 bash[29889]: cluster 2024-09-13T22:32:42.805030+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-13T22:32:44.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:43 smithi178 bash[29889]: audit 2024-09-13T22:32:43.453903+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:44.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:43 smithi192 bash[24608]: cluster 2024-09-13T22:32:42.805030+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-13T22:32:44.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:43 smithi192 bash[24608]: audit 2024-09-13T22:32:43.453903+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:44 smithi178 bash[29889]: cluster 2024-09-13T22:32:44.229131+0000 mgr.y (mgr.14150) 169 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:44.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:44 smithi178 bash[22840]: cluster 2024-09-13T22:32:44.229131+0000 mgr.y (mgr.14150) 169 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:44 smithi192 bash[24608]: cluster 2024-09-13T22:32:44.229131+0000 mgr.y (mgr.14150) 169 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:46.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:46 smithi192 bash[24608]: cluster 2024-09-13T22:32:44.812116+0000 mon.a (mon.0) 419 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-09-13T22:32:46.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:46 smithi192 bash[24608]: audit 2024-09-13T22:32:45.185732+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:46.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:46 smithi192 bash[24608]: audit 2024-09-13T22:32:45.186046+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:46.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:46 smithi192 bash[24608]: audit 2024-09-13T22:32:45.186412+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[22840]: cluster 2024-09-13T22:32:44.812116+0000 mon.a (mon.0) 419 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-09-13T22:32:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[22840]: audit 2024-09-13T22:32:45.185732+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[22840]: audit 2024-09-13T22:32:45.186046+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[22840]: audit 2024-09-13T22:32:45.186412+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[29889]: cluster 2024-09-13T22:32:44.812116+0000 mon.a (mon.0) 419 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-09-13T22:32:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[29889]: audit 2024-09-13T22:32:45.185732+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:46.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[29889]: audit 2024-09-13T22:32:45.186046+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:46.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:46 smithi178 bash[29889]: audit 2024-09-13T22:32:45.186412+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:47.208 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:32:47.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi178:vg_nvme/lv_1 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:45.186546+0000 mon.a (mon.0) 423 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:46.061866+0000 mon.a (mon.0) 424 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:46.069835+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:46.070014+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:46.070170+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:46.070353+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: cluster 2024-09-13T22:32:46.229483+0000 mgr.y (mgr.14150) 170 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:46.968358+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:48.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:47.020524+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:48.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:47.023373+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:48.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:47.023891+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:48.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:47 smithi192 bash[24608]: audit 2024-09-13T22:32:47.024272+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:48.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:45.186546+0000 mon.a (mon.0) 423 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:48.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:46.061866+0000 mon.a (mon.0) 424 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:46.069835+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:46.070014+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:46.070170+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:46.070353+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: cluster 2024-09-13T22:32:46.229483+0000 mgr.y (mgr.14150) 170 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:46.968358+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:47.020524+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:47.023373+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:47.023891+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[22840]: audit 2024-09-13T22:32:47.024272+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:48.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:45.186546+0000 mon.a (mon.0) 423 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:46.061866+0000 mon.a (mon.0) 424 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:46.069835+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:46.070014+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:46.070170+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:46.070353+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: cluster 2024-09-13T22:32:46.229483+0000 mgr.y (mgr.14150) 170 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:46.968358+0000 mon.b (mon.1) 10 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:47.020524+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:47.023373+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:47.023891+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:32:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:47 smithi178 bash[29889]: audit 2024-09-13T22:32:47.024272+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:32:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:48 smithi192 bash[24608]: audit 2024-09-13T22:32:47.023972+0000 mon.c (mon.2) 4 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:48 smithi192 bash[24608]: audit 2024-09-13T22:32:47.902063+0000 mon.c (mon.2) 5 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:48 smithi192 bash[24608]: cluster 2024-09-13T22:32:48.230039+0000 mgr.y (mgr.14150) 171 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 192 KiB/s wr, 12 op/s 2024-09-13T22:32:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:48 smithi178 bash[22840]: audit 2024-09-13T22:32:47.023972+0000 mon.c (mon.2) 4 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:48 smithi178 bash[22840]: audit 2024-09-13T22:32:47.902063+0000 mon.c (mon.2) 5 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:48 smithi178 bash[22840]: cluster 2024-09-13T22:32:48.230039+0000 mgr.y (mgr.14150) 171 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 192 KiB/s wr, 12 op/s 2024-09-13T22:32:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:48 smithi178 bash[29889]: audit 2024-09-13T22:32:47.023972+0000 mon.c (mon.2) 4 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T22:32:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:48 smithi178 bash[29889]: audit 2024-09-13T22:32:47.902063+0000 mon.c (mon.2) 5 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T22:32:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:48 smithi178 bash[29889]: cluster 2024-09-13T22:32:48.230039+0000 mgr.y (mgr.14150) 171 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 192 KiB/s wr, 12 op/s 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: cephadm 2024-09-13T22:32:49.982498+0000 mgr.y (mgr.14150) 172 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:49.990440+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:49.997809+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:49.999849+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:50.001085+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:50.002154+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: cephadm 2024-09-13T22:32:50.002917+0000 mgr.y (mgr.14150) 173 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 5413M 2024-09-13T22:32:51.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:50.008139+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:50.012194+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:51.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:50.013186+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:32:51.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: audit 2024-09-13T22:32:50.019333+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:50 smithi192 bash[24608]: cluster 2024-09-13T22:32:50.230599+0000 mgr.y (mgr.14150) 174 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 141 KiB/s wr, 8 op/s 2024-09-13T22:32:51.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: cephadm 2024-09-13T22:32:49.982498+0000 mgr.y (mgr.14150) 172 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:32:51.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:49.990440+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:49.997809+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:49.999849+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:50.001085+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:50.002154+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: cephadm 2024-09-13T22:32:50.002917+0000 mgr.y (mgr.14150) 173 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 5413M 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:50.008139+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:50.012194+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:50.013186+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: audit 2024-09-13T22:32:50.019333+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[22840]: cluster 2024-09-13T22:32:50.230599+0000 mgr.y (mgr.14150) 174 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 141 KiB/s wr, 8 op/s 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: cephadm 2024-09-13T22:32:49.982498+0000 mgr.y (mgr.14150) 172 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:32:51.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:49.990440+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:49.997809+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:49.999849+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:50.001085+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:50.002154+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: cephadm 2024-09-13T22:32:50.002917+0000 mgr.y (mgr.14150) 173 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 5413M 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:50.008139+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:50.012194+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:50.013186+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: audit 2024-09-13T22:32:50.019333+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:32:51.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:50 smithi178 bash[29889]: cluster 2024-09-13T22:32:50.230599+0000 mgr.y (mgr.14150) 174 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 141 KiB/s wr, 8 op/s 2024-09-13T22:32:52.474 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:32:53.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:53 smithi178 bash[22840]: cluster 2024-09-13T22:32:52.231263+0000 mgr.y (mgr.14150) 175 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 111 KiB/s wr, 6 op/s 2024-09-13T22:32:53.566 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:53 smithi178 bash[29889]: cluster 2024-09-13T22:32:52.231263+0000 mgr.y (mgr.14150) 175 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 111 KiB/s wr, 6 op/s 2024-09-13T22:32:53.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:53 smithi192 bash[24608]: cluster 2024-09-13T22:32:52.231263+0000 mgr.y (mgr.14150) 175 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 111 KiB/s wr, 6 op/s 2024-09-13T22:32:54.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:54 smithi192 bash[24608]: audit 2024-09-13T22:32:53.925474+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:32:54.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:54 smithi192 bash[24608]: audit 2024-09-13T22:32:53.930842+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:32:54.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:54 smithi192 bash[24608]: audit 2024-09-13T22:32:53.931740+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:54.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:54 smithi178 bash[29889]: audit 2024-09-13T22:32:53.925474+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:32:54.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:54 smithi178 bash[29889]: audit 2024-09-13T22:32:53.930842+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:32:54.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:54 smithi178 bash[29889]: audit 2024-09-13T22:32:53.931740+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:54.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:54 smithi178 bash[22840]: audit 2024-09-13T22:32:53.925474+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:32:54.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:54 smithi178 bash[22840]: audit 2024-09-13T22:32:53.930842+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:32:54.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:54 smithi178 bash[22840]: audit 2024-09-13T22:32:53.931740+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:32:55.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:55 smithi192 bash[24608]: audit 2024-09-13T22:32:53.919330+0000 mgr.y (mgr.14150) 176 : audit [DBG] from='client.14298 v1:172.21.15.178:0/2904488949' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:32:55.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:55 smithi192 bash[24608]: cluster 2024-09-13T22:32:54.231808+0000 mgr.y (mgr.14150) 177 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-09-13T22:32:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:55 smithi178 bash[22840]: audit 2024-09-13T22:32:53.919330+0000 mgr.y (mgr.14150) 176 : audit [DBG] from='client.14298 v1:172.21.15.178:0/2904488949' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:32:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:55 smithi178 bash[22840]: cluster 2024-09-13T22:32:54.231808+0000 mgr.y (mgr.14150) 177 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-09-13T22:32:55.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:55 smithi178 bash[29889]: audit 2024-09-13T22:32:53.919330+0000 mgr.y (mgr.14150) 176 : audit [DBG] from='client.14298 v1:172.21.15.178:0/2904488949' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:32:55.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:55 smithi178 bash[29889]: cluster 2024-09-13T22:32:54.231808+0000 mgr.y (mgr.14150) 177 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-09-13T22:32:57.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:57 smithi192 bash[24608]: cluster 2024-09-13T22:32:56.232333+0000 mgr.y (mgr.14150) 178 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-13T22:32:57.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:57 smithi178 bash[22840]: cluster 2024-09-13T22:32:56.232333+0000 mgr.y (mgr.14150) 178 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-13T22:32:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:57 smithi178 bash[29889]: cluster 2024-09-13T22:32:56.232333+0000 mgr.y (mgr.14150) 178 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-13T22:32:59.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:32:59 smithi178 bash[22840]: cluster 2024-09-13T22:32:58.232886+0000 mgr.y (mgr.14150) 179 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-13T22:32:59.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:32:59 smithi178 bash[29889]: cluster 2024-09-13T22:32:58.232886+0000 mgr.y (mgr.14150) 179 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-13T22:32:59.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:32:59 smithi192 bash[24608]: cluster 2024-09-13T22:32:58.232886+0000 mgr.y (mgr.14150) 179 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-13T22:33:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:01 smithi192 bash[24608]: cluster 2024-09-13T22:33:00.233381+0000 mgr.y (mgr.14150) 180 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T22:33:01.662 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:01 smithi178 bash[22840]: cluster 2024-09-13T22:33:00.233381+0000 mgr.y (mgr.14150) 180 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T22:33:01.662 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:01 smithi178 bash[29889]: cluster 2024-09-13T22:33:00.233381+0000 mgr.y (mgr.14150) 180 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T22:33:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:03 smithi192 bash[24608]: cluster 2024-09-13T22:33:02.233730+0000 mgr.y (mgr.14150) 181 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T22:33:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:03 smithi192 bash[24608]: audit 2024-09-13T22:33:02.413224+0000 mon.a (mon.0) 444 : audit [INF] from='client.? v1:172.21.15.178:0/2743858439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad"}]: dispatch 2024-09-13T22:33:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:03 smithi192 bash[24608]: audit 2024-09-13T22:33:02.416251+0000 mon.a (mon.0) 445 : audit [INF] from='client.? v1:172.21.15.178:0/2743858439' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad"}]': finished 2024-09-13T22:33:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:03 smithi192 bash[24608]: cluster 2024-09-13T22:33:02.421419+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-13T22:33:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:03 smithi192 bash[24608]: audit 2024-09-13T22:33:02.436281+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:03.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:03 smithi192 bash[24608]: audit 2024-09-13T22:33:03.045530+0000 mon.a (mon.0) 448 : audit [DBG] from='client.? v1:172.21.15.178:0/903185568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:33:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[22840]: cluster 2024-09-13T22:33:02.233730+0000 mgr.y (mgr.14150) 181 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T22:33:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[22840]: audit 2024-09-13T22:33:02.413224+0000 mon.a (mon.0) 444 : audit [INF] from='client.? v1:172.21.15.178:0/2743858439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad"}]: dispatch 2024-09-13T22:33:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[22840]: audit 2024-09-13T22:33:02.416251+0000 mon.a (mon.0) 445 : audit [INF] from='client.? v1:172.21.15.178:0/2743858439' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad"}]': finished 2024-09-13T22:33:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[22840]: cluster 2024-09-13T22:33:02.421419+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-13T22:33:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[22840]: audit 2024-09-13T22:33:02.436281+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[22840]: audit 2024-09-13T22:33:03.045530+0000 mon.a (mon.0) 448 : audit [DBG] from='client.? v1:172.21.15.178:0/903185568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[29889]: cluster 2024-09-13T22:33:02.233730+0000 mgr.y (mgr.14150) 181 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[29889]: audit 2024-09-13T22:33:02.413224+0000 mon.a (mon.0) 444 : audit [INF] from='client.? v1:172.21.15.178:0/2743858439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad"}]: dispatch 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[29889]: audit 2024-09-13T22:33:02.416251+0000 mon.a (mon.0) 445 : audit [INF] from='client.? v1:172.21.15.178:0/2743858439' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad"}]': finished 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[29889]: cluster 2024-09-13T22:33:02.421419+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[29889]: audit 2024-09-13T22:33:02.436281+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:03.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:03 smithi178 bash[29889]: audit 2024-09-13T22:33:03.045530+0000 mon.a (mon.0) 448 : audit [DBG] from='client.? v1:172.21.15.178:0/903185568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:33:05.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:05 smithi192 bash[24608]: cluster 2024-09-13T22:33:04.234129+0000 mgr.y (mgr.14150) 182 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:05.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:05 smithi178 bash[22840]: cluster 2024-09-13T22:33:04.234129+0000 mgr.y (mgr.14150) 182 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:05.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:05 smithi178 bash[29889]: cluster 2024-09-13T22:33:04.234129+0000 mgr.y (mgr.14150) 182 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:07 smithi192 bash[24608]: cluster 2024-09-13T22:33:06.234616+0000 mgr.y (mgr.14150) 183 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:07 smithi178 bash[22840]: cluster 2024-09-13T22:33:06.234616+0000 mgr.y (mgr.14150) 183 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:07 smithi178 bash[29889]: cluster 2024-09-13T22:33:06.234616+0000 mgr.y (mgr.14150) 183 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:09.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:09 smithi192 bash[24608]: cluster 2024-09-13T22:33:07.787937+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:09.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:09 smithi192 bash[24608]: cluster 2024-09-13T22:33:07.788044+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:09.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:09 smithi192 bash[24608]: cluster 2024-09-13T22:33:08.235272+0000 mgr.y (mgr.14150) 184 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:09.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:09 smithi192 bash[24608]: cluster 2024-09-13T22:33:08.329035+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-13T22:33:09.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:09 smithi192 bash[24608]: audit 2024-09-13T22:33:08.329261+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[22840]: cluster 2024-09-13T22:33:07.787937+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[22840]: cluster 2024-09-13T22:33:07.788044+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[22840]: cluster 2024-09-13T22:33:08.235272+0000 mgr.y (mgr.14150) 184 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[22840]: cluster 2024-09-13T22:33:08.329035+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-13T22:33:09.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[22840]: audit 2024-09-13T22:33:08.329261+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[29889]: cluster 2024-09-13T22:33:07.787937+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[29889]: cluster 2024-09-13T22:33:07.788044+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[29889]: cluster 2024-09-13T22:33:08.235272+0000 mgr.y (mgr.14150) 184 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[29889]: cluster 2024-09-13T22:33:08.329035+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-13T22:33:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:09 smithi178 bash[29889]: audit 2024-09-13T22:33:08.329261+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:11.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:11 smithi192 bash[24608]: cluster 2024-09-13T22:33:10.235840+0000 mgr.y (mgr.14150) 185 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:11.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:11 smithi178 bash[22840]: cluster 2024-09-13T22:33:10.235840+0000 mgr.y (mgr.14150) 185 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:11 smithi178 bash[29889]: cluster 2024-09-13T22:33:10.235840+0000 mgr.y (mgr.14150) 185 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:13 smithi178 bash[29889]: cluster 2024-09-13T22:33:12.236373+0000 mgr.y (mgr.14150) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:13.570 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:13 smithi178 bash[22840]: cluster 2024-09-13T22:33:12.236373+0000 mgr.y (mgr.14150) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:13.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:13 smithi192 bash[24608]: cluster 2024-09-13T22:33:12.236373+0000 mgr.y (mgr.14150) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:15.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:15 smithi192 bash[24608]: cluster 2024-09-13T22:33:14.237021+0000 mgr.y (mgr.14150) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:15 smithi178 bash[22840]: cluster 2024-09-13T22:33:14.237021+0000 mgr.y (mgr.14150) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:15 smithi178 bash[29889]: cluster 2024-09-13T22:33:14.237021+0000 mgr.y (mgr.14150) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:16.617 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:16 smithi178 bash[29889]: audit 2024-09-13T22:33:16.007342+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T22:33:16.617 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:16 smithi178 bash[29889]: audit 2024-09-13T22:33:16.008874+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:16.617 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:16 smithi178 bash[22840]: audit 2024-09-13T22:33:16.007342+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T22:33:16.617 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:16 smithi178 bash[22840]: audit 2024-09-13T22:33:16.008874+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:16.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:16 smithi192 bash[24608]: audit 2024-09-13T22:33:16.007342+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T22:33:16.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:16 smithi192 bash[24608]: audit 2024-09-13T22:33:16.008874+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:17.633 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:17 smithi178 bash[22840]: cephadm 2024-09-13T22:33:16.010181+0000 mgr.y (mgr.14150) 188 : cephadm [INF] Deploying daemon osd.3 on smithi178 2024-09-13T22:33:17.633 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:17 smithi178 bash[22840]: cluster 2024-09-13T22:33:16.237556+0000 mgr.y (mgr.14150) 189 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:17.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:17 smithi178 bash[29889]: cephadm 2024-09-13T22:33:16.010181+0000 mgr.y (mgr.14150) 188 : cephadm [INF] Deploying daemon osd.3 on smithi178 2024-09-13T22:33:17.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:17 smithi178 bash[29889]: cluster 2024-09-13T22:33:16.237556+0000 mgr.y (mgr.14150) 189 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:17 smithi192 bash[24608]: cephadm 2024-09-13T22:33:16.010181+0000 mgr.y (mgr.14150) 188 : cephadm [INF] Deploying daemon osd.3 on smithi178 2024-09-13T22:33:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:17 smithi192 bash[24608]: cluster 2024-09-13T22:33:16.237556+0000 mgr.y (mgr.14150) 189 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:19.585 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:19 smithi178 bash[22840]: cluster 2024-09-13T22:33:18.238059+0000 mgr.y (mgr.14150) 190 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:19.585 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:19 smithi178 bash[29889]: cluster 2024-09-13T22:33:18.238059+0000 mgr.y (mgr.14150) 190 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:19.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:19 smithi192 bash[24608]: cluster 2024-09-13T22:33:18.238059+0000 mgr.y (mgr.14150) 190 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:20.633 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.634 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.634 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.634 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.635 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.635 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.930 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.932 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:20.933 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:33:20 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:33:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:21 smithi192 bash[24608]: cluster 2024-09-13T22:33:20.238575+0000 mgr.y (mgr.14150) 191 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:21 smithi192 bash[24608]: audit 2024-09-13T22:33:20.914806+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:33:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:21 smithi192 bash[24608]: audit 2024-09-13T22:33:20.918747+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:21 smithi192 bash[24608]: audit 2024-09-13T22:33:20.923441+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:21.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[22840]: cluster 2024-09-13T22:33:20.238575+0000 mgr.y (mgr.14150) 191 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:21.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[22840]: audit 2024-09-13T22:33:20.914806+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:33:21.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[22840]: audit 2024-09-13T22:33:20.918747+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:21.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[22840]: audit 2024-09-13T22:33:20.923441+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[29889]: cluster 2024-09-13T22:33:20.238575+0000 mgr.y (mgr.14150) 191 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[29889]: audit 2024-09-13T22:33:20.914806+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:33:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[29889]: audit 2024-09-13T22:33:20.918747+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:21 smithi178 bash[29889]: audit 2024-09-13T22:33:20.923441+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:23.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:23 smithi192 bash[24608]: cluster 2024-09-13T22:33:22.239098+0000 mgr.y (mgr.14150) 192 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:23 smithi178 bash[22840]: cluster 2024-09-13T22:33:22.239098+0000 mgr.y (mgr.14150) 192 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:23 smithi178 bash[29889]: cluster 2024-09-13T22:33:22.239098+0000 mgr.y (mgr.14150) 192 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:25 smithi192 bash[24608]: cluster 2024-09-13T22:33:24.239613+0000 mgr.y (mgr.14150) 193 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:25 smithi192 bash[24608]: cluster 2024-09-13T22:33:24.625926+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:25 smithi192 bash[24608]: cluster 2024-09-13T22:33:24.626074+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:25.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:25 smithi178 bash[29889]: cluster 2024-09-13T22:33:24.239613+0000 mgr.y (mgr.14150) 193 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:25 smithi178 bash[29889]: cluster 2024-09-13T22:33:24.625926+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:25 smithi178 bash[29889]: cluster 2024-09-13T22:33:24.626074+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:25 smithi178 bash[22840]: cluster 2024-09-13T22:33:24.239613+0000 mgr.y (mgr.14150) 193 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:25 smithi178 bash[22840]: cluster 2024-09-13T22:33:24.625926+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:25 smithi178 bash[22840]: cluster 2024-09-13T22:33:24.626074+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:26 smithi192 bash[24608]: cluster 2024-09-13T22:33:25.364310+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-13T22:33:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:26 smithi192 bash[24608]: audit 2024-09-13T22:33:25.365029+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:26 smithi178 bash[29889]: cluster 2024-09-13T22:33:25.364310+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-13T22:33:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:26 smithi178 bash[29889]: audit 2024-09-13T22:33:25.365029+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:26 smithi178 bash[22840]: cluster 2024-09-13T22:33:25.364310+0000 mon.a (mon.0) 456 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-13T22:33:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:26 smithi178 bash[22840]: audit 2024-09-13T22:33:25.365029+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:27 smithi192 bash[24608]: cluster 2024-09-13T22:33:26.240156+0000 mgr.y (mgr.14150) 194 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:27 smithi178 bash[29889]: cluster 2024-09-13T22:33:26.240156+0000 mgr.y (mgr.14150) 194 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:27.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:27 smithi178 bash[22840]: cluster 2024-09-13T22:33:26.240156+0000 mgr.y (mgr.14150) 194 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:28.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[22840]: audit 2024-09-13T22:33:27.568253+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[22840]: audit 2024-09-13T22:33:27.572674+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[22840]: audit 2024-09-13T22:33:27.573564+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:28.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[22840]: audit 2024-09-13T22:33:27.574232+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:33:28.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[22840]: audit 2024-09-13T22:33:27.578224+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[22840]: cluster 2024-09-13T22:33:28.240611+0000 mgr.y (mgr.14150) 195 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[29889]: audit 2024-09-13T22:33:27.568253+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[29889]: audit 2024-09-13T22:33:27.572674+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[29889]: audit 2024-09-13T22:33:27.573564+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[29889]: audit 2024-09-13T22:33:27.574232+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:33:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[29889]: audit 2024-09-13T22:33:27.578224+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:28 smithi178 bash[29889]: cluster 2024-09-13T22:33:28.240611+0000 mgr.y (mgr.14150) 195 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:28.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:28 smithi192 bash[24608]: audit 2024-09-13T22:33:27.568253+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:28 smithi192 bash[24608]: audit 2024-09-13T22:33:27.572674+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:28 smithi192 bash[24608]: audit 2024-09-13T22:33:27.573564+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:28 smithi192 bash[24608]: audit 2024-09-13T22:33:27.574232+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:33:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:28 smithi192 bash[24608]: audit 2024-09-13T22:33:27.578224+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:28 smithi192 bash[24608]: cluster 2024-09-13T22:33:28.240611+0000 mgr.y (mgr.14150) 195 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:30.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:29 smithi178 bash[22840]: audit 2024-09-13T22:33:29.862817+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T22:33:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:29 smithi178 bash[29889]: audit 2024-09-13T22:33:29.862817+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T22:33:30.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:29 smithi192 bash[24608]: audit 2024-09-13T22:33:29.862817+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T22:33:31.146 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 3 on host 'smithi178' 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: audit 2024-09-13T22:33:29.915295+0000 mon.a (mon.0) 464 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: cluster 2024-09-13T22:33:29.919962+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: audit 2024-09-13T22:33:29.920578+0000 mon.a (mon.0) 466 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: audit 2024-09-13T22:33:29.920893+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: cluster 2024-09-13T22:33:30.241028+0000 mgr.y (mgr.14150) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: audit 2024-09-13T22:33:30.917551+0000 mon.a (mon.0) 468 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[29889]: cluster 2024-09-13T22:33:30.923262+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-13T22:33:31.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: audit 2024-09-13T22:33:29.915295+0000 mon.a (mon.0) 464 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T22:33:31.195 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: cluster 2024-09-13T22:33:29.919962+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-13T22:33:31.195 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: audit 2024-09-13T22:33:29.920578+0000 mon.a (mon.0) 466 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:33:31.195 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: audit 2024-09-13T22:33:29.920893+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:31.195 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: cluster 2024-09-13T22:33:30.241028+0000 mgr.y (mgr.14150) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:31.195 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: audit 2024-09-13T22:33:30.917551+0000 mon.a (mon.0) 468 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:33:31.195 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:30 smithi178 bash[22840]: cluster 2024-09-13T22:33:30.923262+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-13T22:33:31.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: audit 2024-09-13T22:33:29.915295+0000 mon.a (mon.0) 464 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T22:33:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: cluster 2024-09-13T22:33:29.919962+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-13T22:33:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: audit 2024-09-13T22:33:29.920578+0000 mon.a (mon.0) 466 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-09-13T22:33:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: audit 2024-09-13T22:33:29.920893+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: cluster 2024-09-13T22:33:30.241028+0000 mgr.y (mgr.14150) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:31.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: audit 2024-09-13T22:33:30.917551+0000 mon.a (mon.0) 468 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-09-13T22:33:31.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:30 smithi192 bash[24608]: cluster 2024-09-13T22:33:30.923262+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-13T22:33:31.775 DEBUG:teuthology.orchestra.run.smithi178:osd.3> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T22:33:31.777 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_4... 2024-09-13T22:33:31.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T22:33:32.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:30.926845+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:30.934637+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:31.131211+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:33:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:31.137873+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:31.144798+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:31.230030+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' 2024-09-13T22:33:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:31 smithi192 bash[24608]: audit 2024-09-13T22:33:31.931908+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:30.926845+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:30.934637+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:31.131211+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:31.137873+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:31.144798+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:31.230030+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' 2024-09-13T22:33:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[22840]: audit 2024-09-13T22:33:31.931908+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:30.926845+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:30.934637+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:31.131211+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:31.137873+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:31.144798+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:31.230030+0000 mon.a (mon.0) 475 : audit [INF] from='osd.3 v1:172.21.15.178:6813/320675660' entity='osd.3' 2024-09-13T22:33:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:31 smithi178 bash[29889]: audit 2024-09-13T22:33:31.931908+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:33 smithi192 bash[24608]: cluster 2024-09-13T22:33:30.894501+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:33 smithi192 bash[24608]: cluster 2024-09-13T22:33:30.894625+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:33 smithi192 bash[24608]: cluster 2024-09-13T22:33:32.235374+0000 mon.a (mon.0) 477 : cluster [INF] osd.3 v1:172.21.15.178:6813/320675660 boot 2024-09-13T22:33:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:33 smithi192 bash[24608]: cluster 2024-09-13T22:33:32.235456+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-13T22:33:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:33 smithi192 bash[24608]: audit 2024-09-13T22:33:32.236005+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:33 smithi192 bash[24608]: cluster 2024-09-13T22:33:32.241403+0000 mgr.y (mgr.14150) 197 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[22840]: cluster 2024-09-13T22:33:30.894501+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[22840]: cluster 2024-09-13T22:33:30.894625+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[22840]: cluster 2024-09-13T22:33:32.235374+0000 mon.a (mon.0) 477 : cluster [INF] osd.3 v1:172.21.15.178:6813/320675660 boot 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[22840]: cluster 2024-09-13T22:33:32.235456+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[22840]: audit 2024-09-13T22:33:32.236005+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[22840]: cluster 2024-09-13T22:33:32.241403+0000 mgr.y (mgr.14150) 197 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[29889]: cluster 2024-09-13T22:33:30.894501+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[29889]: cluster 2024-09-13T22:33:30.894625+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[29889]: cluster 2024-09-13T22:33:32.235374+0000 mon.a (mon.0) 477 : cluster [INF] osd.3 v1:172.21.15.178:6813/320675660 boot 2024-09-13T22:33:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[29889]: cluster 2024-09-13T22:33:32.235456+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-09-13T22:33:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[29889]: audit 2024-09-13T22:33:32.236005+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:33:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:33 smithi178 bash[29889]: cluster 2024-09-13T22:33:32.241403+0000 mgr.y (mgr.14150) 197 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T22:33:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:34 smithi192 bash[24608]: cluster 2024-09-13T22:33:33.245311+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-13T22:33:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:34 smithi178 bash[22840]: cluster 2024-09-13T22:33:33.245311+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-13T22:33:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:34 smithi178 bash[29889]: cluster 2024-09-13T22:33:33.245311+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-13T22:33:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:35 smithi192 bash[24608]: cluster 2024-09-13T22:33:34.241736+0000 mgr.y (mgr.14150) 198 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:35.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:35 smithi178 bash[22840]: cluster 2024-09-13T22:33:34.241736+0000 mgr.y (mgr.14150) 198 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:35 smithi178 bash[29889]: cluster 2024-09-13T22:33:34.241736+0000 mgr.y (mgr.14150) 198 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:36.506 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:33:37.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:37 smithi192 bash[24608]: cluster 2024-09-13T22:33:36.242144+0000 mgr.y (mgr.14150) 199 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:37.532 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:37 smithi178 bash[22840]: cluster 2024-09-13T22:33:36.242144+0000 mgr.y (mgr.14150) 199 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:37.532 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:37 smithi178 bash[29889]: cluster 2024-09-13T22:33:36.242144+0000 mgr.y (mgr.14150) 199 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:38.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:38 smithi192 bash[24608]: cluster 2024-09-13T22:33:38.242657+0000 mgr.y (mgr.14150) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:38.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:38 smithi178 bash[29889]: cluster 2024-09-13T22:33:38.242657+0000 mgr.y (mgr.14150) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:38 smithi178 bash[22840]: cluster 2024-09-13T22:33:38.242657+0000 mgr.y (mgr.14150) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:39.986 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:33:40.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-09-13T22:33:41.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: cluster 2024-09-13T22:33:40.243311+0000 mgr.y (mgr.14150) 201 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:41.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.755533+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.763844+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.766532+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.782039+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.783584+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.785037+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.792604+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.798976+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.800511+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:33:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:41 smithi192 bash[24608]: audit 2024-09-13T22:33:40.810317+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: cluster 2024-09-13T22:33:40.243311+0000 mgr.y (mgr.14150) 201 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:41.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.755533+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.763844+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.766532+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.782039+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.783584+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.785037+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.792604+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.798976+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.800511+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[22840]: audit 2024-09-13T22:33:40.810317+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: cluster 2024-09-13T22:33:40.243311+0000 mgr.y (mgr.14150) 201 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.755533+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.763844+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.766532+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.782039+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.783584+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.785037+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:33:41.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.792604+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:41.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.798976+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:41.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.800511+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:33:41.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:41 smithi178 bash[29889]: audit 2024-09-13T22:33:40.810317+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:33:42.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:42 smithi192 bash[24608]: cephadm 2024-09-13T22:33:40.746558+0000 mgr.y (mgr.14150) 202 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:33:42.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:42 smithi192 bash[24608]: cephadm 2024-09-13T22:33:40.786099+0000 mgr.y (mgr.14150) 203 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 4060M 2024-09-13T22:33:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:42 smithi178 bash[29889]: cephadm 2024-09-13T22:33:40.746558+0000 mgr.y (mgr.14150) 202 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:33:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:42 smithi178 bash[29889]: cephadm 2024-09-13T22:33:40.786099+0000 mgr.y (mgr.14150) 203 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 4060M 2024-09-13T22:33:42.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:42 smithi178 bash[22840]: cephadm 2024-09-13T22:33:40.746558+0000 mgr.y (mgr.14150) 202 : cephadm [INF] Detected new or changed devices on smithi178 2024-09-13T22:33:42.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:42 smithi178 bash[22840]: cephadm 2024-09-13T22:33:40.786099+0000 mgr.y (mgr.14150) 203 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 4060M 2024-09-13T22:33:43.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:43 smithi192 bash[24608]: cluster 2024-09-13T22:33:42.243875+0000 mgr.y (mgr.14150) 204 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:43.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:43 smithi178 bash[22840]: cluster 2024-09-13T22:33:42.243875+0000 mgr.y (mgr.14150) 204 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:43.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:43 smithi178 bash[29889]: cluster 2024-09-13T22:33:42.243875+0000 mgr.y (mgr.14150) 204 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:44.753 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:33:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:45 smithi192 bash[24608]: cluster 2024-09-13T22:33:44.244520+0000 mgr.y (mgr.14150) 205 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:45 smithi178 bash[22840]: cluster 2024-09-13T22:33:44.244520+0000 mgr.y (mgr.14150) 205 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:45.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:45 smithi178 bash[29889]: cluster 2024-09-13T22:33:44.244520+0000 mgr.y (mgr.14150) 205 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:46.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:46 smithi192 bash[24608]: audit 2024-09-13T22:33:46.209057+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:33:46.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:46 smithi192 bash[24608]: audit 2024-09-13T22:33:46.215221+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:33:46.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:46 smithi192 bash[24608]: audit 2024-09-13T22:33:46.216830+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:46 smithi178 bash[22840]: audit 2024-09-13T22:33:46.209057+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:33:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:46 smithi178 bash[22840]: audit 2024-09-13T22:33:46.215221+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:33:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:46 smithi178 bash[22840]: audit 2024-09-13T22:33:46.216830+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:46 smithi178 bash[29889]: audit 2024-09-13T22:33:46.209057+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:33:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:46 smithi178 bash[29889]: audit 2024-09-13T22:33:46.215221+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:33:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:46 smithi178 bash[29889]: audit 2024-09-13T22:33:46.216830+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:33:47.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:47 smithi192 bash[24608]: audit 2024-09-13T22:33:46.202267+0000 mgr.y (mgr.14150) 206 : audit [DBG] from='client.24244 v1:172.21.15.192:0/3544284196' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:33:47.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:47 smithi192 bash[24608]: cluster 2024-09-13T22:33:46.245050+0000 mgr.y (mgr.14150) 207 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:47.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:47 smithi178 bash[22840]: audit 2024-09-13T22:33:46.202267+0000 mgr.y (mgr.14150) 206 : audit [DBG] from='client.24244 v1:172.21.15.192:0/3544284196' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:33:47.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:47 smithi178 bash[22840]: cluster 2024-09-13T22:33:46.245050+0000 mgr.y (mgr.14150) 207 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:47.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:47 smithi178 bash[29889]: audit 2024-09-13T22:33:46.202267+0000 mgr.y (mgr.14150) 206 : audit [DBG] from='client.24244 v1:172.21.15.192:0/3544284196' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:33:47.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:47 smithi178 bash[29889]: cluster 2024-09-13T22:33:46.245050+0000 mgr.y (mgr.14150) 207 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:49.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:49 smithi192 bash[24608]: cluster 2024-09-13T22:33:48.245584+0000 mgr.y (mgr.14150) 208 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:49.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:49 smithi178 bash[22840]: cluster 2024-09-13T22:33:48.245584+0000 mgr.y (mgr.14150) 208 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:49.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:49 smithi178 bash[29889]: cluster 2024-09-13T22:33:48.245584+0000 mgr.y (mgr.14150) 208 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:51.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:51 smithi192 bash[24608]: cluster 2024-09-13T22:33:50.246119+0000 mgr.y (mgr.14150) 209 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:51 smithi178 bash[22840]: cluster 2024-09-13T22:33:50.246119+0000 mgr.y (mgr.14150) 209 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:51 smithi178 bash[29889]: cluster 2024-09-13T22:33:50.246119+0000 mgr.y (mgr.14150) 209 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:53.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:53 smithi192 bash[24608]: cluster 2024-09-13T22:33:52.246643+0000 mgr.y (mgr.14150) 210 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:53 smithi178 bash[22840]: cluster 2024-09-13T22:33:52.246643+0000 mgr.y (mgr.14150) 210 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:53.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:53 smithi178 bash[29889]: cluster 2024-09-13T22:33:52.246643+0000 mgr.y (mgr.14150) 210 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:55.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: cluster 2024-09-13T22:33:54.247315+0000 mgr.y (mgr.14150) 211 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: audit 2024-09-13T22:33:54.424057+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.192:0/1607177651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]: dispatch 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: audit 2024-09-13T22:33:54.424379+0000 mon.a (mon.0) 494 : audit [INF] from='client.24250 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]: dispatch 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: audit 2024-09-13T22:33:54.427541+0000 mon.a (mon.0) 495 : audit [INF] from='client.24250 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]': finished 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: cluster 2024-09-13T22:33:54.434697+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: audit 2024-09-13T22:33:54.435323+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: cluster 2024-09-13T22:33:54.504206+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: cluster 2024-09-13T22:33:54.504331+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:55 smithi192 bash[24608]: audit 2024-09-13T22:33:55.018006+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.192:0/3899759596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:33:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: cluster 2024-09-13T22:33:54.247315+0000 mgr.y (mgr.14150) 211 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: audit 2024-09-13T22:33:54.424057+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.192:0/1607177651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]: dispatch 2024-09-13T22:33:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: audit 2024-09-13T22:33:54.424379+0000 mon.a (mon.0) 494 : audit [INF] from='client.24250 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]: dispatch 2024-09-13T22:33:55.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: audit 2024-09-13T22:33:54.427541+0000 mon.a (mon.0) 495 : audit [INF] from='client.24250 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]': finished 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: cluster 2024-09-13T22:33:54.434697+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: audit 2024-09-13T22:33:54.435323+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: cluster 2024-09-13T22:33:54.504206+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: cluster 2024-09-13T22:33:54.504331+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[22840]: audit 2024-09-13T22:33:55.018006+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.192:0/3899759596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: cluster 2024-09-13T22:33:54.247315+0000 mgr.y (mgr.14150) 211 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: audit 2024-09-13T22:33:54.424057+0000 mon.b (mon.1) 11 : audit [INF] from='client.? v1:172.21.15.192:0/1607177651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]: dispatch 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: audit 2024-09-13T22:33:54.424379+0000 mon.a (mon.0) 494 : audit [INF] from='client.24250 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]: dispatch 2024-09-13T22:33:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: audit 2024-09-13T22:33:54.427541+0000 mon.a (mon.0) 495 : audit [INF] from='client.24250 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b"}]': finished 2024-09-13T22:33:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: cluster 2024-09-13T22:33:54.434697+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-09-13T22:33:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: audit 2024-09-13T22:33:54.435323+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:33:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: cluster 2024-09-13T22:33:54.504206+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:33:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: cluster 2024-09-13T22:33:54.504331+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:33:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:55 smithi178 bash[29889]: audit 2024-09-13T22:33:55.018006+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.192:0/3899759596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:33:56.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:56 smithi192 bash[24608]: cluster 2024-09-13T22:33:55.434403+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-13T22:33:56.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:56 smithi192 bash[24608]: audit 2024-09-13T22:33:55.434737+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:33:56.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:56 smithi178 bash[22840]: cluster 2024-09-13T22:33:55.434403+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-13T22:33:56.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:56 smithi178 bash[22840]: audit 2024-09-13T22:33:55.434737+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:33:56.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:56 smithi178 bash[29889]: cluster 2024-09-13T22:33:55.434403+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-09-13T22:33:56.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:56 smithi178 bash[29889]: audit 2024-09-13T22:33:55.434737+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:33:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:57 smithi192 bash[24608]: cluster 2024-09-13T22:33:56.247896+0000 mgr.y (mgr.14150) 212 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:57.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:57 smithi178 bash[22840]: cluster 2024-09-13T22:33:56.247896+0000 mgr.y (mgr.14150) 212 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:57.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:57 smithi178 bash[29889]: cluster 2024-09-13T22:33:56.247896+0000 mgr.y (mgr.14150) 212 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:58.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:33:58 smithi192 bash[24608]: cluster 2024-09-13T22:33:58.248425+0000 mgr.y (mgr.14150) 213 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:33:58 smithi178 bash[29889]: cluster 2024-09-13T22:33:58.248425+0000 mgr.y (mgr.14150) 213 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:33:58.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:33:58 smithi178 bash[22840]: cluster 2024-09-13T22:33:58.248425+0000 mgr.y (mgr.14150) 213 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:01 smithi192 bash[24608]: cluster 2024-09-13T22:34:00.248956+0000 mgr.y (mgr.14150) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:01.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:01 smithi178 bash[22840]: cluster 2024-09-13T22:34:00.248956+0000 mgr.y (mgr.14150) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:01 smithi178 bash[29889]: cluster 2024-09-13T22:34:00.248956+0000 mgr.y (mgr.14150) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:03 smithi192 bash[24608]: cluster 2024-09-13T22:34:02.249574+0000 mgr.y (mgr.14150) 215 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:03 smithi178 bash[22840]: cluster 2024-09-13T22:34:02.249574+0000 mgr.y (mgr.14150) 215 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:03 smithi178 bash[29889]: cluster 2024-09-13T22:34:02.249574+0000 mgr.y (mgr.14150) 215 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:05.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:05 smithi192 bash[24608]: cluster 2024-09-13T22:34:04.250171+0000 mgr.y (mgr.14150) 216 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:05.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:05 smithi178 bash[22840]: cluster 2024-09-13T22:34:04.250171+0000 mgr.y (mgr.14150) 216 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:05.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:05 smithi178 bash[29889]: cluster 2024-09-13T22:34:04.250171+0000 mgr.y (mgr.14150) 216 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:07 smithi192 bash[24608]: cluster 2024-09-13T22:34:06.250689+0000 mgr.y (mgr.14150) 217 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:07.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:07 smithi178 bash[22840]: cluster 2024-09-13T22:34:06.250689+0000 mgr.y (mgr.14150) 217 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:07 smithi178 bash[29889]: cluster 2024-09-13T22:34:06.250689+0000 mgr.y (mgr.14150) 217 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:08.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:08 smithi192 bash[24608]: cluster 2024-09-13T22:34:07.248042+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:34:08.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:08 smithi192 bash[24608]: cluster 2024-09-13T22:34:07.251813+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:34:08.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:08 smithi192 bash[24608]: audit 2024-09-13T22:34:07.608215+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T22:34:08.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:08 smithi192 bash[24608]: audit 2024-09-13T22:34:07.609916+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[22840]: cluster 2024-09-13T22:34:07.248042+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[22840]: cluster 2024-09-13T22:34:07.251813+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[22840]: audit 2024-09-13T22:34:07.608215+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[22840]: audit 2024-09-13T22:34:07.609916+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[29889]: cluster 2024-09-13T22:34:07.248042+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[29889]: cluster 2024-09-13T22:34:07.251813+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[29889]: audit 2024-09-13T22:34:07.608215+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T22:34:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:08 smithi178 bash[29889]: audit 2024-09-13T22:34:07.609916+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:09.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:09 smithi192 bash[24608]: cephadm 2024-09-13T22:34:07.611440+0000 mgr.y (mgr.14150) 218 : cephadm [INF] Deploying daemon osd.4 on smithi192 2024-09-13T22:34:09.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:09 smithi192 bash[24608]: cluster 2024-09-13T22:34:08.251231+0000 mgr.y (mgr.14150) 219 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:09 smithi178 bash[22840]: cephadm 2024-09-13T22:34:07.611440+0000 mgr.y (mgr.14150) 218 : cephadm [INF] Deploying daemon osd.4 on smithi192 2024-09-13T22:34:09.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:09 smithi178 bash[22840]: cluster 2024-09-13T22:34:08.251231+0000 mgr.y (mgr.14150) 219 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:09.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:09 smithi178 bash[29889]: cephadm 2024-09-13T22:34:07.611440+0000 mgr.y (mgr.14150) 218 : cephadm [INF] Deploying daemon osd.4 on smithi192 2024-09-13T22:34:09.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:09 smithi178 bash[29889]: cluster 2024-09-13T22:34:08.251231+0000 mgr.y (mgr.14150) 219 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:11.627 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:11 smithi192 bash[24608]: cluster 2024-09-13T22:34:10.251752+0000 mgr.y (mgr.14150) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:11.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:11 smithi178 bash[22840]: cluster 2024-09-13T22:34:10.251752+0000 mgr.y (mgr.14150) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:11 smithi178 bash[29889]: cluster 2024-09-13T22:34:10.251752+0000 mgr.y (mgr.14150) 220 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:11.878 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:11 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:34:11.879 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:34:11 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:34:12.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:12 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:34:12.157 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:34:12 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:34:12.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:12 smithi192 bash[24608]: audit 2024-09-13T22:34:12.155296+0000 mon.a (mon.0) 502 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:34:12.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:12 smithi192 bash[24608]: audit 2024-09-13T22:34:12.164907+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:12.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:12 smithi192 bash[24608]: audit 2024-09-13T22:34:12.172782+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:12 smithi178 bash[22840]: audit 2024-09-13T22:34:12.155296+0000 mon.a (mon.0) 502 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:34:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:12 smithi178 bash[22840]: audit 2024-09-13T22:34:12.164907+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:12 smithi178 bash[22840]: audit 2024-09-13T22:34:12.172782+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:12.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:12 smithi178 bash[29889]: audit 2024-09-13T22:34:12.155296+0000 mon.a (mon.0) 502 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:34:12.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:12 smithi178 bash[29889]: audit 2024-09-13T22:34:12.164907+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:12.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:12 smithi178 bash[29889]: audit 2024-09-13T22:34:12.172782+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:13.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:13 smithi192 bash[24608]: cluster 2024-09-13T22:34:12.252277+0000 mgr.y (mgr.14150) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:13.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:13 smithi178 bash[22840]: cluster 2024-09-13T22:34:12.252277+0000 mgr.y (mgr.14150) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:13 smithi178 bash[29889]: cluster 2024-09-13T22:34:12.252277+0000 mgr.y (mgr.14150) 221 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:15 smithi192 bash[24608]: cluster 2024-09-13T22:34:14.252860+0000 mgr.y (mgr.14150) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:15 smithi178 bash[22840]: cluster 2024-09-13T22:34:14.252860+0000 mgr.y (mgr.14150) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:15 smithi178 bash[29889]: cluster 2024-09-13T22:34:14.252860+0000 mgr.y (mgr.14150) 222 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:17 smithi192 bash[24608]: cluster 2024-09-13T22:34:16.253375+0000 mgr.y (mgr.14150) 223 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:17.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:17 smithi178 bash[22840]: cluster 2024-09-13T22:34:16.253375+0000 mgr.y (mgr.14150) 223 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:17 smithi178 bash[29889]: cluster 2024-09-13T22:34:16.253375+0000 mgr.y (mgr.14150) 223 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[22840]: cluster 2024-09-13T22:34:18.253904+0000 mgr.y (mgr.14150) 224 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[22840]: audit 2024-09-13T22:34:18.519473+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[22840]: audit 2024-09-13T22:34:18.527448+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[22840]: audit 2024-09-13T22:34:18.529253+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[22840]: audit 2024-09-13T22:34:18.530989+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[22840]: audit 2024-09-13T22:34:18.540482+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[29889]: cluster 2024-09-13T22:34:18.253904+0000 mgr.y (mgr.14150) 224 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[29889]: audit 2024-09-13T22:34:18.519473+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[29889]: audit 2024-09-13T22:34:18.527448+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[29889]: audit 2024-09-13T22:34:18.529253+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[29889]: audit 2024-09-13T22:34:18.530989+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:34:18.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:18 smithi178 bash[29889]: audit 2024-09-13T22:34:18.540482+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:19.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:18 smithi192 bash[24608]: cluster 2024-09-13T22:34:18.253904+0000 mgr.y (mgr.14150) 224 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:19.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:18 smithi192 bash[24608]: audit 2024-09-13T22:34:18.519473+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:19.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:18 smithi192 bash[24608]: audit 2024-09-13T22:34:18.527448+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:19.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:18 smithi192 bash[24608]: audit 2024-09-13T22:34:18.529253+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:19.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:18 smithi192 bash[24608]: audit 2024-09-13T22:34:18.530989+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:34:19.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:18 smithi192 bash[24608]: audit 2024-09-13T22:34:18.540482+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:21 smithi192 bash[24608]: cluster 2024-09-13T22:34:20.254418+0000 mgr.y (mgr.14150) 225 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:21.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:21 smithi178 bash[22840]: cluster 2024-09-13T22:34:20.254418+0000 mgr.y (mgr.14150) 225 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:21 smithi178 bash[29889]: cluster 2024-09-13T22:34:20.254418+0000 mgr.y (mgr.14150) 225 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:22.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:22 smithi192 bash[24608]: audit 2024-09-13T22:34:21.957762+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.192:6800/3593670527' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T22:34:22.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:22 smithi192 bash[24608]: audit 2024-09-13T22:34:21.957866+0000 mon.a (mon.0) 510 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T22:34:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:22 smithi178 bash[22840]: audit 2024-09-13T22:34:21.957762+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.192:6800/3593670527' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T22:34:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:22 smithi178 bash[22840]: audit 2024-09-13T22:34:21.957866+0000 mon.a (mon.0) 510 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T22:34:22.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:22 smithi178 bash[29889]: audit 2024-09-13T22:34:21.957762+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 v1:172.21.15.192:6800/3593670527' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T22:34:22.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:22 smithi178 bash[29889]: audit 2024-09-13T22:34:21.957866+0000 mon.a (mon.0) 510 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T22:34:22.843 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2024-09-13T22:34:23.409 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T22:34:23.411 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-09-13T22:34:23.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T22:34:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: cluster 2024-09-13T22:34:22.254972+0000 mgr.y (mgr.14150) 226 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.320657+0000 mon.a (mon.0) 511 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T22:34:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.326560+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.192:6800/3593670527' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:34:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: cluster 2024-09-13T22:34:22.327749+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-13T22:34:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.328273+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:23.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.329023+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:34:23.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.824499+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:34:23.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.832532+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:23.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[24608]: audit 2024-09-13T22:34:22.840344+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: cluster 2024-09-13T22:34:22.254972+0000 mgr.y (mgr.14150) 226 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.320657+0000 mon.a (mon.0) 511 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T22:34:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.326560+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.192:6800/3593670527' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:34:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: cluster 2024-09-13T22:34:22.327749+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-13T22:34:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.328273+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.329023+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.824499+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.832532+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[22840]: audit 2024-09-13T22:34:22.840344+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: cluster 2024-09-13T22:34:22.254972+0000 mgr.y (mgr.14150) 226 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.320657+0000 mon.a (mon.0) 511 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.326560+0000 mon.b (mon.1) 14 : audit [INF] from='osd.4 v1:172.21.15.192:6800/3593670527' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: cluster 2024-09-13T22:34:22.327749+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.328273+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.329023+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.824499+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:34:23.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.832532+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:23.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:23 smithi178 bash[29889]: audit 2024-09-13T22:34:22.840344+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:24.160 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[27998]: debug 2024-09-13T22:34:23.766+0000 7f13878cd640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T22:34:24.160 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:34:23 smithi192 bash[27998]: debug 2024-09-13T22:34:23.774+0000 7f137f6ed640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T22:34:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:24 smithi192 bash[24608]: audit 2024-09-13T22:34:23.323185+0000 mon.a (mon.0) 518 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:34:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:24 smithi192 bash[24608]: cluster 2024-09-13T22:34:23.328949+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-13T22:34:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:24 smithi192 bash[24608]: audit 2024-09-13T22:34:23.330042+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:24 smithi192 bash[24608]: audit 2024-09-13T22:34:23.339042+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:24 smithi192 bash[24608]: audit 2024-09-13T22:34:23.779681+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-13T22:34:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:24 smithi192 bash[24608]: cluster 2024-09-13T22:34:24.331423+0000 mon.a (mon.0) 523 : cluster [INF] osd.4 v1:172.21.15.192:6800/3593670527 boot 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[22840]: audit 2024-09-13T22:34:23.323185+0000 mon.a (mon.0) 518 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[22840]: cluster 2024-09-13T22:34:23.328949+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[22840]: audit 2024-09-13T22:34:23.330042+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[22840]: audit 2024-09-13T22:34:23.339042+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[22840]: audit 2024-09-13T22:34:23.779681+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[22840]: cluster 2024-09-13T22:34:24.331423+0000 mon.a (mon.0) 523 : cluster [INF] osd.4 v1:172.21.15.192:6800/3593670527 boot 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[29889]: audit 2024-09-13T22:34:23.323185+0000 mon.a (mon.0) 518 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[29889]: cluster 2024-09-13T22:34:23.328949+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[29889]: audit 2024-09-13T22:34:23.330042+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:24.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[29889]: audit 2024-09-13T22:34:23.339042+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[29889]: audit 2024-09-13T22:34:23.779681+0000 mon.a (mon.0) 522 : audit [INF] from='osd.4 ' entity='osd.4' 2024-09-13T22:34:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:24 smithi178 bash[29889]: cluster 2024-09-13T22:34:24.331423+0000 mon.a (mon.0) 523 : cluster [INF] osd.4 v1:172.21.15.192:6800/3593670527 boot 2024-09-13T22:34:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:25 smithi192 bash[24608]: cluster 2024-09-13T22:34:22.986740+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:25 smithi192 bash[24608]: cluster 2024-09-13T22:34:22.986801+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:25 smithi192 bash[24608]: cluster 2024-09-13T22:34:24.255493+0000 mgr.y (mgr.14150) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:25.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:25 smithi192 bash[24608]: cluster 2024-09-13T22:34:24.331518+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2024-09-13T22:34:25.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:25 smithi192 bash[24608]: audit 2024-09-13T22:34:24.331857+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:25.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:25 smithi192 bash[24608]: cluster 2024-09-13T22:34:25.333056+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[22840]: cluster 2024-09-13T22:34:22.986740+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[22840]: cluster 2024-09-13T22:34:22.986801+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[22840]: cluster 2024-09-13T22:34:24.255493+0000 mgr.y (mgr.14150) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[22840]: cluster 2024-09-13T22:34:24.331518+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[22840]: audit 2024-09-13T22:34:24.331857+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[22840]: cluster 2024-09-13T22:34:25.333056+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[29889]: cluster 2024-09-13T22:34:22.986740+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[29889]: cluster 2024-09-13T22:34:22.986801+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[29889]: cluster 2024-09-13T22:34:24.255493+0000 mgr.y (mgr.14150) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-13T22:34:25.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[29889]: cluster 2024-09-13T22:34:24.331518+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2024-09-13T22:34:25.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[29889]: audit 2024-09-13T22:34:24.331857+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:34:25.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:25 smithi178 bash[29889]: cluster 2024-09-13T22:34:25.333056+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-13T22:34:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:27 smithi192 bash[24608]: cluster 2024-09-13T22:34:26.256023+0000 mgr.y (mgr.14150) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:27 smithi192 bash[24608]: cluster 2024-09-13T22:34:26.334737+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-13T22:34:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:27 smithi178 bash[22840]: cluster 2024-09-13T22:34:26.256023+0000 mgr.y (mgr.14150) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:27.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:27 smithi178 bash[22840]: cluster 2024-09-13T22:34:26.334737+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-13T22:34:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:27 smithi178 bash[29889]: cluster 2024-09-13T22:34:26.256023+0000 mgr.y (mgr.14150) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:27 smithi178 bash[29889]: cluster 2024-09-13T22:34:26.334737+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-13T22:34:28.196 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:34:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:28 smithi192 bash[24608]: cluster 2024-09-13T22:34:27.347536+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-13T22:34:28.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:28 smithi178 bash[22840]: cluster 2024-09-13T22:34:27.347536+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-13T22:34:28.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:28 smithi178 bash[29889]: cluster 2024-09-13T22:34:27.347536+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-13T22:34:29.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:29 smithi192 bash[24608]: cluster 2024-09-13T22:34:28.256584+0000 mgr.y (mgr.14150) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-13T22:34:29.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:29 smithi192 bash[24608]: cluster 2024-09-13T22:34:28.357056+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-13T22:34:29.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:29 smithi178 bash[22840]: cluster 2024-09-13T22:34:28.256584+0000 mgr.y (mgr.14150) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-13T22:34:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:29 smithi178 bash[22840]: cluster 2024-09-13T22:34:28.357056+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-13T22:34:29.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:29 smithi178 bash[29889]: cluster 2024-09-13T22:34:28.256584+0000 mgr.y (mgr.14150) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-13T22:34:29.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:29 smithi178 bash[29889]: cluster 2024-09-13T22:34:28.357056+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-09-13T22:34:31.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:31 smithi192 bash[24608]: cluster 2024-09-13T22:34:30.257149+0000 mgr.y (mgr.14150) 230 : cluster [DBG] pgmap v208: 1 pgs: 1 peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T22:34:31.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:31 smithi178 bash[22840]: cluster 2024-09-13T22:34:30.257149+0000 mgr.y (mgr.14150) 230 : cluster [DBG] pgmap v208: 1 pgs: 1 peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T22:34:31.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:31 smithi178 bash[29889]: cluster 2024-09-13T22:34:30.257149+0000 mgr.y (mgr.14150) 230 : cluster [DBG] pgmap v208: 1 pgs: 1 peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T22:34:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: cluster 2024-09-13T22:34:31.562853+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: cluster 2024-09-13T22:34:31.562975+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: cluster 2024-09-13T22:34:32.257732+0000 mgr.y (mgr.14150) 231 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: cluster 2024-09-13T22:34:32.371907+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: audit 2024-09-13T22:34:33.333536+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: audit 2024-09-13T22:34:33.343470+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: audit 2024-09-13T22:34:33.346086+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: audit 2024-09-13T22:34:33.355244+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: audit 2024-09-13T22:34:33.360716+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:33 smithi192 bash[24608]: audit 2024-09-13T22:34:33.362317+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: cluster 2024-09-13T22:34:31.562853+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: cluster 2024-09-13T22:34:31.562975+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: cluster 2024-09-13T22:34:32.257732+0000 mgr.y (mgr.14150) 231 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: cluster 2024-09-13T22:34:32.371907+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: audit 2024-09-13T22:34:33.333536+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: audit 2024-09-13T22:34:33.343470+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: audit 2024-09-13T22:34:33.346086+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: audit 2024-09-13T22:34:33.355244+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: audit 2024-09-13T22:34:33.360716+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[22840]: audit 2024-09-13T22:34:33.362317+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: cluster 2024-09-13T22:34:31.562853+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: cluster 2024-09-13T22:34:31.562975+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: cluster 2024-09-13T22:34:32.257732+0000 mgr.y (mgr.14150) 231 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: cluster 2024-09-13T22:34:32.371907+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: audit 2024-09-13T22:34:33.333536+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: audit 2024-09-13T22:34:33.343470+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: audit 2024-09-13T22:34:33.346086+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: audit 2024-09-13T22:34:33.355244+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: audit 2024-09-13T22:34:33.360716+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:33.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:33 smithi178 bash[29889]: audit 2024-09-13T22:34:33.362317+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:34:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:34 smithi192 bash[24608]: cephadm 2024-09-13T22:34:33.323329+0000 mgr.y (mgr.14150) 232 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:34:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:34 smithi192 bash[24608]: cephadm 2024-09-13T22:34:33.347315+0000 mgr.y (mgr.14150) 233 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 17265M 2024-09-13T22:34:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:34 smithi192 bash[24608]: audit 2024-09-13T22:34:33.380105+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:34 smithi178 bash[22840]: cephadm 2024-09-13T22:34:33.323329+0000 mgr.y (mgr.14150) 232 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:34:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:34 smithi178 bash[22840]: cephadm 2024-09-13T22:34:33.347315+0000 mgr.y (mgr.14150) 233 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 17265M 2024-09-13T22:34:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:34 smithi178 bash[22840]: audit 2024-09-13T22:34:33.380105+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:34 smithi178 bash[29889]: cephadm 2024-09-13T22:34:33.323329+0000 mgr.y (mgr.14150) 232 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:34:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:34 smithi178 bash[29889]: cephadm 2024-09-13T22:34:33.347315+0000 mgr.y (mgr.14150) 233 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 17265M 2024-09-13T22:34:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:34 smithi178 bash[29889]: audit 2024-09-13T22:34:33.380105+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:34:35.355 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:34:35.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-09-13T22:34:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:35 smithi192 bash[24608]: cluster 2024-09-13T22:34:34.258332+0000 mgr.y (mgr.14150) 234 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T22:34:35.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:35 smithi178 bash[22840]: cluster 2024-09-13T22:34:34.258332+0000 mgr.y (mgr.14150) 234 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T22:34:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:35 smithi178 bash[29889]: cluster 2024-09-13T22:34:34.258332+0000 mgr.y (mgr.14150) 234 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T22:34:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:37 smithi192 bash[24608]: cluster 2024-09-13T22:34:36.258863+0000 mgr.y (mgr.14150) 235 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:37.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:37 smithi178 bash[22840]: cluster 2024-09-13T22:34:36.258863+0000 mgr.y (mgr.14150) 235 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:37 smithi178 bash[29889]: cluster 2024-09-13T22:34:36.258863+0000 mgr.y (mgr.14150) 235 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:38 smithi178 bash[22840]: cluster 2024-09-13T22:34:38.259475+0000 mgr.y (mgr.14150) 236 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:38.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:38 smithi178 bash[29889]: cluster 2024-09-13T22:34:38.259475+0000 mgr.y (mgr.14150) 236 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:38 smithi192 bash[24608]: cluster 2024-09-13T22:34:38.259475+0000 mgr.y (mgr.14150) 236 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:40.144 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:34:41.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:40 smithi192 bash[24608]: cluster 2024-09-13T22:34:39.158483+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:41.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:40 smithi192 bash[24608]: cluster 2024-09-13T22:34:39.158598+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:41.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:40 smithi192 bash[24608]: cluster 2024-09-13T22:34:39.682007+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-09-13T22:34:41.145 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:40 smithi192 bash[24608]: cluster 2024-09-13T22:34:40.260026+0000 mgr.y (mgr.14150) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[22840]: cluster 2024-09-13T22:34:39.158483+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[22840]: cluster 2024-09-13T22:34:39.158598+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[22840]: cluster 2024-09-13T22:34:39.682007+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[22840]: cluster 2024-09-13T22:34:40.260026+0000 mgr.y (mgr.14150) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[29889]: cluster 2024-09-13T22:34:39.158483+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[29889]: cluster 2024-09-13T22:34:39.158598+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[29889]: cluster 2024-09-13T22:34:39.682007+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-09-13T22:34:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:40 smithi178 bash[29889]: cluster 2024-09-13T22:34:40.260026+0000 mgr.y (mgr.14150) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:42.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:41 smithi192 bash[24608]: audit 2024-09-13T22:34:41.479336+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:34:42.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:41 smithi192 bash[24608]: audit 2024-09-13T22:34:41.485233+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:34:42.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:41 smithi192 bash[24608]: audit 2024-09-13T22:34:41.486991+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:41 smithi178 bash[22840]: audit 2024-09-13T22:34:41.479336+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:34:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:41 smithi178 bash[22840]: audit 2024-09-13T22:34:41.485233+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:34:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:41 smithi178 bash[22840]: audit 2024-09-13T22:34:41.486991+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:42.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:41 smithi178 bash[29889]: audit 2024-09-13T22:34:41.479336+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:34:42.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:41 smithi178 bash[29889]: audit 2024-09-13T22:34:41.485233+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:34:42.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:41 smithi178 bash[29889]: audit 2024-09-13T22:34:41.486991+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:34:43.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:42 smithi192 bash[24608]: audit 2024-09-13T22:34:41.472982+0000 mgr.y (mgr.14150) 238 : audit [DBG] from='client.24271 v1:172.21.15.192:0/1715299359' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:34:43.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:42 smithi192 bash[24608]: cluster 2024-09-13T22:34:42.260583+0000 mgr.y (mgr.14150) 239 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:42 smithi178 bash[29889]: audit 2024-09-13T22:34:41.472982+0000 mgr.y (mgr.14150) 238 : audit [DBG] from='client.24271 v1:172.21.15.192:0/1715299359' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:34:43.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:42 smithi178 bash[29889]: cluster 2024-09-13T22:34:42.260583+0000 mgr.y (mgr.14150) 239 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:42 smithi178 bash[22840]: audit 2024-09-13T22:34:41.472982+0000 mgr.y (mgr.14150) 238 : audit [DBG] from='client.24271 v1:172.21.15.192:0/1715299359' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:34:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:42 smithi178 bash[22840]: cluster 2024-09-13T22:34:42.260583+0000 mgr.y (mgr.14150) 239 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:45 smithi192 bash[24608]: cluster 2024-09-13T22:34:44.261131+0000 mgr.y (mgr.14150) 240 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:45.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:45 smithi178 bash[22840]: cluster 2024-09-13T22:34:44.261131+0000 mgr.y (mgr.14150) 240 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:45.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:45 smithi178 bash[29889]: cluster 2024-09-13T22:34:44.261131+0000 mgr.y (mgr.14150) 240 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:47.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:47 smithi192 bash[24608]: cluster 2024-09-13T22:34:46.261674+0000 mgr.y (mgr.14150) 241 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:47.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:47 smithi178 bash[22840]: cluster 2024-09-13T22:34:46.261674+0000 mgr.y (mgr.14150) 241 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:47.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:47 smithi178 bash[29889]: cluster 2024-09-13T22:34:46.261674+0000 mgr.y (mgr.14150) 241 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:49.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:49 smithi192 bash[24608]: cluster 2024-09-13T22:34:48.262214+0000 mgr.y (mgr.14150) 242 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:49.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:49 smithi178 bash[22840]: cluster 2024-09-13T22:34:48.262214+0000 mgr.y (mgr.14150) 242 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:49 smithi178 bash[29889]: cluster 2024-09-13T22:34:48.262214+0000 mgr.y (mgr.14150) 242 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:50.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: cluster 2024-09-13T22:34:49.475563+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:50.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: cluster 2024-09-13T22:34:49.475650+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:50.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: audit 2024-09-13T22:34:49.790467+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.192:0/4145965181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]: dispatch 2024-09-13T22:34:50.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: audit 2024-09-13T22:34:49.790783+0000 mon.a (mon.0) 542 : audit [INF] from='client.24277 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]: dispatch 2024-09-13T22:34:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: audit 2024-09-13T22:34:49.794162+0000 mon.a (mon.0) 543 : audit [INF] from='client.24277 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]': finished 2024-09-13T22:34:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: cluster 2024-09-13T22:34:49.799816+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-13T22:34:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:50 smithi192 bash[24608]: audit 2024-09-13T22:34:49.800394+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: cluster 2024-09-13T22:34:49.475563+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: cluster 2024-09-13T22:34:49.475650+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: audit 2024-09-13T22:34:49.790467+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.192:0/4145965181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]: dispatch 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: audit 2024-09-13T22:34:49.790783+0000 mon.a (mon.0) 542 : audit [INF] from='client.24277 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]: dispatch 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: audit 2024-09-13T22:34:49.794162+0000 mon.a (mon.0) 543 : audit [INF] from='client.24277 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]': finished 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: cluster 2024-09-13T22:34:49.799816+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[22840]: audit 2024-09-13T22:34:49.800394+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:34:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: cluster 2024-09-13T22:34:49.475563+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:34:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: cluster 2024-09-13T22:34:49.475650+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:34:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: audit 2024-09-13T22:34:49.790467+0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.192:0/4145965181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]: dispatch 2024-09-13T22:34:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: audit 2024-09-13T22:34:49.790783+0000 mon.a (mon.0) 542 : audit [INF] from='client.24277 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]: dispatch 2024-09-13T22:34:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: audit 2024-09-13T22:34:49.794162+0000 mon.a (mon.0) 543 : audit [INF] from='client.24277 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8"}]': finished 2024-09-13T22:34:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: cluster 2024-09-13T22:34:49.799816+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-13T22:34:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:50 smithi178 bash[29889]: audit 2024-09-13T22:34:49.800394+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:34:51.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:51 smithi192 bash[24608]: cluster 2024-09-13T22:34:50.262821+0000 mgr.y (mgr.14150) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:51.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:51 smithi192 bash[24608]: audit 2024-09-13T22:34:50.422820+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.192:0/4189879415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:34:51.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:51 smithi178 bash[22840]: cluster 2024-09-13T22:34:50.262821+0000 mgr.y (mgr.14150) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:51 smithi178 bash[22840]: audit 2024-09-13T22:34:50.422820+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.192:0/4189879415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:34:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:51 smithi178 bash[29889]: cluster 2024-09-13T22:34:50.262821+0000 mgr.y (mgr.14150) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:51 smithi178 bash[29889]: audit 2024-09-13T22:34:50.422820+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.192:0/4189879415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:34:53.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:53 smithi192 bash[24608]: cluster 2024-09-13T22:34:52.263497+0000 mgr.y (mgr.14150) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:53.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:53 smithi178 bash[22840]: cluster 2024-09-13T22:34:52.263497+0000 mgr.y (mgr.14150) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:53.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:53 smithi178 bash[29889]: cluster 2024-09-13T22:34:52.263497+0000 mgr.y (mgr.14150) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:55.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:55 smithi192 bash[24608]: cluster 2024-09-13T22:34:54.264181+0000 mgr.y (mgr.14150) 245 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:55.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:55 smithi178 bash[22840]: cluster 2024-09-13T22:34:54.264181+0000 mgr.y (mgr.14150) 245 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:55.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:55 smithi178 bash[29889]: cluster 2024-09-13T22:34:54.264181+0000 mgr.y (mgr.14150) 245 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:57.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:57 smithi192 bash[24608]: cluster 2024-09-13T22:34:56.264769+0000 mgr.y (mgr.14150) 246 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:57.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:57 smithi178 bash[29889]: cluster 2024-09-13T22:34:56.264769+0000 mgr.y (mgr.14150) 246 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:57.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:57 smithi178 bash[22840]: cluster 2024-09-13T22:34:56.264769+0000 mgr.y (mgr.14150) 246 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:34:58 smithi178 bash[22840]: cluster 2024-09-13T22:34:58.265300+0000 mgr.y (mgr.14150) 247 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:34:58 smithi178 bash[29889]: cluster 2024-09-13T22:34:58.265300+0000 mgr.y (mgr.14150) 247 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:34:59.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:34:58 smithi192 bash[24608]: cluster 2024-09-13T22:34:58.265300+0000 mgr.y (mgr.14150) 247 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:01 smithi192 bash[24608]: cluster 2024-09-13T22:35:00.265843+0000 mgr.y (mgr.14150) 248 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:01 smithi178 bash[22840]: cluster 2024-09-13T22:35:00.265843+0000 mgr.y (mgr.14150) 248 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:01 smithi178 bash[29889]: cluster 2024-09-13T22:35:00.265843+0000 mgr.y (mgr.14150) 248 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:03.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:03 smithi192 bash[24608]: cluster 2024-09-13T22:35:02.266377+0000 mgr.y (mgr.14150) 249 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:03.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:03 smithi192 bash[24608]: audit 2024-09-13T22:35:03.034406+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T22:35:03.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:03 smithi192 bash[24608]: audit 2024-09-13T22:35:03.036285+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:03 smithi178 bash[22840]: cluster 2024-09-13T22:35:02.266377+0000 mgr.y (mgr.14150) 249 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:03 smithi178 bash[22840]: audit 2024-09-13T22:35:03.034406+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T22:35:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:03 smithi178 bash[22840]: audit 2024-09-13T22:35:03.036285+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:03 smithi178 bash[29889]: cluster 2024-09-13T22:35:02.266377+0000 mgr.y (mgr.14150) 249 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:03 smithi178 bash[29889]: audit 2024-09-13T22:35:03.034406+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T22:35:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:03 smithi178 bash[29889]: audit 2024-09-13T22:35:03.036285+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:04.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:04 smithi192 bash[24608]: cephadm 2024-09-13T22:35:03.037904+0000 mgr.y (mgr.14150) 250 : cephadm [INF] Deploying daemon osd.5 on smithi192 2024-09-13T22:35:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:04 smithi178 bash[22840]: cephadm 2024-09-13T22:35:03.037904+0000 mgr.y (mgr.14150) 250 : cephadm [INF] Deploying daemon osd.5 on smithi192 2024-09-13T22:35:04.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:04 smithi178 bash[29889]: cephadm 2024-09-13T22:35:03.037904+0000 mgr.y (mgr.14150) 250 : cephadm [INF] Deploying daemon osd.5 on smithi192 2024-09-13T22:35:05.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:05 smithi192 bash[24608]: cluster 2024-09-13T22:35:04.266945+0000 mgr.y (mgr.14150) 251 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:05.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:05 smithi178 bash[22840]: cluster 2024-09-13T22:35:04.266945+0000 mgr.y (mgr.14150) 251 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:05.692 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:05 smithi178 bash[29889]: cluster 2024-09-13T22:35:04.266945+0000 mgr.y (mgr.14150) 251 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:07.268 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:07 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:35:07.268 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:35:07 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:35:07.269 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:35:07 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:35:07.626 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:35:07 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:35:07.626 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:07 smithi192 bash[24608]: cluster 2024-09-13T22:35:06.267460+0000 mgr.y (mgr.14150) 252 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:07.626 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:07 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:35:07.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:35:07 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:35:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:07 smithi178 bash[29889]: cluster 2024-09-13T22:35:06.267460+0000 mgr.y (mgr.14150) 252 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:07 smithi178 bash[22840]: cluster 2024-09-13T22:35:06.267460+0000 mgr.y (mgr.14150) 252 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:08 smithi192 bash[24608]: audit 2024-09-13T22:35:07.558840+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:35:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:08 smithi192 bash[24608]: audit 2024-09-13T22:35:07.566022+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:08 smithi192 bash[24608]: audit 2024-09-13T22:35:07.574175+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:08 smithi178 bash[29889]: audit 2024-09-13T22:35:07.558840+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:35:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:08 smithi178 bash[29889]: audit 2024-09-13T22:35:07.566022+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:08 smithi178 bash[29889]: audit 2024-09-13T22:35:07.574175+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:08 smithi178 bash[22840]: audit 2024-09-13T22:35:07.558840+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:35:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:08 smithi178 bash[22840]: audit 2024-09-13T22:35:07.566022+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:08 smithi178 bash[22840]: audit 2024-09-13T22:35:07.574175+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:09.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:09 smithi192 bash[24608]: cluster 2024-09-13T22:35:08.267968+0000 mgr.y (mgr.14150) 253 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:09 smithi178 bash[22840]: cluster 2024-09-13T22:35:08.267968+0000 mgr.y (mgr.14150) 253 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:09.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:09 smithi178 bash[29889]: cluster 2024-09-13T22:35:08.267968+0000 mgr.y (mgr.14150) 253 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:11.607 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:11 smithi192 bash[24608]: cluster 2024-09-13T22:35:10.268533+0000 mgr.y (mgr.14150) 254 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:11.607 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:11 smithi192 bash[24608]: cluster 2024-09-13T22:35:10.819800+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:11.607 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:11 smithi192 bash[24608]: cluster 2024-09-13T22:35:10.819890+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:11.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:11 smithi178 bash[22840]: cluster 2024-09-13T22:35:10.268533+0000 mgr.y (mgr.14150) 254 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:11.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:11 smithi178 bash[22840]: cluster 2024-09-13T22:35:10.819800+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:11.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:11 smithi178 bash[22840]: cluster 2024-09-13T22:35:10.819890+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:11 smithi178 bash[29889]: cluster 2024-09-13T22:35:10.268533+0000 mgr.y (mgr.14150) 254 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:11 smithi178 bash[29889]: cluster 2024-09-13T22:35:10.819800+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:11 smithi178 bash[29889]: cluster 2024-09-13T22:35:10.819890+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:12.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:12 smithi192 bash[24608]: cluster 2024-09-13T22:35:11.349437+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-13T22:35:12.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:12 smithi192 bash[24608]: audit 2024-09-13T22:35:11.350029+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:12.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:12 smithi178 bash[29889]: cluster 2024-09-13T22:35:11.349437+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-13T22:35:12.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:12 smithi178 bash[29889]: audit 2024-09-13T22:35:11.350029+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:12 smithi178 bash[22840]: cluster 2024-09-13T22:35:11.349437+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-13T22:35:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:12 smithi178 bash[22840]: audit 2024-09-13T22:35:11.350029+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:13.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:13 smithi192 bash[24608]: cluster 2024-09-13T22:35:12.269175+0000 mgr.y (mgr.14150) 255 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:13.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:13 smithi178 bash[22840]: cluster 2024-09-13T22:35:12.269175+0000 mgr.y (mgr.14150) 255 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:13 smithi178 bash[29889]: cluster 2024-09-13T22:35:12.269175+0000 mgr.y (mgr.14150) 255 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:15.008 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:14 smithi192 bash[24608]: audit 2024-09-13T22:35:13.908480+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.008 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:14 smithi192 bash[24608]: audit 2024-09-13T22:35:13.916858+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.008 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:14 smithi192 bash[24608]: audit 2024-09-13T22:35:13.918585+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:15.008 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:14 smithi192 bash[24608]: audit 2024-09-13T22:35:13.920226+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:35:15.008 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:14 smithi192 bash[24608]: audit 2024-09-13T22:35:13.929862+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.009 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:14 smithi192 bash[24608]: cluster 2024-09-13T22:35:14.269774+0000 mgr.y (mgr.14150) 256 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:15.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[29889]: audit 2024-09-13T22:35:13.908480+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[29889]: audit 2024-09-13T22:35:13.916858+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[29889]: audit 2024-09-13T22:35:13.918585+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[29889]: audit 2024-09-13T22:35:13.920226+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[29889]: audit 2024-09-13T22:35:13.929862+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[29889]: cluster 2024-09-13T22:35:14.269774+0000 mgr.y (mgr.14150) 256 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[22840]: audit 2024-09-13T22:35:13.908480+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[22840]: audit 2024-09-13T22:35:13.916858+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[22840]: audit 2024-09-13T22:35:13.918585+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[22840]: audit 2024-09-13T22:35:13.920226+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[22840]: audit 2024-09-13T22:35:13.929862+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:14 smithi178 bash[22840]: cluster 2024-09-13T22:35:14.269774+0000 mgr.y (mgr.14150) 256 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:17 smithi192 bash[24608]: cluster 2024-09-13T22:35:16.270309+0000 mgr.y (mgr.14150) 257 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:17 smithi192 bash[24608]: audit 2024-09-13T22:35:16.661108+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.192:6804/3271425338' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T22:35:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:17 smithi192 bash[24608]: audit 2024-09-13T22:35:16.661154+0000 mon.a (mon.0) 558 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T22:35:17.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:17 smithi178 bash[29889]: cluster 2024-09-13T22:35:16.270309+0000 mgr.y (mgr.14150) 257 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:17.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:17 smithi178 bash[29889]: audit 2024-09-13T22:35:16.661108+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.192:6804/3271425338' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T22:35:17.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:17 smithi178 bash[29889]: audit 2024-09-13T22:35:16.661154+0000 mon.a (mon.0) 558 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T22:35:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:17 smithi178 bash[22840]: cluster 2024-09-13T22:35:16.270309+0000 mgr.y (mgr.14150) 257 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:17 smithi178 bash[22840]: audit 2024-09-13T22:35:16.661108+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 v1:172.21.15.192:6804/3271425338' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T22:35:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:17 smithi178 bash[22840]: audit 2024-09-13T22:35:16.661154+0000 mon.a (mon.0) 558 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T22:35:18.025 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:17.325522+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: cluster 2024-09-13T22:35:17.331375+0000 mon.a (mon.0) 560 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:17.331974+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.192:6804/3271425338' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:17.332423+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:17.333048+0000 mon.a (mon.0) 562 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:18.003025+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:18.011375+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:18.019563+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:18.592 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[24608]: audit 2024-09-13T22:35:18.327450+0000 mon.a (mon.0) 566 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:35:18.594 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T22:35:18.596 INFO:tasks.cephadm:Deploying osd.6 on smithi192 with /dev/vg_nvme/lv_2... 2024-09-13T22:35:18.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T22:35:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:17.325522+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T22:35:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: cluster 2024-09-13T22:35:17.331375+0000 mon.a (mon.0) 560 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-13T22:35:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:17.331974+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.192:6804/3271425338' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:35:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:17.332423+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:17.333048+0000 mon.a (mon.0) 562 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:18.003025+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:18.011375+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:18.019563+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[22840]: audit 2024-09-13T22:35:18.327450+0000 mon.a (mon.0) 566 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:17.325522+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: cluster 2024-09-13T22:35:17.331375+0000 mon.a (mon.0) 560 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:17.331974+0000 mon.b (mon.1) 18 : audit [INF] from='osd.5 v1:172.21.15.192:6804/3271425338' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:17.332423+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:17.333048+0000 mon.a (mon.0) 562 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:35:18.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:18.003025+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:35:18.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:18.011375+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:18.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:18.019563+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:18.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:18 smithi178 bash[29889]: audit 2024-09-13T22:35:18.327450+0000 mon.a (mon.0) 566 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:35:18.907 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[33643]: debug 2024-09-13T22:35:18.785+0000 7fdff9b6d640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T22:35:18.907 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:35:18 smithi192 bash[33643]: debug 2024-09-13T22:35:18.797+0000 7fdff1179640 -1 osd.5 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T22:35:19.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:19 smithi192 bash[24608]: cluster 2024-09-13T22:35:18.270834+0000 mgr.y (mgr.14150) 258 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:19.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:19 smithi192 bash[24608]: cluster 2024-09-13T22:35:18.333475+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-09-13T22:35:19.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:19 smithi192 bash[24608]: audit 2024-09-13T22:35:18.334492+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:19.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:19 smithi192 bash[24608]: audit 2024-09-13T22:35:18.343361+0000 mon.a (mon.0) 569 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:19.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:19 smithi192 bash[24608]: audit 2024-09-13T22:35:18.793238+0000 mon.a (mon.0) 570 : audit [INF] from='osd.5 ' entity='osd.5' 2024-09-13T22:35:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[22840]: cluster 2024-09-13T22:35:18.270834+0000 mgr.y (mgr.14150) 258 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[22840]: cluster 2024-09-13T22:35:18.333475+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[22840]: audit 2024-09-13T22:35:18.334492+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[22840]: audit 2024-09-13T22:35:18.343361+0000 mon.a (mon.0) 569 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[22840]: audit 2024-09-13T22:35:18.793238+0000 mon.a (mon.0) 570 : audit [INF] from='osd.5 ' entity='osd.5' 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[29889]: cluster 2024-09-13T22:35:18.270834+0000 mgr.y (mgr.14150) 258 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[29889]: cluster 2024-09-13T22:35:18.333475+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[29889]: audit 2024-09-13T22:35:18.334492+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[29889]: audit 2024-09-13T22:35:18.343361+0000 mon.a (mon.0) 569 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:19 smithi178 bash[29889]: audit 2024-09-13T22:35:18.793238+0000 mon.a (mon.0) 570 : audit [INF] from='osd.5 ' entity='osd.5' 2024-09-13T22:35:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:20 smithi192 bash[24608]: cluster 2024-09-13T22:35:17.695214+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:20 smithi192 bash[24608]: cluster 2024-09-13T22:35:17.695366+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:20 smithi192 bash[24608]: cluster 2024-09-13T22:35:19.335632+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 v1:172.21.15.192:6804/3271425338 boot 2024-09-13T22:35:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:20 smithi192 bash[24608]: cluster 2024-09-13T22:35:19.335708+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-13T22:35:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:20 smithi192 bash[24608]: audit 2024-09-13T22:35:19.336220+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:20 smithi192 bash[24608]: cluster 2024-09-13T22:35:20.337249+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-13T22:35:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[22840]: cluster 2024-09-13T22:35:17.695214+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[22840]: cluster 2024-09-13T22:35:17.695366+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[22840]: cluster 2024-09-13T22:35:19.335632+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 v1:172.21.15.192:6804/3271425338 boot 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[22840]: cluster 2024-09-13T22:35:19.335708+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[22840]: audit 2024-09-13T22:35:19.336220+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[22840]: cluster 2024-09-13T22:35:20.337249+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[29889]: cluster 2024-09-13T22:35:17.695214+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[29889]: cluster 2024-09-13T22:35:17.695366+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[29889]: cluster 2024-09-13T22:35:19.335632+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 v1:172.21.15.192:6804/3271425338 boot 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[29889]: cluster 2024-09-13T22:35:19.335708+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-13T22:35:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[29889]: audit 2024-09-13T22:35:19.336220+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:35:20.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:20 smithi178 bash[29889]: cluster 2024-09-13T22:35:20.337249+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-13T22:35:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:21 smithi192 bash[24608]: cluster 2024-09-13T22:35:20.271536+0000 mgr.y (mgr.14150) 259 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:21 smithi178 bash[22840]: cluster 2024-09-13T22:35:20.271536+0000 mgr.y (mgr.14150) 259 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:21 smithi178 bash[29889]: cluster 2024-09-13T22:35:20.271536+0000 mgr.y (mgr.14150) 259 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:22.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:22 smithi192 bash[24608]: cluster 2024-09-13T22:35:21.348041+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-13T22:35:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:22 smithi178 bash[22840]: cluster 2024-09-13T22:35:21.348041+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-13T22:35:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:22 smithi178 bash[29889]: cluster 2024-09-13T22:35:21.348041+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-09-13T22:35:23.109 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:35:23.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:23 smithi192 bash[24608]: cluster 2024-09-13T22:35:22.272102+0000 mgr.y (mgr.14150) 260 : cluster [DBG] pgmap v243: 1 pgs: 1 remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:23.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:23 smithi192 bash[24608]: cluster 2024-09-13T22:35:22.365538+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-13T22:35:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:23 smithi178 bash[22840]: cluster 2024-09-13T22:35:22.272102+0000 mgr.y (mgr.14150) 260 : cluster [DBG] pgmap v243: 1 pgs: 1 remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:23 smithi178 bash[22840]: cluster 2024-09-13T22:35:22.365538+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-13T22:35:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:23 smithi178 bash[29889]: cluster 2024-09-13T22:35:22.272102+0000 mgr.y (mgr.14150) 260 : cluster [DBG] pgmap v243: 1 pgs: 1 remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:23 smithi178 bash[29889]: cluster 2024-09-13T22:35:22.365538+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-09-13T22:35:24.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:24 smithi192 bash[24608]: cluster 2024-09-13T22:35:23.367395+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-09-13T22:35:24.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:24 smithi178 bash[22840]: cluster 2024-09-13T22:35:23.367395+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-09-13T22:35:24.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:24 smithi178 bash[29889]: cluster 2024-09-13T22:35:23.367395+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-09-13T22:35:25.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:25 smithi192 bash[24608]: cluster 2024-09-13T22:35:24.272647+0000 mgr.y (mgr.14150) 261 : cluster [DBG] pgmap v246: 1 pgs: 1 remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:25.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:25 smithi178 bash[22840]: cluster 2024-09-13T22:35:24.272647+0000 mgr.y (mgr.14150) 261 : cluster [DBG] pgmap v246: 1 pgs: 1 remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:25.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:25 smithi178 bash[29889]: cluster 2024-09-13T22:35:24.272647+0000 mgr.y (mgr.14150) 261 : cluster [DBG] pgmap v246: 1 pgs: 1 remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:27.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:27 smithi192 bash[24608]: cluster 2024-09-13T22:35:26.273308+0000 mgr.y (mgr.14150) 262 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T22:35:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:27 smithi178 bash[22840]: cluster 2024-09-13T22:35:26.273308+0000 mgr.y (mgr.14150) 262 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T22:35:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:27 smithi178 bash[29889]: cluster 2024-09-13T22:35:26.273308+0000 mgr.y (mgr.14150) 262 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: cephadm 2024-09-13T22:35:28.218919+0000 mgr.y (mgr.14150) 263 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.227356+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.235753+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.238394+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.239990+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: cephadm 2024-09-13T22:35:28.241082+0000 mgr.y (mgr.14150) 264 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 8632M 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.247369+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.252155+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.253717+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:35:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: audit 2024-09-13T22:35:28.262988+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:29 smithi192 bash[24608]: cluster 2024-09-13T22:35:28.274015+0000 mgr.y (mgr.14150) 265 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: cephadm 2024-09-13T22:35:28.218919+0000 mgr.y (mgr.14150) 263 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.227356+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.235753+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.238394+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.239990+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: cephadm 2024-09-13T22:35:28.241082+0000 mgr.y (mgr.14150) 264 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 8632M 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.247369+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.252155+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.253717+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: audit 2024-09-13T22:35:28.262988+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[22840]: cluster 2024-09-13T22:35:28.274015+0000 mgr.y (mgr.14150) 265 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: cephadm 2024-09-13T22:35:28.218919+0000 mgr.y (mgr.14150) 263 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.227356+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.235753+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.238394+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.239990+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: cephadm 2024-09-13T22:35:28.241082+0000 mgr.y (mgr.14150) 264 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 8632M 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.247369+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.252155+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.253717+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:35:29.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: audit 2024-09-13T22:35:28.262988+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:35:29.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:29 smithi178 bash[29889]: cluster 2024-09-13T22:35:28.274015+0000 mgr.y (mgr.14150) 265 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T22:35:30.315 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:35:30.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-09-13T22:35:31.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:31 smithi192 bash[24608]: cluster 2024-09-13T22:35:30.274600+0000 mgr.y (mgr.14150) 266 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:35:31.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:31 smithi178 bash[22840]: cluster 2024-09-13T22:35:30.274600+0000 mgr.y (mgr.14150) 266 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:35:31.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:31 smithi178 bash[29889]: cluster 2024-09-13T22:35:30.274600+0000 mgr.y (mgr.14150) 266 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:35:33.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:33 smithi192 bash[24608]: cluster 2024-09-13T22:35:32.275157+0000 mgr.y (mgr.14150) 267 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:35:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:33 smithi178 bash[22840]: cluster 2024-09-13T22:35:32.275157+0000 mgr.y (mgr.14150) 267 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:35:33.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:33 smithi178 bash[29889]: cluster 2024-09-13T22:35:32.275157+0000 mgr.y (mgr.14150) 267 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:35:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:34 smithi192 bash[24608]: cluster 2024-09-13T22:35:32.542545+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:34 smithi192 bash[24608]: cluster 2024-09-13T22:35:32.542634+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:34 smithi192 bash[24608]: cluster 2024-09-13T22:35:33.344487+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-09-13T22:35:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:34 smithi178 bash[22840]: cluster 2024-09-13T22:35:32.542545+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:34 smithi178 bash[22840]: cluster 2024-09-13T22:35:32.542634+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:34 smithi178 bash[22840]: cluster 2024-09-13T22:35:33.344487+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-09-13T22:35:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:34 smithi178 bash[29889]: cluster 2024-09-13T22:35:32.542545+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:34 smithi178 bash[29889]: cluster 2024-09-13T22:35:32.542634+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:34 smithi178 bash[29889]: cluster 2024-09-13T22:35:33.344487+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-09-13T22:35:35.114 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:35:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:35 smithi192 bash[24608]: cluster 2024-09-13T22:35:34.275727+0000 mgr.y (mgr.14150) 268 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:35:35.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:35 smithi178 bash[22840]: cluster 2024-09-13T22:35:34.275727+0000 mgr.y (mgr.14150) 268 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:35:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:35 smithi178 bash[29889]: cluster 2024-09-13T22:35:34.275727+0000 mgr.y (mgr.14150) 268 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:35:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:37 smithi192 bash[24608]: cluster 2024-09-13T22:35:36.276279+0000 mgr.y (mgr.14150) 269 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:37 smithi192 bash[24608]: audit 2024-09-13T22:35:36.488850+0000 mon.a (mon.0) 587 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:35:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:37 smithi192 bash[24608]: audit 2024-09-13T22:35:36.494786+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:35:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:37 smithi192 bash[24608]: audit 2024-09-13T22:35:36.496528+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:37.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[22840]: cluster 2024-09-13T22:35:36.276279+0000 mgr.y (mgr.14150) 269 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[22840]: audit 2024-09-13T22:35:36.488850+0000 mon.a (mon.0) 587 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[22840]: audit 2024-09-13T22:35:36.494786+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[22840]: audit 2024-09-13T22:35:36.496528+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[29889]: cluster 2024-09-13T22:35:36.276279+0000 mgr.y (mgr.14150) 269 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[29889]: audit 2024-09-13T22:35:36.488850+0000 mon.a (mon.0) 587 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[29889]: audit 2024-09-13T22:35:36.494786+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:35:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:37 smithi178 bash[29889]: audit 2024-09-13T22:35:36.496528+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:38 smithi192 bash[24608]: audit 2024-09-13T22:35:36.482592+0000 mgr.y (mgr.14150) 270 : audit [DBG] from='client.24298 v1:172.21.15.192:0/1120921235' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:35:38.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:38 smithi178 bash[22840]: audit 2024-09-13T22:35:36.482592+0000 mgr.y (mgr.14150) 270 : audit [DBG] from='client.24298 v1:172.21.15.192:0/1120921235' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:35:38.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:38 smithi178 bash[29889]: audit 2024-09-13T22:35:36.482592+0000 mgr.y (mgr.14150) 270 : audit [DBG] from='client.24298 v1:172.21.15.192:0/1120921235' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:35:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:39 smithi192 bash[24608]: cluster 2024-09-13T22:35:38.276834+0000 mgr.y (mgr.14150) 271 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:39.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:39 smithi178 bash[22840]: cluster 2024-09-13T22:35:38.276834+0000 mgr.y (mgr.14150) 271 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:39.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:39 smithi178 bash[29889]: cluster 2024-09-13T22:35:38.276834+0000 mgr.y (mgr.14150) 271 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:40.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:40 smithi192 bash[24608]: cluster 2024-09-13T22:35:38.751541+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 deep-scrub starts 2024-09-13T22:35:40.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:40 smithi192 bash[24608]: cluster 2024-09-13T22:35:38.758943+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 deep-scrub ok 2024-09-13T22:35:40.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:40 smithi178 bash[22840]: cluster 2024-09-13T22:35:38.751541+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 deep-scrub starts 2024-09-13T22:35:40.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:40 smithi178 bash[22840]: cluster 2024-09-13T22:35:38.758943+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 deep-scrub ok 2024-09-13T22:35:40.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:40 smithi178 bash[29889]: cluster 2024-09-13T22:35:38.751541+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 deep-scrub starts 2024-09-13T22:35:40.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:40 smithi178 bash[29889]: cluster 2024-09-13T22:35:38.758943+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 deep-scrub ok 2024-09-13T22:35:41.645 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:41 smithi192 bash[24608]: cluster 2024-09-13T22:35:40.277419+0000 mgr.y (mgr.14150) 272 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:41.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:41 smithi178 bash[22840]: cluster 2024-09-13T22:35:40.277419+0000 mgr.y (mgr.14150) 272 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:41.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:41 smithi178 bash[29889]: cluster 2024-09-13T22:35:40.277419+0000 mgr.y (mgr.14150) 272 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:43.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:43 smithi192 bash[24608]: cluster 2024-09-13T22:35:42.277947+0000 mgr.y (mgr.14150) 273 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:43.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:43 smithi178 bash[22840]: cluster 2024-09-13T22:35:42.277947+0000 mgr.y (mgr.14150) 273 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:43.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:43 smithi178 bash[29889]: cluster 2024-09-13T22:35:42.277947+0000 mgr.y (mgr.14150) 273 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:44.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:44 smithi192 bash[24608]: cluster 2024-09-13T22:35:43.881435+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:44.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:44 smithi192 bash[24608]: cluster 2024-09-13T22:35:43.881553+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:44.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:44 smithi178 bash[22840]: cluster 2024-09-13T22:35:43.881435+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:44.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:44 smithi178 bash[22840]: cluster 2024-09-13T22:35:43.881553+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:44 smithi178 bash[29889]: cluster 2024-09-13T22:35:43.881435+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:44 smithi178 bash[29889]: cluster 2024-09-13T22:35:43.881553+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: cluster 2024-09-13T22:35:44.278480+0000 mgr.y (mgr.14150) 274 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: cluster 2024-09-13T22:35:44.394100+0000 mon.a (mon.0) 590 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: audit 2024-09-13T22:35:44.924485+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.192:0/3403206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]: dispatch 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: audit 2024-09-13T22:35:44.924827+0000 mon.a (mon.0) 591 : audit [INF] from='client.24304 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]: dispatch 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: audit 2024-09-13T22:35:44.928299+0000 mon.a (mon.0) 592 : audit [INF] from='client.24304 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]': finished 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: cluster 2024-09-13T22:35:44.933743+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-13T22:35:45.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:45 smithi192 bash[24608]: audit 2024-09-13T22:35:44.934276+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:35:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: cluster 2024-09-13T22:35:44.278480+0000 mgr.y (mgr.14150) 274 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: cluster 2024-09-13T22:35:44.394100+0000 mon.a (mon.0) 590 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-09-13T22:35:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: audit 2024-09-13T22:35:44.924485+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.192:0/3403206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]: dispatch 2024-09-13T22:35:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: audit 2024-09-13T22:35:44.924827+0000 mon.a (mon.0) 591 : audit [INF] from='client.24304 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]: dispatch 2024-09-13T22:35:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: audit 2024-09-13T22:35:44.928299+0000 mon.a (mon.0) 592 : audit [INF] from='client.24304 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]': finished 2024-09-13T22:35:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: cluster 2024-09-13T22:35:44.933743+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-13T22:35:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[22840]: audit 2024-09-13T22:35:44.934276+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:35:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: cluster 2024-09-13T22:35:44.278480+0000 mgr.y (mgr.14150) 274 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: cluster 2024-09-13T22:35:44.394100+0000 mon.a (mon.0) 590 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-09-13T22:35:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: audit 2024-09-13T22:35:44.924485+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.192:0/3403206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]: dispatch 2024-09-13T22:35:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: audit 2024-09-13T22:35:44.924827+0000 mon.a (mon.0) 591 : audit [INF] from='client.24304 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]: dispatch 2024-09-13T22:35:45.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: audit 2024-09-13T22:35:44.928299+0000 mon.a (mon.0) 592 : audit [INF] from='client.24304 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98"}]': finished 2024-09-13T22:35:45.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: cluster 2024-09-13T22:35:44.933743+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-09-13T22:35:45.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:45 smithi178 bash[29889]: audit 2024-09-13T22:35:44.934276+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:35:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:46 smithi192 bash[24608]: audit 2024-09-13T22:35:45.587851+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.192:0/965237879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:35:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:46 smithi178 bash[22840]: audit 2024-09-13T22:35:45.587851+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.192:0/965237879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:35:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:46 smithi178 bash[29889]: audit 2024-09-13T22:35:45.587851+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.192:0/965237879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:35:47.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:47 smithi192 bash[24608]: cluster 2024-09-13T22:35:46.279120+0000 mgr.y (mgr.14150) 275 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:47.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:47 smithi178 bash[22840]: cluster 2024-09-13T22:35:46.279120+0000 mgr.y (mgr.14150) 275 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:47.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:47 smithi178 bash[29889]: cluster 2024-09-13T22:35:46.279120+0000 mgr.y (mgr.14150) 275 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:49 smithi192 bash[24608]: cluster 2024-09-13T22:35:48.279761+0000 mgr.y (mgr.14150) 276 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:49.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:49 smithi178 bash[22840]: cluster 2024-09-13T22:35:48.279761+0000 mgr.y (mgr.14150) 276 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:49 smithi178 bash[29889]: cluster 2024-09-13T22:35:48.279761+0000 mgr.y (mgr.14150) 276 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:51.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[22840]: cluster 2024-09-13T22:35:50.280443+0000 mgr.y (mgr.14150) 277 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:51.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[22840]: cluster 2024-09-13T22:35:50.375610+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[22840]: cluster 2024-09-13T22:35:50.375700+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[22840]: cluster 2024-09-13T22:35:50.415976+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[22840]: audit 2024-09-13T22:35:50.416226+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[29889]: cluster 2024-09-13T22:35:50.280443+0000 mgr.y (mgr.14150) 277 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[29889]: cluster 2024-09-13T22:35:50.375610+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[29889]: cluster 2024-09-13T22:35:50.375700+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[29889]: cluster 2024-09-13T22:35:50.415976+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-13T22:35:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:51 smithi178 bash[29889]: audit 2024-09-13T22:35:50.416226+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:35:51.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:51 smithi192 bash[24608]: cluster 2024-09-13T22:35:50.280443+0000 mgr.y (mgr.14150) 277 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:51.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:51 smithi192 bash[24608]: cluster 2024-09-13T22:35:50.375610+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:35:51.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:51 smithi192 bash[24608]: cluster 2024-09-13T22:35:50.375700+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:35:51.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:51 smithi192 bash[24608]: cluster 2024-09-13T22:35:50.415976+0000 mon.a (mon.0) 595 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-09-13T22:35:51.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:51 smithi192 bash[24608]: audit 2024-09-13T22:35:50.416226+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:35:53.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:53 smithi178 bash[22840]: cluster 2024-09-13T22:35:52.281094+0000 mgr.y (mgr.14150) 278 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:53.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:53 smithi178 bash[29889]: cluster 2024-09-13T22:35:52.281094+0000 mgr.y (mgr.14150) 278 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:53.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:53 smithi192 bash[24608]: cluster 2024-09-13T22:35:52.281094+0000 mgr.y (mgr.14150) 278 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:55.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:55 smithi192 bash[24608]: cluster 2024-09-13T22:35:54.281748+0000 mgr.y (mgr.14150) 279 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:55.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:55 smithi178 bash[22840]: cluster 2024-09-13T22:35:54.281748+0000 mgr.y (mgr.14150) 279 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:55.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:55 smithi178 bash[29889]: cluster 2024-09-13T22:35:54.281748+0000 mgr.y (mgr.14150) 279 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:57.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:57 smithi192 bash[24608]: cluster 2024-09-13T22:35:56.282420+0000 mgr.y (mgr.14150) 280 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:57.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:57 smithi178 bash[22840]: cluster 2024-09-13T22:35:56.282420+0000 mgr.y (mgr.14150) 280 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:57.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:57 smithi178 bash[29889]: cluster 2024-09-13T22:35:56.282420+0000 mgr.y (mgr.14150) 280 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:59.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:58 smithi192 bash[24608]: audit 2024-09-13T22:35:58.238458+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T22:35:59.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:58 smithi192 bash[24608]: audit 2024-09-13T22:35:58.239990+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:59.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:58 smithi192 bash[24608]: cephadm 2024-09-13T22:35:58.241349+0000 mgr.y (mgr.14150) 281 : cephadm [INF] Deploying daemon osd.6 on smithi192 2024-09-13T22:35:59.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:35:58 smithi192 bash[24608]: cluster 2024-09-13T22:35:58.283122+0000 mgr.y (mgr.14150) 282 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[22840]: audit 2024-09-13T22:35:58.238458+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T22:35:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[22840]: audit 2024-09-13T22:35:58.239990+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[22840]: cephadm 2024-09-13T22:35:58.241349+0000 mgr.y (mgr.14150) 281 : cephadm [INF] Deploying daemon osd.6 on smithi192 2024-09-13T22:35:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[22840]: cluster 2024-09-13T22:35:58.283122+0000 mgr.y (mgr.14150) 282 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:35:59.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[29889]: audit 2024-09-13T22:35:58.238458+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T22:35:59.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[29889]: audit 2024-09-13T22:35:58.239990+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:35:59.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[29889]: cephadm 2024-09-13T22:35:58.241349+0000 mgr.y (mgr.14150) 281 : cephadm [INF] Deploying daemon osd.6 on smithi192 2024-09-13T22:35:59.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:35:58 smithi178 bash[29889]: cluster 2024-09-13T22:35:58.283122+0000 mgr.y (mgr.14150) 282 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:01 smithi192 bash[24608]: cluster 2024-09-13T22:36:00.283717+0000 mgr.y (mgr.14150) 283 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:01 smithi178 bash[22840]: cluster 2024-09-13T22:36:00.283717+0000 mgr.y (mgr.14150) 283 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:01 smithi178 bash[29889]: cluster 2024-09-13T22:36:00.283717+0000 mgr.y (mgr.14150) 283 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.573 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.573 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.907 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.907 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:02.907 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:36:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:03 smithi192 bash[24608]: cluster 2024-09-13T22:36:02.284240+0000 mgr.y (mgr.14150) 284 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:03 smithi192 bash[24608]: audit 2024-09-13T22:36:02.846974+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:03 smithi192 bash[24608]: audit 2024-09-13T22:36:02.854879+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:03 smithi192 bash[24608]: audit 2024-09-13T22:36:02.862973+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[29889]: cluster 2024-09-13T22:36:02.284240+0000 mgr.y (mgr.14150) 284 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[29889]: audit 2024-09-13T22:36:02.846974+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[29889]: audit 2024-09-13T22:36:02.854879+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[29889]: audit 2024-09-13T22:36:02.862973+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[22840]: cluster 2024-09-13T22:36:02.284240+0000 mgr.y (mgr.14150) 284 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[22840]: audit 2024-09-13T22:36:02.846974+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[22840]: audit 2024-09-13T22:36:02.854879+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:03 smithi178 bash[22840]: audit 2024-09-13T22:36:02.862973+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:05.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:05 smithi192 bash[24608]: cluster 2024-09-13T22:36:04.284795+0000 mgr.y (mgr.14150) 285 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:05.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:05 smithi192 bash[24608]: cluster 2024-09-13T22:36:04.723707+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:05.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:05 smithi192 bash[24608]: cluster 2024-09-13T22:36:04.723830+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:05.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:05 smithi178 bash[29889]: cluster 2024-09-13T22:36:04.284795+0000 mgr.y (mgr.14150) 285 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:05.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:05 smithi178 bash[29889]: cluster 2024-09-13T22:36:04.723707+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:05.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:05 smithi178 bash[29889]: cluster 2024-09-13T22:36:04.723830+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:05.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:05 smithi178 bash[22840]: cluster 2024-09-13T22:36:04.284795+0000 mgr.y (mgr.14150) 285 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:05 smithi178 bash[22840]: cluster 2024-09-13T22:36:04.723707+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:05 smithi178 bash[22840]: cluster 2024-09-13T22:36:04.723830+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:06.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:06 smithi192 bash[24608]: cluster 2024-09-13T22:36:05.351178+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-09-13T22:36:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:06 smithi192 bash[24608]: audit 2024-09-13T22:36:05.351761+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:06.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:06 smithi178 bash[22840]: cluster 2024-09-13T22:36:05.351178+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-09-13T22:36:06.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:06 smithi178 bash[22840]: audit 2024-09-13T22:36:05.351761+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:06 smithi178 bash[29889]: cluster 2024-09-13T22:36:05.351178+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-09-13T22:36:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:06 smithi178 bash[29889]: audit 2024-09-13T22:36:05.351761+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:07.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:07 smithi192 bash[24608]: cluster 2024-09-13T22:36:06.285312+0000 mgr.y (mgr.14150) 286 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:07.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:07 smithi178 bash[22840]: cluster 2024-09-13T22:36:06.285312+0000 mgr.y (mgr.14150) 286 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:07 smithi178 bash[29889]: cluster 2024-09-13T22:36:06.285312+0000 mgr.y (mgr.14150) 286 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:09.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:09 smithi192 bash[24608]: cluster 2024-09-13T22:36:08.285825+0000 mgr.y (mgr.14150) 287 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:09 smithi178 bash[22840]: cluster 2024-09-13T22:36:08.285825+0000 mgr.y (mgr.14150) 287 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:09.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:09 smithi178 bash[29889]: cluster 2024-09-13T22:36:08.285825+0000 mgr.y (mgr.14150) 287 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:10 smithi192 bash[24608]: audit 2024-09-13T22:36:09.697132+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:10 smithi192 bash[24608]: audit 2024-09-13T22:36:09.706061+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:10 smithi192 bash[24608]: cluster 2024-09-13T22:36:10.286406+0000 mgr.y (mgr.14150) 288 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:10 smithi192 bash[24608]: audit 2024-09-13T22:36:10.412060+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:10 smithi192 bash[24608]: audit 2024-09-13T22:36:10.413830+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:36:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:10 smithi192 bash[24608]: audit 2024-09-13T22:36:10.423325+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[22840]: audit 2024-09-13T22:36:09.697132+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[22840]: audit 2024-09-13T22:36:09.706061+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[22840]: cluster 2024-09-13T22:36:10.286406+0000 mgr.y (mgr.14150) 288 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[22840]: audit 2024-09-13T22:36:10.412060+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[22840]: audit 2024-09-13T22:36:10.413830+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[22840]: audit 2024-09-13T22:36:10.423325+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[29889]: audit 2024-09-13T22:36:09.697132+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[29889]: audit 2024-09-13T22:36:09.706061+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[29889]: cluster 2024-09-13T22:36:10.286406+0000 mgr.y (mgr.14150) 288 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[29889]: audit 2024-09-13T22:36:10.412060+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[29889]: audit 2024-09-13T22:36:10.413830+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:36:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:10 smithi178 bash[29889]: audit 2024-09-13T22:36:10.423325+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:11 smithi192 bash[24608]: audit 2024-09-13T22:36:11.853519+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.192:6808/684205162' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T22:36:12.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:11 smithi192 bash[24608]: audit 2024-09-13T22:36:11.853702+0000 mon.a (mon.0) 609 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T22:36:12.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:11 smithi178 bash[22840]: audit 2024-09-13T22:36:11.853519+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.192:6808/684205162' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T22:36:12.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:11 smithi178 bash[22840]: audit 2024-09-13T22:36:11.853702+0000 mon.a (mon.0) 609 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T22:36:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:11 smithi178 bash[29889]: audit 2024-09-13T22:36:11.853519+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 v1:172.21.15.192:6808/684205162' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T22:36:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:11 smithi178 bash[29889]: audit 2024-09-13T22:36:11.853702+0000 mon.a (mon.0) 609 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T22:36:13.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: audit 2024-09-13T22:36:11.906439+0000 mon.a (mon.0) 610 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T22:36:13.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: cluster 2024-09-13T22:36:11.912071+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-09-13T22:36:13.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: audit 2024-09-13T22:36:11.912501+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.192:6808/684205162' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:36:13.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: audit 2024-09-13T22:36:11.913058+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:13.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: audit 2024-09-13T22:36:11.913720+0000 mon.a (mon.0) 613 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:36:13.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: cluster 2024-09-13T22:36:12.286961+0000 mgr.y (mgr.14150) 289 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:13.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:12 smithi192 bash[24608]: audit 2024-09-13T22:36:12.909317+0000 mon.a (mon.0) 614 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:36:13.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: audit 2024-09-13T22:36:11.906439+0000 mon.a (mon.0) 610 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T22:36:13.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: cluster 2024-09-13T22:36:11.912071+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: audit 2024-09-13T22:36:11.912501+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.192:6808/684205162' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: audit 2024-09-13T22:36:11.913058+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: audit 2024-09-13T22:36:11.913720+0000 mon.a (mon.0) 613 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: cluster 2024-09-13T22:36:12.286961+0000 mgr.y (mgr.14150) 289 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[22840]: audit 2024-09-13T22:36:12.909317+0000 mon.a (mon.0) 614 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: audit 2024-09-13T22:36:11.906439+0000 mon.a (mon.0) 610 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: cluster 2024-09-13T22:36:11.912071+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: audit 2024-09-13T22:36:11.912501+0000 mon.b (mon.1) 22 : audit [INF] from='osd.6 v1:172.21.15.192:6808/684205162' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: audit 2024-09-13T22:36:11.913058+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:13.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: audit 2024-09-13T22:36:11.913720+0000 mon.a (mon.0) 613 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:36:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: cluster 2024-09-13T22:36:12.286961+0000 mgr.y (mgr.14150) 289 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:12 smithi178 bash[29889]: audit 2024-09-13T22:36:12.909317+0000 mon.a (mon.0) 614 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:36:13.458 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 6 on host 'smithi192' 2024-09-13T22:36:14.023 DEBUG:teuthology.orchestra.run.smithi192:osd.6> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T22:36:14.025 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_1... 2024-09-13T22:36:14.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T22:36:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: cluster 2024-09-13T22:36:12.915211+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-09-13T22:36:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: audit 2024-09-13T22:36:12.916603+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: audit 2024-09-13T22:36:12.925413+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: audit 2024-09-13T22:36:13.433608+0000 mon.a (mon.0) 618 : audit [INF] from='osd.6 ' entity='osd.6' 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: audit 2024-09-13T22:36:13.440183+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: audit 2024-09-13T22:36:13.447330+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[22840]: audit 2024-09-13T22:36:13.455821+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: cluster 2024-09-13T22:36:12.915211+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: audit 2024-09-13T22:36:12.916603+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: audit 2024-09-13T22:36:12.925413+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: audit 2024-09-13T22:36:13.433608+0000 mon.a (mon.0) 618 : audit [INF] from='osd.6 ' entity='osd.6' 2024-09-13T22:36:14.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: audit 2024-09-13T22:36:13.440183+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:14.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: audit 2024-09-13T22:36:13.447330+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:14.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:13 smithi178 bash[29889]: audit 2024-09-13T22:36:13.455821+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: cluster 2024-09-13T22:36:12.915211+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: audit 2024-09-13T22:36:12.916603+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: audit 2024-09-13T22:36:12.925413+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: audit 2024-09-13T22:36:13.433608+0000 mon.a (mon.0) 618 : audit [INF] from='osd.6 ' entity='osd.6' 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: audit 2024-09-13T22:36:13.440183+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: audit 2024-09-13T22:36:13.447330+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:13 smithi192 bash[24608]: audit 2024-09-13T22:36:13.455821+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: cluster 2024-09-13T22:36:12.883396+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: cluster 2024-09-13T22:36:12.883513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: cluster 2024-09-13T22:36:13.927544+0000 mon.a (mon.0) 622 : cluster [INF] osd.6 v1:172.21.15.192:6808/684205162 boot 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: cluster 2024-09-13T22:36:13.927643+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: audit 2024-09-13T22:36:13.928224+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: cluster 2024-09-13T22:36:14.287434+0000 mgr.y (mgr.14150) 290 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:14 smithi192 bash[24608]: cluster 2024-09-13T22:36:14.929818+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-13T22:36:15.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: cluster 2024-09-13T22:36:12.883396+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:15.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: cluster 2024-09-13T22:36:12.883513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:15.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: cluster 2024-09-13T22:36:13.927544+0000 mon.a (mon.0) 622 : cluster [INF] osd.6 v1:172.21.15.192:6808/684205162 boot 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: cluster 2024-09-13T22:36:13.927643+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: audit 2024-09-13T22:36:13.928224+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: cluster 2024-09-13T22:36:14.287434+0000 mgr.y (mgr.14150) 290 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[22840]: cluster 2024-09-13T22:36:14.929818+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: cluster 2024-09-13T22:36:12.883396+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: cluster 2024-09-13T22:36:12.883513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: cluster 2024-09-13T22:36:13.927544+0000 mon.a (mon.0) 622 : cluster [INF] osd.6 v1:172.21.15.192:6808/684205162 boot 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: cluster 2024-09-13T22:36:13.927643+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: audit 2024-09-13T22:36:13.928224+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: cluster 2024-09-13T22:36:14.287434+0000 mgr.y (mgr.14150) 290 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-13T22:36:15.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:14 smithi178 bash[29889]: cluster 2024-09-13T22:36:14.929818+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-09-13T22:36:17.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:16 smithi192 bash[24608]: cluster 2024-09-13T22:36:15.931170+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-09-13T22:36:17.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:16 smithi192 bash[24608]: cluster 2024-09-13T22:36:16.287925+0000 mgr.y (mgr.14150) 291 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:17.445 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:16 smithi178 bash[22840]: cluster 2024-09-13T22:36:15.931170+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-09-13T22:36:17.445 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:16 smithi178 bash[22840]: cluster 2024-09-13T22:36:16.287925+0000 mgr.y (mgr.14150) 291 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:17.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:16 smithi178 bash[29889]: cluster 2024-09-13T22:36:15.931170+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-09-13T22:36:17.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:16 smithi178 bash[29889]: cluster 2024-09-13T22:36:16.287925+0000 mgr.y (mgr.14150) 291 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:17 smithi192 bash[24608]: cluster 2024-09-13T22:36:16.944746+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-09-13T22:36:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:17 smithi178 bash[22840]: cluster 2024-09-13T22:36:16.944746+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-09-13T22:36:18.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:17 smithi178 bash[29889]: cluster 2024-09-13T22:36:16.944746+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-09-13T22:36:18.487 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:36:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:18 smithi192 bash[24608]: cluster 2024-09-13T22:36:17.944412+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-09-13T22:36:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:18 smithi192 bash[24608]: cluster 2024-09-13T22:36:18.288470+0000 mgr.y (mgr.14150) 292 : cluster [DBG] pgmap v285: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:19.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:18 smithi178 bash[22840]: cluster 2024-09-13T22:36:17.944412+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-09-13T22:36:19.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:18 smithi178 bash[22840]: cluster 2024-09-13T22:36:18.288470+0000 mgr.y (mgr.14150) 292 : cluster [DBG] pgmap v285: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:19.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:18 smithi178 bash[29889]: cluster 2024-09-13T22:36:17.944412+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-09-13T22:36:19.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:18 smithi178 bash[29889]: cluster 2024-09-13T22:36:18.288470+0000 mgr.y (mgr.14150) 292 : cluster [DBG] pgmap v285: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:21 smithi192 bash[24608]: cluster 2024-09-13T22:36:20.289076+0000 mgr.y (mgr.14150) 293 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-13T22:36:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:21 smithi178 bash[22840]: cluster 2024-09-13T22:36:20.289076+0000 mgr.y (mgr.14150) 293 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-13T22:36:21.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:21 smithi178 bash[29889]: cluster 2024-09-13T22:36:20.289076+0000 mgr.y (mgr.14150) 293 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-13T22:36:23.360 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-13T22:36:23.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-09-13T22:36:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:23 smithi192 bash[24608]: cluster 2024-09-13T22:36:22.289604+0000 mgr.y (mgr.14150) 294 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-13T22:36:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:23 smithi178 bash[22840]: cluster 2024-09-13T22:36:22.289604+0000 mgr.y (mgr.14150) 294 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-13T22:36:23.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:23 smithi178 bash[29889]: cluster 2024-09-13T22:36:22.289604+0000 mgr.y (mgr.14150) 294 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-13T22:36:25.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:25 smithi192 bash[24608]: cluster 2024-09-13T22:36:24.290196+0000 mgr.y (mgr.14150) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:36:25.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:25 smithi178 bash[22840]: cluster 2024-09-13T22:36:24.290196+0000 mgr.y (mgr.14150) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:36:25.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:25 smithi178 bash[29889]: cluster 2024-09-13T22:36:24.290196+0000 mgr.y (mgr.14150) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:36:26.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: cluster 2024-09-13T22:36:24.771984+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: cluster 2024-09-13T22:36:24.772055+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: cluster 2024-09-13T22:36:25.356809+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.205986+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.214399+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.217257+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.219001+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.220582+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.228403+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.233271+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.234831+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:36:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:26 smithi192 bash[24608]: audit 2024-09-13T22:36:26.244523+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: cluster 2024-09-13T22:36:24.771984+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: cluster 2024-09-13T22:36:24.772055+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: cluster 2024-09-13T22:36:25.356809+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.205986+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.214399+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.217257+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.219001+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.220582+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.228403+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.233271+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.234831+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[22840]: audit 2024-09-13T22:36:26.244523+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: cluster 2024-09-13T22:36:24.771984+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: cluster 2024-09-13T22:36:24.772055+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: cluster 2024-09-13T22:36:25.356809+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.205986+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.214399+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.217257+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.219001+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.220582+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.228403+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.233271+0000 mon.a (mon.0) 636 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.234831+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:36:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:26 smithi178 bash[29889]: audit 2024-09-13T22:36:26.244523+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:27.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:27 smithi192 bash[24608]: cephadm 2024-09-13T22:36:26.197351+0000 mgr.y (mgr.14150) 296 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:36:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:27 smithi192 bash[24608]: cephadm 2024-09-13T22:36:26.221800+0000 mgr.y (mgr.14150) 297 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 5755M 2024-09-13T22:36:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:27 smithi192 bash[24608]: cluster 2024-09-13T22:36:26.290839+0000 mgr.y (mgr.14150) 298 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-13T22:36:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:27 smithi178 bash[22840]: cephadm 2024-09-13T22:36:26.197351+0000 mgr.y (mgr.14150) 296 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:36:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:27 smithi178 bash[22840]: cephadm 2024-09-13T22:36:26.221800+0000 mgr.y (mgr.14150) 297 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 5755M 2024-09-13T22:36:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:27 smithi178 bash[22840]: cluster 2024-09-13T22:36:26.290839+0000 mgr.y (mgr.14150) 298 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-13T22:36:27.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:27 smithi178 bash[29889]: cephadm 2024-09-13T22:36:26.197351+0000 mgr.y (mgr.14150) 296 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:36:27.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:27 smithi178 bash[29889]: cephadm 2024-09-13T22:36:26.221800+0000 mgr.y (mgr.14150) 297 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 5755M 2024-09-13T22:36:27.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:27 smithi178 bash[29889]: cluster 2024-09-13T22:36:26.290839+0000 mgr.y (mgr.14150) 298 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-13T22:36:28.657 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:36:29.627 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:29 smithi192 bash[24608]: cluster 2024-09-13T22:36:28.291554+0000 mgr.y (mgr.14150) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:36:29.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:29 smithi178 bash[22840]: cluster 2024-09-13T22:36:28.291554+0000 mgr.y (mgr.14150) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:36:29.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:29 smithi178 bash[29889]: cluster 2024-09-13T22:36:28.291554+0000 mgr.y (mgr.14150) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:36:30.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:30 smithi192 bash[24608]: audit 2024-09-13T22:36:29.956948+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:36:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:30 smithi192 bash[24608]: audit 2024-09-13T22:36:29.962793+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:36:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:30 smithi192 bash[24608]: audit 2024-09-13T22:36:29.964357+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:30.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:30 smithi178 bash[22840]: audit 2024-09-13T22:36:29.956948+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:36:30.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:30 smithi178 bash[22840]: audit 2024-09-13T22:36:29.962793+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:36:30.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:30 smithi178 bash[22840]: audit 2024-09-13T22:36:29.964357+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:30.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:30 smithi178 bash[29889]: audit 2024-09-13T22:36:29.956948+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T22:36:30.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:30 smithi178 bash[29889]: audit 2024-09-13T22:36:29.962793+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T22:36:30.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:30 smithi178 bash[29889]: audit 2024-09-13T22:36:29.964357+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:31.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:31 smithi192 bash[24608]: audit 2024-09-13T22:36:29.950707+0000 mgr.y (mgr.14150) 300 : audit [DBG] from='client.24325 v1:172.21.15.192:0/2569738202' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:36:31.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:31 smithi192 bash[24608]: cluster 2024-09-13T22:36:30.292228+0000 mgr.y (mgr.14150) 301 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:31.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:31 smithi178 bash[22840]: audit 2024-09-13T22:36:29.950707+0000 mgr.y (mgr.14150) 300 : audit [DBG] from='client.24325 v1:172.21.15.192:0/2569738202' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:36:31.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:31 smithi178 bash[22840]: cluster 2024-09-13T22:36:30.292228+0000 mgr.y (mgr.14150) 301 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:31.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:31 smithi178 bash[29889]: audit 2024-09-13T22:36:29.950707+0000 mgr.y (mgr.14150) 300 : audit [DBG] from='client.24325 v1:172.21.15.192:0/2569738202' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:36:31.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:31 smithi178 bash[29889]: cluster 2024-09-13T22:36:30.292228+0000 mgr.y (mgr.14150) 301 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:33 smithi192 bash[24608]: cluster 2024-09-13T22:36:32.292869+0000 mgr.y (mgr.14150) 302 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:33 smithi178 bash[22840]: cluster 2024-09-13T22:36:32.292869+0000 mgr.y (mgr.14150) 302 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:33.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:33 smithi178 bash[29889]: cluster 2024-09-13T22:36:32.292869+0000 mgr.y (mgr.14150) 302 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:35.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:35 smithi192 bash[24608]: cluster 2024-09-13T22:36:34.293425+0000 mgr.y (mgr.14150) 303 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:35.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:35 smithi178 bash[22840]: cluster 2024-09-13T22:36:34.293425+0000 mgr.y (mgr.14150) 303 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:35.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:35 smithi178 bash[29889]: cluster 2024-09-13T22:36:34.293425+0000 mgr.y (mgr.14150) 303 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:37 smithi192 bash[24608]: cluster 2024-09-13T22:36:36.293985+0000 mgr.y (mgr.14150) 304 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:37.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:37 smithi178 bash[22840]: cluster 2024-09-13T22:36:36.293985+0000 mgr.y (mgr.14150) 304 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:37.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:37 smithi178 bash[29889]: cluster 2024-09-13T22:36:36.293985+0000 mgr.y (mgr.14150) 304 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:38 smithi192 bash[24608]: cluster 2024-09-13T22:36:38.294513+0000 mgr.y (mgr.14150) 305 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:38 smithi192 bash[24608]: audit 2024-09-13T22:36:38.467805+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.192:0/2061631431' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]: dispatch 2024-09-13T22:36:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:38 smithi192 bash[24608]: audit 2024-09-13T22:36:38.468200+0000 mon.a (mon.0) 642 : audit [INF] from='client.24331 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]: dispatch 2024-09-13T22:36:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:38 smithi192 bash[24608]: audit 2024-09-13T22:36:38.471615+0000 mon.a (mon.0) 643 : audit [INF] from='client.24331 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]': finished 2024-09-13T22:36:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:38 smithi192 bash[24608]: cluster 2024-09-13T22:36:38.477597+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-13T22:36:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:38 smithi192 bash[24608]: audit 2024-09-13T22:36:38.478137+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:36:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[22840]: cluster 2024-09-13T22:36:38.294513+0000 mgr.y (mgr.14150) 305 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[22840]: audit 2024-09-13T22:36:38.467805+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.192:0/2061631431' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]: dispatch 2024-09-13T22:36:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[22840]: audit 2024-09-13T22:36:38.468200+0000 mon.a (mon.0) 642 : audit [INF] from='client.24331 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]: dispatch 2024-09-13T22:36:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[22840]: audit 2024-09-13T22:36:38.471615+0000 mon.a (mon.0) 643 : audit [INF] from='client.24331 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]': finished 2024-09-13T22:36:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[22840]: cluster 2024-09-13T22:36:38.477597+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-13T22:36:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[22840]: audit 2024-09-13T22:36:38.478137+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:36:39.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[29889]: cluster 2024-09-13T22:36:38.294513+0000 mgr.y (mgr.14150) 305 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:39.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[29889]: audit 2024-09-13T22:36:38.467805+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.192:0/2061631431' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]: dispatch 2024-09-13T22:36:39.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[29889]: audit 2024-09-13T22:36:38.468200+0000 mon.a (mon.0) 642 : audit [INF] from='client.24331 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]: dispatch 2024-09-13T22:36:39.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[29889]: audit 2024-09-13T22:36:38.471615+0000 mon.a (mon.0) 643 : audit [INF] from='client.24331 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52"}]': finished 2024-09-13T22:36:39.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[29889]: cluster 2024-09-13T22:36:38.477597+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-09-13T22:36:39.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:38 smithi178 bash[29889]: audit 2024-09-13T22:36:38.478137+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:36:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:39 smithi192 bash[24608]: cluster 2024-09-13T22:36:38.768283+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:39 smithi192 bash[24608]: cluster 2024-09-13T22:36:38.768379+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:39 smithi192 bash[24608]: cluster 2024-09-13T22:36:39.108237+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:40.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:39 smithi192 bash[24608]: cluster 2024-09-13T22:36:39.108319+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:40.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:39 smithi192 bash[24608]: audit 2024-09-13T22:36:39.110972+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.192:0/3711191253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:36:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[22840]: cluster 2024-09-13T22:36:38.768283+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[22840]: cluster 2024-09-13T22:36:38.768379+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[22840]: cluster 2024-09-13T22:36:39.108237+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[22840]: cluster 2024-09-13T22:36:39.108319+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[22840]: audit 2024-09-13T22:36:39.110972+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.192:0/3711191253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:36:40.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[29889]: cluster 2024-09-13T22:36:38.768283+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:40.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[29889]: cluster 2024-09-13T22:36:38.768379+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:40.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[29889]: cluster 2024-09-13T22:36:39.108237+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:36:40.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[29889]: cluster 2024-09-13T22:36:39.108319+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:36:40.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:39 smithi178 bash[29889]: audit 2024-09-13T22:36:39.110972+0000 mon.b (mon.1) 24 : audit [DBG] from='client.? v1:172.21.15.192:0/3711191253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T22:36:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:40 smithi192 bash[24608]: cluster 2024-09-13T22:36:39.713444+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-09-13T22:36:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:40 smithi192 bash[24608]: audit 2024-09-13T22:36:39.713780+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:36:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:40 smithi192 bash[24608]: cluster 2024-09-13T22:36:40.295140+0000 mgr.y (mgr.14150) 306 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:40 smithi178 bash[22840]: cluster 2024-09-13T22:36:39.713444+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-09-13T22:36:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:40 smithi178 bash[22840]: audit 2024-09-13T22:36:39.713780+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:36:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:40 smithi178 bash[22840]: cluster 2024-09-13T22:36:40.295140+0000 mgr.y (mgr.14150) 306 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:40 smithi178 bash[29889]: cluster 2024-09-13T22:36:39.713444+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-09-13T22:36:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:40 smithi178 bash[29889]: audit 2024-09-13T22:36:39.713780+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:36:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:40 smithi178 bash[29889]: cluster 2024-09-13T22:36:40.295140+0000 mgr.y (mgr.14150) 306 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:43.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:43 smithi192 bash[24608]: cluster 2024-09-13T22:36:42.295695+0000 mgr.y (mgr.14150) 307 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:43.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:43 smithi178 bash[22840]: cluster 2024-09-13T22:36:42.295695+0000 mgr.y (mgr.14150) 307 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:43.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:43 smithi178 bash[29889]: cluster 2024-09-13T22:36:42.295695+0000 mgr.y (mgr.14150) 307 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:45.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:45 smithi192 bash[24608]: cluster 2024-09-13T22:36:44.296250+0000 mgr.y (mgr.14150) 308 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:45.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:45 smithi178 bash[22840]: cluster 2024-09-13T22:36:44.296250+0000 mgr.y (mgr.14150) 308 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:45.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:45 smithi178 bash[29889]: cluster 2024-09-13T22:36:44.296250+0000 mgr.y (mgr.14150) 308 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:47.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:47 smithi192 bash[24608]: cluster 2024-09-13T22:36:46.296818+0000 mgr.y (mgr.14150) 309 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:47.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:47 smithi178 bash[22840]: cluster 2024-09-13T22:36:46.296818+0000 mgr.y (mgr.14150) 309 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:47.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:47 smithi178 bash[29889]: cluster 2024-09-13T22:36:46.296818+0000 mgr.y (mgr.14150) 309 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:49 smithi192 bash[24608]: cluster 2024-09-13T22:36:48.297361+0000 mgr.y (mgr.14150) 310 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:49.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:49 smithi178 bash[22840]: cluster 2024-09-13T22:36:48.297361+0000 mgr.y (mgr.14150) 310 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:49 smithi178 bash[29889]: cluster 2024-09-13T22:36:48.297361+0000 mgr.y (mgr.14150) 310 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:51.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:51 smithi192 bash[24608]: cluster 2024-09-13T22:36:50.297907+0000 mgr.y (mgr.14150) 311 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:51.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:51 smithi178 bash[22840]: cluster 2024-09-13T22:36:50.297907+0000 mgr.y (mgr.14150) 311 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:51.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:51 smithi178 bash[29889]: cluster 2024-09-13T22:36:50.297907+0000 mgr.y (mgr.14150) 311 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:52.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:52 smithi192 bash[24608]: audit 2024-09-13T22:36:51.555850+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T22:36:52.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:52 smithi192 bash[24608]: audit 2024-09-13T22:36:51.557541+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:52.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:52 smithi178 bash[22840]: audit 2024-09-13T22:36:51.555850+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T22:36:52.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:52 smithi178 bash[22840]: audit 2024-09-13T22:36:51.557541+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:52.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:52 smithi178 bash[29889]: audit 2024-09-13T22:36:51.555850+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T22:36:52.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:52 smithi178 bash[29889]: audit 2024-09-13T22:36:51.557541+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:36:53.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:53 smithi192 bash[24608]: cephadm 2024-09-13T22:36:51.559076+0000 mgr.y (mgr.14150) 312 : cephadm [INF] Deploying daemon osd.7 on smithi192 2024-09-13T22:36:53.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:53 smithi192 bash[24608]: cluster 2024-09-13T22:36:52.298473+0000 mgr.y (mgr.14150) 313 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:53.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:53 smithi178 bash[22840]: cephadm 2024-09-13T22:36:51.559076+0000 mgr.y (mgr.14150) 312 : cephadm [INF] Deploying daemon osd.7 on smithi192 2024-09-13T22:36:53.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:53 smithi178 bash[22840]: cluster 2024-09-13T22:36:52.298473+0000 mgr.y (mgr.14150) 313 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:53.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:53 smithi178 bash[29889]: cephadm 2024-09-13T22:36:51.559076+0000 mgr.y (mgr.14150) 312 : cephadm [INF] Deploying daemon osd.7 on smithi192 2024-09-13T22:36:53.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:53 smithi178 bash[29889]: cluster 2024-09-13T22:36:52.298473+0000 mgr.y (mgr.14150) 313 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:55.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:55 smithi192 bash[24608]: cluster 2024-09-13T22:36:54.299109+0000 mgr.y (mgr.14150) 314 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:55.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:55 smithi178 bash[22840]: cluster 2024-09-13T22:36:54.299109+0000 mgr.y (mgr.14150) 314 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:55.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:55 smithi178 bash[29889]: cluster 2024-09-13T22:36:54.299109+0000 mgr.y (mgr.14150) 314 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:55.915 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:55 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:55.916 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:36:55 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:55.917 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:36:55 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:55.917 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:36:55 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:55.918 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:36:55 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:56.263 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:36:56 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:56.263 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:36:56 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:56.264 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:36:56 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:56.264 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:36:56 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:56.264 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:56 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:36:56.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:56 smithi192 bash[24608]: audit 2024-09-13T22:36:56.207001+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:56.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:56 smithi192 bash[24608]: audit 2024-09-13T22:36:56.213060+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:56.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:56 smithi192 bash[24608]: audit 2024-09-13T22:36:56.222405+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:56.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:56 smithi178 bash[22840]: audit 2024-09-13T22:36:56.207001+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:56.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:56 smithi178 bash[22840]: audit 2024-09-13T22:36:56.213060+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:56.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:56 smithi178 bash[22840]: audit 2024-09-13T22:36:56.222405+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:56.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:56 smithi178 bash[29889]: audit 2024-09-13T22:36:56.207001+0000 mon.a (mon.0) 650 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:36:56.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:56 smithi178 bash[29889]: audit 2024-09-13T22:36:56.213060+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:56 smithi178 bash[29889]: audit 2024-09-13T22:36:56.222405+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:36:57.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:57 smithi192 bash[24608]: cluster 2024-09-13T22:36:56.299606+0000 mgr.y (mgr.14150) 315 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:57.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:57 smithi192 bash[24608]: cluster 2024-09-13T22:36:56.544958+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:36:57.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:57 smithi192 bash[24608]: cluster 2024-09-13T22:36:56.547543+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:36:57.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:57 smithi178 bash[22840]: cluster 2024-09-13T22:36:56.299606+0000 mgr.y (mgr.14150) 315 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:57.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:57 smithi178 bash[22840]: cluster 2024-09-13T22:36:56.544958+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:36:57.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:57 smithi178 bash[22840]: cluster 2024-09-13T22:36:56.547543+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:36:57.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:57 smithi178 bash[29889]: cluster 2024-09-13T22:36:56.299606+0000 mgr.y (mgr.14150) 315 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:57.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:57 smithi178 bash[29889]: cluster 2024-09-13T22:36:56.544958+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:36:57.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:57 smithi178 bash[29889]: cluster 2024-09-13T22:36:56.547543+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:36:59.098 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:36:58 smithi192 bash[24608]: cluster 2024-09-13T22:36:58.300137+0000 mgr.y (mgr.14150) 316 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:36:58 smithi178 bash[22840]: cluster 2024-09-13T22:36:58.300137+0000 mgr.y (mgr.14150) 316 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:36:59.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:36:58 smithi178 bash[29889]: cluster 2024-09-13T22:36:58.300137+0000 mgr.y (mgr.14150) 316 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:01.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:01 smithi192 bash[24608]: cluster 2024-09-13T22:37:00.300727+0000 mgr.y (mgr.14150) 317 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:01 smithi178 bash[22840]: cluster 2024-09-13T22:37:00.300727+0000 mgr.y (mgr.14150) 317 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:01 smithi178 bash[29889]: cluster 2024-09-13T22:37:00.300727+0000 mgr.y (mgr.14150) 317 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:03 smithi192 bash[24608]: cluster 2024-09-13T22:37:02.301298+0000 mgr.y (mgr.14150) 318 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:03 smithi192 bash[24608]: audit 2024-09-13T22:37:02.832241+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:03 smithi192 bash[24608]: audit 2024-09-13T22:37:02.843163+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:03 smithi192 bash[24608]: audit 2024-09-13T22:37:02.845274+0000 mon.a (mon.0) 655 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:03 smithi192 bash[24608]: audit 2024-09-13T22:37:02.847035+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:03 smithi192 bash[24608]: audit 2024-09-13T22:37:02.858275+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[29889]: cluster 2024-09-13T22:37:02.301298+0000 mgr.y (mgr.14150) 318 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[29889]: audit 2024-09-13T22:37:02.832241+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[29889]: audit 2024-09-13T22:37:02.843163+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[29889]: audit 2024-09-13T22:37:02.845274+0000 mon.a (mon.0) 655 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[29889]: audit 2024-09-13T22:37:02.847035+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[29889]: audit 2024-09-13T22:37:02.858275+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[22840]: cluster 2024-09-13T22:37:02.301298+0000 mgr.y (mgr.14150) 318 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[22840]: audit 2024-09-13T22:37:02.832241+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[22840]: audit 2024-09-13T22:37:02.843163+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[22840]: audit 2024-09-13T22:37:02.845274+0000 mon.a (mon.0) 655 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[22840]: audit 2024-09-13T22:37:02.847035+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:03 smithi178 bash[22840]: audit 2024-09-13T22:37:02.858275+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:05.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:05 smithi192 bash[24608]: cluster 2024-09-13T22:37:04.301863+0000 mgr.y (mgr.14150) 319 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:05.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:05 smithi192 bash[24608]: audit 2024-09-13T22:37:05.000559+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.192:6812/1303971738' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T22:37:05.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:05 smithi192 bash[24608]: audit 2024-09-13T22:37:05.000856+0000 mon.a (mon.0) 658 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T22:37:05.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:05 smithi178 bash[29889]: cluster 2024-09-13T22:37:04.301863+0000 mgr.y (mgr.14150) 319 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:05.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:05 smithi178 bash[29889]: audit 2024-09-13T22:37:05.000559+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.192:6812/1303971738' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T22:37:05.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:05 smithi178 bash[29889]: audit 2024-09-13T22:37:05.000856+0000 mon.a (mon.0) 658 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T22:37:05.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:05 smithi178 bash[22840]: cluster 2024-09-13T22:37:04.301863+0000 mgr.y (mgr.14150) 319 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:05.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:05 smithi178 bash[22840]: audit 2024-09-13T22:37:05.000559+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 v1:172.21.15.192:6812/1303971738' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T22:37:05.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:05 smithi178 bash[22840]: audit 2024-09-13T22:37:05.000856+0000 mon.a (mon.0) 658 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T22:37:06.531 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2024-09-13T22:37:06.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: audit 2024-09-13T22:37:05.369878+0000 mon.a (mon.0) 659 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T22:37:06.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: audit 2024-09-13T22:37:05.373460+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.192:6812/1303971738' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:37:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: cluster 2024-09-13T22:37:05.375679+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-09-13T22:37:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: audit 2024-09-13T22:37:05.376381+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: audit 2024-09-13T22:37:05.376958+0000 mon.a (mon.0) 662 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:37:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: cluster 2024-09-13T22:37:05.953828+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:06 smithi192 bash[24608]: cluster 2024-09-13T22:37:05.953951+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: audit 2024-09-13T22:37:05.369878+0000 mon.a (mon.0) 659 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T22:37:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: audit 2024-09-13T22:37:05.373460+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.192:6812/1303971738' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:37:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: cluster 2024-09-13T22:37:05.375679+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-09-13T22:37:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: audit 2024-09-13T22:37:05.376381+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:06.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: audit 2024-09-13T22:37:05.376958+0000 mon.a (mon.0) 662 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: cluster 2024-09-13T22:37:05.953828+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[29889]: cluster 2024-09-13T22:37:05.953951+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: audit 2024-09-13T22:37:05.369878+0000 mon.a (mon.0) 659 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: audit 2024-09-13T22:37:05.373460+0000 mon.b (mon.1) 26 : audit [INF] from='osd.7 v1:172.21.15.192:6812/1303971738' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: cluster 2024-09-13T22:37:05.375679+0000 mon.a (mon.0) 660 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: audit 2024-09-13T22:37:05.376381+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: audit 2024-09-13T22:37:05.376958+0000 mon.a (mon.0) 662 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: cluster 2024-09-13T22:37:05.953828+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:06 smithi178 bash[22840]: cluster 2024-09-13T22:37:05.953951+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:07.188 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T22:37:07.190 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T22:37:07.190 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: cluster 2024-09-13T22:37:06.302395+0000 mgr.y (mgr.14150) 320 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.372236+0000 mon.a (mon.0) 663 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: cluster 2024-09-13T22:37:06.378257+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.379333+0000 mon.a (mon.0) 665 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.388589+0000 mon.a (mon.0) 666 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.455071+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:07.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.482006+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:07.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.528958+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:07.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:07 smithi192 bash[24608]: audit 2024-09-13T22:37:06.825235+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' 2024-09-13T22:37:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: cluster 2024-09-13T22:37:06.302395+0000 mgr.y (mgr.14150) 320 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.372236+0000 mon.a (mon.0) 663 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:37:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: cluster 2024-09-13T22:37:06.378257+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-09-13T22:37:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.379333+0000 mon.a (mon.0) 665 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:07.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.388589+0000 mon.a (mon.0) 666 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.455071+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.482006+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.528958+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[29889]: audit 2024-09-13T22:37:06.825235+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: cluster 2024-09-13T22:37:06.302395+0000 mgr.y (mgr.14150) 320 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.372236+0000 mon.a (mon.0) 663 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: cluster 2024-09-13T22:37:06.378257+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.379333+0000 mon.a (mon.0) 665 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:07.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.388589+0000 mon.a (mon.0) 666 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.455071+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.482006+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.528958+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:07 smithi178 bash[22840]: audit 2024-09-13T22:37:06.825235+0000 mon.a (mon.0) 670 : audit [INF] from='osd.7 ' entity='osd.7' 2024-09-13T22:37:08.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:08 smithi192 bash[24608]: cluster 2024-09-13T22:37:05.967591+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:08 smithi192 bash[24608]: cluster 2024-09-13T22:37:05.967726+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:08 smithi192 bash[24608]: cluster 2024-09-13T22:37:07.381116+0000 mon.a (mon.0) 671 : cluster [INF] osd.7 v1:172.21.15.192:6812/1303971738 boot 2024-09-13T22:37:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:08 smithi192 bash[24608]: cluster 2024-09-13T22:37:07.381224+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-13T22:37:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:08 smithi192 bash[24608]: audit 2024-09-13T22:37:07.381563+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:08 smithi192 bash[24608]: cluster 2024-09-13T22:37:07.704292+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[29889]: cluster 2024-09-13T22:37:05.967591+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[29889]: cluster 2024-09-13T22:37:05.967726+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[29889]: cluster 2024-09-13T22:37:07.381116+0000 mon.a (mon.0) 671 : cluster [INF] osd.7 v1:172.21.15.192:6812/1303971738 boot 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[29889]: cluster 2024-09-13T22:37:07.381224+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[29889]: audit 2024-09-13T22:37:07.381563+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[29889]: cluster 2024-09-13T22:37:07.704292+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-13T22:37:08.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[22840]: cluster 2024-09-13T22:37:05.967591+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[22840]: cluster 2024-09-13T22:37:05.967726+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[22840]: cluster 2024-09-13T22:37:07.381116+0000 mon.a (mon.0) 671 : cluster [INF] osd.7 v1:172.21.15.192:6812/1303971738 boot 2024-09-13T22:37:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[22840]: cluster 2024-09-13T22:37:07.381224+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-13T22:37:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[22840]: audit 2024-09-13T22:37:07.381563+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:37:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:08 smithi178 bash[22840]: cluster 2024-09-13T22:37:07.704292+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-13T22:37:09.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:09 smithi192 bash[24608]: cluster 2024-09-13T22:37:08.303011+0000 mgr.y (mgr.14150) 321 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:09.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:09 smithi192 bash[24608]: cluster 2024-09-13T22:37:08.721075+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-13T22:37:09.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:09 smithi178 bash[29889]: cluster 2024-09-13T22:37:08.303011+0000 mgr.y (mgr.14150) 321 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:09.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:09 smithi178 bash[29889]: cluster 2024-09-13T22:37:08.721075+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-13T22:37:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:09 smithi178 bash[22840]: cluster 2024-09-13T22:37:08.303011+0000 mgr.y (mgr.14150) 321 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:09 smithi178 bash[22840]: cluster 2024-09-13T22:37:08.721075+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-13T22:37:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:10 smithi192 bash[24608]: cluster 2024-09-13T22:37:09.722756+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-13T22:37:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:10 smithi192 bash[24608]: cluster 2024-09-13T22:37:10.303606+0000 mgr.y (mgr.14150) 322 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:10 smithi178 bash[29889]: cluster 2024-09-13T22:37:09.722756+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-13T22:37:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:10 smithi178 bash[29889]: cluster 2024-09-13T22:37:10.303606+0000 mgr.y (mgr.14150) 322 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:10 smithi178 bash[22840]: cluster 2024-09-13T22:37:09.722756+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-13T22:37:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:10 smithi178 bash[22840]: cluster 2024-09-13T22:37:10.303606+0000 mgr.y (mgr.14150) 322 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:11.967 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:37:12.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:11 smithi178 bash[29889]: cluster 2024-09-13T22:37:10.735806+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-13T22:37:12.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:11 smithi178 bash[22840]: cluster 2024-09-13T22:37:10.735806+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-13T22:37:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:11 smithi192 bash[24608]: cluster 2024-09-13T22:37:10.735806+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-13T22:37:13.033 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:12 smithi178 bash[29889]: cluster 2024-09-13T22:37:12.304191+0000 mgr.y (mgr.14150) 323 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:13.034 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:12 smithi178 bash[22840]: cluster 2024-09-13T22:37:12.304191+0000 mgr.y (mgr.14150) 323 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:13.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:12 smithi192 bash[24608]: cluster 2024-09-13T22:37:12.304191+0000 mgr.y (mgr.14150) 323 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:13.354 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:37:13.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:13 smithi192 bash[24608]: audit 2024-09-13T22:37:13.356409+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? v1:172.21.15.178:0/3757143192' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T22:37:14.043 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:13 smithi178 bash[22840]: audit 2024-09-13T22:37:13.356409+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? v1:172.21.15.178:0/3757143192' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T22:37:14.044 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:13 smithi178 bash[29889]: audit 2024-09-13T22:37:13.356409+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? v1:172.21.15.178:0/3757143192' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T22:37:14.044 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":71,"num_osds":8,"num_up_osds":8,"osd_up_since":1726267027,"num_in_osds":8,"osd_in_since":1726266998,"num_remapped_pgs":0} 2024-09-13T22:37:14.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T22:37:14.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:14 smithi192 bash[24608]: cluster 2024-09-13T22:37:14.304635+0000 mgr.y (mgr.14150) 324 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T22:37:15.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:14 smithi178 bash[22840]: cluster 2024-09-13T22:37:14.304635+0000 mgr.y (mgr.14150) 324 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T22:37:15.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:14 smithi178 bash[29889]: cluster 2024-09-13T22:37:14.304635+0000 mgr.y (mgr.14150) 324 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T22:37:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:15 smithi192 bash[24608]: cluster 2024-09-13T22:37:14.357966+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:15.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:15 smithi192 bash[24608]: cluster 2024-09-13T22:37:14.358050+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:15.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:15 smithi192 bash[24608]: cluster 2024-09-13T22:37:14.755140+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-13T22:37:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:15 smithi178 bash[22840]: cluster 2024-09-13T22:37:14.357966+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:15 smithi178 bash[22840]: cluster 2024-09-13T22:37:14.358050+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:15 smithi178 bash[22840]: cluster 2024-09-13T22:37:14.755140+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-13T22:37:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:15 smithi178 bash[29889]: cluster 2024-09-13T22:37:14.357966+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:15 smithi178 bash[29889]: cluster 2024-09-13T22:37:14.358050+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:15 smithi178 bash[29889]: cluster 2024-09-13T22:37:14.755140+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: cephadm 2024-09-13T22:37:16.302984+0000 mgr.y (mgr.14150) 325 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: cluster 2024-09-13T22:37:16.305245+0000 mgr.y (mgr.14150) 326 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.311423+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.319284+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.321761+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.323202+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.324688+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.326101+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: cephadm 2024-09-13T22:37:16.327129+0000 mgr.y (mgr.14150) 327 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 4316M 2024-09-13T22:37:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.332753+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.336677+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.338327+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:17 smithi192 bash[24608]: audit 2024-09-13T22:37:16.347755+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: cephadm 2024-09-13T22:37:16.302984+0000 mgr.y (mgr.14150) 325 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:37:17.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: cluster 2024-09-13T22:37:16.305245+0000 mgr.y (mgr.14150) 326 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-13T22:37:17.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.311423+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.319284+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.321761+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.323202+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.324688+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.326101+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: cephadm 2024-09-13T22:37:16.327129+0000 mgr.y (mgr.14150) 327 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 4316M 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.332753+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.336677+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.338327+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[22840]: audit 2024-09-13T22:37:16.347755+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: cephadm 2024-09-13T22:37:16.302984+0000 mgr.y (mgr.14150) 325 : cephadm [INF] Detected new or changed devices on smithi192 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: cluster 2024-09-13T22:37:16.305245+0000 mgr.y (mgr.14150) 326 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.311423+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.319284+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.321761+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.323202+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.324688+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.326101+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: cephadm 2024-09-13T22:37:16.327129+0000 mgr.y (mgr.14150) 327 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 4316M 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.332753+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.336677+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.338327+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:17.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:17 smithi178 bash[29889]: audit 2024-09-13T22:37:16.347755+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:18.821 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:37:19.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:18 smithi192 bash[24608]: cluster 2024-09-13T22:37:18.305876+0000 mgr.y (mgr.14150) 328 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:37:19.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:18 smithi178 bash[22840]: cluster 2024-09-13T22:37:18.305876+0000 mgr.y (mgr.14150) 328 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:37:19.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:18 smithi178 bash[29889]: cluster 2024-09-13T22:37:18.305876+0000 mgr.y (mgr.14150) 328 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T22:37:20.325 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:37:20.326 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":72,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","created":"2024-09-13T22:27:31.500176+0000","modified":"2024-09-13T22:37:14.740676+0000","last_up_change":"2024-09-13T22:37:07.372146+0000","last_in_change":"2024-09-13T22:36:38.469358+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T22:32:40.780155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"394f2664-bba1-4b15-ace8-78f70f970cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6801","nonce":1489719354}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6802","nonce":1489719354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6804","nonce":1489719354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6803","nonce":1489719354}]},"public_addr":"172.21.15.178:6801/1489719354","cluster_addr":"172.21.15.178:6802/1489719354","heartbeat_back_addr":"172.21.15.178:6804/1489719354","heartbeat_front_addr":"172.21.15.178:6803/1489719354","state":["exists","up"]},{"osd":1,"uuid":"8e65c08b-6431-4849-a6af-2da9cb1efdfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6805","nonce":4187084287}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6806","nonce":4187084287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6808","nonce":4187084287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6807","nonce":4187084287}]},"public_addr":"172.21.15.178:6805/4187084287","cluster_addr":"172.21.15.178:6806/4187084287","heartbeat_back_addr":"172.21.15.178:6808/4187084287","heartbeat_front_addr":"172.21.15.178:6807/4187084287","state":["exists","up"]},{"osd":2,"uuid":"40a79075-5102-4f6c-b41a-255102064624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6809","nonce":2303802730}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6810","nonce":2303802730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6812","nonce":2303802730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6811","nonce":2303802730}]},"public_addr":"172.21.15.178:6809/2303802730","cluster_addr":"172.21.15.178:6810/2303802730","heartbeat_back_addr":"172.21.15.178:6812/2303802730","heartbeat_front_addr":"172.21.15.178:6811/2303802730","state":["exists","up"]},{"osd":3,"uuid":"3962c796-6f97-4cc6-a186-4b6ef58d67ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6813","nonce":320675660}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6814","nonce":320675660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6816","nonce":320675660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6815","nonce":320675660}]},"public_addr":"172.21.15.178:6813/320675660","cluster_addr":"172.21.15.178:6814/320675660","heartbeat_back_addr":"172.21.15.178:6816/320675660","heartbeat_front_addr":"172.21.15.178:6815/320675660","state":["exists","up"]},{"osd":4,"uuid":"1b89db2f-9e81-40b1-89bd-8de1dbff133b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":3593670527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":3593670527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":3593670527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":3593670527}]},"public_addr":"172.21.15.192:6800/3593670527","cluster_addr":"172.21.15.192:6801/3593670527","heartbeat_back_addr":"172.21.15.192:6803/3593670527","heartbeat_front_addr":"172.21.15.192:6802/3593670527","state":["exists","up"]},{"osd":5,"uuid":"efa849a2-d789-4fdd-acc7-04f2c1bebfa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":3271425338}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":3271425338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":3271425338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":3271425338}]},"public_addr":"172.21.15.192:6804/3271425338","cluster_addr":"172.21.15.192:6805/3271425338","heartbeat_back_addr":"172.21.15.192:6807/3271425338","heartbeat_front_addr":"172.21.15.192:6806/3271425338","state":["exists","up"]},{"osd":6,"uuid":"54a26ae9-bf31-49cf-a924-901eed4a0c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":684205162}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":684205162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":684205162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":684205162}]},"public_addr":"172.21.15.192:6808/684205162","cluster_addr":"172.21.15.192:6809/684205162","heartbeat_back_addr":"172.21.15.192:6811/684205162","heartbeat_front_addr":"172.21.15.192:6810/684205162","state":["exists","up"]},{"osd":7,"uuid":"13e146db-61ac-4fe6-8f26-cbf8d1623a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":1303971738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":1303971738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":1303971738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":1303971738}]},"public_addr":"172.21.15.192:6812/1303971738","cluster_addr":"172.21.15.192:6813/1303971738","heartbeat_back_addr":"172.21.15.192:6815/1303971738","heartbeat_front_addr":"172.21.15.192:6814/1303971738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:37:14.358054+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:36:38.768384+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:36:39.108323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:36:04.723839+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:37:05.953957+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:36:24.772058+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:36:12.883530+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:37:05.967734+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:0/2675812306":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/2835351236":"2024-09-14T22:28:30.168687+0000","172.21.15.178:6800/388635774":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3632788039":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/669877409":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/2959920382":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/268210503":"2024-09-14T22:27:58.436396+0000","172.21.15.178:6800/2039160385":"2024-09-14T22:27:58.436396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T22:37:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:20 smithi192 bash[24608]: audit 2024-09-13T22:37:20.326763+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.178:0/2993274818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:37:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:20 smithi178 bash[22840]: audit 2024-09-13T22:37:20.326763+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.178:0/2993274818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:37:20.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:20 smithi178 bash[29889]: audit 2024-09-13T22:37:20.326763+0000 mon.a (mon.0) 690 : audit [DBG] from='client.? v1:172.21.15.178:0/2993274818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:37:21.024 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T22:32:40.780155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T22:37:21.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T22:37:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:21 smithi192 bash[24608]: cluster 2024-09-13T22:37:20.306432+0000 mgr.y (mgr.14150) 329 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-13T22:37:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:21 smithi178 bash[22840]: cluster 2024-09-13T22:37:20.306432+0000 mgr.y (mgr.14150) 329 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-13T22:37:21.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:21 smithi178 bash[29889]: cluster 2024-09-13T22:37:20.306432+0000 mgr.y (mgr.14150) 329 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-13T22:37:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:23 smithi192 bash[24608]: cluster 2024-09-13T22:37:22.307065+0000 mgr.y (mgr.14150) 330 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:37:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:23 smithi178 bash[22840]: cluster 2024-09-13T22:37:22.307065+0000 mgr.y (mgr.14150) 330 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:37:23.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:23 smithi178 bash[29889]: cluster 2024-09-13T22:37:22.307065+0000 mgr.y (mgr.14150) 330 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T22:37:25.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:25 smithi192 bash[24608]: cluster 2024-09-13T22:37:24.307618+0000 mgr.y (mgr.14150) 331 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:25.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:25 smithi178 bash[22840]: cluster 2024-09-13T22:37:24.307618+0000 mgr.y (mgr.14150) 331 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:25.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:25 smithi178 bash[29889]: cluster 2024-09-13T22:37:24.307618+0000 mgr.y (mgr.14150) 331 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:25.800 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:37:27.224 INFO:teuthology.orchestra.run.smithi178.stdout:pg_num: 1 2024-09-13T22:37:27.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:27 smithi192 bash[24608]: cluster 2024-09-13T22:37:26.308192+0000 mgr.y (mgr.14150) 332 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:27.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:27 smithi192 bash[24608]: audit 2024-09-13T22:37:27.226645+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.178:0/4292003872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T22:37:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:27 smithi178 bash[22840]: cluster 2024-09-13T22:37:26.308192+0000 mgr.y (mgr.14150) 332 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:27.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:27 smithi178 bash[22840]: audit 2024-09-13T22:37:27.226645+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.178:0/4292003872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T22:37:27.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:27 smithi178 bash[29889]: cluster 2024-09-13T22:37:26.308192+0000 mgr.y (mgr.14150) 332 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:27.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:27 smithi178 bash[29889]: audit 2024-09-13T22:37:27.226645+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.178:0/4292003872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T22:37:27.828 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi178 2024-09-13T22:37:27.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi178=foo.a' 2024-09-13T22:37:29.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:29 smithi192 bash[24608]: cluster 2024-09-13T22:37:28.308783+0000 mgr.y (mgr.14150) 333 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:29.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:29 smithi178 bash[22840]: cluster 2024-09-13T22:37:28.308783+0000 mgr.y (mgr.14150) 333 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:29.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:29 smithi178 bash[29889]: cluster 2024-09-13T22:37:28.308783+0000 mgr.y (mgr.14150) 333 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:31.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:31 smithi192 bash[24608]: cluster 2024-09-13T22:37:30.309321+0000 mgr.y (mgr.14150) 334 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:31.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:31 smithi178 bash[22840]: cluster 2024-09-13T22:37:30.309321+0000 mgr.y (mgr.14150) 334 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:31.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:31 smithi178 bash[29889]: cluster 2024-09-13T22:37:30.309321+0000 mgr.y (mgr.14150) 334 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:32.591 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:37:32.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:32 smithi178 bash[22840]: cluster 2024-09-13T22:37:31.281405+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:32.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:32 smithi178 bash[22840]: cluster 2024-09-13T22:37:31.281513+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:32.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:32 smithi178 bash[22840]: cluster 2024-09-13T22:37:31.413903+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-13T22:37:32.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:32 smithi178 bash[29889]: cluster 2024-09-13T22:37:31.281405+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:32.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:32 smithi178 bash[29889]: cluster 2024-09-13T22:37:31.281513+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:32.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:32 smithi178 bash[29889]: cluster 2024-09-13T22:37:31.413903+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-13T22:37:32.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:32 smithi192 bash[24608]: cluster 2024-09-13T22:37:31.281405+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:32.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:32 smithi192 bash[24608]: cluster 2024-09-13T22:37:31.281513+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:32.782 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:32 smithi192 bash[24608]: cluster 2024-09-13T22:37:31.413903+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-13T22:37:33.672 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:33 smithi192 bash[24608]: cluster 2024-09-13T22:37:32.309908+0000 mgr.y (mgr.14150) 335 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:33.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:33 smithi178 bash[22840]: cluster 2024-09-13T22:37:32.309908+0000 mgr.y (mgr.14150) 335 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:33.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:33 smithi178 bash[29889]: cluster 2024-09-13T22:37:32.309908+0000 mgr.y (mgr.14150) 335 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:34.002 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled rgw.foo.a update... 2024-09-13T22:37:34.683 DEBUG:teuthology.orchestra.run.smithi178:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-13T22:37:34.685 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi192 2024-09-13T22:37:34.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-13T22:37:35.319 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:35 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:35.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:33.991531+0000 mgr.y (mgr.14150) 336 : audit [DBG] from='client.24370 v1:172.21.15.192:0/1876047094' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi178=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:37:35.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: cephadm 2024-09-13T22:37:33.994992+0000 mgr.y (mgr.14150) 337 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi178=foo.a;count:1 2024-09-13T22:37:35.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.003941+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.005686+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: cluster 2024-09-13T22:37:34.310478+0000 mgr.y (mgr.14150) 338 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.688049+0000 mon.a (mon.0) 694 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.689888+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.697093+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.700182+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.702816+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.713306+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[22840]: audit 2024-09-13T22:37:34.716983+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:33.991531+0000 mgr.y (mgr.14150) 336 : audit [DBG] from='client.24370 v1:172.21.15.192:0/1876047094' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi178=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: cephadm 2024-09-13T22:37:33.994992+0000 mgr.y (mgr.14150) 337 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi178=foo.a;count:1 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.003941+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.320 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.005686+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: cluster 2024-09-13T22:37:34.310478+0000 mgr.y (mgr.14150) 338 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.688049+0000 mon.a (mon.0) 694 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.689888+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.697093+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.700182+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.702816+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.713306+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:35 smithi178 bash[29889]: audit 2024-09-13T22:37:34.716983+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:33.991531+0000 mgr.y (mgr.14150) 336 : audit [DBG] from='client.24370 v1:172.21.15.192:0/1876047094' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi178=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: cephadm 2024-09-13T22:37:33.994992+0000 mgr.y (mgr.14150) 337 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi178=foo.a;count:1 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.003941+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.005686+0000 mon.a (mon.0) 693 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: cluster 2024-09-13T22:37:34.310478+0000 mgr.y (mgr.14150) 338 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.688049+0000 mon.a (mon.0) 694 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.689888+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.697093+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.700182+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.702816+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.713306+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:35 smithi192 bash[24608]: audit 2024-09-13T22:37:34.716983+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:36.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:36 smithi178 bash[22840]: cephadm 2024-09-13T22:37:34.718391+0000 mgr.y (mgr.14150) 339 : cephadm [INF] Deploying daemon rgw.foo.a on smithi178 2024-09-13T22:37:36.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:36 smithi178 bash[29889]: cephadm 2024-09-13T22:37:34.718391+0000 mgr.y (mgr.14150) 339 : cephadm [INF] Deploying daemon rgw.foo.a on smithi178 2024-09-13T22:37:36.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:36 smithi192 bash[24608]: cephadm 2024-09-13T22:37:34.718391+0000 mgr.y (mgr.14150) 339 : cephadm [INF] Deploying daemon rgw.foo.a on smithi178 2024-09-13T22:37:37.346 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.347 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.347 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.347 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:37 smithi178 bash[29889]: cluster 2024-09-13T22:37:36.310977+0000 mgr.y (mgr.14150) 340 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:37.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.348 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:37 smithi178 bash[22840]: cluster 2024-09-13T22:37:36.310977+0000 mgr.y (mgr.14150) 340 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:37.348 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.348 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:37 smithi192 bash[24608]: cluster 2024-09-13T22:37:36.310977+0000 mgr.y (mgr.14150) 340 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:37.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.679 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.679 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.680 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.680 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.680 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.680 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.681 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.681 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.681 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:37:37.681 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:37 smithi178 systemd[1]: Started Ceph rgw.foo.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:37:38.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: audit 2024-09-13T22:37:37.629282+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: audit 2024-09-13T22:37:37.635024+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: audit 2024-09-13T22:37:37.640891+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: cephadm 2024-09-13T22:37:37.641507+0000 mgr.y (mgr.14150) 341 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi178=foo.a;count:1 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: audit 2024-09-13T22:37:37.645246+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: audit 2024-09-13T22:37:37.649784+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: audit 2024-09-13T22:37:37.680628+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:38 smithi192 bash[24608]: cluster 2024-09-13T22:37:38.311469+0000 mgr.y (mgr.14150) 342 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: audit 2024-09-13T22:37:37.629282+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: audit 2024-09-13T22:37:37.635024+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: audit 2024-09-13T22:37:37.640891+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: cephadm 2024-09-13T22:37:37.641507+0000 mgr.y (mgr.14150) 341 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi178=foo.a;count:1 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: audit 2024-09-13T22:37:37.645246+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: audit 2024-09-13T22:37:37.649784+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: audit 2024-09-13T22:37:37.680628+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[29889]: cluster 2024-09-13T22:37:38.311469+0000 mgr.y (mgr.14150) 342 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: audit 2024-09-13T22:37:37.629282+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: audit 2024-09-13T22:37:37.635024+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: audit 2024-09-13T22:37:37.640891+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: cephadm 2024-09-13T22:37:37.641507+0000 mgr.y (mgr.14150) 341 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi178=foo.a;count:1 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: audit 2024-09-13T22:37:37.645246+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: audit 2024-09-13T22:37:37.649784+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: audit 2024-09-13T22:37:37.680628+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:38 smithi178 bash[22840]: cluster 2024-09-13T22:37:38.311469+0000 mgr.y (mgr.14150) 342 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:39.429 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:37:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: cluster 2024-09-13T22:37:38.877347+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: cluster 2024-09-13T22:37:38.877437+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: cluster 2024-09-13T22:37:39.727874+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-13T22:37:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: audit 2024-09-13T22:37:39.730265+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.178:0/4067660848' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-13T22:37:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: audit 2024-09-13T22:37:39.731381+0000 mon.a (mon.0) 708 : audit [INF] from='client.24353 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-13T22:37:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: audit 2024-09-13T22:37:40.288818+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:40 smithi192 bash[24608]: cluster 2024-09-13T22:37:40.312141+0000 mgr.y (mgr.14150) 343 : cluster [DBG] pgmap v339: 33 pgs: 5 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: cluster 2024-09-13T22:37:38.877347+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: cluster 2024-09-13T22:37:38.877437+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: cluster 2024-09-13T22:37:39.727874+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: audit 2024-09-13T22:37:39.730265+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.178:0/4067660848' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: audit 2024-09-13T22:37:39.731381+0000 mon.a (mon.0) 708 : audit [INF] from='client.24353 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: audit 2024-09-13T22:37:40.288818+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[29889]: cluster 2024-09-13T22:37:40.312141+0000 mgr.y (mgr.14150) 343 : cluster [DBG] pgmap v339: 33 pgs: 5 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: cluster 2024-09-13T22:37:38.877347+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: cluster 2024-09-13T22:37:38.877437+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: cluster 2024-09-13T22:37:39.727874+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: audit 2024-09-13T22:37:39.730265+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.178:0/4067660848' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: audit 2024-09-13T22:37:39.731381+0000 mon.a (mon.0) 708 : audit [INF] from='client.24353 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: audit 2024-09-13T22:37:40.288818+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:40 smithi178 bash[22840]: cluster 2024-09-13T22:37:40.312141+0000 mgr.y (mgr.14150) 343 : cluster [DBG] pgmap v339: 33 pgs: 5 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:41.750 INFO:teuthology.orchestra.run.smithi192.stderr:pool 'datapool' created 2024-09-13T22:37:41.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:41 smithi192 bash[24608]: audit 2024-09-13T22:37:40.722706+0000 mon.a (mon.0) 710 : audit [INF] from='client.24353 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-13T22:37:41.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:41 smithi192 bash[24608]: cluster 2024-09-13T22:37:40.726938+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-13T22:37:41.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:41 smithi192 bash[24608]: audit 2024-09-13T22:37:40.843227+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.192:0/89318003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-13T22:37:41.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:41 smithi192 bash[24608]: audit 2024-09-13T22:37:41.230461+0000 mon.a (mon.0) 712 : audit [INF] from='client.24385 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-13T22:37:41.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:41 smithi192 bash[24608]: cluster 2024-09-13T22:37:41.624752+0000 mon.a (mon.0) 713 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:37:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[29889]: audit 2024-09-13T22:37:40.722706+0000 mon.a (mon.0) 710 : audit [INF] from='client.24353 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-13T22:37:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[29889]: cluster 2024-09-13T22:37:40.726938+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-13T22:37:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[29889]: audit 2024-09-13T22:37:40.843227+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.192:0/89318003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-13T22:37:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[29889]: audit 2024-09-13T22:37:41.230461+0000 mon.a (mon.0) 712 : audit [INF] from='client.24385 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-13T22:37:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[29889]: cluster 2024-09-13T22:37:41.624752+0000 mon.a (mon.0) 713 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:37:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[22840]: audit 2024-09-13T22:37:40.722706+0000 mon.a (mon.0) 710 : audit [INF] from='client.24353 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-13T22:37:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[22840]: cluster 2024-09-13T22:37:40.726938+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-13T22:37:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[22840]: audit 2024-09-13T22:37:40.843227+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.192:0/89318003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-13T22:37:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[22840]: audit 2024-09-13T22:37:41.230461+0000 mon.a (mon.0) 712 : audit [INF] from='client.24385 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-13T22:37:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:41 smithi178 bash[22840]: cluster 2024-09-13T22:37:41.624752+0000 mon.a (mon.0) 713 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:37:42.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-13T22:37:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:42 smithi192 bash[24608]: audit 2024-09-13T22:37:41.742831+0000 mon.a (mon.0) 714 : audit [INF] from='client.24385 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-13T22:37:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:42 smithi192 bash[24608]: cluster 2024-09-13T22:37:41.748182+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-13T22:37:43.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:42 smithi192 bash[24608]: audit 2024-09-13T22:37:41.749470+0000 mon.a (mon.0) 716 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-13T22:37:43.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:42 smithi192 bash[24608]: cluster 2024-09-13T22:37:42.312842+0000 mgr.y (mgr.14150) 344 : cluster [DBG] pgmap v342: 68 pgs: 7 creating+peering, 60 unknown, 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[22840]: audit 2024-09-13T22:37:41.742831+0000 mon.a (mon.0) 714 : audit [INF] from='client.24385 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[22840]: cluster 2024-09-13T22:37:41.748182+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[22840]: audit 2024-09-13T22:37:41.749470+0000 mon.a (mon.0) 716 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[22840]: cluster 2024-09-13T22:37:42.312842+0000 mgr.y (mgr.14150) 344 : cluster [DBG] pgmap v342: 68 pgs: 7 creating+peering, 60 unknown, 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[29889]: audit 2024-09-13T22:37:41.742831+0000 mon.a (mon.0) 714 : audit [INF] from='client.24385 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[29889]: cluster 2024-09-13T22:37:41.748182+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[29889]: audit 2024-09-13T22:37:41.749470+0000 mon.a (mon.0) 716 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-13T22:37:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:42 smithi178 bash[29889]: cluster 2024-09-13T22:37:42.312842+0000 mgr.y (mgr.14150) 344 : cluster [DBG] pgmap v342: 68 pgs: 7 creating+peering, 60 unknown, 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:37:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:43 smithi192 bash[24608]: audit 2024-09-13T22:37:42.744578+0000 mon.a (mon.0) 717 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T22:37:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:43 smithi192 bash[24608]: cluster 2024-09-13T22:37:42.748236+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-13T22:37:44.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:43 smithi192 bash[24608]: audit 2024-09-13T22:37:43.205750+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:44.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:43 smithi192 bash[24608]: audit 2024-09-13T22:37:43.211362+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:44.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:43 smithi192 bash[24608]: audit 2024-09-13T22:37:43.212527+0000 mon.a (mon.0) 721 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:44.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:43 smithi192 bash[24608]: audit 2024-09-13T22:37:43.213448+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[29889]: audit 2024-09-13T22:37:42.744578+0000 mon.a (mon.0) 717 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[29889]: cluster 2024-09-13T22:37:42.748236+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[29889]: audit 2024-09-13T22:37:43.205750+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[29889]: audit 2024-09-13T22:37:43.211362+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[29889]: audit 2024-09-13T22:37:43.212527+0000 mon.a (mon.0) 721 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[29889]: audit 2024-09-13T22:37:43.213448+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[22840]: audit 2024-09-13T22:37:42.744578+0000 mon.a (mon.0) 717 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[22840]: cluster 2024-09-13T22:37:42.748236+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-13T22:37:44.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[22840]: audit 2024-09-13T22:37:43.205750+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:44.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[22840]: audit 2024-09-13T22:37:43.211362+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:44.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[22840]: audit 2024-09-13T22:37:43.212527+0000 mon.a (mon.0) 721 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:44.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:43 smithi178 bash[22840]: audit 2024-09-13T22:37:43.213448+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:45.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: cephadm 2024-09-13T22:37:43.219055+0000 mgr.y (mgr.14150) 345 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-13T22:37:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: cluster 2024-09-13T22:37:43.842187+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-13T22:37:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: audit 2024-09-13T22:37:43.845547+0000 mon.a (mon.0) 724 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-13T22:37:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: audit 2024-09-13T22:37:43.846598+0000 mon.a (mon.0) 725 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-13T22:37:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: cluster 2024-09-13T22:37:44.225338+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: cluster 2024-09-13T22:37:44.225438+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:44 smithi192 bash[24608]: cluster 2024-09-13T22:37:44.313718+0000 mgr.y (mgr.14150) 346 : cluster [DBG] pgmap v345: 100 pgs: 26 active+clean, 1 creating+activating, 17 creating+peering, 56 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1023 B/s wr, 1 op/s 2024-09-13T22:37:45.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: cephadm 2024-09-13T22:37:43.219055+0000 mgr.y (mgr.14150) 345 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: cluster 2024-09-13T22:37:43.842187+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: audit 2024-09-13T22:37:43.845547+0000 mon.a (mon.0) 724 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: audit 2024-09-13T22:37:43.846598+0000 mon.a (mon.0) 725 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: cluster 2024-09-13T22:37:44.225338+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: cluster 2024-09-13T22:37:44.225438+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[29889]: cluster 2024-09-13T22:37:44.313718+0000 mgr.y (mgr.14150) 346 : cluster [DBG] pgmap v345: 100 pgs: 26 active+clean, 1 creating+activating, 17 creating+peering, 56 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1023 B/s wr, 1 op/s 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: cephadm 2024-09-13T22:37:43.219055+0000 mgr.y (mgr.14150) 345 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: cluster 2024-09-13T22:37:43.842187+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: audit 2024-09-13T22:37:43.845547+0000 mon.a (mon.0) 724 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-13T22:37:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: audit 2024-09-13T22:37:43.846598+0000 mon.a (mon.0) 725 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-13T22:37:45.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: cluster 2024-09-13T22:37:44.225338+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:45.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: cluster 2024-09-13T22:37:44.225438+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:45.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:44 smithi178 bash[22840]: cluster 2024-09-13T22:37:44.313718+0000 mgr.y (mgr.14150) 346 : cluster [DBG] pgmap v345: 100 pgs: 26 active+clean, 1 creating+activating, 17 creating+peering, 56 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1023 B/s wr, 1 op/s 2024-09-13T22:37:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:45 smithi192 bash[24608]: audit 2024-09-13T22:37:44.840564+0000 mon.a (mon.0) 726 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T22:37:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:45 smithi192 bash[24608]: audit 2024-09-13T22:37:44.840718+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T22:37:46.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:45 smithi192 bash[24608]: cluster 2024-09-13T22:37:44.847708+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-13T22:37:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:45 smithi178 bash[29889]: audit 2024-09-13T22:37:44.840564+0000 mon.a (mon.0) 726 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T22:37:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:45 smithi178 bash[29889]: audit 2024-09-13T22:37:44.840718+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T22:37:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:45 smithi178 bash[29889]: cluster 2024-09-13T22:37:44.847708+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-13T22:37:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:45 smithi178 bash[22840]: audit 2024-09-13T22:37:44.840564+0000 mon.a (mon.0) 726 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T22:37:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:45 smithi178 bash[22840]: audit 2024-09-13T22:37:44.840718+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T22:37:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:45 smithi178 bash[22840]: cluster 2024-09-13T22:37:44.847708+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-13T22:37:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:46 smithi192 bash[24608]: cluster 2024-09-13T22:37:45.086489+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:47.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:46 smithi192 bash[24608]: cluster 2024-09-13T22:37:45.086574+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:47.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:46 smithi192 bash[24608]: cluster 2024-09-13T22:37:45.871971+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-13T22:37:47.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:46 smithi192 bash[24608]: audit 2024-09-13T22:37:45.874064+0000 mon.a (mon.0) 730 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-13T22:37:47.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:46 smithi192 bash[24608]: audit 2024-09-13T22:37:45.876618+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-13T22:37:47.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:46 smithi192 bash[24608]: cluster 2024-09-13T22:37:46.314506+0000 mgr.y (mgr.14150) 347 : cluster [DBG] pgmap v348: 132 pgs: 63 active+clean, 1 creating+activating, 19 creating+peering, 49 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[22840]: cluster 2024-09-13T22:37:45.086489+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[22840]: cluster 2024-09-13T22:37:45.086574+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[22840]: cluster 2024-09-13T22:37:45.871971+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[22840]: audit 2024-09-13T22:37:45.874064+0000 mon.a (mon.0) 730 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[22840]: audit 2024-09-13T22:37:45.876618+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[22840]: cluster 2024-09-13T22:37:46.314506+0000 mgr.y (mgr.14150) 347 : cluster [DBG] pgmap v348: 132 pgs: 63 active+clean, 1 creating+activating, 19 creating+peering, 49 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[29889]: cluster 2024-09-13T22:37:45.086489+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[29889]: cluster 2024-09-13T22:37:45.086574+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:37:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[29889]: cluster 2024-09-13T22:37:45.871971+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-13T22:37:47.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[29889]: audit 2024-09-13T22:37:45.874064+0000 mon.a (mon.0) 730 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-13T22:37:47.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[29889]: audit 2024-09-13T22:37:45.876618+0000 mon.a (mon.0) 731 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-13T22:37:47.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:46 smithi178 bash[29889]: cluster 2024-09-13T22:37:46.314506+0000 mgr.y (mgr.14150) 347 : cluster [DBG] pgmap v348: 132 pgs: 63 active+clean, 1 creating+activating, 19 creating+peering, 49 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-13T22:37:47.252 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:37:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:47 smithi192 bash[24608]: audit 2024-09-13T22:37:46.880969+0000 mon.a (mon.0) 732 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T22:37:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:47 smithi192 bash[24608]: audit 2024-09-13T22:37:46.881105+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T22:37:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:47 smithi192 bash[24608]: cluster 2024-09-13T22:37:46.888210+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-13T22:37:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:47 smithi192 bash[24608]: audit 2024-09-13T22:37:46.901342+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-13T22:37:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:47 smithi192 bash[24608]: audit 2024-09-13T22:37:46.902103+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-13T22:37:48.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:47 smithi192 bash[24608]: cluster 2024-09-13T22:37:47.708090+0000 mon.a (mon.0) 737 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:37:48.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[29889]: audit 2024-09-13T22:37:46.880969+0000 mon.a (mon.0) 732 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T22:37:48.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[29889]: audit 2024-09-13T22:37:46.881105+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T22:37:48.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[29889]: cluster 2024-09-13T22:37:46.888210+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[29889]: audit 2024-09-13T22:37:46.901342+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[29889]: audit 2024-09-13T22:37:46.902103+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[29889]: cluster 2024-09-13T22:37:47.708090+0000 mon.a (mon.0) 737 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:37:48.179 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[52826]: debug 2024-09-13T22:37:48.012+0000 7f3599045880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[22840]: audit 2024-09-13T22:37:46.880969+0000 mon.a (mon.0) 732 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[22840]: audit 2024-09-13T22:37:46.881105+0000 mon.a (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[22840]: cluster 2024-09-13T22:37:46.888210+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-13T22:37:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[22840]: audit 2024-09-13T22:37:46.901342+0000 mon.a (mon.0) 735 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-13T22:37:48.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[22840]: audit 2024-09-13T22:37:46.902103+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-13T22:37:48.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:47 smithi178 bash[22840]: cluster 2024-09-13T22:37:47.708090+0000 mon.a (mon.0) 737 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:37:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:47.891672+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:47.891835+0000 mon.a (mon.0) 739 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: cluster 2024-09-13T22:37:47.898078+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.167285+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.172192+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.182200+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.235732+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.237779+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.238519+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: cephadm 2024-09-13T22:37:48.243075+0000 mgr.y (mgr.14150) 348 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: cluster 2024-09-13T22:37:48.315727+0000 mgr.y (mgr.14150) 349 : cluster [DBG] pgmap v351: 132 pgs: 98 active+clean, 4 creating+activating, 9 creating+peering, 21 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-13T22:37:49.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.514503+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.596548+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.192:0/3438385389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-13T22:37:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:48 smithi192 bash[24608]: audit 2024-09-13T22:37:48.596674+0000 mon.a (mon.0) 748 : audit [INF] from='client.24475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-13T22:37:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:47.891672+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T22:37:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:47.891835+0000 mon.a (mon.0) 739 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T22:37:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: cluster 2024-09-13T22:37:47.898078+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-13T22:37:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.167285+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.172192+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.182200+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.235732+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.237779+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.238519+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: cephadm 2024-09-13T22:37:48.243075+0000 mgr.y (mgr.14150) 348 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: cluster 2024-09-13T22:37:48.315727+0000 mgr.y (mgr.14150) 349 : cluster [DBG] pgmap v351: 132 pgs: 98 active+clean, 4 creating+activating, 9 creating+peering, 21 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.514503+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.596548+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.192:0/3438385389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-13T22:37:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[29889]: audit 2024-09-13T22:37:48.596674+0000 mon.a (mon.0) 748 : audit [INF] from='client.24475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:47.891672+0000 mon.a (mon.0) 738 : audit [INF] from='client.? v1:172.21.15.178:0/1055532036' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:47.891835+0000 mon.a (mon.0) 739 : audit [INF] from='client.? v1:172.21.15.178:0/1037408744' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: cluster 2024-09-13T22:37:47.898078+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.167285+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.172192+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.182200+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.235732+0000 mon.a (mon.0) 744 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.237779+0000 mon.a (mon.0) 745 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:37:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.238519+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:37:49.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: cephadm 2024-09-13T22:37:48.243075+0000 mgr.y (mgr.14150) 348 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-13T22:37:49.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: cluster 2024-09-13T22:37:48.315727+0000 mgr.y (mgr.14150) 349 : cluster [DBG] pgmap v351: 132 pgs: 98 active+clean, 4 creating+activating, 9 creating+peering, 21 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-13T22:37:49.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.514503+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:37:49.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.596548+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.192:0/3438385389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-13T22:37:49.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:48 smithi178 bash[22840]: audit 2024-09-13T22:37:48.596674+0000 mon.a (mon.0) 748 : audit [INF] from='client.24475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-13T22:37:50.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:50 smithi192 bash[24608]: audit 2024-09-13T22:37:49.514536+0000 mon.a (mon.0) 749 : audit [INF] from='client.24475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-13T22:37:50.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:50 smithi192 bash[24608]: cluster 2024-09-13T22:37:49.520790+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-13T22:37:50.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:50 smithi192 bash[24608]: cluster 2024-09-13T22:37:50.316622+0000 mgr.y (mgr.14150) 350 : cluster [DBG] pgmap v353: 132 pgs: 123 active+clean, 4 creating+activating, 5 unknown; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.6 KiB/s wr, 94 op/s 2024-09-13T22:37:50.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:50 smithi178 bash[29889]: audit 2024-09-13T22:37:49.514536+0000 mon.a (mon.0) 749 : audit [INF] from='client.24475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-13T22:37:50.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:50 smithi178 bash[29889]: cluster 2024-09-13T22:37:49.520790+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-13T22:37:50.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:50 smithi178 bash[29889]: cluster 2024-09-13T22:37:50.316622+0000 mgr.y (mgr.14150) 350 : cluster [DBG] pgmap v353: 132 pgs: 123 active+clean, 4 creating+activating, 5 unknown; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.6 KiB/s wr, 94 op/s 2024-09-13T22:37:50.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:50 smithi178 bash[22840]: audit 2024-09-13T22:37:49.514536+0000 mon.a (mon.0) 749 : audit [INF] from='client.24475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-13T22:37:50.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:50 smithi178 bash[22840]: cluster 2024-09-13T22:37:49.520790+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-13T22:37:50.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:50 smithi178 bash[22840]: cluster 2024-09-13T22:37:50.316622+0000 mgr.y (mgr.14150) 350 : cluster [DBG] pgmap v353: 132 pgs: 123 active+clean, 4 creating+activating, 5 unknown; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.6 KiB/s wr, 94 op/s 2024-09-13T22:37:51.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:51 smithi192 bash[24608]: cluster 2024-09-13T22:37:50.514520+0000 mon.a (mon.0) 751 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:37:51.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:51 smithi192 bash[24608]: cluster 2024-09-13T22:37:50.514563+0000 mon.a (mon.0) 752 : cluster [INF] Cluster is now healthy 2024-09-13T22:37:51.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:51 smithi192 bash[24608]: cluster 2024-09-13T22:37:50.530511+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-13T22:37:51.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:51 smithi178 bash[22840]: cluster 2024-09-13T22:37:50.514520+0000 mon.a (mon.0) 751 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:37:51.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:51 smithi178 bash[22840]: cluster 2024-09-13T22:37:50.514563+0000 mon.a (mon.0) 752 : cluster [INF] Cluster is now healthy 2024-09-13T22:37:51.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:51 smithi178 bash[22840]: cluster 2024-09-13T22:37:50.530511+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-13T22:37:51.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:51 smithi178 bash[29889]: cluster 2024-09-13T22:37:50.514520+0000 mon.a (mon.0) 751 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:37:51.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:51 smithi178 bash[29889]: cluster 2024-09-13T22:37:50.514563+0000 mon.a (mon.0) 752 : cluster [INF] Cluster is now healthy 2024-09-13T22:37:51.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:51 smithi178 bash[29889]: cluster 2024-09-13T22:37:50.530511+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-13T22:37:52.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.192 --placement '1;smithi192=iscsi.a' 2024-09-13T22:37:52.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:52 smithi192 bash[24608]: cluster 2024-09-13T22:37:51.539882+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-13T22:37:52.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:52 smithi192 bash[24608]: cluster 2024-09-13T22:37:52.317375+0000 mgr.y (mgr.14150) 351 : cluster [DBG] pgmap v356: 132 pgs: 128 active+clean, 4 creating+activating; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.1 KiB/s wr, 128 op/s 2024-09-13T22:37:52.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:52 smithi178 bash[29889]: cluster 2024-09-13T22:37:51.539882+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-13T22:37:52.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:52 smithi178 bash[29889]: cluster 2024-09-13T22:37:52.317375+0000 mgr.y (mgr.14150) 351 : cluster [DBG] pgmap v356: 132 pgs: 128 active+clean, 4 creating+activating; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.1 KiB/s wr, 128 op/s 2024-09-13T22:37:52.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:52 smithi178 bash[22840]: cluster 2024-09-13T22:37:51.539882+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-13T22:37:52.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:52 smithi178 bash[22840]: cluster 2024-09-13T22:37:52.317375+0000 mgr.y (mgr.14150) 351 : cluster [DBG] pgmap v356: 132 pgs: 128 active+clean, 4 creating+activating; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.1 KiB/s wr, 128 op/s 2024-09-13T22:37:55.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:55 smithi192 bash[24608]: cluster 2024-09-13T22:37:54.318598+0000 mgr.y (mgr.14150) 352 : cluster [DBG] pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 188 op/s 2024-09-13T22:37:55.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:55 smithi178 bash[22840]: cluster 2024-09-13T22:37:54.318598+0000 mgr.y (mgr.14150) 352 : cluster [DBG] pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 188 op/s 2024-09-13T22:37:55.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:55 smithi178 bash[29889]: cluster 2024-09-13T22:37:54.318598+0000 mgr.y (mgr.14150) 352 : cluster [DBG] pgmap v357: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 188 op/s 2024-09-13T22:37:57.227 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:37:57.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:57 smithi192 bash[24608]: cluster 2024-09-13T22:37:56.319733+0000 mgr.y (mgr.14150) 353 : cluster [DBG] pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 3.4 KiB/s wr, 116 op/s 2024-09-13T22:37:57.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:57 smithi178 bash[22840]: cluster 2024-09-13T22:37:56.319733+0000 mgr.y (mgr.14150) 353 : cluster [DBG] pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 3.4 KiB/s wr, 116 op/s 2024-09-13T22:37:57.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:57 smithi178 bash[29889]: cluster 2024-09-13T22:37:56.319733+0000 mgr.y (mgr.14150) 353 : cluster [DBG] pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 3.4 KiB/s wr, 116 op/s 2024-09-13T22:37:58.755 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled iscsi.datapool update... 2024-09-13T22:37:58.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:58 smithi192 bash[24608]: cluster 2024-09-13T22:37:57.719541+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-13T22:37:58.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:58 smithi192 bash[24608]: cluster 2024-09-13T22:37:58.320641+0000 mgr.y (mgr.14150) 354 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 74 op/s 2024-09-13T22:37:59.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:58 smithi178 bash[22840]: cluster 2024-09-13T22:37:57.719541+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-13T22:37:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:58 smithi178 bash[22840]: cluster 2024-09-13T22:37:58.320641+0000 mgr.y (mgr.14150) 354 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 74 op/s 2024-09-13T22:37:59.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:58 smithi178 bash[29889]: cluster 2024-09-13T22:37:57.719541+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-13T22:37:59.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:58 smithi178 bash[29889]: cluster 2024-09-13T22:37:58.320641+0000 mgr.y (mgr.14150) 354 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 74 op/s 2024-09-13T22:37:59.407 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-13T22:37:59.407 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:37:59.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-13T22:37:59.420 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:37:59.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-13T22:37:59.434 DEBUG:teuthology.orchestra.run.smithi192:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-13T22:37:59.480 INFO:tasks.cephadm:Adding prometheus.a on smithi192 2024-09-13T22:37:59.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi192=a' 2024-09-13T22:38:00.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:37:59 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:58.745891+0000 mgr.y (mgr.14150) 355 : audit [DBG] from='client.24481 v1:172.21.15.192:0/3614821510' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: cephadm 2024-09-13T22:37:58.751386+0000 mgr.y (mgr.14150) 356 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:58.757643+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:58.759434+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:59.454760+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:59.455681+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:59.461801+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:00.104 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:59.464780+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-13T22:38:00.105 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:59.467090+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-13T22:38:00.105 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: audit 2024-09-13T22:37:59.481933+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:00.105 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:37:59 smithi192 bash[24608]: cephadm 2024-09-13T22:37:59.483576+0000 mgr.y (mgr.14150) 357 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi192 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:58.745891+0000 mgr.y (mgr.14150) 355 : audit [DBG] from='client.24481 v1:172.21.15.192:0/3614821510' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: cephadm 2024-09-13T22:37:58.751386+0000 mgr.y (mgr.14150) 356 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:58.757643+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:58.759434+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:59.454760+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:59.455681+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:00.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:59.461801+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:59.464780+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:59.467090+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: audit 2024-09-13T22:37:59.481933+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[22840]: cephadm 2024-09-13T22:37:59.483576+0000 mgr.y (mgr.14150) 357 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi192 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:58.745891+0000 mgr.y (mgr.14150) 355 : audit [DBG] from='client.24481 v1:172.21.15.192:0/3614821510' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: cephadm 2024-09-13T22:37:58.751386+0000 mgr.y (mgr.14150) 356 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:58.757643+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:58.759434+0000 mon.a (mon.0) 757 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:59.454760+0000 mon.a (mon.0) 758 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:59.455681+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:59.461801+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:59.464780+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-13T22:38:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:59.467090+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-13T22:38:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: audit 2024-09-13T22:37:59.481933+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:37:59 smithi178 bash[29889]: cephadm 2024-09-13T22:37:59.483576+0000 mgr.y (mgr.14150) 357 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi192 2024-09-13T22:38:01.088 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:00.321957+0000 mgr.y (mgr.14150) 358 : cluster [DBG] pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 65 op/s 2024-09-13T22:38:01.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:00.321957+0000 mgr.y (mgr.14150) 358 : cluster [DBG] pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 65 op/s 2024-09-13T22:38:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:00.321957+0000 mgr.y (mgr.14150) 358 : cluster [DBG] pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 65 op/s 2024-09-13T22:38:02.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.238 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.238 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.239 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.239 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.239 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.504 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.504 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.505 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.506 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.506 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:02.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:02 smithi192 systemd[1]: Started Ceph iscsi.iscsi.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:38:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: cluster 2024-09-13T22:38:02.322713+0000 mgr.y (mgr.14150) 359 : cluster [DBG] pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.9 KiB/s wr, 57 op/s 2024-09-13T22:38:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: audit 2024-09-13T22:38:02.570705+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: audit 2024-09-13T22:38:02.579688+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: audit 2024-09-13T22:38:02.588732+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: audit 2024-09-13T22:38:02.603451+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: audit 2024-09-13T22:38:02.637193+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:03.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:03 smithi192 bash[24608]: cluster 2024-09-13T22:38:02.722148+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: cluster 2024-09-13T22:38:02.322713+0000 mgr.y (mgr.14150) 359 : cluster [DBG] pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.9 KiB/s wr, 57 op/s 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: audit 2024-09-13T22:38:02.570705+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: audit 2024-09-13T22:38:02.579688+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: audit 2024-09-13T22:38:02.588732+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: audit 2024-09-13T22:38:02.603451+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: audit 2024-09-13T22:38:02.637193+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[22840]: cluster 2024-09-13T22:38:02.722148+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: cluster 2024-09-13T22:38:02.322713+0000 mgr.y (mgr.14150) 359 : cluster [DBG] pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.9 KiB/s wr, 57 op/s 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: audit 2024-09-13T22:38:02.570705+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: audit 2024-09-13T22:38:02.579688+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: audit 2024-09-13T22:38:02.588732+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: audit 2024-09-13T22:38:02.603451+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: audit 2024-09-13T22:38:02.637193+0000 mon.a (mon.0) 768 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:03 smithi178 bash[29889]: cluster 2024-09-13T22:38:02.722148+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-13T22:38:04.289 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:38:04.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:04 smithi192 bash[24608]: cephadm 2024-09-13T22:38:02.590236+0000 mgr.y (mgr.14150) 360 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-13T22:38:04.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:04 smithi192 bash[24608]: cluster 2024-09-13T22:38:03.745552+0000 mon.a (mon.0) 770 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-09-13T22:38:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:04 smithi178 bash[22840]: cephadm 2024-09-13T22:38:02.590236+0000 mgr.y (mgr.14150) 360 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-13T22:38:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:04 smithi178 bash[22840]: cluster 2024-09-13T22:38:03.745552+0000 mon.a (mon.0) 770 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-09-13T22:38:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:04 smithi178 bash[29889]: cephadm 2024-09-13T22:38:02.590236+0000 mgr.y (mgr.14150) 360 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-13T22:38:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:04 smithi178 bash[29889]: cluster 2024-09-13T22:38:03.745552+0000 mon.a (mon.0) 770 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-09-13T22:38:04.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:04 smithi192 bash[48621]: debug Started the configuration object watcher 2024-09-13T22:38:04.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:04 smithi192 bash[48621]: debug Checking for config object changes every 1s 2024-09-13T22:38:04.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:04 smithi192 bash[48621]: debug Processing osd blocklist entries for this node 2024-09-13T22:38:05.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: debug Reading the configuration object to update local LIO configuration 2024-09-13T22:38:05.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: debug Configuration does not have an entry for this host(smithi192.front.sepia.ceph.com) - nothing to define to LIO 2024-09-13T22:38:05.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-13T22:38:05.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: * Environment: production 2024-09-13T22:38:05.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: Use a production WSGI server instead. 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: * Debug mode: off 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: debug * Running on all addresses. 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: * Running on all addresses. 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: debug * Running on http://[fe80::ec4:7aff:fe88:8049]:5000/ (Press CTRL+C to quit) 2024-09-13T22:38:05.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[48621]: * Running on http://[fe80::ec4:7aff:fe88:8049]:5000/ (Press CTRL+C to quit) 2024-09-13T22:38:05.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[24608]: cluster 2024-09-13T22:38:04.323651+0000 mgr.y (mgr.14150) 361 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:05.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[24608]: audit 2024-09-13T22:38:05.043265+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.192:0/2592490089' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T22:38:05.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:05 smithi192 bash[24608]: audit 2024-09-13T22:38:05.298962+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:05.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:05 smithi178 bash[22840]: cluster 2024-09-13T22:38:04.323651+0000 mgr.y (mgr.14150) 361 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:05.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:05 smithi178 bash[22840]: audit 2024-09-13T22:38:05.043265+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.192:0/2592490089' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T22:38:05.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:05 smithi178 bash[22840]: audit 2024-09-13T22:38:05.298962+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:05.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:05 smithi178 bash[29889]: cluster 2024-09-13T22:38:04.323651+0000 mgr.y (mgr.14150) 361 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:05.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:05 smithi178 bash[29889]: audit 2024-09-13T22:38:05.043265+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.192:0/2592490089' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T22:38:05.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:05 smithi178 bash[29889]: audit 2024-09-13T22:38:05.298962+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:05.978 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled prometheus update... 2024-09-13T22:38:06.635 DEBUG:teuthology.orchestra.run.smithi192:prometheus.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-13T22:38:06.638 INFO:tasks.cephadm:Adding node-exporter.a on smithi178 2024-09-13T22:38:06.638 INFO:tasks.cephadm:Adding node-exporter.b on smithi192 2024-09-13T22:38:06.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi178=a;smithi192=b' 2024-09-13T22:38:07.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: cluster 2024-09-13T22:38:05.712675+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:07.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: cluster 2024-09-13T22:38:05.712787+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:07.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: audit 2024-09-13T22:38:05.969434+0000 mgr.y (mgr.14150) 362 : audit [DBG] from='client.24517 v1:172.21.15.192:0/4162288515' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:07.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: cephadm 2024-09-13T22:38:05.972557+0000 mgr.y (mgr.14150) 363 : cephadm [INF] Saving service prometheus spec with placement smithi192=a;count:1 2024-09-13T22:38:07.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: audit 2024-09-13T22:38:05.979779+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:07.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: cluster 2024-09-13T22:38:06.324754+0000 mgr.y (mgr.14150) 364 : cluster [DBG] pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:07.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:06 smithi192 bash[24608]: cluster 2024-09-13T22:38:06.391530+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: cluster 2024-09-13T22:38:05.712675+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: cluster 2024-09-13T22:38:05.712787+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: audit 2024-09-13T22:38:05.969434+0000 mgr.y (mgr.14150) 362 : audit [DBG] from='client.24517 v1:172.21.15.192:0/4162288515' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: cephadm 2024-09-13T22:38:05.972557+0000 mgr.y (mgr.14150) 363 : cephadm [INF] Saving service prometheus spec with placement smithi192=a;count:1 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: audit 2024-09-13T22:38:05.979779+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: cluster 2024-09-13T22:38:06.324754+0000 mgr.y (mgr.14150) 364 : cluster [DBG] pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[29889]: cluster 2024-09-13T22:38:06.391530+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-13T22:38:07.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: cluster 2024-09-13T22:38:05.712675+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: cluster 2024-09-13T22:38:05.712787+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: audit 2024-09-13T22:38:05.969434+0000 mgr.y (mgr.14150) 362 : audit [DBG] from='client.24517 v1:172.21.15.192:0/4162288515' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: cephadm 2024-09-13T22:38:05.972557+0000 mgr.y (mgr.14150) 363 : cephadm [INF] Saving service prometheus spec with placement smithi192=a;count:1 2024-09-13T22:38:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: audit 2024-09-13T22:38:05.979779+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: cluster 2024-09-13T22:38:06.324754+0000 mgr.y (mgr.14150) 364 : cluster [DBG] pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:06 smithi178 bash[22840]: cluster 2024-09-13T22:38:06.391530+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-13T22:38:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:08 smithi192 bash[24608]: cluster 2024-09-13T22:38:07.724288+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-13T22:38:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:08 smithi192 bash[24608]: cluster 2024-09-13T22:38:08.325694+0000 mgr.y (mgr.14150) 365 : cluster [DBG] pgmap v368: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-09-13T22:38:09.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:08 smithi178 bash[22840]: cluster 2024-09-13T22:38:07.724288+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-13T22:38:09.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:08 smithi178 bash[22840]: cluster 2024-09-13T22:38:08.325694+0000 mgr.y (mgr.14150) 365 : cluster [DBG] pgmap v368: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-09-13T22:38:09.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:08 smithi178 bash[29889]: cluster 2024-09-13T22:38:07.724288+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-13T22:38:09.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:08 smithi178 bash[29889]: cluster 2024-09-13T22:38:08.325694+0000 mgr.y (mgr.14150) 365 : cluster [DBG] pgmap v368: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-09-13T22:38:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: audit 2024-09-13T22:38:09.865768+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: audit 2024-09-13T22:38:09.877105+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: audit 2024-09-13T22:38:09.879353+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: audit 2024-09-13T22:38:09.881060+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: audit 2024-09-13T22:38:09.890766+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: cephadm 2024-09-13T22:38:10.188751+0000 mgr.y (mgr.14150) 366 : cephadm [INF] Deploying daemon prometheus.a on smithi192 2024-09-13T22:38:11.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:10 smithi192 bash[24608]: cluster 2024-09-13T22:38:10.326834+0000 mgr.y (mgr.14150) 367 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 134 B/s wr, 1 op/s 2024-09-13T22:38:11.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: audit 2024-09-13T22:38:09.865768+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: audit 2024-09-13T22:38:09.877105+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: audit 2024-09-13T22:38:09.879353+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: audit 2024-09-13T22:38:09.881060+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: audit 2024-09-13T22:38:09.890766+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: cephadm 2024-09-13T22:38:10.188751+0000 mgr.y (mgr.14150) 366 : cephadm [INF] Deploying daemon prometheus.a on smithi192 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[29889]: cluster 2024-09-13T22:38:10.326834+0000 mgr.y (mgr.14150) 367 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 134 B/s wr, 1 op/s 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: audit 2024-09-13T22:38:09.865768+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: audit 2024-09-13T22:38:09.877105+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: audit 2024-09-13T22:38:09.879353+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: audit 2024-09-13T22:38:09.881060+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: audit 2024-09-13T22:38:09.890766+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: cephadm 2024-09-13T22:38:10.188751+0000 mgr.y (mgr.14150) 366 : cephadm [INF] Deploying daemon prometheus.a on smithi192 2024-09-13T22:38:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:10 smithi178 bash[22840]: cluster 2024-09-13T22:38:10.326834+0000 mgr.y (mgr.14150) 367 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 134 B/s wr, 1 op/s 2024-09-13T22:38:11.470 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:38:13.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:13 smithi192 bash[24608]: cluster 2024-09-13T22:38:12.327499+0000 mgr.y (mgr.14150) 368 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-09-13T22:38:13.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:13 smithi192 bash[24608]: cluster 2024-09-13T22:38:12.726240+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-13T22:38:13.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:13 smithi178 bash[22840]: cluster 2024-09-13T22:38:12.327499+0000 mgr.y (mgr.14150) 368 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-09-13T22:38:13.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:13 smithi178 bash[22840]: cluster 2024-09-13T22:38:12.726240+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-13T22:38:13.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:13 smithi178 bash[29889]: cluster 2024-09-13T22:38:12.327499+0000 mgr.y (mgr.14150) 368 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-09-13T22:38:13.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:13 smithi178 bash[29889]: cluster 2024-09-13T22:38:12.726240+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-13T22:38:13.948 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled node-exporter update... 2024-09-13T22:38:15.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:14 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:38:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:14 smithi192 bash[24608]: audit 2024-09-13T22:38:13.939348+0000 mgr.y (mgr.14150) 369 : audit [DBG] from='client.24523 v1:172.21.15.192:0/4061845752' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi178=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:15.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:14 smithi192 bash[24608]: cephadm 2024-09-13T22:38:13.942524+0000 mgr.y (mgr.14150) 370 : cephadm [INF] Saving service node-exporter spec with placement smithi178=a;smithi192=b;count:2 2024-09-13T22:38:15.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:14 smithi192 bash[24608]: audit 2024-09-13T22:38:13.949884+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:15.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:14 smithi192 bash[24608]: cluster 2024-09-13T22:38:14.328434+0000 mgr.y (mgr.14150) 371 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 128 B/s wr, 2 op/s 2024-09-13T22:38:15.306 DEBUG:teuthology.orchestra.run.smithi178:node-exporter.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-13T22:38:15.308 DEBUG:teuthology.orchestra.run.smithi192:node-exporter.b> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-13T22:38:15.310 INFO:tasks.cephadm:Adding alertmanager.a on smithi178 2024-09-13T22:38:15.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi178=a' 2024-09-13T22:38:15.316 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[29889]: audit 2024-09-13T22:38:13.939348+0000 mgr.y (mgr.14150) 369 : audit [DBG] from='client.24523 v1:172.21.15.192:0/4061845752' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi178=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[29889]: cephadm 2024-09-13T22:38:13.942524+0000 mgr.y (mgr.14150) 370 : cephadm [INF] Saving service node-exporter spec with placement smithi178=a;smithi192=b;count:2 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[29889]: audit 2024-09-13T22:38:13.949884+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[29889]: cluster 2024-09-13T22:38:14.328434+0000 mgr.y (mgr.14150) 371 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 128 B/s wr, 2 op/s 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[22840]: audit 2024-09-13T22:38:13.939348+0000 mgr.y (mgr.14150) 369 : audit [DBG] from='client.24523 v1:172.21.15.192:0/4061845752' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi178=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[22840]: cephadm 2024-09-13T22:38:13.942524+0000 mgr.y (mgr.14150) 370 : cephadm [INF] Saving service node-exporter spec with placement smithi178=a;smithi192=b;count:2 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[22840]: audit 2024-09-13T22:38:13.949884+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:15.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:14 smithi178 bash[22840]: cluster 2024-09-13T22:38:14.328434+0000 mgr.y (mgr.14150) 371 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 128 B/s wr, 2 op/s 2024-09-13T22:38:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:15 smithi192 bash[24608]: cluster 2024-09-13T22:38:14.770600+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:38:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:15 smithi192 bash[24608]: cluster 2024-09-13T22:38:14.773559+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:38:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:15 smithi192 bash[24608]: audit 2024-09-13T22:38:14.789946+0000 mgr.y (mgr.14150) 372 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:16.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:15 smithi178 bash[29889]: cluster 2024-09-13T22:38:14.770600+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:38:16.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:15 smithi178 bash[29889]: cluster 2024-09-13T22:38:14.773559+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:38:16.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:15 smithi178 bash[29889]: audit 2024-09-13T22:38:14.789946+0000 mgr.y (mgr.14150) 372 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:16.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:15 smithi178 bash[22840]: cluster 2024-09-13T22:38:14.770600+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:38:16.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:15 smithi178 bash[22840]: cluster 2024-09-13T22:38:14.773559+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:38:16.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:15 smithi178 bash[22840]: audit 2024-09-13T22:38:14.789946+0000 mgr.y (mgr.14150) 372 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:17.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:16 smithi192 bash[24608]: cluster 2024-09-13T22:38:16.329522+0000 mgr.y (mgr.14150) 373 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 118 B/s wr, 2 op/s 2024-09-13T22:38:17.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:16 smithi178 bash[29889]: cluster 2024-09-13T22:38:16.329522+0000 mgr.y (mgr.14150) 373 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 118 B/s wr, 2 op/s 2024-09-13T22:38:17.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:16 smithi178 bash[22840]: cluster 2024-09-13T22:38:16.329522+0000 mgr.y (mgr.14150) 373 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 118 B/s wr, 2 op/s 2024-09-13T22:38:19.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:18 smithi192 bash[24608]: cluster 2024-09-13T22:38:17.140393+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:19.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:18 smithi192 bash[24608]: cluster 2024-09-13T22:38:17.140500+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:19.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:18 smithi192 bash[24608]: cluster 2024-09-13T22:38:17.727997+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-13T22:38:19.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:18 smithi192 bash[24608]: cluster 2024-09-13T22:38:18.330357+0000 mgr.y (mgr.14150) 374 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[29889]: cluster 2024-09-13T22:38:17.140393+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[29889]: cluster 2024-09-13T22:38:17.140500+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[29889]: cluster 2024-09-13T22:38:17.727997+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[29889]: cluster 2024-09-13T22:38:18.330357+0000 mgr.y (mgr.14150) 374 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[22840]: cluster 2024-09-13T22:38:17.140393+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[22840]: cluster 2024-09-13T22:38:17.140500+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[22840]: cluster 2024-09-13T22:38:17.727997+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-13T22:38:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:18 smithi178 bash[22840]: cluster 2024-09-13T22:38:18.330357+0000 mgr.y (mgr.14150) 374 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:38:20.145 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:38:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:21 smithi192 bash[24608]: cluster 2024-09-13T22:38:20.331527+0000 mgr.y (mgr.14150) 375 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:38:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:21 smithi192 bash[24608]: cluster 2024-09-13T22:38:20.438964+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:21 smithi192 bash[24608]: cluster 2024-09-13T22:38:20.439077+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:21.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:21 smithi178 bash[29889]: cluster 2024-09-13T22:38:20.331527+0000 mgr.y (mgr.14150) 375 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:38:21.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:21 smithi178 bash[29889]: cluster 2024-09-13T22:38:20.438964+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:21.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:21 smithi178 bash[29889]: cluster 2024-09-13T22:38:20.439077+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:21 smithi178 bash[22840]: cluster 2024-09-13T22:38:20.331527+0000 mgr.y (mgr.14150) 375 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:38:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:21 smithi178 bash[22840]: cluster 2024-09-13T22:38:20.438964+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:21 smithi178 bash[22840]: cluster 2024-09-13T22:38:20.439077+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:22.156 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled alertmanager update... 2024-09-13T22:38:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:22 smithi192 bash[24608]: cluster 2024-09-13T22:38:21.400086+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-13T22:38:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:22 smithi192 bash[24608]: audit 2024-09-13T22:38:22.157656+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:22 smithi178 bash[29889]: cluster 2024-09-13T22:38:21.400086+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-13T22:38:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:22 smithi178 bash[29889]: audit 2024-09-13T22:38:22.157656+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:22 smithi178 bash[22840]: cluster 2024-09-13T22:38:21.400086+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-13T22:38:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:22 smithi178 bash[22840]: audit 2024-09-13T22:38:22.157656+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:23.329 DEBUG:teuthology.orchestra.run.smithi178:alertmanager.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-13T22:38:23.332 INFO:tasks.cephadm:Adding grafana.a on smithi192 2024-09-13T22:38:23.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi192=a' 2024-09-13T22:38:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:23 smithi192 bash[24608]: audit 2024-09-13T22:38:22.147691+0000 mgr.y (mgr.14150) 376 : audit [DBG] from='client.24529 v1:172.21.15.192:0/501622195' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi178=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:23 smithi192 bash[24608]: cephadm 2024-09-13T22:38:22.150514+0000 mgr.y (mgr.14150) 377 : cephadm [INF] Saving service alertmanager spec with placement smithi178=a;count:1 2024-09-13T22:38:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:23 smithi192 bash[24608]: cluster 2024-09-13T22:38:22.332228+0000 mgr.y (mgr.14150) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:23 smithi178 bash[29889]: audit 2024-09-13T22:38:22.147691+0000 mgr.y (mgr.14150) 376 : audit [DBG] from='client.24529 v1:172.21.15.192:0/501622195' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi178=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:23.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:23 smithi178 bash[29889]: cephadm 2024-09-13T22:38:22.150514+0000 mgr.y (mgr.14150) 377 : cephadm [INF] Saving service alertmanager spec with placement smithi178=a;count:1 2024-09-13T22:38:23.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:23 smithi178 bash[29889]: cluster 2024-09-13T22:38:22.332228+0000 mgr.y (mgr.14150) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:23 smithi178 bash[22840]: audit 2024-09-13T22:38:22.147691+0000 mgr.y (mgr.14150) 376 : audit [DBG] from='client.24529 v1:172.21.15.192:0/501622195' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi178=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:38:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:23 smithi178 bash[22840]: cephadm 2024-09-13T22:38:22.150514+0000 mgr.y (mgr.14150) 377 : cephadm [INF] Saving service alertmanager spec with placement smithi178=a;count:1 2024-09-13T22:38:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:23 smithi178 bash[22840]: cluster 2024-09-13T22:38:22.332228+0000 mgr.y (mgr.14150) 378 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:24.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:24 smithi192 bash[24608]: cluster 2024-09-13T22:38:22.894313+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:24.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:24 smithi192 bash[24608]: cluster 2024-09-13T22:38:22.894415+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:24.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:24 smithi192 bash[24608]: cluster 2024-09-13T22:38:23.411870+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-13T22:38:24.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:24 smithi178 bash[29889]: cluster 2024-09-13T22:38:22.894313+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:24.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:24 smithi178 bash[29889]: cluster 2024-09-13T22:38:22.894415+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:24.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:24 smithi178 bash[29889]: cluster 2024-09-13T22:38:23.411870+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-13T22:38:24.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:24 smithi178 bash[22840]: cluster 2024-09-13T22:38:22.894313+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:24.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:24 smithi178 bash[22840]: cluster 2024-09-13T22:38:22.894415+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:24.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:24 smithi178 bash[22840]: cluster 2024-09-13T22:38:23.411870+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-13T22:38:25.142 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.143 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:24 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:38:25.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.144 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.144 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.144 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.396 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.397 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.397 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.397 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.397 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.398 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.398 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.398 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.398 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:25.656 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:25 smithi192 systemd[1]: Started Ceph prometheus.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:38:25.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:25 smithi192 bash[24608]: cluster 2024-09-13T22:38:24.332900+0000 mgr.y (mgr.14150) 379 : cluster [DBG] pgmap v380: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:38:25.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:25 smithi178 bash[22840]: cluster 2024-09-13T22:38:24.332900+0000 mgr.y (mgr.14150) 379 : cluster [DBG] pgmap v380: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:38:25.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:25 smithi178 bash[29889]: cluster 2024-09-13T22:38:24.332900+0000 mgr.y (mgr.14150) 379 : cluster [DBG] pgmap v380: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:38:26.906 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[25468]: ignoring --setuser ceph since I am not root 2024-09-13T22:38:26.906 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[25468]: ignoring --setgroup ceph since I am not root 2024-09-13T22:38:26.906 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[25468]: debug 2024-09-13T22:38:26.646+0000 7f3a71b2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T22:38:26.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[24608]: audit 2024-09-13T22:38:24.798353+0000 mgr.y (mgr.14150) 380 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:26.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[24608]: audit 2024-09-13T22:38:25.451602+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[24608]: audit 2024-09-13T22:38:25.461326+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[24608]: audit 2024-09-13T22:38:25.469118+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[24608]: audit 2024-09-13T22:38:25.473505+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T22:38:26.907 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.554Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T22:38:26.907 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.554Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.554Z caller=main.go:570 level=info host_details="(Linux 5.15.0-121-generic #131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024 x86_64 smithi192 (none))" 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.554Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.554Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.556Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.556Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.559Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.559Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.561Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.561Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.867µs 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.561Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.561Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-13T22:38:26.908 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.561Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.541µs wal_replay_duration=304.589µs wbl_replay_duration=139ns total_replay_duration=351.41µs 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.562Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.562Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.562Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.563Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.563Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.563Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.589Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.064804ms db_storage=1.232µs remote_storage=1.246µs web_handler=405ns query_engine=680ns scrape=296.87µs scrape_sd=103.941µs notify=1.055µs notify_sd=2.923µs rules=26.264408ms tracing=7.71µs 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.589Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T22:38:26.909 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:26 smithi192 bash[49678]: ts=2024-09-13T22:38:26.589Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T22:38:26.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[23095]: ignoring --setuser ceph since I am not root 2024-09-13T22:38:26.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[23095]: ignoring --setgroup ceph since I am not root 2024-09-13T22:38:26.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[23095]: debug 2024-09-13T22:38:26.656+0000 7f08ffba6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T22:38:26.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[29889]: audit 2024-09-13T22:38:24.798353+0000 mgr.y (mgr.14150) 380 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:26.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[29889]: audit 2024-09-13T22:38:25.451602+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[29889]: audit 2024-09-13T22:38:25.461326+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[29889]: audit 2024-09-13T22:38:25.469118+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[29889]: audit 2024-09-13T22:38:25.473505+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[22840]: audit 2024-09-13T22:38:24.798353+0000 mgr.y (mgr.14150) 380 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[22840]: audit 2024-09-13T22:38:25.451602+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[22840]: audit 2024-09-13T22:38:25.461326+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[22840]: audit 2024-09-13T22:38:25.469118+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' 2024-09-13T22:38:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:26 smithi178 bash[22840]: audit 2024-09-13T22:38:25.473505+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T22:38:27.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:27 smithi192 bash[24608]: audit 2024-09-13T22:38:26.474406+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T22:38:27.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:27 smithi192 bash[24608]: cluster 2024-09-13T22:38:26.489592+0000 mon.a (mon.0) 791 : cluster [DBG] mgrmap e17: y(active, since 9m), standbys: x 2024-09-13T22:38:27.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:27 smithi178 bash[29889]: audit 2024-09-13T22:38:26.474406+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T22:38:27.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:27 smithi178 bash[29889]: cluster 2024-09-13T22:38:26.489592+0000 mon.a (mon.0) 791 : cluster [DBG] mgrmap e17: y(active, since 9m), standbys: x 2024-09-13T22:38:27.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:27 smithi178 bash[22840]: audit 2024-09-13T22:38:26.474406+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14150 v1:172.21.15.178:0/1306140760' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T22:38:27.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:27 smithi178 bash[22840]: cluster 2024-09-13T22:38:26.489592+0000 mon.a (mon.0) 791 : cluster [DBG] mgrmap e17: y(active, since 9m), standbys: x 2024-09-13T22:38:28.104 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:38:28.156 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:27 smithi192 bash[25468]: debug 2024-09-13T22:38:27.866+0000 7f3a71b2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T22:38:28.156 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:27 smithi192 bash[25468]: debug 2024-09-13T22:38:27.934+0000 7f3a71b2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T22:38:28.156 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:28 smithi192 bash[25468]: debug 2024-09-13T22:38:28.082+0000 7f3a71b2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T22:38:28.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:27 smithi178 bash[23095]: debug 2024-09-13T22:38:27.884+0000 7f08ffba6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T22:38:28.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:27 smithi178 bash[23095]: debug 2024-09-13T22:38:27.948+0000 7f08ffba6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T22:38:28.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:28 smithi178 bash[23095]: debug 2024-09-13T22:38:28.100+0000 7f08ffba6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T22:38:29.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:28 smithi178 bash[29889]: cluster 2024-09-13T22:38:27.731790+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-13T22:38:29.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:28 smithi178 bash[22840]: cluster 2024-09-13T22:38:27.731790+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-13T22:38:29.080 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:28 smithi178 bash[23095]: debug 2024-09-13T22:38:28.752+0000 7f08ffba6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T22:38:29.080 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:28 smithi178 bash[23095]: debug 2024-09-13T22:38:28.824+0000 7f08ffba6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T22:38:29.080 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:28 smithi178 bash[23095]: debug 2024-09-13T22:38:28.892+0000 7f08ffba6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T22:38:29.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:28 smithi192 bash[24608]: cluster 2024-09-13T22:38:27.731790+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-13T22:38:29.084 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:28 smithi192 bash[25468]: debug 2024-09-13T22:38:28.750+0000 7f3a71b2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T22:38:29.084 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:28 smithi192 bash[25468]: debug 2024-09-13T22:38:28.822+0000 7f3a71b2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T22:38:29.084 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:28 smithi192 bash[25468]: debug 2024-09-13T22:38:28.890+0000 7f3a71b2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T22:38:29.084 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.082+0000 7f3a71b2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T22:38:29.406 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.162+0000 7f3a71b2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T22:38:29.406 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.234+0000 7f3a71b2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T22:38:29.428 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.076+0000 7f08ffba6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T22:38:29.429 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.156+0000 7f08ffba6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T22:38:29.429 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.224+0000 7f08ffba6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T22:38:29.743 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.480+0000 7f08ffba6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T22:38:29.743 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.548+0000 7f08ffba6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T22:38:29.764 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.498+0000 7f3a71b2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T22:38:29.764 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.566+0000 7f3a71b2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T22:38:29.764 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.762+0000 7f3a71b2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T22:38:30.070 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.740+0000 7f08ffba6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T22:38:30.070 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:29 smithi178 bash[23095]: debug 2024-09-13T22:38:29.820+0000 7f08ffba6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T22:38:30.093 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:29 smithi192 bash[25468]: debug 2024-09-13T22:38:29.838+0000 7f3a71b2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T22:38:30.406 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:30 smithi192 bash[25468]: debug 2024-09-13T22:38:30.090+0000 7f3a71b2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T22:38:30.406 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:30 smithi192 bash[25468]: debug 2024-09-13T22:38:30.266+0000 7f3a71b2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:38:30.408 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:30 smithi178 bash[23095]: debug 2024-09-13T22:38:30.068+0000 7f08ffba6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T22:38:30.408 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:30 smithi178 bash[23095]: debug 2024-09-13T22:38:30.244+0000 7f08ffba6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:38:30.668 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:30 smithi178 bash[23095]: debug 2024-09-13T22:38:30.404+0000 7f08ffba6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T22:38:30.668 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:30 smithi178 bash[23095]: debug 2024-09-13T22:38:30.484+0000 7f08ffba6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T22:38:30.695 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:30 smithi192 bash[25468]: debug 2024-09-13T22:38:30.430+0000 7f3a71b2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T22:38:30.695 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:30 smithi192 bash[25468]: debug 2024-09-13T22:38:30.510+0000 7f3a71b2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T22:38:30.695 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:30 smithi192 bash[25468]: debug 2024-09-13T22:38:30.694+0000 7f3a71b2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:38:30.928 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:30 smithi178 bash[23095]: debug 2024-09-13T22:38:30.668+0000 7f08ffba6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T22:38:31.811 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:31 smithi192 bash[25468]: debug 2024-09-13T22:38:31.510+0000 7f3a71b2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T22:38:31.811 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:31 smithi192 bash[25468]: debug 2024-09-13T22:38:31.586+0000 7f3a71b2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T22:38:31.812 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:31 smithi192 bash[25468]: debug 2024-09-13T22:38:31.810+0000 7f3a71b2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T22:38:31.834 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:31 smithi178 bash[23095]: debug 2024-09-13T22:38:31.532+0000 7f08ffba6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T22:38:31.834 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:31 smithi178 bash[23095]: debug 2024-09-13T22:38:31.608+0000 7f08ffba6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T22:38:32.156 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:31 smithi192 bash[25468]: debug 2024-09-13T22:38:31.882+0000 7f3a71b2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T22:38:32.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:31 smithi178 bash[23095]: debug 2024-09-13T22:38:31.832+0000 7f08ffba6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T22:38:32.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:31 smithi178 bash[23095]: debug 2024-09-13T22:38:31.904+0000 7f08ffba6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T22:38:32.609 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: debug 2024-09-13T22:38:32.292+0000 7f08ffba6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T22:38:32.609 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: debug 2024-09-13T22:38:32.372+0000 7f08ffba6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T22:38:32.609 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: [13/Sep/2024:22:38:32] ENGINE Bus STARTING 2024-09-13T22:38:32.609 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: CherryPy Checker: 2024-09-13T22:38:32.609 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: The Application mounted at '' has an empty config. 2024-09-13T22:38:32.609 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.376673+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.376989+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.379893+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.380400+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.382070+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.382679+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.396553+0000 mon.a (mon.0) 795 : cluster [INF] Active manager daemon y restarted 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.397120+0000 mon.a (mon.0) 796 : cluster [INF] Activating manager daemon y 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.407460+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.407900+0000 mon.a (mon.0) 798 : cluster [DBG] mgrmap e18: y(active, starting, since 0.0109915s), standbys: x 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.411833+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.412022+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:38:32.610 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.412183+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.413498+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.413712+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.413920+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.414182+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.414426+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.414670+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.376673+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.376989+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-13T22:38:32.611 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.379893+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.380400+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.382070+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.382679+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.396553+0000 mon.a (mon.0) 795 : cluster [INF] Active manager daemon y restarted 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.397120+0000 mon.a (mon.0) 796 : cluster [INF] Activating manager daemon y 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.407460+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.407900+0000 mon.a (mon.0) 798 : cluster [DBG] mgrmap e18: y(active, starting, since 0.0109915s), standbys: x 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.411833+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.412022+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.412183+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.413498+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.413712+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.413920+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:38:32.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.414182+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.414426+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.414670+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.414896+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.415129+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.415358+0000 mon.b (mon.1) 45 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.415590+0000 mon.b (mon.1) 46 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.415831+0000 mon.b (mon.1) 47 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.415997+0000 mon.b (mon.1) 48 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:38:32.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[29889]: audit 2024-09-13T22:38:32.416543+0000 mon.b (mon.1) 49 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.414896+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.415129+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.415358+0000 mon.b (mon.1) 45 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.415590+0000 mon.b (mon.1) 46 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.415831+0000 mon.b (mon.1) 47 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.415997+0000 mon.b (mon.1) 48 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:38:32.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[22840]: audit 2024-09-13T22:38:32.416543+0000 mon.b (mon.1) 49 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: debug 2024-09-13T22:38:32.270+0000 7f3a71b2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: debug 2024-09-13T22:38:32.350+0000 7f3a71b2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: [13/Sep/2024:22:38:32] ENGINE Bus STARTING 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: CherryPy Checker: 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: The Application mounted at '' has an empty config. 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: [13/Sep/2024:22:38:32] ENGINE Serving on http://:::9283 2024-09-13T22:38:32.627 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[25468]: [13/Sep/2024:22:38:32] ENGINE Bus STARTED 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.376673+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.376989+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.379893+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.380400+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.382070+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.382679+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.192:0/2700554849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.396553+0000 mon.a (mon.0) 795 : cluster [INF] Active manager daemon y restarted 2024-09-13T22:38:32.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.397120+0000 mon.a (mon.0) 796 : cluster [INF] Activating manager daemon y 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.407460+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.407900+0000 mon.a (mon.0) 798 : cluster [DBG] mgrmap e18: y(active, starting, since 0.0109915s), standbys: x 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.411833+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.412022+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.412183+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.413498+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.413712+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.413920+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.414182+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.414426+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.414670+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.414896+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T22:38:32.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.415129+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T22:38:32.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.415358+0000 mon.b (mon.1) 45 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T22:38:32.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.415590+0000 mon.b (mon.1) 46 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T22:38:32.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.415831+0000 mon.b (mon.1) 47 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T22:38:32.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.415997+0000 mon.b (mon.1) 48 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T22:38:32.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:32 smithi192 bash[24608]: audit 2024-09-13T22:38:32.416543+0000 mon.b (mon.1) 49 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T22:38:32.928 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: [13/Sep/2024:22:38:32] ENGINE Serving on http://:::9283 2024-09-13T22:38:32.928 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:32 smithi178 bash[23095]: [13/Sep/2024:22:38:32] ENGINE Bus STARTED 2024-09-13T22:38:33.457 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled grafana update... 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.441943+0000 mon.a (mon.0) 799 : cluster [INF] Manager daemon y is now available 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.469301+0000 mon.b (mon.1) 50 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.485398+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.493064+0000 mon.b (mon.1) 51 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.508379+0000 mon.b (mon.1) 52 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.508486+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.530229+0000 mon.b (mon.1) 53 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: audit 2024-09-13T22:38:32.530328+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[22840]: cluster 2024-09-13T22:38:32.729231+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-13T22:38:33.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.441943+0000 mon.a (mon.0) 799 : cluster [INF] Manager daemon y is now available 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.469301+0000 mon.b (mon.1) 50 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.485398+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.493064+0000 mon.b (mon.1) 51 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.508379+0000 mon.b (mon.1) 52 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.508486+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.530229+0000 mon.b (mon.1) 53 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: audit 2024-09-13T22:38:32.530328+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:33 smithi192 bash[24608]: cluster 2024-09-13T22:38:32.729231+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.441943+0000 mon.a (mon.0) 799 : cluster [INF] Manager daemon y is now available 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.469301+0000 mon.b (mon.1) 50 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:38:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.485398+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.493064+0000 mon.b (mon.1) 51 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:38:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.508379+0000 mon.b (mon.1) 52 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:38:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.508486+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T22:38:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.530229+0000 mon.b (mon.1) 53 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:38:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: audit 2024-09-13T22:38:32.530328+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T22:38:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:33 smithi178 bash[29889]: cluster 2024-09-13T22:38:32.729231+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-13T22:38:34.265 DEBUG:teuthology.orchestra.run.smithi192:grafana.a> sudo journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@grafana.a.service 2024-09-13T22:38:34.268 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T22:38:34.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T22:38:34.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:34 smithi192 bash[24608]: cluster 2024-09-13T22:38:33.445027+0000 mon.a (mon.0) 804 : cluster [DBG] mgrmap e19: y(active, since 1.04812s), standbys: x 2024-09-13T22:38:34.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:34 smithi192 bash[24608]: cephadm 2024-09-13T22:38:33.452351+0000 mgr.y (mgr.14613) 3 : cephadm [INF] Saving service grafana spec with placement smithi192=a;count:1 2024-09-13T22:38:34.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:34 smithi192 bash[24608]: audit 2024-09-13T22:38:33.458813+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:34 smithi192 bash[24608]: audit 2024-09-13T22:38:33.461936+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:34 smithi192 bash[24608]: audit 2024-09-13T22:38:33.467260+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[29889]: cluster 2024-09-13T22:38:33.445027+0000 mon.a (mon.0) 804 : cluster [DBG] mgrmap e19: y(active, since 1.04812s), standbys: x 2024-09-13T22:38:34.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[29889]: cephadm 2024-09-13T22:38:33.452351+0000 mgr.y (mgr.14613) 3 : cephadm [INF] Saving service grafana spec with placement smithi192=a;count:1 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[29889]: audit 2024-09-13T22:38:33.458813+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[29889]: audit 2024-09-13T22:38:33.461936+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[29889]: audit 2024-09-13T22:38:33.467260+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[22840]: cluster 2024-09-13T22:38:33.445027+0000 mon.a (mon.0) 804 : cluster [DBG] mgrmap e19: y(active, since 1.04812s), standbys: x 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[22840]: cephadm 2024-09-13T22:38:33.452351+0000 mgr.y (mgr.14613) 3 : cephadm [INF] Saving service grafana spec with placement smithi192=a;count:1 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[22840]: audit 2024-09-13T22:38:33.458813+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[22840]: audit 2024-09-13T22:38:33.461936+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:34 smithi178 bash[22840]: audit 2024-09-13T22:38:33.467260+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:35.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:34 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:38:35.904 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[29889]: cluster 2024-09-13T22:38:34.414822+0000 mgr.y (mgr.14613) 4 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:35.904 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[29889]: cluster 2024-09-13T22:38:34.474160+0000 mon.a (mon.0) 808 : cluster [DBG] mgrmap e20: y(active, since 2s), standbys: x 2024-09-13T22:38:35.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[22840]: cluster 2024-09-13T22:38:34.414822+0000 mgr.y (mgr.14613) 4 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:35.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[22840]: cluster 2024-09-13T22:38:34.474160+0000 mon.a (mon.0) 808 : cluster [DBG] mgrmap e20: y(active, since 2s), standbys: x 2024-09-13T22:38:35.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:35 smithi192 bash[24608]: cluster 2024-09-13T22:38:34.414822+0000 mgr.y (mgr.14613) 4 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:35.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:35 smithi192 bash[24608]: cluster 2024-09-13T22:38:34.474160+0000 mon.a (mon.0) 808 : cluster [DBG] mgrmap e20: y(active, since 2s), standbys: x 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: debug 2024-09-13T22:38:35.904+0000 7f0858318640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:22:38:35] ENGINE Error in HTTPServer.serve 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: Traceback (most recent call last): 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: self._connections.run(self.expiration_interval) 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: self._run(expiration_interval) 2024-09-13T22:38:36.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: s = self.context.wrap_socket( 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: return self.sslsocket_class._create( 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: self.do_handshake() 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: self._sslobj.do_handshake() 2024-09-13T22:38:36.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:35 smithi178 bash[23095]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:38:36.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:36 smithi192 bash[24608]: audit 2024-09-13T22:38:34.809531+0000 mgr.y (mgr.14613) 5 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:36.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:36 smithi178 bash[29889]: audit 2024-09-13T22:38:34.809531+0000 mgr.y (mgr.14613) 5 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:36.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:36 smithi178 bash[22840]: audit 2024-09-13T22:38:34.809531+0000 mgr.y (mgr.14613) 5 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cephadm 2024-09-13T22:38:35.783709+0000 mgr.y (mgr.14613) 6 : cephadm [INF] [13/Sep/2024:22:38:35] ENGINE Bus STARTING 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cephadm 2024-09-13T22:38:35.903954+0000 mgr.y (mgr.14613) 7 : cephadm [INF] [13/Sep/2024:22:38:35] ENGINE Serving on https://172.21.15.178:7150 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cephadm 2024-09-13T22:38:35.907851+0000 mgr.y (mgr.14613) 8 : cephadm [ERR] [13/Sep/2024:22:38:35] ENGINE Error in HTTPServer.serve 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: Traceback (most recent call last): 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: self._connections.run(self.expiration_interval) 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: self._run(expiration_interval) 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:38:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: s = self.context.wrap_socket( 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: return self.sslsocket_class._create( 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: self.do_handshake() 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: self._sslobj.do_handshake() 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cephadm 2024-09-13T22:38:36.005307+0000 mgr.y (mgr.14613) 9 : cephadm [INF] [13/Sep/2024:22:38:36] ENGINE Serving on http://172.21.15.178:8765 2024-09-13T22:38:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cephadm 2024-09-13T22:38:36.005421+0000 mgr.y (mgr.14613) 10 : cephadm [INF] [13/Sep/2024:22:38:36] ENGINE Bus STARTED 2024-09-13T22:38:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cluster 2024-09-13T22:38:36.415440+0000 mgr.y (mgr.14613) 11 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:37 smithi192 bash[24608]: cluster 2024-09-13T22:38:36.487686+0000 mon.a (mon.0) 809 : cluster [DBG] mgrmap e21: y(active, since 4s), standbys: x 2024-09-13T22:38:37.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cephadm 2024-09-13T22:38:35.783709+0000 mgr.y (mgr.14613) 6 : cephadm [INF] [13/Sep/2024:22:38:35] ENGINE Bus STARTING 2024-09-13T22:38:37.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cephadm 2024-09-13T22:38:35.903954+0000 mgr.y (mgr.14613) 7 : cephadm [INF] [13/Sep/2024:22:38:35] ENGINE Serving on https://172.21.15.178:7150 2024-09-13T22:38:37.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cephadm 2024-09-13T22:38:35.907851+0000 mgr.y (mgr.14613) 8 : cephadm [ERR] [13/Sep/2024:22:38:35] ENGINE Error in HTTPServer.serve 2024-09-13T22:38:37.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: Traceback (most recent call last): 2024-09-13T22:38:37.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: self._connections.run(self.expiration_interval) 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: self._run(expiration_interval) 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: s = self.context.wrap_socket( 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: return self.sslsocket_class._create( 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: self.do_handshake() 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:38:37.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: self._sslobj.do_handshake() 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cephadm 2024-09-13T22:38:36.005307+0000 mgr.y (mgr.14613) 9 : cephadm [INF] [13/Sep/2024:22:38:36] ENGINE Serving on http://172.21.15.178:8765 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cephadm 2024-09-13T22:38:36.005421+0000 mgr.y (mgr.14613) 10 : cephadm [INF] [13/Sep/2024:22:38:36] ENGINE Bus STARTED 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cluster 2024-09-13T22:38:36.415440+0000 mgr.y (mgr.14613) 11 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[22840]: cluster 2024-09-13T22:38:36.487686+0000 mon.a (mon.0) 809 : cluster [DBG] mgrmap e21: y(active, since 4s), standbys: x 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cephadm 2024-09-13T22:38:35.783709+0000 mgr.y (mgr.14613) 6 : cephadm [INF] [13/Sep/2024:22:38:35] ENGINE Bus STARTING 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cephadm 2024-09-13T22:38:35.903954+0000 mgr.y (mgr.14613) 7 : cephadm [INF] [13/Sep/2024:22:38:35] ENGINE Serving on https://172.21.15.178:7150 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cephadm 2024-09-13T22:38:35.907851+0000 mgr.y (mgr.14613) 8 : cephadm [ERR] [13/Sep/2024:22:38:35] ENGINE Error in HTTPServer.serve 2024-09-13T22:38:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: Traceback (most recent call last): 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: self._connections.run(self.expiration_interval) 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: self._run(expiration_interval) 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T22:38:37.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T22:38:37.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: s = self.context.wrap_socket( 2024-09-13T22:38:37.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T22:38:37.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: return self.sslsocket_class._create( 2024-09-13T22:38:37.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T22:38:37.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: self.do_handshake() 2024-09-13T22:38:37.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T22:38:37.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: self._sslobj.do_handshake() 2024-09-13T22:38:37.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T22:38:37.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cephadm 2024-09-13T22:38:36.005307+0000 mgr.y (mgr.14613) 9 : cephadm [INF] [13/Sep/2024:22:38:36] ENGINE Serving on http://172.21.15.178:8765 2024-09-13T22:38:37.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cephadm 2024-09-13T22:38:36.005421+0000 mgr.y (mgr.14613) 10 : cephadm [INF] [13/Sep/2024:22:38:36] ENGINE Bus STARTED 2024-09-13T22:38:37.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cluster 2024-09-13T22:38:36.415440+0000 mgr.y (mgr.14613) 11 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:37.935 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:37 smithi178 bash[29889]: cluster 2024-09-13T22:38:36.487686+0000 mon.a (mon.0) 809 : cluster [DBG] mgrmap e21: y(active, since 4s), standbys: x 2024-09-13T22:38:38.506 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:38:39.832 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[29889]: cluster 2024-09-13T22:38:38.416153+0000 mgr.y (mgr.14613) 12 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:39.832 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[29889]: audit 2024-09-13T22:38:38.598622+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[29889]: audit 2024-09-13T22:38:38.605335+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[29889]: audit 2024-09-13T22:38:38.867054+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[29889]: audit 2024-09-13T22:38:38.875564+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[22840]: cluster 2024-09-13T22:38:38.416153+0000 mgr.y (mgr.14613) 12 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[22840]: audit 2024-09-13T22:38:38.598622+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[22840]: audit 2024-09-13T22:38:38.605335+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[22840]: audit 2024-09-13T22:38:38.867054+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.833 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:39 smithi178 bash[22840]: audit 2024-09-13T22:38:38.875564+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:39 smithi192 bash[24608]: cluster 2024-09-13T22:38:38.416153+0000 mgr.y (mgr.14613) 12 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-09-13T22:38:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:39 smithi192 bash[24608]: audit 2024-09-13T22:38:38.598622+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:39 smithi192 bash[24608]: audit 2024-09-13T22:38:38.605335+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:39 smithi192 bash[24608]: audit 2024-09-13T22:38:38.867054+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:39.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:39 smithi192 bash[24608]: audit 2024-09-13T22:38:38.875564+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:40.237 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-09-13T22:38:40.238 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDwvuRmDGQKDhAAC0ezQpuPFwLOKV8mtKmb8Q== 2024-09-13T22:38:40.934 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-13T22:38:40.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T22:38:40.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T22:38:40.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T22:38:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.732735+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.744164+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.752085+0000 mon.b (mon.1) 54 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.752329+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.754108+0000 mon.b (mon.1) 55 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.754157+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.755691+0000 mon.b (mon.1) 56 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.755741+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.757263+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.757267+0000 mon.b (mon.1) 57 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:39.758142+0000 mgr.y (mgr.14613) 13 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 3804M 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:39.764200+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cluster 2024-09-13T22:38:39.827424+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cluster 2024-09-13T22:38:39.827521+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.021825+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.030693+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.035559+0000 mon.b (mon.1) 58 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.035745+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.037470+0000 mon.b (mon.1) 59 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.037569+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.039425+0000 mon.b (mon.1) 60 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.039476+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.041637+0000 mon.b (mon.1) 61 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.041956+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.043475+0000 mgr.y (mgr.14613) 14 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 3804M 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.048891+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.054728+0000 mon.b (mon.1) 62 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.055987+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.057057+0000 mgr.y (mgr.14613) 15 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:38:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.057361+0000 mgr.y (mgr.14613) 16 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.200700+0000 mgr.y (mgr.14613) 17 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.209329+0000 mgr.y (mgr.14613) 18 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.234958+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.178:0/3212410132' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.235340+0000 mon.a (mon.0) 828 : audit [INF] from='client.24542 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.237644+0000 mon.a (mon.0) 829 : audit [INF] from='client.24542 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.313977+0000 mgr.y (mgr.14613) 19 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.355973+0000 mgr.y (mgr.14613) 20 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cluster 2024-09-13T22:38:40.417625+0000 mgr.y (mgr.14613) 21 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.421252+0000 mgr.y (mgr.14613) 22 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:38:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:40 smithi192 bash[24608]: audit 2024-09-13T22:38:40.728108+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.732735+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.744164+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.752085+0000 mon.b (mon.1) 54 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.752329+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.754108+0000 mon.b (mon.1) 55 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.754157+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.755691+0000 mon.b (mon.1) 56 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.755741+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.757263+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.757267+0000 mon.b (mon.1) 57 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:39.758142+0000 mgr.y (mgr.14613) 13 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 3804M 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:39.764200+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cluster 2024-09-13T22:38:39.827424+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cluster 2024-09-13T22:38:39.827521+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.021825+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.030693+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.035559+0000 mon.b (mon.1) 58 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.035745+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.037470+0000 mon.b (mon.1) 59 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.037569+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.732735+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.744164+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.752085+0000 mon.b (mon.1) 54 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.752329+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.754108+0000 mon.b (mon.1) 55 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.754157+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.755691+0000 mon.b (mon.1) 56 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.755741+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.757263+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.757267+0000 mon.b (mon.1) 57 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:39.758142+0000 mgr.y (mgr.14613) 13 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 3804M 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:39.764200+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cluster 2024-09-13T22:38:39.827424+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cluster 2024-09-13T22:38:39.827521+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.021825+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.030693+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.035559+0000 mon.b (mon.1) 58 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.035745+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.037470+0000 mon.b (mon.1) 59 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.037569+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.039425+0000 mon.b (mon.1) 60 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.039476+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.041637+0000 mon.b (mon.1) 61 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.041956+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.043475+0000 mgr.y (mgr.14613) 14 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 3804M 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.048891+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.054728+0000 mon.b (mon.1) 62 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.055987+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.057057+0000 mgr.y (mgr.14613) 15 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.057361+0000 mgr.y (mgr.14613) 16 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.200700+0000 mgr.y (mgr.14613) 17 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.209329+0000 mgr.y (mgr.14613) 18 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.234958+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.178:0/3212410132' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.235340+0000 mon.a (mon.0) 828 : audit [INF] from='client.24542 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:41.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.237644+0000 mon.a (mon.0) 829 : audit [INF] from='client.24542 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.313977+0000 mgr.y (mgr.14613) 19 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.355973+0000 mgr.y (mgr.14613) 20 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cluster 2024-09-13T22:38:40.417625+0000 mgr.y (mgr.14613) 21 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.421252+0000 mgr.y (mgr.14613) 22 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[22840]: audit 2024-09-13T22:38:40.728108+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.039425+0000 mon.b (mon.1) 60 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.184 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.039476+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.041637+0000 mon.b (mon.1) 61 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.041956+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.043475+0000 mgr.y (mgr.14613) 14 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 3804M 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.048891+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.054728+0000 mon.b (mon.1) 62 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.055987+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.057057+0000 mgr.y (mgr.14613) 15 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.conf 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.057361+0000 mgr.y (mgr.14613) 16 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.200700+0000 mgr.y (mgr.14613) 17 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.209329+0000 mgr.y (mgr.14613) 18 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.234958+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.178:0/3212410132' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.235340+0000 mon.a (mon.0) 828 : audit [INF] from='client.24542 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:41.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.237644+0000 mon.a (mon.0) 829 : audit [INF] from='client.24542 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T22:38:41.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.313977+0000 mgr.y (mgr.14613) 19 : cephadm [INF] Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:38:41.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.355973+0000 mgr.y (mgr.14613) 20 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-09-13T22:38:41.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cluster 2024-09-13T22:38:40.417625+0000 mgr.y (mgr.14613) 21 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-13T22:38:41.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.421252+0000 mgr.y (mgr.14613) 22 : cephadm [INF] Updating smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:38:41.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:40 smithi178 bash[29889]: audit 2024-09-13T22:38:40.728108+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:41.678 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:41 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.667628+0000 mgr.y (mgr.14613) 23 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:38:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: audit 2024-09-13T22:38:40.742609+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: cluster 2024-09-13T22:38:40.768157+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-13T22:38:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: audit 2024-09-13T22:38:40.795292+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: audit 2024-09-13T22:38:40.806653+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: audit 2024-09-13T22:38:40.815856+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:41 smithi192 bash[24608]: cephadm 2024-09-13T22:38:40.845052+0000 mgr.y (mgr.14613) 24 : cephadm [INF] Deploying daemon alertmanager.a on smithi178 2024-09-13T22:38:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.667628+0000 mgr.y (mgr.14613) 23 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:38:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: audit 2024-09-13T22:38:40.742609+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: cluster 2024-09-13T22:38:40.768157+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-13T22:38:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: audit 2024-09-13T22:38:40.795292+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: audit 2024-09-13T22:38:40.806653+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: audit 2024-09-13T22:38:40.815856+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[29889]: cephadm 2024-09-13T22:38:40.845052+0000 mgr.y (mgr.14613) 24 : cephadm [INF] Deploying daemon alertmanager.a on smithi178 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.667628+0000 mgr.y (mgr.14613) 23 : cephadm [INF] Updating smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: audit 2024-09-13T22:38:40.742609+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: cluster 2024-09-13T22:38:40.768157+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: audit 2024-09-13T22:38:40.795292+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: audit 2024-09-13T22:38:40.806653+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: audit 2024-09-13T22:38:40.815856+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:42.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:41 smithi178 bash[22840]: cephadm 2024-09-13T22:38:40.845052+0000 mgr.y (mgr.14613) 24 : cephadm [INF] Deploying daemon alertmanager.a on smithi178 2024-09-13T22:38:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:42 smithi192 bash[24608]: cluster 2024-09-13T22:38:42.418316+0000 mgr.y (mgr.14613) 25 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-13T22:38:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:42 smithi192 bash[24608]: cluster 2024-09-13T22:38:42.734582+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-13T22:38:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:42 smithi178 bash[29889]: cluster 2024-09-13T22:38:42.418316+0000 mgr.y (mgr.14613) 25 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-13T22:38:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:42 smithi178 bash[29889]: cluster 2024-09-13T22:38:42.734582+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-13T22:38:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:42 smithi178 bash[22840]: cluster 2024-09-13T22:38:42.418316+0000 mgr.y (mgr.14613) 25 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-13T22:38:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:42 smithi178 bash[22840]: cluster 2024-09-13T22:38:42.734582+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-13T22:38:44.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:43 smithi192 bash[24608]: cluster 2024-09-13T22:38:43.748264+0000 osd.3 (osd.3) 9 : cluster [DBG] 3.b scrub starts 2024-09-13T22:38:44.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:43 smithi192 bash[24608]: cluster 2024-09-13T22:38:43.751966+0000 osd.3 (osd.3) 10 : cluster [DBG] 3.b scrub ok 2024-09-13T22:38:44.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:43 smithi178 bash[29889]: cluster 2024-09-13T22:38:43.748264+0000 osd.3 (osd.3) 9 : cluster [DBG] 3.b scrub starts 2024-09-13T22:38:44.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:43 smithi178 bash[29889]: cluster 2024-09-13T22:38:43.751966+0000 osd.3 (osd.3) 10 : cluster [DBG] 3.b scrub ok 2024-09-13T22:38:44.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:43 smithi178 bash[22840]: cluster 2024-09-13T22:38:43.748264+0000 osd.3 (osd.3) 9 : cluster [DBG] 3.b scrub starts 2024-09-13T22:38:44.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:43 smithi178 bash[22840]: cluster 2024-09-13T22:38:43.751966+0000 osd.3 (osd.3) 10 : cluster [DBG] 3.b scrub ok 2024-09-13T22:38:45.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:44 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:38:45.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:44 smithi192 bash[24608]: cluster 2024-09-13T22:38:44.419839+0000 mgr.y (mgr.14613) 26 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:38:45.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:44 smithi178 bash[29889]: cluster 2024-09-13T22:38:44.419839+0000 mgr.y (mgr.14613) 26 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:38:45.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:44 smithi178 bash[22840]: cluster 2024-09-13T22:38:44.419839+0000 mgr.y (mgr.14613) 26 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:38:45.744 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:38:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:45 smithi192 bash[24608]: cluster 2024-09-13T22:38:44.580121+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-13T22:38:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:45 smithi192 bash[24608]: cluster 2024-09-13T22:38:44.582271+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-13T22:38:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:45 smithi192 bash[24608]: cluster 2024-09-13T22:38:44.648551+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:38:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:45 smithi192 bash[24608]: cluster 2024-09-13T22:38:44.652572+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:38:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:45 smithi192 bash[24608]: audit 2024-09-13T22:38:44.820894+0000 mgr.y (mgr.14613) 27 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:46.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[29889]: cluster 2024-09-13T22:38:44.580121+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-13T22:38:46.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[29889]: cluster 2024-09-13T22:38:44.582271+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-13T22:38:46.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[29889]: cluster 2024-09-13T22:38:44.648551+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:38:46.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[29889]: cluster 2024-09-13T22:38:44.652572+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:38:46.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[29889]: audit 2024-09-13T22:38:44.820894+0000 mgr.y (mgr.14613) 27 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[22840]: cluster 2024-09-13T22:38:44.580121+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.c scrub starts 2024-09-13T22:38:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[22840]: cluster 2024-09-13T22:38:44.582271+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.c scrub ok 2024-09-13T22:38:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[22840]: cluster 2024-09-13T22:38:44.648551+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:38:46.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[22840]: cluster 2024-09-13T22:38:44.652572+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:38:46.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:45 smithi178 bash[22840]: audit 2024-09-13T22:38:44.820894+0000 mgr.y (mgr.14613) 27 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:47.075 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[22840]: cluster 2024-09-13T22:38:46.003530+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:38:47.075 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[22840]: cluster 2024-09-13T22:38:46.006063+0000 osd.1 (osd.1) 16 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:38:47.075 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[22840]: cluster 2024-09-13T22:38:46.420597+0000 mgr.y (mgr.14613) 28 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:38:47.075 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[22840]: cluster 2024-09-13T22:38:46.611123+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T22:38:47.075 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[22840]: cluster 2024-09-13T22:38:46.613456+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T22:38:47.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[29889]: cluster 2024-09-13T22:38:46.003530+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:38:47.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[29889]: cluster 2024-09-13T22:38:46.006063+0000 osd.1 (osd.1) 16 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:38:47.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[29889]: cluster 2024-09-13T22:38:46.420597+0000 mgr.y (mgr.14613) 28 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:38:47.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[29889]: cluster 2024-09-13T22:38:46.611123+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T22:38:47.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:46 smithi178 bash[29889]: cluster 2024-09-13T22:38:46.613456+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T22:38:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:46 smithi192 bash[24608]: cluster 2024-09-13T22:38:46.003530+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:38:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:46 smithi192 bash[24608]: cluster 2024-09-13T22:38:46.006063+0000 osd.1 (osd.1) 16 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:38:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:46 smithi192 bash[24608]: cluster 2024-09-13T22:38:46.420597+0000 mgr.y (mgr.14613) 28 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:38:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:46 smithi192 bash[24608]: cluster 2024-09-13T22:38:46.611123+0000 osd.0 (osd.0) 15 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T22:38:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:46 smithi192 bash[24608]: cluster 2024-09-13T22:38:46.613456+0000 osd.0 (osd.0) 16 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T22:38:47.278 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-09-13T22:38:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD3vuRmgHNpEBAA1PcLzN0SADun/hCBs3zX0w== 2024-09-13T22:38:47.983 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-13T22:38:47.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T22:38:47.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T22:38:48.006 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T22:38:48.006 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T22:38:48.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T22:38:48.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[22840]: audit 2024-09-13T22:38:47.274590+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.192:0/2171388379' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[22840]: audit 2024-09-13T22:38:47.275025+0000 mon.a (mon.0) 837 : audit [INF] from='client.24580 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[22840]: audit 2024-09-13T22:38:47.278225+0000 mon.a (mon.0) 838 : audit [INF] from='client.24580 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[22840]: audit 2024-09-13T22:38:47.516141+0000 mon.b (mon.1) 65 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[29889]: audit 2024-09-13T22:38:47.274590+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.192:0/2171388379' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[29889]: audit 2024-09-13T22:38:47.275025+0000 mon.a (mon.0) 837 : audit [INF] from='client.24580 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[29889]: audit 2024-09-13T22:38:47.278225+0000 mon.a (mon.0) 838 : audit [INF] from='client.24580 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T22:38:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:47 smithi178 bash[29889]: audit 2024-09-13T22:38:47.516141+0000 mon.b (mon.1) 65 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:38:48.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:47 smithi192 bash[24608]: audit 2024-09-13T22:38:47.274590+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.192:0/2171388379' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:48.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:47 smithi192 bash[24608]: audit 2024-09-13T22:38:47.275025+0000 mon.a (mon.0) 837 : audit [INF] from='client.24580 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T22:38:48.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:47 smithi192 bash[24608]: audit 2024-09-13T22:38:47.278225+0000 mon.a (mon.0) 838 : audit [INF] from='client.24580 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T22:38:48.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:47 smithi192 bash[24608]: audit 2024-09-13T22:38:47.516141+0000 mon.b (mon.1) 65 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:38:49.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:48 smithi178 bash[29889]: cluster 2024-09-13T22:38:48.421308+0000 mgr.y (mgr.14613) 29 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:48 smithi178 bash[22840]: cluster 2024-09-13T22:38:48.421308+0000 mgr.y (mgr.14613) 29 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:49.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:48 smithi192 bash[24608]: cluster 2024-09-13T22:38:48.421308+0000 mgr.y (mgr.14613) 29 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:49.732 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.733 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.733 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.733 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.734 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.734 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.734 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:49.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.052 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.052 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.053 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.053 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.053 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.053 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.053 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.053 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: Started Ceph alertmanager.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[29889]: cluster 2024-09-13T22:38:48.604891+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.e scrub starts 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[29889]: cluster 2024-09-13T22:38:48.606724+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.e scrub ok 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[29889]: cluster 2024-09-13T22:38:49.538726+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[29889]: cluster 2024-09-13T22:38:49.543000+0000 osd.5 (osd.5) 10 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[29889]: cluster 2024-09-13T22:38:49.747897+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:38:50.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[29889]: cluster 2024-09-13T22:38:49.749296+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:38:50.054 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.055 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.055 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:50.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[22840]: cluster 2024-09-13T22:38:48.604891+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.e scrub starts 2024-09-13T22:38:50.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[22840]: cluster 2024-09-13T22:38:48.606724+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.e scrub ok 2024-09-13T22:38:50.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[22840]: cluster 2024-09-13T22:38:49.538726+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:38:50.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[22840]: cluster 2024-09-13T22:38:49.543000+0000 osd.5 (osd.5) 10 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:38:50.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[22840]: cluster 2024-09-13T22:38:49.747897+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:38:50.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:49 smithi178 bash[22840]: cluster 2024-09-13T22:38:49.749296+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:38:50.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:49 smithi192 bash[24608]: cluster 2024-09-13T22:38:48.604891+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.e scrub starts 2024-09-13T22:38:50.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:49 smithi192 bash[24608]: cluster 2024-09-13T22:38:48.606724+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.e scrub ok 2024-09-13T22:38:50.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:49 smithi192 bash[24608]: cluster 2024-09-13T22:38:49.538726+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:38:50.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:49 smithi192 bash[24608]: cluster 2024-09-13T22:38:49.543000+0000 osd.5 (osd.5) 10 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:38:50.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:49 smithi192 bash[24608]: cluster 2024-09-13T22:38:49.747897+0000 osd.3 (osd.3) 11 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:38:50.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:49 smithi192 bash[24608]: cluster 2024-09-13T22:38:49.749296+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:38:51.100 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.100 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.101 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.101 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.101 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.101 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.102 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: cluster 2024-09-13T22:38:49.652055+0000 osd.4 (osd.4) 13 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: cluster 2024-09-13T22:38:49.656083+0000 osd.4 (osd.4) 14 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: audit 2024-09-13T22:38:49.994126+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: audit 2024-09-13T22:38:50.004304+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: audit 2024-09-13T22:38:50.010476+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: audit 2024-09-13T22:38:50.016944+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: cephadm 2024-09-13T22:38:50.022085+0000 mgr.y (mgr.14613) 30 : cephadm [INF] Deploying daemon node-exporter.a on smithi178 2024-09-13T22:38:51.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: cluster 2024-09-13T22:38:50.422770+0000 mgr.y (mgr.14613) 31 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:38:51.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: cluster 2024-09-13T22:38:50.495227+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:38:51.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[22840]: cluster 2024-09-13T22:38:50.497557+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:38:51.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.103 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.103 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.103 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: cluster 2024-09-13T22:38:49.652055+0000 osd.4 (osd.4) 13 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:38:51.103 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: cluster 2024-09-13T22:38:49.656083+0000 osd.4 (osd.4) 14 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: audit 2024-09-13T22:38:49.994126+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: audit 2024-09-13T22:38:50.004304+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: audit 2024-09-13T22:38:50.010476+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: audit 2024-09-13T22:38:50.016944+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: cephadm 2024-09-13T22:38:50.022085+0000 mgr.y (mgr.14613) 30 : cephadm [INF] Deploying daemon node-exporter.a on smithi178 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: cluster 2024-09-13T22:38:50.422770+0000 mgr.y (mgr.14613) 31 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: cluster 2024-09-13T22:38:50.495227+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:38:51.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:50 smithi178 bash[29889]: cluster 2024-09-13T22:38:50.497557+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:38:51.377 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.378 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.378 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.378 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.379 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.379 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.379 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.380 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.380 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.380 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.380 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:51.380 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 systemd[1]: Started Ceph node-exporter.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: cluster 2024-09-13T22:38:49.652055+0000 osd.4 (osd.4) 13 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: cluster 2024-09-13T22:38:49.656083+0000 osd.4 (osd.4) 14 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: audit 2024-09-13T22:38:49.994126+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: audit 2024-09-13T22:38:50.004304+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: audit 2024-09-13T22:38:50.010476+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: audit 2024-09-13T22:38:50.016944+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: cephadm 2024-09-13T22:38:50.022085+0000 mgr.y (mgr.14613) 30 : cephadm [INF] Deploying daemon node-exporter.a on smithi178 2024-09-13T22:38:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: cluster 2024-09-13T22:38:50.422770+0000 mgr.y (mgr.14613) 31 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:38:51.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: cluster 2024-09-13T22:38:50.495227+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:38:51.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:50 smithi192 bash[24608]: cluster 2024-09-13T22:38:50.497557+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:38:51.678 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57765]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.410Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.410Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.412Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.178 port=9094 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.412Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.446Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.447Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.450Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T22:38:51.679 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57602]: ts=2024-09-13T22:38:51.450Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T22:38:52.155 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.156 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.156 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.157 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.157 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.157 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.178 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:51 smithi178 bash[57765]: v1.5.0: Pulling from prometheus/node-exporter 2024-09-13T22:38:52.471 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.472 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.473 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.473 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.473 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.474 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: audit 2024-09-13T22:38:51.385088+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: audit 2024-09-13T22:38:51.393911+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: audit 2024-09-13T22:38:51.402109+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: cephadm 2024-09-13T22:38:51.405140+0000 mgr.y (mgr.14613) 32 : cephadm [INF] Deploying daemon node-exporter.b on smithi192 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: cluster 2024-09-13T22:38:51.541080+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.f scrub starts 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: cluster 2024-09-13T22:38:51.542751+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.f scrub ok 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: cluster 2024-09-13T22:38:51.574227+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:38:52.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[24608]: cluster 2024-09-13T22:38:51.576330+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:38:52.521 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 22b70bddd3ac: Pulling fs layer 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 5c12815fee55: Pulling fs layer 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: c0e87333d380: Pulling fs layer 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 5c12815fee55: Verifying Checksum 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 5c12815fee55: Download complete 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 22b70bddd3ac: Verifying Checksum 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 22b70bddd3ac: Download complete 2024-09-13T22:38:52.522 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: c0e87333d380: Verifying Checksum 2024-09-13T22:38:52.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: audit 2024-09-13T22:38:51.385088+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: audit 2024-09-13T22:38:51.393911+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: audit 2024-09-13T22:38:51.402109+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: cephadm 2024-09-13T22:38:51.405140+0000 mgr.y (mgr.14613) 32 : cephadm [INF] Deploying daemon node-exporter.b on smithi192 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: cluster 2024-09-13T22:38:51.541080+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.f scrub starts 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: cluster 2024-09-13T22:38:51.542751+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.f scrub ok 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: cluster 2024-09-13T22:38:51.574227+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[22840]: cluster 2024-09-13T22:38:51.576330+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: audit 2024-09-13T22:38:51.385088+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: audit 2024-09-13T22:38:51.393911+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: audit 2024-09-13T22:38:51.402109+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: cephadm 2024-09-13T22:38:51.405140+0000 mgr.y (mgr.14613) 32 : cephadm [INF] Deploying daemon node-exporter.b on smithi192 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: cluster 2024-09-13T22:38:51.541080+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.f scrub starts 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: cluster 2024-09-13T22:38:51.542751+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.f scrub ok 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: cluster 2024-09-13T22:38:51.574227+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:38:52.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[29889]: cluster 2024-09-13T22:38:51.576330+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:38:52.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:52 smithi192 systemd[1]: Started Ceph node-exporter.b for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:38:52.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:52 smithi192 bash[50503]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-09-13T22:38:52.928 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[23095]: [13/Sep/2024:22:38:52] ENGINE Bus STOPPING 2024-09-13T22:38:52.928 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: c0e87333d380: Download complete 2024-09-13T22:38:52.928 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:52 smithi178 bash[57765]: 22b70bddd3ac: Pull complete 2024-09-13T22:38:53.325 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[57765]: 5c12815fee55: Pull complete 2024-09-13T22:38:53.325 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[23095]: [13/Sep/2024:22:38:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T22:38:53.325 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[23095]: [13/Sep/2024:22:38:53] ENGINE Bus STOPPED 2024-09-13T22:38:53.325 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[23095]: [13/Sep/2024:22:38:53] ENGINE Bus STARTING 2024-09-13T22:38:53.405 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: v1.5.0: Pulling from prometheus/node-exporter 2024-09-13T22:38:53.406 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 22b70bddd3ac: Pulling fs layer 2024-09-13T22:38:53.406 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 5c12815fee55: Pulling fs layer 2024-09-13T22:38:53.406 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: c0e87333d380: Pulling fs layer 2024-09-13T22:38:53.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:38:53 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:38:53.656 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[23095]: [13/Sep/2024:22:38:53] ENGINE Serving on http://:::9283 2024-09-13T22:38:53.657 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[23095]: [13/Sep/2024:22:38:53] ENGINE Bus STARTED 2024-09-13T22:38:53.657 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[57602]: ts=2024-09-13T22:38:53.413Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000888547s 2024-09-13T22:38:53.657 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[57765]: c0e87333d380: Pull complete 2024-09-13T22:38:53.657 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[57765]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-09-13T22:38:53.657 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[57765]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.098246+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.d scrub starts 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.100367+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.d scrub ok 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.423526+0000 mgr.y (mgr.14613) 33 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.503380+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.522443+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.529785+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.536132+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.541533+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.550221+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.668176+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.676203+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.680516+0000 mon.b (mon.1) 66 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: audit 2024-09-13T22:38:52.687857+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.758996+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.760261+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.098246+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.d scrub starts 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.100367+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.d scrub ok 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.423526+0000 mgr.y (mgr.14613) 33 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:53.659 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.503380+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.522443+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.529785+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.536132+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.541533+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.550221+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.668176+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.676203+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.680516+0000 mon.b (mon.1) 66 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: audit 2024-09-13T22:38:52.687857+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.758996+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:38:53.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:53 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.760261+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:38:53.784 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 5c12815fee55: Verifying Checksum 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 5c12815fee55: Download complete 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 22b70bddd3ac: Verifying Checksum 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 22b70bddd3ac: Download complete 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: c0e87333d380: Verifying Checksum 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: c0e87333d380: Download complete 2024-09-13T22:38:53.906 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[50503]: 22b70bddd3ac: Pull complete 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.098246+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.d scrub starts 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.100367+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.d scrub ok 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.423526+0000 mgr.y (mgr.14613) 33 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.503380+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.522443+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.529785+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.536132+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.541533+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.550221+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.668176+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.676203+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.680516+0000 mon.b (mon.1) 66 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T22:38:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: audit 2024-09-13T22:38:52.687857+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.758996+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:38:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:53 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.760261+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:38:54.655 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[50503]: 5c12815fee55: Pull complete 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cephadm 2024-09-13T22:38:52.563135+0000 mgr.y (mgr.14613) 34 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: audit 2024-09-13T22:38:52.681154+0000 mgr.y (mgr.14613) 35 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cephadm 2024-09-13T22:38:52.709047+0000 mgr.y (mgr.14613) 36 : cephadm [INF] Deploying daemon grafana.a on smithi192 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.919952+0000 osd.5 (osd.5) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:52.920048+0000 osd.5 (osd.5) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:53.552262+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:53.554274+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:53.708488+0000 osd.7 (osd.7) 7 : cluster [DBG] 3.e scrub starts 2024-09-13T22:38:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:53.711122+0000 osd.7 (osd.7) 8 : cluster [DBG] 3.e scrub ok 2024-09-13T22:38:54.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[24608]: cluster 2024-09-13T22:38:53.742119+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cephadm 2024-09-13T22:38:52.563135+0000 mgr.y (mgr.14613) 34 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: audit 2024-09-13T22:38:52.681154+0000 mgr.y (mgr.14613) 35 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cephadm 2024-09-13T22:38:52.709047+0000 mgr.y (mgr.14613) 36 : cephadm [INF] Deploying daemon grafana.a on smithi192 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.919952+0000 osd.5 (osd.5) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:52.920048+0000 osd.5 (osd.5) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:53.552262+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:53.554274+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:53.708488+0000 osd.7 (osd.7) 7 : cluster [DBG] 3.e scrub starts 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:53.711122+0000 osd.7 (osd.7) 8 : cluster [DBG] 3.e scrub ok 2024-09-13T22:38:54.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[22840]: cluster 2024-09-13T22:38:53.742119+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cephadm 2024-09-13T22:38:52.563135+0000 mgr.y (mgr.14613) 34 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: audit 2024-09-13T22:38:52.681154+0000 mgr.y (mgr.14613) 35 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cephadm 2024-09-13T22:38:52.709047+0000 mgr.y (mgr.14613) 36 : cephadm [INF] Deploying daemon grafana.a on smithi192 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.919952+0000 osd.5 (osd.5) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:52.920048+0000 osd.5 (osd.5) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:53.552262+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:53.554274+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:53.708488+0000 osd.7 (osd.7) 7 : cluster [DBG] 3.e scrub starts 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:53.711122+0000 osd.7 (osd.7) 8 : cluster [DBG] 3.e scrub ok 2024-09-13T22:38:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:54 smithi178 bash[29889]: cluster 2024-09-13T22:38:53.742119+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-13T22:38:55.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:38:55.156 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:54 smithi192 bash[50503]: c0e87333d380: Pull complete 2024-09-13T22:38:55.502 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[50503]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-09-13T22:38:55.502 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[50503]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T22:38:55.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.424310+0000 mgr.y (mgr.14613) 37 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:55.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.476145+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:38:55.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.478412+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:38:55.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.596434+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:38:55.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.600558+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:38:55.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.687647+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:38:55.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.689109+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:38:55.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.766824+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.1e deep-scrub starts 2024-09-13T22:38:55.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:55 smithi192 bash[24608]: cluster 2024-09-13T22:38:54.776228+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.1e deep-scrub ok 2024-09-13T22:38:55.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.424310+0000 mgr.y (mgr.14613) 37 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.476145+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.478412+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.596434+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.600558+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.687647+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.689109+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.766824+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.1e deep-scrub starts 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[22840]: cluster 2024-09-13T22:38:54.776228+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.1e deep-scrub ok 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.424310+0000 mgr.y (mgr.14613) 37 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.476145+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:38:55.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.478412+0000 osd.2 (osd.2) 14 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:38:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.596434+0000 osd.0 (osd.0) 19 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:38:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.600558+0000 osd.0 (osd.0) 20 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:38:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.687647+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:38:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.689109+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:38:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.766824+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.1e deep-scrub starts 2024-09-13T22:38:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:55 smithi178 bash[29889]: cluster 2024-09-13T22:38:54.776228+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.1e deep-scrub ok 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.049Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.049Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.050Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.050Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.050Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.050Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T22:38:56.429 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T22:38:56.430 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T22:38:56.431 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T22:38:56.432 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T22:38:56.433 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T22:38:56.433 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.051Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T22:38:56.433 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.052Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T22:38:56.433 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[57765]: ts=2024-09-13T22:38:56.052Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T22:38:56.495 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:38:56.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:56 smithi192 bash[24608]: audit 2024-09-13T22:38:54.830032+0000 mgr.y (mgr.14613) 38 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:56.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:56 smithi192 bash[24608]: audit 2024-09-13T22:38:56.491110+0000 mon.a (mon.0) 856 : audit [DBG] from='client.? v1:172.21.15.178:0/1128322865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T22:38:56.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[29889]: audit 2024-09-13T22:38:54.830032+0000 mgr.y (mgr.14613) 38 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:56.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[29889]: audit 2024-09-13T22:38:56.491110+0000 mon.a (mon.0) 856 : audit [DBG] from='client.? v1:172.21.15.178:0/1128322865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T22:38:56.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[22840]: audit 2024-09-13T22:38:54.830032+0000 mgr.y (mgr.14613) 38 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:38:56.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:56 smithi178 bash[22840]: audit 2024-09-13T22:38:56.491110+0000 mon.a (mon.0) 856 : audit [DBG] from='client.? v1:172.21.15.178:0/1128322865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T22:38:57.250 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":21,"flags":0,"active_gid":14613,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6800","nonce":114721772}]},"active_addr":"172.21.15.178:6800/114721772","active_change":"2024-09-13T22:38:32.396896+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24535,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.178:8443/","prometheus":"http://172.21.15.178:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":95,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1942966429}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":887594350}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":3677235059}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1715602669}]}]} 2024-09-13T22:38:57.254 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T22:38:57.254 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T22:38:57.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T22:38:57.512 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.424979+0000 mgr.y (mgr.14613) 39 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:57.512 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.493196+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.494409+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.424979+0000 mgr.y (mgr.14613) 39 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.493196+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.494409+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.498011+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.500041+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.673936+0000 osd.7 (osd.7) 11 : cluster [DBG] 5.4 deep-scrub starts 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.676163+0000 osd.7 (osd.7) 12 : cluster [DBG] 5.4 deep-scrub ok 2024-09-13T22:38:57.513 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.771580+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:57.514 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.771679+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:57.514 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[22840]: audit 2024-09-13T22:38:57.511271+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:57.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.424979+0000 mgr.y (mgr.14613) 39 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:38:57.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.493196+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:38:57.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.494409+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:38:57.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.498011+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T22:38:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.500041+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T22:38:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.673936+0000 osd.7 (osd.7) 11 : cluster [DBG] 5.4 deep-scrub starts 2024-09-13T22:38:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.676163+0000 osd.7 (osd.7) 12 : cluster [DBG] 5.4 deep-scrub ok 2024-09-13T22:38:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.771580+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.771679+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:57.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:57 smithi192 bash[24608]: audit 2024-09-13T22:38:57.511271+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:57.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.498011+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T22:38:57.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.500041+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T22:38:57.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.673936+0000 osd.7 (osd.7) 11 : cluster [DBG] 5.4 deep-scrub starts 2024-09-13T22:38:57.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.676163+0000 osd.7 (osd.7) 12 : cluster [DBG] 5.4 deep-scrub ok 2024-09-13T22:38:57.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.771580+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:38:57.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.771679+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:38:57.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:57 smithi178 bash[29889]: audit 2024-09-13T22:38:57.511271+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:38:58.406 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.056Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T22:38:58.406 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.056Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.057Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.057Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.057Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T22:38:58.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T22:38:58.408 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T22:38:58.409 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T22:38:58.410 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[50503]: ts=2024-09-13T22:38:58.058Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T22:38:58.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.894755+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:38:58.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:56.896820+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:38:58.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.015687+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:38:58.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.017911+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:38:58.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.524345+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-13T22:38:58.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.660801+0000 osd.4 (osd.4) 15 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:38:58.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.663224+0000 osd.4 (osd.4) 16 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:38:58.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.826732+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:38:58.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:58 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.829455+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.894755+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:56.896820+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.015687+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.017911+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.524345+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.660801+0000 osd.4 (osd.4) 15 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.663224+0000 osd.4 (osd.4) 16 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.826732+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:38:58.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.829455+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.894755+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:56.896820+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.015687+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.017911+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.524345+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.660801+0000 osd.4 (osd.4) 15 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.663224+0000 osd.4 (osd.4) 16 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.826732+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:38:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:58 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.829455+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:38:59.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.968738+0000 osd.1 (osd.1) 21 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:38:59.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:57.970840+0000 osd.1 (osd.1) 22 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:38:59.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.425680+0000 mgr.y (mgr.14613) 40 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:59.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.566412+0000 osd.0 (osd.0) 21 : cluster [DBG] 3.1b deep-scrub starts 2024-09-13T22:38:59.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.572631+0000 osd.0 (osd.0) 22 : cluster [DBG] 3.1b deep-scrub ok 2024-09-13T22:38:59.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.702447+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:38:59.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.704185+0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:38:59.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.830951+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:38:59.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:38:59 smithi192 bash[24608]: cluster 2024-09-13T22:38:58.835264+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:38:59.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.968738+0000 osd.1 (osd.1) 21 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:57.970840+0000 osd.1 (osd.1) 22 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.425680+0000 mgr.y (mgr.14613) 40 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.566412+0000 osd.0 (osd.0) 21 : cluster [DBG] 3.1b deep-scrub starts 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.572631+0000 osd.0 (osd.0) 22 : cluster [DBG] 3.1b deep-scrub ok 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.702447+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.704185+0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.830951+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[29889]: cluster 2024-09-13T22:38:58.835264+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.968738+0000 osd.1 (osd.1) 21 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:57.970840+0000 osd.1 (osd.1) 22 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:38:59.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.425680+0000 mgr.y (mgr.14613) 40 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:38:59.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.566412+0000 osd.0 (osd.0) 21 : cluster [DBG] 3.1b deep-scrub starts 2024-09-13T22:38:59.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.572631+0000 osd.0 (osd.0) 22 : cluster [DBG] 3.1b deep-scrub ok 2024-09-13T22:38:59.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.702447+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:38:59.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.704185+0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:38:59.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.830951+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:38:59.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:38:59 smithi178 bash[22840]: cluster 2024-09-13T22:38:58.835264+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.473673+0000 osd.5 (osd.5) 21 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.476998+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.521726+0000 osd.2 (osd.2) 17 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.523806+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.655283+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.657628+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.861247+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:39:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:00 smithi192 bash[24608]: cluster 2024-09-13T22:38:59.864327+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:39:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.473673+0000 osd.5 (osd.5) 21 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.476998+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.521726+0000 osd.2 (osd.2) 17 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.523806+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.655283+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.657628+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.861247+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[29889]: cluster 2024-09-13T22:38:59.864327+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.473673+0000 osd.5 (osd.5) 21 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.476998+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.521726+0000 osd.2 (osd.2) 17 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.523806+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:39:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.655283+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:39:00.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.657628+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:39:00.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.861247+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:39:00.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:00 smithi178 bash[22840]: cluster 2024-09-13T22:38:59.864327+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:39:01.678 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[57602]: ts=2024-09-13T22:39:01.416Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003672284s 2024-09-13T22:39:01.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.426729+0000 mgr.y (mgr.14613) 41 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.429173+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.430975+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.538678+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.a scrub starts 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.541539+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.a scrub ok 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.650644+0000 osd.4 (osd.4) 21 : cluster [DBG] 2.f scrub starts 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[22840]: cluster 2024-09-13T22:39:00.654129+0000 osd.4 (osd.4) 22 : cluster [DBG] 2.f scrub ok 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.426729+0000 mgr.y (mgr.14613) 41 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.429173+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.430975+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.538678+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.a scrub starts 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.541539+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.a scrub ok 2024-09-13T22:39:01.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.650644+0000 osd.4 (osd.4) 21 : cluster [DBG] 2.f scrub starts 2024-09-13T22:39:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:01 smithi178 bash[29889]: cluster 2024-09-13T22:39:00.654129+0000 osd.4 (osd.4) 22 : cluster [DBG] 2.f scrub ok 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.426729+0000 mgr.y (mgr.14613) 41 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.429173+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.430975+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.538678+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.a scrub starts 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.541539+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.a scrub ok 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.650644+0000 osd.4 (osd.4) 21 : cluster [DBG] 2.f scrub starts 2024-09-13T22:39:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:01 smithi192 bash[24608]: cluster 2024-09-13T22:39:00.654129+0000 osd.4 (osd.4) 22 : cluster [DBG] 2.f scrub ok 2024-09-13T22:39:02.059 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:02.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:02 smithi192 bash[24608]: cluster 2024-09-13T22:39:01.650723+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.3 deep-scrub starts 2024-09-13T22:39:02.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:02 smithi192 bash[24608]: cluster 2024-09-13T22:39:01.655938+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.3 deep-scrub ok 2024-09-13T22:39:02.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:02 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.006498+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:02.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:02 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.006586+0000 osd.2 (osd.2) 22 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:02.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[22840]: cluster 2024-09-13T22:39:01.650723+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.3 deep-scrub starts 2024-09-13T22:39:02.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[22840]: cluster 2024-09-13T22:39:01.655938+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.3 deep-scrub ok 2024-09-13T22:39:02.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.006498+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:02.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.006586+0000 osd.2 (osd.2) 22 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:02.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[29889]: cluster 2024-09-13T22:39:01.650723+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.3 deep-scrub starts 2024-09-13T22:39:02.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[29889]: cluster 2024-09-13T22:39:01.655938+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.3 deep-scrub ok 2024-09-13T22:39:02.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.006498+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:02.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:02 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.006586+0000 osd.2 (osd.2) 22 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:03.662 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:39:03.662 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":102,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","created":"2024-09-13T22:27:31.500176+0000","modified":"2024-09-13T22:39:02.734418+0000","last_up_change":"2024-09-13T22:37:07.372146+0000","last_in_change":"2024-09-13T22:36:38.469358+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T22:32:40.780155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-13T22:37:39.227580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-13T22:37:41.229251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-09-13T22:37:41.624329+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T22:37:43.199974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T22:37:45.423695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"394f2664-bba1-4b15-ace8-78f70f970cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6801","nonce":1489719354}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6802","nonce":1489719354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6804","nonce":1489719354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6803","nonce":1489719354}]},"public_addr":"172.21.15.178:6801/1489719354","cluster_addr":"172.21.15.178:6802/1489719354","heartbeat_back_addr":"172.21.15.178:6804/1489719354","heartbeat_front_addr":"172.21.15.178:6803/1489719354","state":["exists","up"]},{"osd":1,"uuid":"8e65c08b-6431-4849-a6af-2da9cb1efdfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6805","nonce":4187084287}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6806","nonce":4187084287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6808","nonce":4187084287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6807","nonce":4187084287}]},"public_addr":"172.21.15.178:6805/4187084287","cluster_addr":"172.21.15.178:6806/4187084287","heartbeat_back_addr":"172.21.15.178:6808/4187084287","heartbeat_front_addr":"172.21.15.178:6807/4187084287","state":["exists","up"]},{"osd":2,"uuid":"40a79075-5102-4f6c-b41a-255102064624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6809","nonce":2303802730}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6810","nonce":2303802730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6812","nonce":2303802730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6811","nonce":2303802730}]},"public_addr":"172.21.15.178:6809/2303802730","cluster_addr":"172.21.15.178:6810/2303802730","heartbeat_back_addr":"172.21.15.178:6812/2303802730","heartbeat_front_addr":"172.21.15.178:6811/2303802730","state":["exists","up"]},{"osd":3,"uuid":"3962c796-6f97-4cc6-a186-4b6ef58d67ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6813","nonce":320675660}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6814","nonce":320675660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6816","nonce":320675660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6815","nonce":320675660}]},"public_addr":"172.21.15.178:6813/320675660","cluster_addr":"172.21.15.178:6814/320675660","heartbeat_back_addr":"172.21.15.178:6816/320675660","heartbeat_front_addr":"172.21.15.178:6815/320675660","state":["exists","up"]},{"osd":4,"uuid":"1b89db2f-9e81-40b1-89bd-8de1dbff133b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":3593670527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":3593670527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":3593670527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":3593670527}]},"public_addr":"172.21.15.192:6800/3593670527","cluster_addr":"172.21.15.192:6801/3593670527","heartbeat_back_addr":"172.21.15.192:6803/3593670527","heartbeat_front_addr":"172.21.15.192:6802/3593670527","state":["exists","up"]},{"osd":5,"uuid":"efa849a2-d789-4fdd-acc7-04f2c1bebfa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":3271425338}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":3271425338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":3271425338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":3271425338}]},"public_addr":"172.21.15.192:6804/3271425338","cluster_addr":"172.21.15.192:6805/3271425338","heartbeat_back_addr":"172.21.15.192:6807/3271425338","heartbeat_front_addr":"172.21.15.192:6806/3271425338","state":["exists","up"]},{"osd":6,"uuid":"54a26ae9-bf31-49cf-a924-901eed4a0c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":684205162}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":684205162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":684205162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":684205162}]},"public_addr":"172.21.15.192:6808/684205162","cluster_addr":"172.21.15.192:6809/684205162","heartbeat_back_addr":"172.21.15.192:6811/684205162","heartbeat_front_addr":"172.21.15.192:6810/684205162","state":["exists","up"]},{"osd":7,"uuid":"13e146db-61ac-4fe6-8f26-cbf8d1623a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":1303971738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":1303971738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":1303971738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":1303971738}]},"public_addr":"172.21.15.192:6812/1303971738","cluster_addr":"172.21.15.192:6813/1303971738","heartbeat_back_addr":"172.21.15.192:6815/1303971738","heartbeat_front_addr":"172.21.15.192:6814/1303971738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:22.894419+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:05.712792+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:39:02.006590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:56.771685+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:17.140503+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:52.920052+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:39.827526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:20.439105+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:0/268210503":"2024-09-14T22:27:58.436396+0000","172.21.15.178:6800/2039160385":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/669877409":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/129655740":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/3632788039":"2024-09-14T22:28:30.168687+0000","172.21.15.178:6800/388635774":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3265105147":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2959920382":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/2494442689":"2024-09-14T22:38:32.396576+0000","172.21.15.178:6800/2676020773":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2675812306":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/1834442338":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2835351236":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3836928220":"2024-09-14T22:38:32.396576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:01.943608+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.a scrub starts 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:01.945721+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.a scrub ok 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.427410+0000 mgr.y (mgr.14613) 42 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.559472+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: audit 2024-09-13T22:39:02.562252+0000 mon.b (mon.1) 67 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.570686+0000 osd.4 (osd.4) 23 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.572690+0000 osd.4 (osd.4) 24 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.679996+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.682160+0000 osd.0 (osd.0) 26 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:39:03.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.693761+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.697562+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[22840]: cluster 2024-09-13T22:39:02.744367+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:01.943608+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.a scrub starts 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:01.945721+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.a scrub ok 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.427410+0000 mgr.y (mgr.14613) 42 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.559472+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: audit 2024-09-13T22:39:02.562252+0000 mon.b (mon.1) 67 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.570686+0000 osd.4 (osd.4) 23 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.572690+0000 osd.4 (osd.4) 24 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.679996+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:39:03.676 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.682160+0000 osd.0 (osd.0) 26 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:39:03.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.693761+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:39:03.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.697562+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:39:03.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:03 smithi178 bash[29889]: cluster 2024-09-13T22:39:02.744367+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:01.943608+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.a scrub starts 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:01.945721+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.a scrub ok 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.427410+0000 mgr.y (mgr.14613) 42 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.559472+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: audit 2024-09-13T22:39:02.562252+0000 mon.b (mon.1) 67 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.570686+0000 osd.4 (osd.4) 23 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.572690+0000 osd.4 (osd.4) 24 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.679996+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.682160+0000 osd.0 (osd.0) 26 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:39:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.693761+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:39:03.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.697562+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:39:03.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:03 smithi192 bash[24608]: cluster 2024-09-13T22:39:02.744367+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-13T22:39:04.423 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T22:39:04.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T22:39:04.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[22840]: audit 2024-09-13T22:39:03.663106+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.178:0/2687354235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[22840]: cluster 2024-09-13T22:39:03.675430+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.1e deep-scrub starts 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[22840]: cluster 2024-09-13T22:39:03.677114+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.1e deep-scrub ok 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[22840]: cluster 2024-09-13T22:39:03.879136+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[22840]: cluster 2024-09-13T22:39:03.881483+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[29889]: audit 2024-09-13T22:39:03.663106+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.178:0/2687354235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[29889]: cluster 2024-09-13T22:39:03.675430+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.1e deep-scrub starts 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[29889]: cluster 2024-09-13T22:39:03.677114+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.1e deep-scrub ok 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[29889]: cluster 2024-09-13T22:39:03.879136+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:39:04.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:04 smithi178 bash[29889]: cluster 2024-09-13T22:39:03.881483+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:39:04.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:04 smithi192 bash[24608]: audit 2024-09-13T22:39:03.663106+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.178:0/2687354235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:39:04.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:04 smithi192 bash[24608]: cluster 2024-09-13T22:39:03.675430+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.1e deep-scrub starts 2024-09-13T22:39:04.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:04 smithi192 bash[24608]: cluster 2024-09-13T22:39:03.677114+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.1e deep-scrub ok 2024-09-13T22:39:04.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:04 smithi192 bash[24608]: cluster 2024-09-13T22:39:03.879136+0000 osd.3 (osd.3) 25 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:39:04.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:04 smithi192 bash[24608]: cluster 2024-09-13T22:39:03.881483+0000 osd.3 (osd.3) 26 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:39:05.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:04 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:39:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.428842+0000 mgr.y (mgr.14613) 43 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:39:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.515916+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.e scrub starts 2024-09-13T22:39:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.517651+0000 osd.2 (osd.2) 24 : cluster [DBG] 2.e scrub ok 2024-09-13T22:39:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.722230+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:39:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.726095+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:39:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: audit 2024-09-13T22:39:04.838056+0000 mgr.y (mgr.14613) 44 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:05.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.838408+0000 osd.3 (osd.3) 27 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:39:05.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.842744+0000 osd.3 (osd.3) 28 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:39:05.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.872685+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:39:05.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:05 smithi192 bash[24608]: cluster 2024-09-13T22:39:04.875947+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:39:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.428842+0000 mgr.y (mgr.14613) 43 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.515916+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.e scrub starts 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.517651+0000 osd.2 (osd.2) 24 : cluster [DBG] 2.e scrub ok 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.722230+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.726095+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: audit 2024-09-13T22:39:04.838056+0000 mgr.y (mgr.14613) 44 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.838408+0000 osd.3 (osd.3) 27 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.842744+0000 osd.3 (osd.3) 28 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.872685+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[22840]: cluster 2024-09-13T22:39:04.875947+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.428842+0000 mgr.y (mgr.14613) 43 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.515916+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.e scrub starts 2024-09-13T22:39:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.517651+0000 osd.2 (osd.2) 24 : cluster [DBG] 2.e scrub ok 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.722230+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.726095+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: audit 2024-09-13T22:39:04.838056+0000 mgr.y (mgr.14613) 44 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.838408+0000 osd.3 (osd.3) 27 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.842744+0000 osd.3 (osd.3) 28 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.872685+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:39:05.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:05 smithi178 bash[29889]: cluster 2024-09-13T22:39:04.875947+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:05.559071+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:05.561479+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:05.721046+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:05.725558+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:05.852379+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:05.854883+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T22:39:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:06 smithi192 bash[24608]: cluster 2024-09-13T22:39:06.429534+0000 mgr.y (mgr.14613) 45 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:05.559071+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:39:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:05.561479+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:39:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:05.721046+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:39:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:05.725558+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:05.852379+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:05.854883+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[22840]: cluster 2024-09-13T22:39:06.429534+0000 mgr.y (mgr.14613) 45 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:05.559071+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:05.561479+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:05.721046+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:05.725558+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:05.852379+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:05.854883+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T22:39:06.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:06 smithi178 bash[29889]: cluster 2024-09-13T22:39:06.429534+0000 mgr.y (mgr.14613) 45 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:07.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:07 smithi192 bash[24608]: cluster 2024-09-13T22:39:06.022521+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:39:07.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:07 smithi192 bash[24608]: cluster 2024-09-13T22:39:06.025478+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:39:07.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:07 smithi192 bash[24608]: cluster 2024-09-13T22:39:07.045917+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:39:07.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:07 smithi192 bash[24608]: cluster 2024-09-13T22:39:07.049071+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:39:07.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[29889]: cluster 2024-09-13T22:39:06.022521+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:39:07.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[29889]: cluster 2024-09-13T22:39:06.025478+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:39:07.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[29889]: cluster 2024-09-13T22:39:07.045917+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:39:07.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[29889]: cluster 2024-09-13T22:39:07.049071+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:39:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[22840]: cluster 2024-09-13T22:39:06.022521+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:39:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[22840]: cluster 2024-09-13T22:39:06.025478+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:39:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[22840]: cluster 2024-09-13T22:39:07.045917+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:39:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:07 smithi178 bash[22840]: cluster 2024-09-13T22:39:07.049071+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:39:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:08 smithi192 bash[24608]: cluster 2024-09-13T22:39:07.739033+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:39:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:08 smithi192 bash[24608]: cluster 2024-09-13T22:39:07.744099+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:39:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:08 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.430232+0000 mgr.y (mgr.14613) 46 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:39:09.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:08 smithi178 bash[22840]: cluster 2024-09-13T22:39:07.739033+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:39:09.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:08 smithi178 bash[22840]: cluster 2024-09-13T22:39:07.744099+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:39:09.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:08 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.430232+0000 mgr.y (mgr.14613) 46 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:39:09.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:08 smithi178 bash[29889]: cluster 2024-09-13T22:39:07.739033+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:39:09.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:08 smithi178 bash[29889]: cluster 2024-09-13T22:39:07.744099+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:39:09.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:08 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.430232+0000 mgr.y (mgr.14613) 46 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:39:09.228 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:09.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:07.880048+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:39:09.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:07.882451+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.747695+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.1f deep-scrub starts 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.754225+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.1f deep-scrub ok 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.788866+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.790722+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.856759+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:08.858477+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:09.329275+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.c scrub starts 2024-09-13T22:39:09.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:09 smithi192 bash[24608]: cluster 2024-09-13T22:39:09.333348+0000 osd.5 (osd.5) 26 : cluster [DBG] 3.c scrub ok 2024-09-13T22:39:10.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:07.880048+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:39:10.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:07.882451+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.747695+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.1f deep-scrub starts 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.754225+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.1f deep-scrub ok 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.788866+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.790722+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.856759+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:08.858477+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:09.329275+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.c scrub starts 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[22840]: cluster 2024-09-13T22:39:09.333348+0000 osd.5 (osd.5) 26 : cluster [DBG] 3.c scrub ok 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:07.880048+0000 osd.6 (osd.6) 15 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:39:10.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:07.882451+0000 osd.6 (osd.6) 16 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.747695+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.1f deep-scrub starts 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.754225+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.1f deep-scrub ok 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.788866+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.790722+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.856759+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:08.858477+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:09.329275+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.c scrub starts 2024-09-13T22:39:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:09 smithi178 bash[29889]: cluster 2024-09-13T22:39:09.333348+0000 osd.5 (osd.5) 26 : cluster [DBG] 3.c scrub ok 2024-09-13T22:39:10.690 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.691 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.691 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.692 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.692 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.693 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.693 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.774 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:39:10.775 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":102,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","created":"2024-09-13T22:27:31.500176+0000","modified":"2024-09-13T22:39:02.734418+0000","last_up_change":"2024-09-13T22:37:07.372146+0000","last_in_change":"2024-09-13T22:36:38.469358+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T22:32:40.780155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-13T22:37:39.227580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-13T22:37:41.229251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-09-13T22:37:41.624329+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T22:37:43.199974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T22:37:45.423695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"394f2664-bba1-4b15-ace8-78f70f970cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6801","nonce":1489719354}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6802","nonce":1489719354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6804","nonce":1489719354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6803","nonce":1489719354}]},"public_addr":"172.21.15.178:6801/1489719354","cluster_addr":"172.21.15.178:6802/1489719354","heartbeat_back_addr":"172.21.15.178:6804/1489719354","heartbeat_front_addr":"172.21.15.178:6803/1489719354","state":["exists","up"]},{"osd":1,"uuid":"8e65c08b-6431-4849-a6af-2da9cb1efdfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6805","nonce":4187084287}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6806","nonce":4187084287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6808","nonce":4187084287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6807","nonce":4187084287}]},"public_addr":"172.21.15.178:6805/4187084287","cluster_addr":"172.21.15.178:6806/4187084287","heartbeat_back_addr":"172.21.15.178:6808/4187084287","heartbeat_front_addr":"172.21.15.178:6807/4187084287","state":["exists","up"]},{"osd":2,"uuid":"40a79075-5102-4f6c-b41a-255102064624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6809","nonce":2303802730}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6810","nonce":2303802730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6812","nonce":2303802730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6811","nonce":2303802730}]},"public_addr":"172.21.15.178:6809/2303802730","cluster_addr":"172.21.15.178:6810/2303802730","heartbeat_back_addr":"172.21.15.178:6812/2303802730","heartbeat_front_addr":"172.21.15.178:6811/2303802730","state":["exists","up"]},{"osd":3,"uuid":"3962c796-6f97-4cc6-a186-4b6ef58d67ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6813","nonce":320675660}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6814","nonce":320675660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6816","nonce":320675660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6815","nonce":320675660}]},"public_addr":"172.21.15.178:6813/320675660","cluster_addr":"172.21.15.178:6814/320675660","heartbeat_back_addr":"172.21.15.178:6816/320675660","heartbeat_front_addr":"172.21.15.178:6815/320675660","state":["exists","up"]},{"osd":4,"uuid":"1b89db2f-9e81-40b1-89bd-8de1dbff133b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":3593670527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":3593670527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":3593670527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":3593670527}]},"public_addr":"172.21.15.192:6800/3593670527","cluster_addr":"172.21.15.192:6801/3593670527","heartbeat_back_addr":"172.21.15.192:6803/3593670527","heartbeat_front_addr":"172.21.15.192:6802/3593670527","state":["exists","up"]},{"osd":5,"uuid":"efa849a2-d789-4fdd-acc7-04f2c1bebfa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":3271425338}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":3271425338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":3271425338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":3271425338}]},"public_addr":"172.21.15.192:6804/3271425338","cluster_addr":"172.21.15.192:6805/3271425338","heartbeat_back_addr":"172.21.15.192:6807/3271425338","heartbeat_front_addr":"172.21.15.192:6806/3271425338","state":["exists","up"]},{"osd":6,"uuid":"54a26ae9-bf31-49cf-a924-901eed4a0c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":684205162}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":684205162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":684205162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":684205162}]},"public_addr":"172.21.15.192:6808/684205162","cluster_addr":"172.21.15.192:6809/684205162","heartbeat_back_addr":"172.21.15.192:6811/684205162","heartbeat_front_addr":"172.21.15.192:6810/684205162","state":["exists","up"]},{"osd":7,"uuid":"13e146db-61ac-4fe6-8f26-cbf8d1623a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":1303971738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":1303971738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":1303971738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":1303971738}]},"public_addr":"172.21.15.192:6812/1303971738","cluster_addr":"172.21.15.192:6813/1303971738","heartbeat_back_addr":"172.21.15.192:6815/1303971738","heartbeat_front_addr":"172.21.15.192:6814/1303971738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:22.894419+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:05.712792+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:39:02.006590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:56.771685+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:17.140503+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:52.920052+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:39.827526+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T22:38:20.439105+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:0/268210503":"2024-09-14T22:27:58.436396+0000","172.21.15.178:6800/2039160385":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/669877409":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/129655740":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/3632788039":"2024-09-14T22:28:30.168687+0000","172.21.15.178:6800/388635774":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3265105147":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2959920382":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/2494442689":"2024-09-14T22:38:32.396576+0000","172.21.15.178:6800/2676020773":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2675812306":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/1834442338":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2835351236":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3836928220":"2024-09-14T22:38:32.396576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T22:39:10.947 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.948 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.948 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.948 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.949 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.949 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.949 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:10 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.294870+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.a scrub starts 2024-09-13T22:39:10.949 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:10 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.297209+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2024-09-13T22:39:10.949 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:10 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.431902+0000 mgr.y (mgr.14613) 47 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:10.949 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.950 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.950 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.950 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.950 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.950 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:10.950 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: Started Ceph grafana.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:39:10.951 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 22:39:10 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T22:39:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:10 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.294870+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.a scrub starts 2024-09-13T22:39:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:10 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.297209+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2024-09-13T22:39:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:10 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.431902+0000 mgr.y (mgr.14613) 47 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:11.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:10 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.294870+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.a scrub starts 2024-09-13T22:39:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:10 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.297209+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2024-09-13T22:39:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:10 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.431902+0000 mgr.y (mgr.14613) 47 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:11.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T22:39:11.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T22:39:11.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T22:39:11.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T22:39:11.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T22:39:11.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T22:39:11.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T22:39:11.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T22:39:11.767 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:09.957259+0000 osd.3 (osd.3) 29 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:39:11.767 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:09.960539+0000 osd.3 (osd.3) 30 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:39:11.768 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.615769+0000 osd.4 (osd.4) 27 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:39:12.026 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:09.957259+0000 osd.3 (osd.3) 29 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:39:12.026 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:09.960539+0000 osd.3 (osd.3) 30 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:39:12.026 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.615769+0000 osd.4 (osd.4) 27 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:39:12.026 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.618585+0000 osd.4 (osd.4) 28 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:39:12.026 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:10.774460+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? v1:172.21.15.178:0/3122435132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:39:12.026 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.821434+0000 osd.7 (osd.7) 17 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:10.823170+0000 osd.7 (osd.7) 18 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:10.998845+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:11.009580+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:11.017540+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:11.025488+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:11.030768+0000 osd.1 (osd.1) 29 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: cluster 2024-09-13T22:39:11.035776+0000 osd.1 (osd.1) 30 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:11.083601+0000 mon.b (mon.1) 68 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:11.614650+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:11 smithi192 bash[24608]: audit 2024-09-13T22:39:11.625920+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.618585+0000 osd.4 (osd.4) 28 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:10.774460+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? v1:172.21.15.178:0/3122435132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.821434+0000 osd.7 (osd.7) 17 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:10.823170+0000 osd.7 (osd.7) 18 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:10.998845+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:11.009580+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:11.017540+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:11.025488+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:11.030768+0000 osd.1 (osd.1) 29 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: cluster 2024-09-13T22:39:11.035776+0000 osd.1 (osd.1) 30 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:39:12.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:11.083601+0000 mon.b (mon.1) 68 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:11.614650+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[29889]: audit 2024-09-13T22:39:11.625920+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:09.957259+0000 osd.3 (osd.3) 29 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:09.960539+0000 osd.3 (osd.3) 30 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.615769+0000 osd.4 (osd.4) 27 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.618585+0000 osd.4 (osd.4) 28 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:10.774460+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? v1:172.21.15.178:0/3122435132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.821434+0000 osd.7 (osd.7) 17 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:10.823170+0000 osd.7 (osd.7) 18 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:10.998845+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:11.009580+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:11.017540+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:11.025488+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:11.030768+0000 osd.1 (osd.1) 29 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: cluster 2024-09-13T22:39:11.035776+0000 osd.1 (osd.1) 30 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:11.083601+0000 mon.b (mon.1) 68 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:11.614650+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:11 smithi178 bash[22840]: audit 2024-09-13T22:39:11.625920+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256055176Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256547537Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256583539Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256604016Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256632933Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256649159Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256665033Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-13T22:39:12.288 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256680238Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256696532Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256713387Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256727169Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256741886Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256757492Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256779477Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256794497Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256807565Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256822288Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256839748Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=settings t=2024-09-13T22:39:12.256853128Z level=info msg="App mode production" 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=sqlstore t=2024-09-13T22:39:12.256980783Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T22:39:12.289 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=sqlstore t=2024-09-13T22:39:12.25702434Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T22:39:12.290 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.25962624Z level=info msg="Starting DB migrations" 2024-09-13T22:39:12.290 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.259817386Z level=info msg="Executing migration" id="create migration_log table" 2024-09-13T22:39:12.290 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.290752724Z level=info msg="Executing migration" id="create user table" 2024-09-13T22:39:12.573 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.332634516Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-13T22:39:12.573 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.357944842Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-13T22:39:12.573 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.383204559Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-13T22:39:12.573 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.408280464Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-13T22:39:12.573 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.45036207Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-13T22:39:12.574 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.492527387Z level=info msg="Executing migration" id="create user table v2" 2024-09-13T22:39:12.574 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.534479328Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-13T22:39:12.574 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.576487068Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.618837331Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.643947376Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.669249427Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.711214974Z level=info msg="Executing migration" id="Update user table charset" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.753228977Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.786821452Z level=info msg="Executing migration" id="Add missing user data" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.828990817Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-13T22:39:12.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:11.832519+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:39:12.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:11.834622+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:11.902426+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:11.903754+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.039234+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.041243+0000 osd.1 (osd.1) 32 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.299655+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.301900+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.432688+0000 mgr.y (mgr.14613) 48 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:39:12.827 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[24608]: audit 2024-09-13T22:39:12.523218+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.870883688Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.912993447Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.938273113Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:12 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:12.963624616Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.005578267Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.047686233Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.072944809Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-13T22:39:13.095 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.098258663Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-13T22:39:13.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:11.832519+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:11.834622+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:11.902426+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:11.903754+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.039234+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.041243+0000 osd.1 (osd.1) 32 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.299655+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.301900+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.432688+0000 mgr.y (mgr.14613) 48 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[29889]: audit 2024-09-13T22:39:12.523218+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:13.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:11.832519+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:11.834622+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:11.902426+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:11.903754+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.039234+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.041243+0000 osd.1 (osd.1) 32 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.299655+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.301900+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.432688+0000 mgr.y (mgr.14613) 48 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:39:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:12 smithi178 bash[22840]: audit 2024-09-13T22:39:12.523218+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:13.355 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.139983198Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-13T22:39:13.355 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.165326565Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.190559082Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.215649397Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.240957285Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.266249778Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.291629731Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.316745999Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-13T22:39:13.356 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.358685193Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-13T22:39:13.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.384111352Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-13T22:39:13.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.409303805Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-13T22:39:13.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.451406238Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-13T22:39:13.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.493398179Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-13T22:39:13.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.518921247Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-13T22:39:13.608 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.561040459Z level=info msg="Executing migration" id="create star table" 2024-09-13T22:39:13.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.586286093Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-13T22:39:13.609 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.611550918Z level=info msg="Executing migration" id="create org table v1" 2024-09-13T22:39:13.868 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.653421779Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.678740176Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.720737507Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.745955741Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.771235168Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.830085474Z level=info msg="Executing migration" id="Update org table charset" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.87190887Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-13T22:39:13.869 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.434964+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.b scrub starts 2024-09-13T22:39:13.869 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.437289+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.b scrub ok 2024-09-13T22:39:13.869 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.587253+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e deep-scrub starts 2024-09-13T22:39:13.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:12.592123+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e deep-scrub ok 2024-09-13T22:39:13.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:13.006279+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:39:13.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:13.008754+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:39:13.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:13.334802+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:39:13.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[24608]: cluster 2024-09-13T22:39:13.338958+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:39:14.148 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.897239313Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-13T22:39:14.148 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.922546966Z level=info msg="Executing migration" id="create dashboard table" 2024-09-13T22:39:14.148 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.947840846Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-13T22:39:14.148 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:13 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:13.973123759Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-13T22:39:14.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.014974099Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-13T22:39:14.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.040308954Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-13T22:39:14.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.074008346Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-13T22:39:14.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.107600942Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-13T22:39:14.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.149480937Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-13T22:39:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.434964+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.b scrub starts 2024-09-13T22:39:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.437289+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.b scrub ok 2024-09-13T22:39:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.587253+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e deep-scrub starts 2024-09-13T22:39:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:12.592123+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e deep-scrub ok 2024-09-13T22:39:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:13.006279+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:13.008754+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:13.334802+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[29889]: cluster 2024-09-13T22:39:13.338958+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.434964+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.b scrub starts 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.437289+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.b scrub ok 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.587253+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e deep-scrub starts 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:12.592123+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e deep-scrub ok 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:13.006279+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:13.008754+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:13.334802+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:39:14.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:13 smithi178 bash[22840]: cluster 2024-09-13T22:39:13.338958+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:39:14.399 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.191505368Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-13T22:39:14.399 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.216938083Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-13T22:39:14.399 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.242205736Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-13T22:39:14.399 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.259039885Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-13T22:39:14.399 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.292653397Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-13T22:39:14.400 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.326342441Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-13T22:39:14.400 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.351669435Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-13T22:39:14.400 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.376908083Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-13T22:39:14.400 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.402362481Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.444473968Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.48641659Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.511759744Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.553726414Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.578970779Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.604232563Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-13T22:39:14.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.646186658Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-13T22:39:14.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:39:14.946 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.032723+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:14.946 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.032829+0000 osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:14.946 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.434168+0000 mgr.y (mgr.14613) 49 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.688052663Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.746397326Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.788689115Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.830987726Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.87295131Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.914886048Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-13T22:39:14.946 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.948742691Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-13T22:39:15.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:14 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.032723+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:15.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:14 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.032829+0000 osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:15.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:14 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.434168+0000 mgr.y (mgr.14613) 49 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:39:15.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:14 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.032723+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:15.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:14 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.032829+0000 osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:15.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:14 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.434168+0000 mgr.y (mgr.14613) 49 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:14 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:14.97400049Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.01599866Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.0412196Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.0664641Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.091761811Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.133695598Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.158939016Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.184053259Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-13T22:39:15.206 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.209456261Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-13T22:39:15.458 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.234571954Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-13T22:39:15.458 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.259779379Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-13T22:39:15.459 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.28499064Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-13T22:39:15.459 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.327288722Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-13T22:39:15.459 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.361094894Z level=info msg="Executing migration" id="create data_source table" 2024-09-13T22:39:15.459 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.394658488Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-13T22:39:15.459 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.41982812Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-13T22:39:15.459 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.461831168Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-13T22:39:15.710 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.503762077Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-13T22:39:15.711 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.529092661Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-13T22:39:15.711 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.554360145Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-13T22:39:15.711 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.579669665Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-13T22:39:15.711 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.604802224Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-13T22:39:15.711 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.646845646Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-13T22:39:15.711 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.672153564Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.434434+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.436736+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.750490+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.752670+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:14.800919+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: audit 2024-09-13T22:39:14.842160+0000 mgr.y (mgr.14613) 50 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.303811+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:39:15.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.307185+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:39:15.989 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.714181454Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.739379199Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.764660513Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.798267012Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.832016091Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.865520049Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.899322824Z level=info msg="Executing migration" id="Add uid column" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.924584255Z level=info msg="Executing migration" id="Update uid value" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.949926183Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-13T22:39:15.990 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:15 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:15.991863495Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-13T22:39:16.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.434434+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-13T22:39:16.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.436736+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.750490+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.752670+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:14.800919+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: audit 2024-09-13T22:39:14.842160+0000 mgr.y (mgr.14613) 50 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.303811+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.307185+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.434434+0000 osd.2 (osd.2) 27 : cluster [DBG] 6.f scrub starts 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.436736+0000 osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.750490+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.752670+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:14.800919+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-13T22:39:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: audit 2024-09-13T22:39:14.842160+0000 mgr.y (mgr.14613) 50 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:16.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.303811+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:39:16.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:15 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.307185+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.033962362Z level=info msg="Executing migration" id="create api_key table" 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.075841892Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.101088891Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.126349906Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.168246927Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.193554501Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-13T22:39:16.240 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.218626745Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-13T22:39:16.241 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.244077077Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-13T22:39:16.339 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.339 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.340 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.340 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.340 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.344 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.345 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.347 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:16.518 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.269530573Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-13T22:39:16.518 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.294748244Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-13T22:39:16.518 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.33676292Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-13T22:39:16.519 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.362183541Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-13T22:39:16.519 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.387439485Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-13T22:39:16.519 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.412578795Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-13T22:39:16.519 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.454587754Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-13T22:39:16.519 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.479813691Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-13T22:39:16.519 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.52175801Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-13T22:39:16.804 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.563731379Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-13T22:39:16.804 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.630888244Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-13T22:39:16.804 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.672862098Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-13T22:39:16.804 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.698142399Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-13T22:39:16.804 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.72319764Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-13T22:39:16.805 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.765264089Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-13T22:39:16.805 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.807911233Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-13T22:39:17.056 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.84969403Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-13T22:39:17.056 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.950389655Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-13T22:39:17.056 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:16.983953299Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-13T22:39:17.056 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.017775142Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-13T22:39:17.056 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.468740+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:39:17.056 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.472766+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:39:17.056 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.621952+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:39:17.056 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.627254+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:39:17.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.851016+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:39:17.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.852590+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:39:17.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.969725+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.c scrub starts 2024-09-13T22:39:17.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:15.971891+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.c scrub ok 2024-09-13T22:39:17.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:16 smithi192 bash[24608]: cluster 2024-09-13T22:39:16.434608+0000 mgr.y (mgr.14613) 51 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:17.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.468740+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:39:17.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.472766+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:39:17.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.621952+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:39:17.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.627254+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:39:17.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.851016+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:39:17.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.852590+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.969725+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.c scrub starts 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:15.971891+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.c scrub ok 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[22840]: cluster 2024-09-13T22:39:16.434608+0000 mgr.y (mgr.14613) 51 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.468740+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.472766+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.621952+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.627254+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.851016+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.852590+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.969725+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.c scrub starts 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:15.971891+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.c scrub ok 2024-09-13T22:39:17.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:16 smithi178 bash[29889]: cluster 2024-09-13T22:39:16.434608+0000 mgr.y (mgr.14613) 51 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:17.333 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.059682415Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-13T22:39:17.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.085023411Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-13T22:39:17.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.14368175Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-13T22:39:17.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.186032669Z level=info msg="Executing migration" id="create quota table v1" 2024-09-13T22:39:17.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.228103212Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-13T22:39:17.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.25339264Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-13T22:39:17.334 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.295420525Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-13T22:39:17.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.33756985Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-13T22:39:17.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.379561815Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-13T22:39:17.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.421667586Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-13T22:39:17.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.463703662Z level=info msg="Executing migration" id="create session table" 2024-09-13T22:39:17.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.489208967Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-13T22:39:17.638 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.514312077Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-13T22:39:17.639 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.556267931Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-13T22:39:17.639 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.581485761Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-13T22:39:17.639 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.641870754Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-13T22:39:17.897 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.690479901Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-13T22:39:17.897 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.715706243Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-13T22:39:17.897 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.740978311Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-13T22:39:17.897 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.783076316Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-13T22:39:17.897 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.82513844Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-13T22:39:17.898 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.850265677Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-13T22:39:17.898 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.87553601Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-13T22:39:17.898 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.90071095Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.926022482Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.951334362Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:17 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:17.976639958Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.018597148Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.060715024Z level=info msg="Executing migration" id="create alert table v1" 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.102599662Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-13T22:39:18.156 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.144962222Z level=info msg="Executing migration" id="add index alert state" 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: cluster 2024-09-13T22:39:16.959721+0000 osd.1 (osd.1) 39 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: cluster 2024-09-13T22:39:16.963631+0000 osd.1 (osd.1) 40 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: cluster 2024-09-13T22:39:16.968823+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.7 deep-scrub starts 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: cluster 2024-09-13T22:39:16.975240+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.7 deep-scrub ok 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.123553+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.128875+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.594536+0000 mon.b (mon.1) 69 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: cluster 2024-09-13T22:39:17.627268+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: cluster 2024-09-13T22:39:17.628963+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.631825+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.644271+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.647907+0000 mon.b (mon.1) 70 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:39:18.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.649858+0000 mon.b (mon.1) 71 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:39:18.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[24608]: audit 2024-09-13T22:39:17.660383+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: cluster 2024-09-13T22:39:16.959721+0000 osd.1 (osd.1) 39 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:39:18.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: cluster 2024-09-13T22:39:16.963631+0000 osd.1 (osd.1) 40 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:39:18.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: cluster 2024-09-13T22:39:16.968823+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.7 deep-scrub starts 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: cluster 2024-09-13T22:39:16.975240+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.7 deep-scrub ok 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.123553+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.128875+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.594536+0000 mon.b (mon.1) 69 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: cluster 2024-09-13T22:39:17.627268+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: cluster 2024-09-13T22:39:17.628963+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.631825+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.644271+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.647907+0000 mon.b (mon.1) 70 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.649858+0000 mon.b (mon.1) 71 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:39:18.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[22840]: audit 2024-09-13T22:39:17.660383+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: cluster 2024-09-13T22:39:16.959721+0000 osd.1 (osd.1) 39 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: cluster 2024-09-13T22:39:16.963631+0000 osd.1 (osd.1) 40 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: cluster 2024-09-13T22:39:16.968823+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.7 deep-scrub starts 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: cluster 2024-09-13T22:39:16.975240+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.7 deep-scrub ok 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.123553+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.128875+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.594536+0000 mon.b (mon.1) 69 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: cluster 2024-09-13T22:39:17.627268+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: cluster 2024-09-13T22:39:17.628963+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.631825+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.644271+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.647907+0000 mon.b (mon.1) 70 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:39:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.649858+0000 mon.b (mon.1) 71 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:39:18.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:18 smithi178 bash[29889]: audit 2024-09-13T22:39:17.660383+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.186955765Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.212192514Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.229032935Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.246039893Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.262890363Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.304801275Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.321761983Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.33862719Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-13T22:39:18.436 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.355610649Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-13T22:39:18.437 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.397655612Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-13T22:39:18.437 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.439536307Z level=info msg="Executing migration" id="Add column is_default" 2024-09-13T22:39:18.713 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.481660706Z level=info msg="Executing migration" id="Add column frequency" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.506948285Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.540488507Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.582414085Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.607753583Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.632953542Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.674999555Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-13T22:39:18.714 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.716946009Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.742229565Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.784209938Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.826354254Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.868244892Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.910349375Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.935705976Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:18 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:18.961004504Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-13T22:39:19.002 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.005223858Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-13T22:39:19.254 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.030441064Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-13T22:39:19.254 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.072445304Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-13T22:39:19.254 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.097812616Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-13T22:39:19.255 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.123109841Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-13T22:39:19.255 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.165378912Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-13T22:39:19.255 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.190464707Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-13T22:39:19.255 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.215871708Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-13T22:39:19.255 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.257782326Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-13T22:39:19.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cephadm 2024-09-13T22:39:17.727380+0000 mgr.y (mgr.14613) 52 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T22:39:19.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cephadm 2024-09-13T22:39:17.735897+0000 mgr.y (mgr.14613) 53 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi178 2024-09-13T22:39:19.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.435286+0000 mgr.y (mgr.14613) 54 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:19.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.685882+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:39:19.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.688131+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:39:19.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.862284+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:39:19.256 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.864322+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:39:19.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cephadm 2024-09-13T22:39:17.727380+0000 mgr.y (mgr.14613) 52 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T22:39:19.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cephadm 2024-09-13T22:39:17.735897+0000 mgr.y (mgr.14613) 53 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi178 2024-09-13T22:39:19.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.435286+0000 mgr.y (mgr.14613) 54 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:19.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.685882+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:39:19.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.688131+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:39:19.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.862284+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.864322+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cephadm 2024-09-13T22:39:17.727380+0000 mgr.y (mgr.14613) 52 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cephadm 2024-09-13T22:39:17.735897+0000 mgr.y (mgr.14613) 53 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi178 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.435286+0000 mgr.y (mgr.14613) 54 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.685882+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.688131+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.862284+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:39:19.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:19 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.864322+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.299878576Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.342008318Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.367214945Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.392478195Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.434357923Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.476340948Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-13T22:39:19.515 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.518344702Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-13T22:39:19.792 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.560343544Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-13T22:39:19.792 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.585706474Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-13T22:39:19.793 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.610992373Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-13T22:39:19.793 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.652884289Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-13T22:39:19.793 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.686436125Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-13T22:39:19.793 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.728512572Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-13T22:39:19.793 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.753872906Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-13T22:39:19.793 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.795788305Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.821123833Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.84632004Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.888654083Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.914019691Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.939243844Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:19 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:19.981193879Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.02324997Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-13T22:39:20.062 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.06531171Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.107402008Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.13296399Z level=info msg="Executing migration" id="Move region to single row" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.174759821Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.200020933Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.225306392Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.250481713Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.275793419Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.301042322Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-13T22:39:20.323 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.326294891Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.972819+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.975451+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.996175+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:18.998402+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.313375+0000 osd.5 (osd.5) 35 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.315694+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.634938+0000 osd.4 (osd.4) 35 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.638115+0000 osd.4 (osd.4) 36 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.776150+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.778581+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.866935+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:39:20.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.869963+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.972819+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.975451+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.996175+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:18.998402+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.313375+0000 osd.5 (osd.5) 35 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.315694+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:39:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.634938+0000 osd.4 (osd.4) 35 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.638115+0000 osd.4 (osd.4) 36 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.776150+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.778581+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.866935+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.869963+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.972819+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.975451+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.996175+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:18.998402+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T22:39:20.442 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.313375+0000 osd.5 (osd.5) 35 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.315694+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.634938+0000 osd.4 (osd.4) 35 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.638115+0000 osd.4 (osd.4) 36 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.776150+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.778581+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.866935+0000 osd.6 (osd.6) 23 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:39:20.443 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:20 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.869963+0000 osd.6 (osd.6) 24 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:39:20.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.351579254Z level=info msg="Executing migration" id="create test_data table" 2024-09-13T22:39:20.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.393660768Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-13T22:39:20.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.435607153Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-13T22:39:20.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.477587591Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-13T22:39:20.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.502914251Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-13T22:39:20.592 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.528054484Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-13T22:39:20.593 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.570098703Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-13T22:39:20.593 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.595361119Z level=info msg="Executing migration" id="create team table" 2024-09-13T22:39:20.853 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.637409523Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-13T22:39:20.853 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.654342639Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-13T22:39:20.853 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.687889832Z level=info msg="Executing migration" id="create team member table" 2024-09-13T22:39:20.853 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.704829082Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-13T22:39:20.853 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.730366931Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-13T22:39:20.853 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.772308787Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-13T22:39:20.854 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.81445889Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-13T22:39:20.854 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.856386192Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-13T22:39:21.114 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.898430607Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-13T22:39:21.114 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.923846734Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.949159807Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:20 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:20.974327831Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.016383678Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.041815093Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.067141805Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.092343259Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-13T22:39:21.115 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.117596469Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.159607904Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.184966136Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.210144577Z level=info msg="Executing migration" id="create tag table" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.2354538Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.269047156Z level=info msg="Executing migration" id="create login attempt table" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.29426774Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.319458047Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-13T22:39:21.392 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.344667511Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.486440+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.d scrub starts 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.489538+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.d scrub ok 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.961461+0000 osd.1 (osd.1) 43 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:19.966029+0000 osd.1 (osd.1) 44 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:20.436207+0000 mgr.y (mgr.14613) 55 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:20.598368+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:20.600560+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:20.745756+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:39:21.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[24608]: cluster 2024-09-13T22:39:20.748069+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:39:21.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.486440+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.d scrub starts 2024-09-13T22:39:21.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.489538+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.d scrub ok 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.961461+0000 osd.1 (osd.1) 43 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:19.966029+0000 osd.1 (osd.1) 44 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:20.436207+0000 mgr.y (mgr.14613) 55 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:20.598368+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:20.600560+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:20.745756+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[29889]: cluster 2024-09-13T22:39:20.748069+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.486440+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.d scrub starts 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.489538+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.d scrub ok 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.961461+0000 osd.1 (osd.1) 43 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:39:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:19.966029+0000 osd.1 (osd.1) 44 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:39:21.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:20.436207+0000 mgr.y (mgr.14613) 55 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:21.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:20.598368+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T22:39:21.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:20.600560+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T22:39:21.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:20.745756+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:39:21.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:21 smithi178 bash[22840]: cluster 2024-09-13T22:39:20.748069+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.395135221Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.43709764Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.504115685Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.529334418Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.571292894Z level=info msg="Executing migration" id="create user auth table" 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.596661353Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-13T22:39:21.656 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.638917155Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-13T22:39:21.938 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.680814292Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-13T22:39:21.938 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.722816914Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-13T22:39:21.938 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.748109937Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-13T22:39:21.938 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.790057793Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-13T22:39:21.939 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.832131144Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-13T22:39:21.939 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.857374711Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-13T22:39:21.939 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.899284088Z level=info msg="Executing migration" id="create server_lock table" 2024-09-13T22:39:21.939 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:21 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:21.941406153Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-13T22:39:22.159 INFO:teuthology.orchestra.run.smithi178.stdout:193273528370 2024-09-13T22:39:22.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T22:39:22.222 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.035794415Z level=info msg="Executing migration" id="create user auth token table" 2024-09-13T22:39:22.223 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.133970197Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-13T22:39:22.223 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.375435+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:39:22.223 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.377816+0000 osd.5 (osd.5) 38 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:39:22.223 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.652072+0000 osd.0 (osd.0) 41 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:39:22.223 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.654264+0000 osd.0 (osd.0) 42 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:39:22.223 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.868877+0000 osd.6 (osd.6) 25 : cluster [DBG] 3.a scrub starts 2024-09-13T22:39:22.223 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.873735+0000 osd.6 (osd.6) 26 : cluster [DBG] 3.a scrub ok 2024-09-13T22:39:22.285 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.375435+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:39:22.285 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.377816+0000 osd.5 (osd.5) 38 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:39:22.285 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.652072+0000 osd.0 (osd.0) 41 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:39:22.285 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.654264+0000 osd.0 (osd.0) 42 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.868877+0000 osd.6 (osd.6) 25 : cluster [DBG] 3.a scrub starts 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.873735+0000 osd.6 (osd.6) 26 : cluster [DBG] 3.a scrub ok 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.375435+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.377816+0000 osd.5 (osd.5) 38 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.652072+0000 osd.0 (osd.0) 41 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.654264+0000 osd.0 (osd.0) 42 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.868877+0000 osd.6 (osd.6) 25 : cluster [DBG] 3.a scrub starts 2024-09-13T22:39:22.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:22 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.873735+0000 osd.6 (osd.6) 26 : cluster [DBG] 3.a scrub ok 2024-09-13T22:39:22.482 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.226156827Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-13T22:39:22.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.376885388Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-13T22:39:22.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.418914644Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-13T22:39:22.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.460795291Z level=info msg="Executing migration" id="create cache_data table" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.486162944Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.511409547Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.55336073Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.595365663Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.620601716Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.645947946Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.671184109Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-13T22:39:22.735 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.69645899Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-13T22:39:22.736 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.738442736Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.763968203Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.789275336Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.831274385Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.8733408Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.90694186Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.949003396Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:22 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:22.974268482Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-13T22:39:23.013 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.016185703Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-13T22:39:23.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.521298+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:39:23.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[24608]: cluster 2024-09-13T22:39:21.523614+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:39:23.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[24608]: cluster 2024-09-13T22:39:22.437151+0000 mgr.y (mgr.14613) 56 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.04151745Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.083508076Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.108779053Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.134029437Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.176087043Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.201364597Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.226515748Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-13T22:39:23.274 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.251925584Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-13T22:39:23.275 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.277109856Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-13T22:39:23.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:23 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.521298+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:39:23.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:23 smithi178 bash[29889]: cluster 2024-09-13T22:39:21.523614+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:39:23.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:23 smithi178 bash[29889]: cluster 2024-09-13T22:39:22.437151+0000 mgr.y (mgr.14613) 56 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:23.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:23 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.521298+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:39:23.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:23 smithi178 bash[22840]: cluster 2024-09-13T22:39:21.523614+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:39:23.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:23 smithi178 bash[22840]: cluster 2024-09-13T22:39:22.437151+0000 mgr.y (mgr.14613) 56 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:23.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.302390023Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-13T22:39:23.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.327671066Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-13T22:39:23.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.386320806Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-13T22:39:23.559 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.428265584Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-13T22:39:23.560 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.47032722Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-13T22:39:23.560 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.495658042Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-13T22:39:23.560 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.521026215Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-13T22:39:23.560 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.562875497Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-13T22:39:23.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.588190593Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-13T22:39:23.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.630279106Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-13T22:39:23.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.672517652Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-13T22:39:23.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.71440745Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-13T22:39:23.820 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.756520119Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-13T22:39:23.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.781803716Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-13T22:39:23.821 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.823785133Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-13T22:39:23.991 INFO:teuthology.orchestra.run.smithi178.stdout:34359738471 2024-09-13T22:39:23.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T22:39:24.069 INFO:teuthology.orchestra.run.smithi178.stdout:287762808860 2024-09-13T22:39:24.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.840741776Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.857585228Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.874486521Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.91663311Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.933651953Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.958872497Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:23 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:23.984110328Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.026158127Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.068163728Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-13T22:39:24.107 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.110189641Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-13T22:39:24.287 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:24 smithi178 bash[22840]: cluster 2024-09-13T22:39:23.070302+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T22:39:24.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:24 smithi178 bash[22840]: cluster 2024-09-13T22:39:23.072578+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T22:39:24.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:24 smithi178 bash[29889]: cluster 2024-09-13T22:39:23.070302+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T22:39:24.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:24 smithi178 bash[29889]: cluster 2024-09-13T22:39:23.072578+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T22:39:24.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[24608]: cluster 2024-09-13T22:39:23.070302+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T22:39:24.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[24608]: cluster 2024-09-13T22:39:23.072578+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.152193599Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.194229109Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.219517766Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.244770203Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.270052908Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.295330861Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-13T22:39:24.385 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.320547469Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-13T22:39:24.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.345762741Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-13T22:39:24.386 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.387741868Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-13T22:39:24.389 INFO:teuthology.orchestra.run.smithi178.stdout:146028888124 2024-09-13T22:39:24.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.413052388Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.454938587Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.480351809Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.505899359Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.547855747Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.58993708Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.615178632Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-13T22:39:24.637 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.640437976Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.665703843Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.691031213Z level=info msg="Executing migration" id="create alert_image table" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.732953561Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.775071107Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.800416514Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.825675285Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.867681492Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T22:39:24.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.867964979Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.909721639Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.935028665Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.960305746Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.960703564Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.961093549Z level=info msg="alerts found to migrate" alerts=0 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:24 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:24.985583897Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.027605304Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-13T22:39:25.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.061327178Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-13T22:39:25.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.078262004Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-13T22:39:25.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.09513419Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-13T22:39:25.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.112095137Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-13T22:39:25.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.128962045Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-13T22:39:25.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.145945539Z level=info msg="Executing migration" id="create data_keys table" 2024-09-13T22:39:25.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.031494+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:39:25.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.033217+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:39:25.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.355424+0000 osd.5 (osd.5) 39 : cluster [DBG] 3.5 deep-scrub starts 2024-09-13T22:39:25.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.367830+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.5 deep-scrub ok 2024-09-13T22:39:25.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.437998+0000 mgr.y (mgr.14613) 57 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:25.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.759738+0000 osd.6 (osd.6) 27 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:39:25.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[24608]: cluster 2024-09-13T22:39:24.762112+0000 osd.6 (osd.6) 28 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:39:25.420 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.196351475Z level=info msg="Executing migration" id="create secrets table" 2024-09-13T22:39:25.420 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.221643447Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-13T22:39:25.420 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.263475733Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-13T22:39:25.420 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.329473907Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-13T22:39:25.420 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.381086143Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-13T22:39:25.421 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.423033395Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-13T22:39:25.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.031494+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:39:25.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.033217+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:39:25.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.355424+0000 osd.5 (osd.5) 39 : cluster [DBG] 3.5 deep-scrub starts 2024-09-13T22:39:25.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.367830+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.5 deep-scrub ok 2024-09-13T22:39:25.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.437998+0000 mgr.y (mgr.14613) 57 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:25.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.759738+0000 osd.6 (osd.6) 27 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[29889]: cluster 2024-09-13T22:39:24.762112+0000 osd.6 (osd.6) 28 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.031494+0000 osd.1 (osd.1) 47 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.033217+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.355424+0000 osd.5 (osd.5) 39 : cluster [DBG] 3.5 deep-scrub starts 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.367830+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.5 deep-scrub ok 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.437998+0000 mgr.y (mgr.14613) 57 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.759738+0000 osd.6 (osd.6) 27 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:39:25.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:25 smithi178 bash[22840]: cluster 2024-09-13T22:39:24.762112+0000 osd.6 (osd.6) 28 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:39:25.688 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.481681033Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.50701126Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.532260435Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.574145921Z level=info msg="Executing migration" id="create permission table" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.599512214Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.641500222Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.666799006Z level=info msg="Executing migration" id="create role table" 2024-09-13T22:39:25.689 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.692032127Z level=info msg="Executing migration" id="add column display_name" 2024-09-13T22:39:25.738 INFO:teuthology.orchestra.run.smithi178.stdout:81604378706 2024-09-13T22:39:25.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T22:39:25.958 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.717262573Z level=info msg="Executing migration" id="add column group_name" 2024-09-13T22:39:25.958 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.759334706Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-13T22:39:25.958 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.776395768Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-13T22:39:25.958 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.80163942Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-13T22:39:25.958 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.843722417Z level=info msg="Executing migration" id="create team role table" 2024-09-13T22:39:25.959 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.868989178Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-13T22:39:25.959 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.911045748Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-13T22:39:25.959 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.93630732Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-13T22:39:25.959 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.961530058Z level=info msg="Executing migration" id="create user role table" 2024-09-13T22:39:26.227 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:25 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:25.986729187Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-13T22:39:26.227 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.028736449Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-13T22:39:26.227 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.070791576Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-13T22:39:26.227 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.096115973Z level=info msg="Executing migration" id="create builtin role table" 2024-09-13T22:39:26.227 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.138147272Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-13T22:39:26.227 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.180064518Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-13T22:39:26.228 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.205465278Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-13T22:39:26.228 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.230601094Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: audit 2024-09-13T22:39:24.850091+0000 mgr.y (mgr.14613) 58 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: cluster 2024-09-13T22:39:25.016772+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: cluster 2024-09-13T22:39:25.019159+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: cluster 2024-09-13T22:39:25.683068+0000 osd.0 (osd.0) 43 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: cluster 2024-09-13T22:39:25.685419+0000 osd.0 (osd.0) 44 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: cluster 2024-09-13T22:39:25.741694+0000 osd.7 (osd.7) 25 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:39:26.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[24608]: cluster 2024-09-13T22:39:25.745601+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:39:26.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: audit 2024-09-13T22:39:24.850091+0000 mgr.y (mgr.14613) 58 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:26.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: cluster 2024-09-13T22:39:25.016772+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: cluster 2024-09-13T22:39:25.019159+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: cluster 2024-09-13T22:39:25.683068+0000 osd.0 (osd.0) 43 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: cluster 2024-09-13T22:39:25.685419+0000 osd.0 (osd.0) 44 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: cluster 2024-09-13T22:39:25.741694+0000 osd.7 (osd.7) 25 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[22840]: cluster 2024-09-13T22:39:25.745601+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: audit 2024-09-13T22:39:24.850091+0000 mgr.y (mgr.14613) 58 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: cluster 2024-09-13T22:39:25.016772+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: cluster 2024-09-13T22:39:25.019159+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: cluster 2024-09-13T22:39:25.683068+0000 osd.0 (osd.0) 43 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: cluster 2024-09-13T22:39:25.685419+0000 osd.0 (osd.0) 44 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:39:26.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: cluster 2024-09-13T22:39:25.741694+0000 osd.7 (osd.7) 25 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:39:26.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:26 smithi178 bash[29889]: cluster 2024-09-13T22:39:25.745601+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.256246864Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.281451819Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.32334516Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.348802748Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.374109713Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.416034335Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-13T22:39:26.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.458042697Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-13T22:39:26.657 INFO:teuthology.orchestra.run.smithi178.stdout:55834574941 2024-09-13T22:39:26.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T22:39:26.741 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.483320652Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-13T22:39:26.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.508599846Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-13T22:39:26.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.533779032Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-13T22:39:26.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.575731561Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-13T22:39:26.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.601136591Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-13T22:39:26.742 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.634793755Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-13T22:39:26.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.651700464Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-13T22:39:26.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.668682834Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-13T22:39:26.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.685567942Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-13T22:39:26.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.719338986Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-13T22:39:26.743 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.744543834Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-13T22:39:27.010 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.78647181Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-13T22:39:27.010 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.828694966Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-13T22:39:27.010 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.895676112Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-13T22:39:27.010 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.921025509Z level=info msg="Executing migration" id="add correlation config column" 2024-09-13T22:39:27.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.946238905Z level=info msg="Executing migration" id="create entity_events table" 2024-09-13T22:39:27.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:26 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:26.971540556Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-13T22:39:27.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.013528471Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T22:39:27.011 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.013871221Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T22:39:27.262 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[24608]: cluster 2024-09-13T22:39:26.305965+0000 osd.5 (osd.5) 41 : cluster [DBG] 3.1c deep-scrub starts 2024-09-13T22:39:27.262 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[24608]: cluster 2024-09-13T22:39:26.318739+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.1c deep-scrub ok 2024-09-13T22:39:27.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[24608]: cluster 2024-09-13T22:39:26.438817+0000 mgr.y (mgr.14613) 59 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-13T22:39:27.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[24608]: cluster 2024-09-13T22:39:26.770300+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:39:27.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[24608]: cluster 2024-09-13T22:39:26.772336+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:39:27.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.055498964Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T22:39:27.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.05582475Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T22:39:27.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.172759158Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-13T22:39:27.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.215421865Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-13T22:39:27.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.240628522Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-13T22:39:27.263 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.265954309Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T22:39:27.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[22840]: cluster 2024-09-13T22:39:26.305965+0000 osd.5 (osd.5) 41 : cluster [DBG] 3.1c deep-scrub starts 2024-09-13T22:39:27.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[22840]: cluster 2024-09-13T22:39:26.318739+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.1c deep-scrub ok 2024-09-13T22:39:27.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[22840]: cluster 2024-09-13T22:39:26.438817+0000 mgr.y (mgr.14613) 59 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-13T22:39:27.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[22840]: cluster 2024-09-13T22:39:26.770300+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:39:27.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[22840]: cluster 2024-09-13T22:39:26.772336+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:39:27.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[29889]: cluster 2024-09-13T22:39:26.305965+0000 osd.5 (osd.5) 41 : cluster [DBG] 3.1c deep-scrub starts 2024-09-13T22:39:27.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[29889]: cluster 2024-09-13T22:39:26.318739+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.1c deep-scrub ok 2024-09-13T22:39:27.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[29889]: cluster 2024-09-13T22:39:26.438817+0000 mgr.y (mgr.14613) 59 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-13T22:39:27.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[29889]: cluster 2024-09-13T22:39:26.770300+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:39:27.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:27 smithi178 bash[29889]: cluster 2024-09-13T22:39:26.772336+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:39:27.494 INFO:teuthology.orchestra.run.smithi178.stdout:120259084359 2024-09-13T22:39:27.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T22:39:27.531 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.291154515Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.333171994Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.391877323Z level=info msg="Executing migration" id="Drop public config table" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.41728569Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.459219393Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.484486331Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.509748287Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-13T22:39:27.532 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.535056491Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.585285088Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.627270524Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.652585639Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.669494194Z level=info msg="Executing migration" id="add share column" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.694716033Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.71169775Z level=info msg="Executing migration" id="create file table" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.745452078Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.77925165Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-13T22:39:27.801 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.804439624Z level=info msg="Executing migration" id="create file_meta table" 2024-09-13T22:39:27.934 INFO:teuthology.orchestra.run.smithi178.stdout:244813135912 2024-09-13T22:39:27.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.829676336Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.85497011Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.88014772Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.905433138Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.947468783Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:27 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:27.973018288Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.015009167Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.040358129Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-13T22:39:28.079 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.082350014Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-13T22:39:28.347 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.107620443Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.132839436Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.174797227Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.208436966Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.233685471Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.258931404Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.325979464Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-13T22:39:28.348 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.35127312Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.39323174Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.435250036Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.477252492Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.502500554Z level=info msg="Executing migration" id="create folder table" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.5277274Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.552904613Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.594981029Z level=info msg="Executing migration" id="Update folder title length" 2024-09-13T22:39:28.617 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.620274041Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-13T22:39:28.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[24608]: cluster 2024-09-13T22:39:27.115727+0000 osd.3 (osd.3) 37 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:39:28.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[24608]: cluster 2024-09-13T22:39:27.118147+0000 osd.3 (osd.3) 38 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:39:28.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[24608]: cluster 2024-09-13T22:39:27.603743+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:39:28.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[24608]: cluster 2024-09-13T22:39:27.605601+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:39:28.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[24608]: cluster 2024-09-13T22:39:28.439286+0000 mgr.y (mgr.14613) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-13T22:39:28.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=migrator t=2024-09-13T22:39:28.645506597Z level=info msg="migrations completed" performed=485 skipped=0 duration=16.385730649s 2024-09-13T22:39:28.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=sqlstore t=2024-09-13T22:39:28.646983965Z level=info msg="Created default organization" 2024-09-13T22:39:28.906 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=plugin.finder t=2024-09-13T22:39:28.756760455Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-13T22:39:28.907 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=secrets t=2024-09-13T22:39:28.756977868Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T22:39:28.907 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=query_data t=2024-09-13T22:39:28.761285963Z level=info msg="Query Service initialization" 2024-09-13T22:39:28.907 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:28 smithi192 bash[50858]: logger=live.push_http t=2024-09-13T22:39:28.773756627Z level=info msg="Live Push Gateway initialization" 2024-09-13T22:39:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[29889]: cluster 2024-09-13T22:39:27.115727+0000 osd.3 (osd.3) 37 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:39:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[29889]: cluster 2024-09-13T22:39:27.118147+0000 osd.3 (osd.3) 38 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:39:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[29889]: cluster 2024-09-13T22:39:27.603743+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:39:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[29889]: cluster 2024-09-13T22:39:27.605601+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:39:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[29889]: cluster 2024-09-13T22:39:28.439286+0000 mgr.y (mgr.14613) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-13T22:39:28.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[22840]: cluster 2024-09-13T22:39:27.115727+0000 osd.3 (osd.3) 37 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:39:28.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[22840]: cluster 2024-09-13T22:39:27.118147+0000 osd.3 (osd.3) 38 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:39:28.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[22840]: cluster 2024-09-13T22:39:27.603743+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:39:28.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[22840]: cluster 2024-09-13T22:39:27.605601+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:39:28.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:28 smithi178 bash[22840]: cluster 2024-09-13T22:39:28.439286+0000 mgr.y (mgr.14613) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-13T22:39:30.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:29 smithi178 bash[22840]: cluster 2024-09-13T22:39:28.139732+0000 osd.3 (osd.3) 39 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:39:30.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:29 smithi178 bash[22840]: cluster 2024-09-13T22:39:28.141323+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:39:30.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:29 smithi178 bash[29889]: cluster 2024-09-13T22:39:28.139732+0000 osd.3 (osd.3) 39 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:39:30.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:29 smithi178 bash[29889]: cluster 2024-09-13T22:39:28.141323+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:39:30.008 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:29 smithi178 systemd[1]: Stopping Ceph alertmanager.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T22:39:30.009 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:29 smithi178 bash[57602]: ts=2024-09-13T22:39:29.960Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T22:39:30.013 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[24608]: cluster 2024-09-13T22:39:28.139732+0000 osd.3 (osd.3) 39 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:39:30.014 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[24608]: cluster 2024-09-13T22:39:28.141323+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=infra.usagestats.collector t=2024-09-13T22:39:29.911564905Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=provisioning.datasources t=2024-09-13T22:39:29.912846205Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=provisioning.datasources t=2024-09-13T22:39:29.946697603Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=provisioning.alerting t=2024-09-13T22:39:29.980214662Z level=info msg="starting to provision alerting" 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=provisioning.alerting t=2024-09-13T22:39:29.980236535Z level=info msg="finished to provision alerting" 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=grafanaStorageLogger t=2024-09-13T22:39:29.980995253Z level=info msg="storage starting" 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=ngalert.state.manager t=2024-09-13T22:39:29.98112185Z level=info msg="Warming state cache for startup" 2024-09-13T22:39:30.014 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:29 smithi192 bash[50858]: logger=http.server t=2024-09-13T22:39:29.983293647Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T22:39:30.298 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:30.299 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:30.299 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:30.299 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:30.300 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=ngalert.state.manager t=2024-09-13T22:39:30.017769914Z level=info msg="State cache has been initialized" states=0 duration=36.64683ms 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=ticker t=2024-09-13T22:39:30.017856007Z level=info msg=starting first_tick=2024-09-13T22:39:40Z 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=ngalert.multiorg.alertmanager t=2024-09-13T22:39:30.017822028Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=sqlstore.transactions t=2024-09-13T22:39:30.029016037Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=sqlstore.transactions t=2024-09-13T22:39:30.039498481Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=sqlstore.transactions t=2024-09-13T22:39:30.050265679Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-13T22:39:30.406 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[50858]: logger=sqlstore.transactions t=2024-09-13T22:39:30.114594005Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-13T22:39:30.763 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[59962]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T22:39:31.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[29889]: cluster 2024-09-13T22:39:29.135224+0000 osd.3 (osd.3) 41 : cluster [DBG] 5.13 deep-scrub starts 2024-09-13T22:39:31.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[29889]: cluster 2024-09-13T22:39:29.136351+0000 osd.3 (osd.3) 42 : cluster [DBG] 5.13 deep-scrub ok 2024-09-13T22:39:31.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[29889]: cluster 2024-09-13T22:39:29.663496+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:39:31.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[29889]: cluster 2024-09-13T22:39:29.665834+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:39:31.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[29889]: cluster 2024-09-13T22:39:30.440745+0000 mgr.y (mgr.14613) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:31.133 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[22840]: cluster 2024-09-13T22:39:29.135224+0000 osd.3 (osd.3) 41 : cluster [DBG] 5.13 deep-scrub starts 2024-09-13T22:39:31.134 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[22840]: cluster 2024-09-13T22:39:29.136351+0000 osd.3 (osd.3) 42 : cluster [DBG] 5.13 deep-scrub ok 2024-09-13T22:39:31.134 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[22840]: cluster 2024-09-13T22:39:29.663496+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:39:31.134 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[22840]: cluster 2024-09-13T22:39:29.665834+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:39:31.134 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:30 smithi178 bash[22840]: cluster 2024-09-13T22:39:30.440745+0000 mgr.y (mgr.14613) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[24608]: cluster 2024-09-13T22:39:29.135224+0000 osd.3 (osd.3) 41 : cluster [DBG] 5.13 deep-scrub starts 2024-09-13T22:39:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[24608]: cluster 2024-09-13T22:39:29.136351+0000 osd.3 (osd.3) 42 : cluster [DBG] 5.13 deep-scrub ok 2024-09-13T22:39:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[24608]: cluster 2024-09-13T22:39:29.663496+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:39:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[24608]: cluster 2024-09-13T22:39:29.665834+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:39:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:30 smithi192 bash[24608]: cluster 2024-09-13T22:39:30.440745+0000 mgr.y (mgr.14613) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:39:31.385 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[60092]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T22:39:31.630 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:31.669 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:31 smithi178 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T22:39:31.669 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:31 smithi178 systemd[1]: Stopped Ceph alertmanager.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:39:31.669 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:31 smithi178 systemd[1]: Started Ceph alertmanager.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: cluster 2024-09-13T22:39:30.702465+0000 osd.7 (osd.7) 31 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: cluster 2024-09-13T22:39:30.706875+0000 osd.7 (osd.7) 32 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.437077+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.438859+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: audit 2024-09-13T22:39:31.443459+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: audit 2024-09-13T22:39:31.452770+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:31.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[29889]: cephadm 2024-09-13T22:39:31.457295+0000 mgr.y (mgr.14613) 62 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: cluster 2024-09-13T22:39:30.702465+0000 osd.7 (osd.7) 31 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: cluster 2024-09-13T22:39:30.706875+0000 osd.7 (osd.7) 32 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.437077+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.438859+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: audit 2024-09-13T22:39:31.443459+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: audit 2024-09-13T22:39:31.452770+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:31.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:31 smithi178 bash[22840]: cephadm 2024-09-13T22:39:31.457295+0000 mgr.y (mgr.14613) 62 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T22:39:32.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: cluster 2024-09-13T22:39:30.702465+0000 osd.7 (osd.7) 31 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:39:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: cluster 2024-09-13T22:39:30.706875+0000 osd.7 (osd.7) 32 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:39:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.437077+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:39:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.438859+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:39:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: audit 2024-09-13T22:39:31.443459+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: audit 2024-09-13T22:39:31.452770+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:31 smithi192 bash[24608]: cephadm 2024-09-13T22:39:31.457295+0000 mgr.y (mgr.14613) 62 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T22:39:32.652 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:32.653 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:32.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.088105+0000 osd.0 (osd.0) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.088239+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.668642+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.670811+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cephadm 2024-09-13T22:39:31.697629+0000 mgr.y (mgr.14613) 63 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cluster 2024-09-13T22:39:31.780711+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: cluster 2024-09-13T22:39:32.441464+0000 mgr.y (mgr.14613) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: audit 2024-09-13T22:39:32.453551+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: audit 2024-09-13T22:39:32.454033+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-09-13T22:39:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[22840]: audit 2024-09-13T22:39:32.619996+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.088105+0000 osd.0 (osd.0) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.088239+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.668642+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.670811+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cephadm 2024-09-13T22:39:31.697629+0000 mgr.y (mgr.14613) 63 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cluster 2024-09-13T22:39:31.780711+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: cluster 2024-09-13T22:39:32.441464+0000 mgr.y (mgr.14613) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: audit 2024-09-13T22:39:32.453551+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: audit 2024-09-13T22:39:32.454033+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-09-13T22:39:32.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:32 smithi178 bash[29889]: audit 2024-09-13T22:39:32.619996+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.088105+0000 osd.0 (osd.0) 47 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.088239+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.668642+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.670811+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cephadm 2024-09-13T22:39:31.697629+0000 mgr.y (mgr.14613) 63 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi192 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cluster 2024-09-13T22:39:31.780711+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: cluster 2024-09-13T22:39:32.441464+0000 mgr.y (mgr.14613) 64 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: audit 2024-09-13T22:39:32.453551+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-09-13T22:39:33.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: audit 2024-09-13T22:39:32.454033+0000 mon.a (mon.0) 877 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-09-13T22:39:33.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:32 smithi192 bash[24608]: audit 2024-09-13T22:39:32.619996+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:34.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:33 smithi192 bash[24608]: audit 2024-09-13T22:39:32.783938+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-09-13T22:39:34.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:33 smithi192 bash[24608]: cluster 2024-09-13T22:39:32.788957+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-13T22:39:34.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:33 smithi192 bash[24608]: cluster 2024-09-13T22:39:33.389420+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T22:39:34.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:33 smithi192 bash[24608]: cluster 2024-09-13T22:39:33.392301+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T22:39:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[29889]: audit 2024-09-13T22:39:32.783938+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-09-13T22:39:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[29889]: cluster 2024-09-13T22:39:32.788957+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-13T22:39:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[29889]: cluster 2024-09-13T22:39:33.389420+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T22:39:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[29889]: cluster 2024-09-13T22:39:33.392301+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T22:39:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[22840]: audit 2024-09-13T22:39:32.783938+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-09-13T22:39:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[22840]: cluster 2024-09-13T22:39:32.788957+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-13T22:39:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[22840]: cluster 2024-09-13T22:39:33.389420+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T22:39:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:33 smithi178 bash[22840]: cluster 2024-09-13T22:39:33.392301+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T22:39:35.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:34 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:39:35.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:34 smithi192 bash[24608]: cluster 2024-09-13T22:39:33.802151+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-13T22:39:35.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:34 smithi192 bash[24608]: cluster 2024-09-13T22:39:34.442291+0000 mgr.y (mgr.14613) 65 : cluster [DBG] pgmap v45: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:39:35.162 INFO:teuthology.orchestra.run.smithi178.stdout:81604378709 2024-09-13T22:39:35.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:34 smithi178 bash[29889]: cluster 2024-09-13T22:39:33.802151+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-13T22:39:35.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:34 smithi178 bash[29889]: cluster 2024-09-13T22:39:34.442291+0000 mgr.y (mgr.14613) 65 : cluster [DBG] pgmap v45: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:39:35.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:34 smithi178 bash[22840]: cluster 2024-09-13T22:39:33.802151+0000 mon.a (mon.0) 880 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-13T22:39:35.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:34 smithi178 bash[22840]: cluster 2024-09-13T22:39:34.442291+0000 mgr.y (mgr.14613) 65 : cluster [DBG] pgmap v45: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:39:36.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[24608]: cluster 2024-09-13T22:39:34.794717+0000 mon.a (mon.0) 881 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T22:39:36.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[24608]: audit 2024-09-13T22:39:34.861211+0000 mgr.y (mgr.14613) 66 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:36.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[24608]: audit 2024-09-13T22:39:35.165957+0000 mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.178:0/1445058573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T22:39:36.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 systemd[1]: Stopping Ceph prometheus.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T22:39:36.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T22:39:36.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T22:39:36.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T22:39:36.062 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.876Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.877Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.877Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T22:39:36.063 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:35 smithi192 bash[49678]: ts=2024-09-13T22:39:35.877Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T22:39:36.073 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:35 smithi178 bash[29889]: cluster 2024-09-13T22:39:34.794717+0000 mon.a (mon.0) 881 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T22:39:36.073 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:35 smithi178 bash[29889]: audit 2024-09-13T22:39:34.861211+0000 mgr.y (mgr.14613) 66 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:36.073 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:35 smithi178 bash[29889]: audit 2024-09-13T22:39:35.165957+0000 mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.178:0/1445058573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T22:39:36.074 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:35 smithi178 bash[22840]: cluster 2024-09-13T22:39:34.794717+0000 mon.a (mon.0) 881 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T22:39:36.074 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:35 smithi178 bash[22840]: audit 2024-09-13T22:39:34.861211+0000 mgr.y (mgr.14613) 66 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:36.074 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:35 smithi178 bash[22840]: audit 2024-09-13T22:39:35.165957+0000 mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.178:0/1445058573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T22:39:36.084 INFO:teuthology.orchestra.run.smithi178.stdout:146028888127 2024-09-13T22:39:36.313 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[51511]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T22:39:36.385 INFO:teuthology.orchestra.run.smithi178.stdout:34359738474 2024-09-13T22:39:36.641 INFO:teuthology.orchestra.run.smithi178.stdout:287762808863 2024-09-13T22:39:36.809 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[51572]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T22:39:36.809 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:36 smithi192 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T22:39:36.809 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:36 smithi192 systemd[1]: Stopped Ceph prometheus.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:39:36.809 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:36 smithi192 systemd[1]: Started Ceph prometheus.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T22:39:36.881 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE Bus STOPPING 2024-09-13T22:39:36.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: cluster 2024-09-13T22:39:35.659911+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: cluster 2024-09-13T22:39:35.662387+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.088574+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.178:0/746798214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.389402+0000 mon.a (mon.0) 883 : audit [DBG] from='client.? v1:172.21.15.178:0/1841292628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: cluster 2024-09-13T22:39:36.442734+0000 mgr.y (mgr.14613) 67 : cluster [DBG] pgmap v46: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.589531+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.599754+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.606078+0000 mon.b (mon.1) 74 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.607722+0000 mon.b (mon.1) 75 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi178.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.614481+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.630671+0000 mon.b (mon.1) 76 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.635083+0000 mon.b (mon.1) 77 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T22:39:36.882 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.640586+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.644366+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.178:0/2041128975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.652216+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.656907+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.661776+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[29889]: audit 2024-09-13T22:39:36.733232+0000 mon.b (mon.1) 80 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: cluster 2024-09-13T22:39:35.659911+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: cluster 2024-09-13T22:39:35.662387+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.088574+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.178:0/746798214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.389402+0000 mon.a (mon.0) 883 : audit [DBG] from='client.? v1:172.21.15.178:0/1841292628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T22:39:36.883 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: cluster 2024-09-13T22:39:36.442734+0000 mgr.y (mgr.14613) 67 : cluster [DBG] pgmap v46: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.589531+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.599754+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.606078+0000 mon.b (mon.1) 74 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.607722+0000 mon.b (mon.1) 75 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi178.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.614481+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.630671+0000 mon.b (mon.1) 76 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.635083+0000 mon.b (mon.1) 77 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.640586+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.644366+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.178:0/2041128975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.652216+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.656907+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.661776+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:36.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[22840]: audit 2024-09-13T22:39:36.733232+0000 mon.b (mon.1) 80 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: cluster 2024-09-13T22:39:35.659911+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: cluster 2024-09-13T22:39:35.662387+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.088574+0000 mon.a (mon.0) 882 : audit [DBG] from='client.? v1:172.21.15.178:0/746798214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.389402+0000 mon.a (mon.0) 883 : audit [DBG] from='client.? v1:172.21.15.178:0/1841292628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: cluster 2024-09-13T22:39:36.442734+0000 mgr.y (mgr.14613) 67 : cluster [DBG] pgmap v46: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.589531+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.599754+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.606078+0000 mon.b (mon.1) 74 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T22:39:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.607722+0000 mon.b (mon.1) 75 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi178.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.614481+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.630671+0000 mon.b (mon.1) 76 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.635083+0000 mon.b (mon.1) 77 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.640586+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.644366+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.178:0/2041128975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.652216+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.656907+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.661776+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:37.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:36 smithi192 bash[24608]: audit 2024-09-13T22:39:36.733232+0000 mon.b (mon.1) 80 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:39:37.159 INFO:teuthology.orchestra.run.smithi178.stdout:193273528373 2024-09-13T22:39:37.172 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T22:39:37.172 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE Bus STOPPED 2024-09-13T22:39:37.172 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE Bus STARTING 2024-09-13T22:39:37.173 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE Serving on http://:::9283 2024-09-13T22:39:37.173 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE Bus STARTED 2024-09-13T22:39:37.173 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:36 smithi178 bash[23095]: [13/Sep/2024:22:39:36] ENGINE Bus STOPPING 2024-09-13T22:39:37.732 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[23095]: [13/Sep/2024:22:39:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T22:39:37.732 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[23095]: [13/Sep/2024:22:39:37] ENGINE Bus STOPPED 2024-09-13T22:39:37.732 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[23095]: [13/Sep/2024:22:39:37] ENGINE Bus STARTING 2024-09-13T22:39:37.732 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[23095]: [13/Sep/2024:22:39:37] ENGINE Serving on http://:::9283 2024-09-13T22:39:37.733 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[23095]: [13/Sep/2024:22:39:37] ENGINE Bus STARTED 2024-09-13T22:39:37.733 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[23095]: [13/Sep/2024:22:39:37] ENGINE Bus STOPPING 2024-09-13T22:39:37.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378706 got 81604378709 for osd.2 2024-09-13T22:39:37.774 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:38.099 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:36.606272+0000 mgr.y (mgr.14613) 68 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T22:39:38.099 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:36.607770+0000 mgr.y (mgr.14613) 69 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi178.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:36.631032+0000 mgr.y (mgr.14613) 70 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:36.635320+0000 mgr.y (mgr.14613) 71 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:36.652505+0000 mgr.y (mgr.14613) 72 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:36.657129+0000 mgr.y (mgr.14613) 73 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: audit 2024-09-13T22:39:37.163214+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.178:0/1692665989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: cluster 2024-09-13T22:39:37.423171+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[29889]: cluster 2024-09-13T22:39:37.426086+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:39:38.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:36.606272+0000 mgr.y (mgr.14613) 68 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:36.607770+0000 mgr.y (mgr.14613) 69 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi178.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:36.631032+0000 mgr.y (mgr.14613) 70 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:36.635320+0000 mgr.y (mgr.14613) 71 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:36.652505+0000 mgr.y (mgr.14613) 72 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:36.657129+0000 mgr.y (mgr.14613) 73 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: audit 2024-09-13T22:39:37.163214+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.178:0/1692665989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: cluster 2024-09-13T22:39:37.423171+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:39:38.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:37 smithi178 bash[22840]: cluster 2024-09-13T22:39:37.426086+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:36.606272+0000 mgr.y (mgr.14613) 68 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:36.607770+0000 mgr.y (mgr.14613) 69 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi178.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:36.631032+0000 mgr.y (mgr.14613) 70 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:36.635320+0000 mgr.y (mgr.14613) 71 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:36.652505+0000 mgr.y (mgr.14613) 72 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:36.657129+0000 mgr.y (mgr.14613) 73 : audit [DBG] from='mon.? v1:172.21.15.192:0/3641649214' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: audit 2024-09-13T22:39:37.163214+0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.178:0/1692665989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: cluster 2024-09-13T22:39:37.423171+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:39:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[24608]: cluster 2024-09-13T22:39:37.426086+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.789Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.789Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.789Z caller=main.go:570 level=info host_details="(Linux 5.15.0-121-generic #131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024 x86_64 smithi192 (none))" 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.789Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.790Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.791Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.792Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.793Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.793Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.795Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T22:39:38.157 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.795Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.511µs 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.795Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.796Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.796Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.796Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=52.17µs wal_replay_duration=941.055µs wbl_replay_duration=122ns total_replay_duration=1.128887ms 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.797Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.797Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.797Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.823Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.392239ms db_storage=872ns remote_storage=1.181µs web_handler=403ns query_engine=569ns scrape=281.39µs scrape_sd=106.944µs notify=19.915µs notify_sd=19.563µs rules=24.549956ms tracing=6.569µs 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.823Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T22:39:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 22:39:37 smithi192 bash[51599]: ts=2024-09-13T22:39:37.823Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T22:39:38.401 INFO:teuthology.orchestra.run.smithi178.stdout:55834574943 2024-09-13T22:39:38.441 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[23095]: [13/Sep/2024:22:39:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T22:39:38.442 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[23095]: [13/Sep/2024:22:39:38] ENGINE Bus STOPPED 2024-09-13T22:39:38.442 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[23095]: [13/Sep/2024:22:39:38] ENGINE Bus STARTING 2024-09-13T22:39:38.442 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[23095]: [13/Sep/2024:22:39:38] ENGINE Serving on http://:::9283 2024-09-13T22:39:38.442 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[23095]: [13/Sep/2024:22:39:38] ENGINE Bus STARTED 2024-09-13T22:39:38.611 INFO:teuthology.orchestra.run.smithi178.stdout:244813135914 2024-09-13T22:39:39.016 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[29889]: audit 2024-09-13T22:39:38.405330+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.178:0/155107045' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T22:39:39.016 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[29889]: cluster 2024-09-13T22:39:38.443118+0000 mgr.y (mgr.14613) 74 : cluster [DBG] pgmap v47: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-13T22:39:39.016 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[29889]: audit 2024-09-13T22:39:38.615107+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.178:0/974405475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T22:39:39.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[22840]: audit 2024-09-13T22:39:38.405330+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.178:0/155107045' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T22:39:39.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[22840]: cluster 2024-09-13T22:39:38.443118+0000 mgr.y (mgr.14613) 74 : cluster [DBG] pgmap v47: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-13T22:39:39.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:38 smithi178 bash[22840]: audit 2024-09-13T22:39:38.615107+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.178:0/974405475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T22:39:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:38 smithi192 bash[24608]: audit 2024-09-13T22:39:38.405330+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.178:0/155107045' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T22:39:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:38 smithi192 bash[24608]: cluster 2024-09-13T22:39:38.443118+0000 mgr.y (mgr.14613) 74 : cluster [DBG] pgmap v47: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-13T22:39:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:38 smithi192 bash[24608]: audit 2024-09-13T22:39:38.615107+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.178:0/974405475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T22:39:39.462 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.036Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T22:39:39.462 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.036Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T22:39:39.463 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.038Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.178 port=9094 2024-09-13T22:39:39.463 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.039Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T22:39:39.463 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.070Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T22:39:39.463 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.070Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T22:39:39.463 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.073Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T22:39:39.463 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[60120]: ts=2024-09-13T22:39:39.073Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T22:39:39.788 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808860 got 287762808863 for osd.7 2024-09-13T22:39:39.789 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:39.864 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738471 got 34359738474 for osd.0 2024-09-13T22:39:39.865 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:40.003 INFO:teuthology.orchestra.run.smithi178.stdout:120259084361 2024-09-13T22:39:40.144 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[29889]: cluster 2024-09-13T22:39:38.222478+0000 osd.3 (osd.3) 43 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[29889]: cluster 2024-09-13T22:39:38.224362+0000 osd.3 (osd.3) 44 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[29889]: cluster 2024-09-13T22:39:38.910007+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[29889]: cluster 2024-09-13T22:39:38.910135+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[22840]: cluster 2024-09-13T22:39:38.222478+0000 osd.3 (osd.3) 43 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[22840]: cluster 2024-09-13T22:39:38.224362+0000 osd.3 (osd.3) 44 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[22840]: cluster 2024-09-13T22:39:38.910007+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:40.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:39 smithi178 bash[22840]: cluster 2024-09-13T22:39:38.910135+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:39 smithi192 bash[24608]: cluster 2024-09-13T22:39:38.222478+0000 osd.3 (osd.3) 43 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:39:40.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:39 smithi192 bash[24608]: cluster 2024-09-13T22:39:38.224362+0000 osd.3 (osd.3) 44 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:39:40.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:39 smithi192 bash[24608]: cluster 2024-09-13T22:39:38.910007+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:40.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:39 smithi192 bash[24608]: cluster 2024-09-13T22:39:38.910135+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:41.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:40 smithi178 bash[29889]: cluster 2024-09-13T22:39:39.828735+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-13T22:39:41.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:40 smithi178 bash[29889]: audit 2024-09-13T22:39:40.007103+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? v1:172.21.15.178:0/4189843658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T22:39:41.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:40 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.443731+0000 mgr.y (mgr.14613) 75 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:39:41.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:40 smithi178 bash[22840]: cluster 2024-09-13T22:39:39.828735+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-13T22:39:41.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:40 smithi178 bash[22840]: audit 2024-09-13T22:39:40.007103+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? v1:172.21.15.178:0/4189843658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T22:39:41.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:40 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.443731+0000 mgr.y (mgr.14613) 75 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:39:41.016 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528370 got 193273528373 for osd.5 2024-09-13T22:39:41.016 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:41.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888124 got 146028888127 for osd.4 2024-09-13T22:39:41.045 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:41.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:40 smithi192 bash[24608]: cluster 2024-09-13T22:39:39.828735+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-13T22:39:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:40 smithi192 bash[24608]: audit 2024-09-13T22:39:40.007103+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? v1:172.21.15.178:0/4189843658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T22:39:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:40 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.443731+0000 mgr.y (mgr.14613) 75 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:39:41.428 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[60120]: ts=2024-09-13T22:39:41.040Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000437029s 2024-09-13T22:39:41.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574941 got 55834574943 for osd.1 2024-09-13T22:39:41.755 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.262719+0000 osd.3 (osd.3) 45 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.265341+0000 osd.3 (osd.3) 46 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.609617+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.611985+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.826097+0000 mon.a (mon.0) 890 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[29889]: cluster 2024-09-13T22:39:40.826150+0000 mon.a (mon.0) 891 : cluster [INF] Cluster is now healthy 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.262719+0000 osd.3 (osd.3) 45 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.265341+0000 osd.3 (osd.3) 46 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.609617+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.611985+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.826097+0000 mon.a (mon.0) 890 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T22:39:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:41 smithi178 bash[22840]: cluster 2024-09-13T22:39:40.826150+0000 mon.a (mon.0) 891 : cluster [INF] Cluster is now healthy 2024-09-13T22:39:41.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135912 got 244813135914 for osd.6 2024-09-13T22:39:41.942 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:41.976 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084359 got 120259084361 for osd.3 2024-09-13T22:39:41.977 DEBUG:teuthology.parallel:result is None 2024-09-13T22:39:41.977 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T22:39:41.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:39:42.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:41 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.262719+0000 osd.3 (osd.3) 45 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:39:42.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:41 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.265341+0000 osd.3 (osd.3) 46 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:39:42.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:41 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.609617+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:39:42.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:41 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.611985+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:39:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:41 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.826097+0000 mon.a (mon.0) 890 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T22:39:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:41 smithi192 bash[24608]: cluster 2024-09-13T22:39:40.826150+0000 mon.a (mon.0) 891 : cluster [INF] Cluster is now healthy 2024-09-13T22:39:43.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:42 smithi192 bash[24608]: cluster 2024-09-13T22:39:42.444172+0000 mgr.y (mgr.14613) 76 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-09-13T22:39:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:42 smithi192 bash[24608]: audit 2024-09-13T22:39:42.756712+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:42 smithi192 bash[24608]: audit 2024-09-13T22:39:42.772308+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:42 smithi178 bash[29889]: cluster 2024-09-13T22:39:42.444172+0000 mgr.y (mgr.14613) 76 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-09-13T22:39:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:42 smithi178 bash[29889]: audit 2024-09-13T22:39:42.756712+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:42 smithi178 bash[29889]: audit 2024-09-13T22:39:42.772308+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:42 smithi178 bash[22840]: cluster 2024-09-13T22:39:42.444172+0000 mgr.y (mgr.14613) 76 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-09-13T22:39:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:42 smithi178 bash[22840]: audit 2024-09-13T22:39:42.756712+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:42 smithi178 bash[22840]: audit 2024-09-13T22:39:42.772308+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:44.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:43 smithi192 bash[24608]: cluster 2024-09-13T22:39:42.661962+0000 osd.7 (osd.7) 39 : cluster [DBG] 3.d scrub starts 2024-09-13T22:39:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:43 smithi192 bash[24608]: cluster 2024-09-13T22:39:42.666492+0000 osd.7 (osd.7) 40 : cluster [DBG] 3.d scrub ok 2024-09-13T22:39:44.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:43 smithi178 bash[29889]: cluster 2024-09-13T22:39:42.661962+0000 osd.7 (osd.7) 39 : cluster [DBG] 3.d scrub starts 2024-09-13T22:39:44.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:43 smithi178 bash[29889]: cluster 2024-09-13T22:39:42.666492+0000 osd.7 (osd.7) 40 : cluster [DBG] 3.d scrub ok 2024-09-13T22:39:44.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:43 smithi178 bash[22840]: cluster 2024-09-13T22:39:42.661962+0000 osd.7 (osd.7) 39 : cluster [DBG] 3.d scrub starts 2024-09-13T22:39:44.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:43 smithi178 bash[22840]: cluster 2024-09-13T22:39:42.666492+0000 osd.7 (osd.7) 40 : cluster [DBG] 3.d scrub ok 2024-09-13T22:39:45.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:44 smithi192 bash[24608]: cluster 2024-09-13T22:39:44.445584+0000 mgr.y (mgr.14613) 77 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:45.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:44 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:39:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:44 smithi178 bash[22840]: cluster 2024-09-13T22:39:44.445584+0000 mgr.y (mgr.14613) 77 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:45.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:44 smithi178 bash[29889]: cluster 2024-09-13T22:39:44.445584+0000 mgr.y (mgr.14613) 77 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:45.804 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: audit 2024-09-13T22:39:44.869911+0000 mgr.y (mgr.14613) 78 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: cluster 2024-09-13T22:39:45.279801+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: cluster 2024-09-13T22:39:45.281508+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: audit 2024-09-13T22:39:45.328352+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: audit 2024-09-13T22:39:45.339243+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: audit 2024-09-13T22:39:45.932826+0000 mon.b (mon.1) 81 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: audit 2024-09-13T22:39:45.934290+0000 mon.b (mon.1) 82 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:39:46.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:46 smithi192 bash[24608]: audit 2024-09-13T22:39:45.940150+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: audit 2024-09-13T22:39:44.869911+0000 mgr.y (mgr.14613) 78 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:46.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: cluster 2024-09-13T22:39:45.279801+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: cluster 2024-09-13T22:39:45.281508+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: audit 2024-09-13T22:39:45.328352+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: audit 2024-09-13T22:39:45.339243+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: audit 2024-09-13T22:39:45.932826+0000 mon.b (mon.1) 81 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: audit 2024-09-13T22:39:45.934290+0000 mon.b (mon.1) 82 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[22840]: audit 2024-09-13T22:39:45.940150+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: audit 2024-09-13T22:39:44.869911+0000 mgr.y (mgr.14613) 78 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: cluster 2024-09-13T22:39:45.279801+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: cluster 2024-09-13T22:39:45.281508+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: audit 2024-09-13T22:39:45.328352+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: audit 2024-09-13T22:39:45.339243+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: audit 2024-09-13T22:39:45.932826+0000 mon.b (mon.1) 81 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:39:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: audit 2024-09-13T22:39:45.934290+0000 mon.b (mon.1) 82 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:39:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:46 smithi178 bash[29889]: audit 2024-09-13T22:39:45.940150+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:39:47.321 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:39:47.324 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:39:47.333 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:47 smithi178 bash[22840]: cluster 2024-09-13T22:39:46.446255+0000 mgr.y (mgr.14613) 79 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:47.333 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:47 smithi178 bash[22840]: cluster 2024-09-13T22:39:46.673080+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T22:39:47.333 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:47 smithi178 bash[22840]: cluster 2024-09-13T22:39:46.684503+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T22:39:47.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:47 smithi192 bash[24608]: cluster 2024-09-13T22:39:46.446255+0000 mgr.y (mgr.14613) 79 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:47.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:47 smithi192 bash[24608]: cluster 2024-09-13T22:39:46.673080+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T22:39:47.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:47 smithi192 bash[24608]: cluster 2024-09-13T22:39:46.684503+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T22:39:47.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:47 smithi178 bash[29889]: cluster 2024-09-13T22:39:46.446255+0000 mgr.y (mgr.14613) 79 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:47.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:47 smithi178 bash[29889]: cluster 2024-09-13T22:39:46.673080+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T22:39:47.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:47 smithi178 bash[29889]: cluster 2024-09-13T22:39:46.684503+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T22:39:48.016 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-13T22:39:46.445886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1054,"num_read_kb":900,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":663,"ondisk_log_size":663,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244024,"kb_used_data":7004,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749717192,"statfs":{"total":767960285184,"available":767710404608,"internally_reserved":0,"allocated":7172096,"data_stored":4258105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003374"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827732+0000","last_change":"2024-09-13T22:37:47.692367+0000","last_active":"2024-09-13T22:39:39.827732+0000","last_peered":"2024-09-13T22:39:39.827732+0000","last_clean":"2024-09-13T22:39:39.827732+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:39:39.827732+0000","last_undegraded":"2024-09-13T22:39:39.827732+0000","last_fullsized":"2024-09-13T22:39:39.827732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932812+0000","last_change":"2024-09-13T22:38:46.613749+0000","last_active":"2024-09-13T22:39:39.932812+0000","last_peered":"2024-09-13T22:39:39.932812+0000","last_clean":"2024-09-13T22:39:39.932812+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:39:39.932812+0000","last_undegraded":"2024-09-13T22:39:39.932812+0000","last_fullsized":"2024-09-13T22:39:39.932812+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.613466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:15.391727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828413+0000","last_change":"2024-09-13T22:38:54.776466+0000","last_active":"2024-09-13T22:39:39.828413+0000","last_peered":"2024-09-13T22:39:39.828413+0000","last_clean":"2024-09-13T22:39:39.828413+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:39:39.828413+0000","last_undegraded":"2024-09-13T22:39:39.828413+0000","last_fullsized":"2024-09-13T22:39:39.828413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.776238+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.515503+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827931+0000","last_change":"2024-09-13T22:39:20.600784+0000","last_active":"2024-09-13T22:39:39.827931+0000","last_peered":"2024-09-13T22:39:39.827931+0000","last_clean":"2024-09-13T22:39:39.827931+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:39:39.827931+0000","last_undegraded":"2024-09-13T22:39:39.827931+0000","last_fullsized":"2024-09-13T22:39:39.827931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.600571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.907326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828404+0000","last_change":"2024-09-13T22:39:27.118459+0000","last_active":"2024-09-13T22:39:39.828404+0000","last_peered":"2024-09-13T22:39:39.828404+0000","last_clean":"2024-09-13T22:39:39.828404+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:39:39.828404+0000","last_undegraded":"2024-09-13T22:39:39.828404+0000","last_fullsized":"2024-09-13T22:39:39.828404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.118162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.118162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.139079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":102,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932605+0000","last_change":"2024-09-13T22:39:08.754354+0000","last_active":"2024-09-13T22:39:39.932605+0000","last_peered":"2024-09-13T22:39:39.932605+0000","last_clean":"2024-09-13T22:39:39.932605+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:39:39.932605+0000","last_undegraded":"2024-09-13T22:39:39.932605+0000","last_fullsized":"2024-09-13T22:39:39.932605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.754231+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:14.661781+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":75,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933019+0000","last_change":"2024-09-13T22:39:24.033368+0000","last_active":"2024-09-13T22:39:39.933019+0000","last_peered":"2024-09-13T22:39:39.933019+0000","last_clean":"2024-09-13T22:39:39.933019+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:39:39.933019+0000","last_undegraded":"2024-09-13T22:39:39.933019+0000","last_fullsized":"2024-09-13T22:39:39.933019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.033224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.033224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.108639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828626+0000","last_change":"2024-09-13T22:38:59.657956+0000","last_active":"2024-09-13T22:39:39.828626+0000","last_peered":"2024-09-13T22:39:39.828626+0000","last_clean":"2024-09-13T22:39:39.828626+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:39:39.828626+0000","last_undegraded":"2024-09-13T22:39:39.828626+0000","last_fullsized":"2024-09-13T22:39:39.828626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:59.657646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.657646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.987419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935008+0000","last_change":"2024-09-13T22:39:19.778773+0000","last_active":"2024-09-13T22:39:39.935008+0000","last_peered":"2024-09-13T22:39:39.935008+0000","last_clean":"2024-09-13T22:39:39.935008+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:39:39.935008+0000","last_undegraded":"2024-09-13T22:39:39.935008+0000","last_fullsized":"2024-09-13T22:39:39.935008+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.778590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.778590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.572565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":110,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936101+0000","last_change":"2024-09-13T22:39:26.318996+0000","last_active":"2024-09-13T22:39:39.936101+0000","last_peered":"2024-09-13T22:39:39.936101+0000","last_clean":"2024-09-13T22:39:39.936101+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:39:39.936101+0000","last_undegraded":"2024-09-13T22:39:39.936101+0000","last_fullsized":"2024-09-13T22:39:39.936101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.318753+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.307721+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":66,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933916+0000","last_change":"2024-09-13T22:37:44.878512+0000","last_active":"2024-09-13T22:39:39.933916+0000","last_peered":"2024-09-13T22:39:39.933916+0000","last_clean":"2024-09-13T22:39:39.933916+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:39:39.933916+0000","last_undegraded":"2024-09-13T22:39:39.933916+0000","last_fullsized":"2024-09-13T22:39:39.933916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:37:43.825402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935657+0000","last_change":"2024-09-13T22:39:37.426328+0000","last_active":"2024-09-13T22:39:39.935657+0000","last_peered":"2024-09-13T22:39:39.935657+0000","last_clean":"2024-09-13T22:39:39.935657+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:39:39.935657+0000","last_undegraded":"2024-09-13T22:39:39.935657+0000","last_fullsized":"2024-09-13T22:39:39.935657+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:37.426096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:37.426096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.076012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934940+0000","last_change":"2024-09-13T22:38:54.689196+0000","last_active":"2024-09-13T22:39:39.934940+0000","last_peered":"2024-09-13T22:39:39.934940+0000","last_clean":"2024-09-13T22:39:39.934940+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:39:39.934940+0000","last_undegraded":"2024-09-13T22:39:39.934940+0000","last_fullsized":"2024-09-13T22:39:39.934940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:54.689114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.689114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:59.621133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":101,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.937705+0000","last_change":"2024-09-13T22:38:49.543240+0000","last_active":"2024-09-13T22:39:39.937705+0000","last_peered":"2024-09-13T22:39:39.937705+0000","last_clean":"2024-09-13T22:39:39.937705+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:39:39.937705+0000","last_undegraded":"2024-09-13T22:39:39.937705+0000","last_fullsized":"2024-09-13T22:39:39.937705+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.543010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.543010+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.890250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936233+0000","last_change":"2024-09-13T22:38:50.497874+0000","last_active":"2024-09-13T22:39:39.936233+0000","last_peered":"2024-09-13T22:39:39.936233+0000","last_clean":"2024-09-13T22:39:39.936233+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:39:39.936233+0000","last_undegraded":"2024-09-13T22:39:39.936233+0000","last_fullsized":"2024-09-13T22:39:39.936233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:50.497567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:50.497567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.010369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933375+0000","last_change":"2024-09-13T22:39:14.752874+0000","last_active":"2024-09-13T22:39:39.933375+0000","last_peered":"2024-09-13T22:39:39.933375+0000","last_clean":"2024-09-13T22:39:39.933375+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:39:39.933375+0000","last_undegraded":"2024-09-13T22:39:39.933375+0000","last_fullsized":"2024-09-13T22:39:39.933375+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.752683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.752683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.453233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"82'19","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.894932+0000","last_change":"2024-09-13T22:39:21.873986+0000","last_active":"2024-09-13T22:39:39.894932+0000","last_peered":"2024-09-13T22:39:39.894932+0000","last_clean":"2024-09-13T22:39:39.894932+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:39:39.894932+0000","last_undegraded":"2024-09-13T22:39:39.894932+0000","last_fullsized":"2024-09-13T22:39:39.894932+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:21.873747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.873747+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.726394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935064+0000","last_change":"2024-09-13T22:39:35.662591+0000","last_active":"2024-09-13T22:39:39.935064+0000","last_peered":"2024-09-13T22:39:39.935064+0000","last_clean":"2024-09-13T22:39:39.935064+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:39:39.935064+0000","last_undegraded":"2024-09-13T22:39:39.935064+0000","last_fullsized":"2024-09-13T22:39:39.935064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:35.662396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:35.662396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:48.563351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":75,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932078+0000","last_change":"2024-09-13T22:39:15.972091+0000","last_active":"2024-09-13T22:39:39.932078+0000","last_peered":"2024-09-13T22:39:39.932078+0000","last_clean":"2024-09-13T22:39:39.932078+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:39:39.932078+0000","last_undegraded":"2024-09-13T22:39:39.932078+0000","last_fullsized":"2024-09-13T22:39:39.932078+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.971904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.971904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:40.478614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934489+0000","last_change":"2024-09-13T22:39:14.436930+0000","last_active":"2024-09-13T22:39:39.934489+0000","last_peered":"2024-09-13T22:39:39.934489+0000","last_clean":"2024-09-13T22:39:39.934489+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:39:39.934489+0000","last_undegraded":"2024-09-13T22:39:39.934489+0000","last_fullsized":"2024-09-13T22:39:39.934489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.436746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.436746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.039947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'9","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827925+0000","last_change":"2024-09-13T22:38:43.752187+0000","last_active":"2024-09-13T22:39:39.827925+0000","last_peered":"2024-09-13T22:39:39.827925+0000","last_clean":"2024-09-13T22:39:39.827925+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:39:39.827925+0000","last_undegraded":"2024-09-13T22:39:39.827925+0000","last_fullsized":"2024-09-13T22:39:39.827925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:43.751981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:43.751981+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.845059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933118+0000","last_change":"2024-09-13T22:39:01.945920+0000","last_active":"2024-09-13T22:39:39.933118+0000","last_peered":"2024-09-13T22:39:39.933118+0000","last_clean":"2024-09-13T22:39:39.933118+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:39:39.933118+0000","last_undegraded":"2024-09-13T22:39:39.933118+0000","last_fullsized":"2024-09-13T22:39:39.933118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:01.945733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.945733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.448893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934406+0000","last_change":"2024-09-13T22:39:19.489746+0000","last_active":"2024-09-13T22:39:39.934406+0000","last_peered":"2024-09-13T22:39:39.934406+0000","last_clean":"2024-09-13T22:39:39.934406+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:39:39.934406+0000","last_undegraded":"2024-09-13T22:39:39.934406+0000","last_fullsized":"2024-09-13T22:39:39.934406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.489549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.489549+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:47.360174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828590+0000","last_change":"2024-09-13T22:38:48.606917+0000","last_active":"2024-09-13T22:39:39.828590+0000","last_peered":"2024-09-13T22:39:39.828590+0000","last_clean":"2024-09-13T22:39:39.828590+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:39:39.828590+0000","last_undegraded":"2024-09-13T22:39:39.828590+0000","last_fullsized":"2024-09-13T22:39:39.828590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:48.606733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:48.606733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:50.950898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":103,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827414+0000","last_change":"2024-09-13T22:38:58.835483+0000","last_active":"2024-09-13T22:39:39.827414+0000","last_peered":"2024-09-13T22:39:39.827414+0000","last_clean":"2024-09-13T22:39:39.827414+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:39:39.827414+0000","last_undegraded":"2024-09-13T22:39:39.827414+0000","last_fullsized":"2024-09-13T22:39:39.827414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:38:58.835275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.835275+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.927373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933179+0000","last_change":"2024-09-13T22:38:46.006268+0000","last_active":"2024-09-13T22:39:39.933179+0000","last_peered":"2024-09-13T22:39:39.933179+0000","last_clean":"2024-09-13T22:39:39.933179+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:39:39.933179+0000","last_undegraded":"2024-09-13T22:39:39.933179+0000","last_fullsized":"2024-09-13T22:39:39.933179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.006072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.006072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:13.250082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":110,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828784+0000","last_change":"2024-09-13T22:39:12.592344+0000","last_active":"2024-09-13T22:39:39.828784+0000","last_peered":"2024-09-13T22:39:39.828784+0000","last_clean":"2024-09-13T22:39:39.828784+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:39:39.828784+0000","last_undegraded":"2024-09-13T22:39:39.828784+0000","last_fullsized":"2024-09-13T22:39:39.828784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.592134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.802063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936418+0000","last_change":"2024-09-13T22:39:33.392634+0000","last_active":"2024-09-13T22:39:39.936418+0000","last_peered":"2024-09-13T22:39:39.936418+0000","last_clean":"2024-09-13T22:39:39.936418+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:39:39.936418+0000","last_undegraded":"2024-09-13T22:39:39.936418+0000","last_fullsized":"2024-09-13T22:39:39.936418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_clean_scrub_stamp":"2024-09-13T22:39:33.392318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.641551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":101,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828504+0000","last_change":"2024-09-13T22:38:49.656312+0000","last_active":"2024-09-13T22:39:39.828504+0000","last_peered":"2024-09-13T22:39:39.828504+0000","last_clean":"2024-09-13T22:39:39.828504+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:39:39.828504+0000","last_undegraded":"2024-09-13T22:39:39.828504+0000","last_fullsized":"2024-09-13T22:39:39.828504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.656093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.656093+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.977202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:40.612137+0000","last_change":"2024-09-13T22:39:40.612195+0000","last_active":"2024-09-13T22:39:40.612137+0000","last_peered":"2024-09-13T22:39:40.612137+0000","last_clean":"2024-09-13T22:39:40.612137+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:39:40.612137+0000","last_undegraded":"2024-09-13T22:39:40.612137+0000","last_fullsized":"2024-09-13T22:39:40.612137+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:40.611997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.611997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:10.478412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936733+0000","last_change":"2024-09-13T22:38:51.542882+0000","last_active":"2024-09-13T22:39:39.936733+0000","last_peered":"2024-09-13T22:39:39.936733+0000","last_clean":"2024-09-13T22:39:39.936733+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:39:39.936733+0000","last_undegraded":"2024-09-13T22:39:39.936733+0000","last_fullsized":"2024-09-13T22:39:39.936733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:51.542757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:51.542757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:05.178866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827881+0000","last_change":"2024-09-13T22:37:47.692397+0000","last_active":"2024-09-13T22:39:39.827881+0000","last_peered":"2024-09-13T22:39:39.827881+0000","last_clean":"2024-09-13T22:39:39.827881+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:39:39.827881+0000","last_undegraded":"2024-09-13T22:39:39.827881+0000","last_fullsized":"2024-09-13T22:39:39.827881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":96,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932943+0000","last_change":"2024-09-13T22:39:04.726321+0000","last_active":"2024-09-13T22:39:39.932943+0000","last_peered":"2024-09-13T22:39:39.932943+0000","last_clean":"2024-09-13T22:39:39.932943+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:39:39.932943+0000","last_undegraded":"2024-09-13T22:39:39.932943+0000","last_fullsized":"2024-09-13T22:39:39.932943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:39:04.726105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.726105+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.304382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.896071+0000","last_change":"2024-09-13T22:39:08.858599+0000","last_active":"2024-09-13T22:39:39.896071+0000","last_peered":"2024-09-13T22:39:39.896071+0000","last_clean":"2024-09-13T22:39:39.896071+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:39:39.896071+0000","last_undegraded":"2024-09-13T22:39:39.896071+0000","last_fullsized":"2024-09-13T22:39:39.896071+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:08.858481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.858481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.756681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":86,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828323+0000","last_change":"2024-09-13T22:39:15.627485+0000","last_active":"2024-09-13T22:39:39.828323+0000","last_peered":"2024-09-13T22:39:39.828323+0000","last_clean":"2024-09-13T22:39:39.828323+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:39:39.828323+0000","last_undegraded":"2024-09-13T22:39:39.828323+0000","last_fullsized":"2024-09-13T22:39:39.828323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:39:15.627264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.627264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.016937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:40.265509+0000","last_change":"2024-09-13T22:39:40.265509+0000","last_active":"2024-09-13T22:39:40.265509+0000","last_peered":"2024-09-13T22:39:40.265509+0000","last_clean":"2024-09-13T22:39:40.265509+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:39:40.265509+0000","last_undegraded":"2024-09-13T22:39:40.265509+0000","last_fullsized":"2024-09-13T22:39:40.265509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:40.265350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.265350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:00.451629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933820+0000","last_change":"2024-09-13T22:37:47.689527+0000","last_active":"2024-09-13T22:39:39.933820+0000","last_peered":"2024-09-13T22:39:39.933820+0000","last_clean":"2024-09-13T22:39:39.933820+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:39:39.933820+0000","last_undegraded":"2024-09-13T22:39:39.933820+0000","last_fullsized":"2024-09-13T22:39:39.933820+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":110,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828589+0000","last_change":"2024-09-13T22:39:16.975346+0000","last_active":"2024-09-13T22:39:39.828589+0000","last_peered":"2024-09-13T22:39:39.828589+0000","last_clean":"2024-09-13T22:39:39.828589+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:39:39.828589+0000","last_undegraded":"2024-09-13T22:39:39.828589+0000","last_fullsized":"2024-09-13T22:39:39.828589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.975244+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.689091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":84,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932701+0000","last_change":"2024-09-13T22:39:07.049286+0000","last_active":"2024-09-13T22:39:39.932701+0000","last_peered":"2024-09-13T22:39:39.932701+0000","last_clean":"2024-09-13T22:39:39.932701+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:39:39.932701+0000","last_undegraded":"2024-09-13T22:39:39.932701+0000","last_fullsized":"2024-09-13T22:39:39.932701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:07.049084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.049084+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:35.995865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":193,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:42.986698+0000","last_change":"2024-09-13T22:39:38.224453+0000","last_active":"2024-09-13T22:39:42.986698+0000","last_peered":"2024-09-13T22:39:42.986698+0000","last_clean":"2024-09-13T22:39:42.986698+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:39:42.986698+0000","last_undegraded":"2024-09-13T22:39:42.986698+0000","last_fullsized":"2024-09-13T22:39:42.986698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:39:38.224366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:38.224366+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.543802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":104,"num_read_kb":99,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827986+0000","last_change":"2024-09-13T22:38:58.704351+0000","last_active":"2024-09-13T22:39:39.827986+0000","last_peered":"2024-09-13T22:39:39.827986+0000","last_clean":"2024-09-13T22:39:39.827986+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:39:39.827986+0000","last_undegraded":"2024-09-13T22:39:39.827986+0000","last_fullsized":"2024-09-13T22:39:39.827986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:58.704193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.704193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:04.200792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827997+0000","last_change":"2024-09-13T22:39:05.561714+0000","last_active":"2024-09-13T22:39:39.827997+0000","last_peered":"2024-09-13T22:39:39.827997+0000","last_clean":"2024-09-13T22:39:39.827997+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:39:39.827997+0000","last_undegraded":"2024-09-13T22:39:39.827997+0000","last_fullsized":"2024-09-13T22:39:39.827997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.561489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.561489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.439424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":154,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932270+0000","last_change":"2024-09-13T22:39:11.036017+0000","last_active":"2024-09-13T22:39:39.932270+0000","last_peered":"2024-09-13T22:39:39.932270+0000","last_clean":"2024-09-13T22:39:39.932270+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:39:39.932270+0000","last_undegraded":"2024-09-13T22:39:39.932270+0000","last_fullsized":"2024-09-13T22:39:39.932270+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:39:11.035787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.035787+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.606987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935272+0000","last_change":"2024-09-13T22:39:31.671011+0000","last_active":"2024-09-13T22:39:39.935272+0000","last_peered":"2024-09-13T22:39:39.935272+0000","last_clean":"2024-09-13T22:39:39.935272+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:39:39.935272+0000","last_undegraded":"2024-09-13T22:39:39.935272+0000","last_fullsized":"2024-09-13T22:39:39.935272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.670821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.670821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.397166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.894654+0000","last_change":"2024-09-13T22:39:24.762341+0000","last_active":"2024-09-13T22:39:39.894654+0000","last_peered":"2024-09-13T22:39:39.894654+0000","last_clean":"2024-09-13T22:39:39.894654+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:39:39.894654+0000","last_undegraded":"2024-09-13T22:39:39.894654+0000","last_fullsized":"2024-09-13T22:39:39.894654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.762123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.762123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.895897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":71,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932927+0000","last_change":"2024-09-13T22:39:25.019347+0000","last_active":"2024-09-13T22:39:39.932927+0000","last_peered":"2024-09-13T22:39:39.932927+0000","last_clean":"2024-09-13T22:39:39.932927+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:39:39.932927+0000","last_undegraded":"2024-09-13T22:39:39.932927+0000","last_fullsized":"2024-09-13T22:39:39.932927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.019168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.019168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.669620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":133,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.937903+0000","last_change":"2024-09-13T22:39:24.368074+0000","last_active":"2024-09-13T22:39:39.937903+0000","last_peered":"2024-09-13T22:39:39.937903+0000","last_clean":"2024-09-13T22:39:39.937903+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:39:39.937903+0000","last_undegraded":"2024-09-13T22:39:39.937903+0000","last_fullsized":"2024-09-13T22:39:39.937903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.367843+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.291892+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":91,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932323+0000","last_change":"2024-09-13T22:39:06.025706+0000","last_active":"2024-09-13T22:39:39.932323+0000","last_peered":"2024-09-13T22:39:39.932323+0000","last_clean":"2024-09-13T22:39:39.932323+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:39:39.932323+0000","last_undegraded":"2024-09-13T22:39:39.932323+0000","last_fullsized":"2024-09-13T22:39:39.932323+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:06.025488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:06.025488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.356278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":88,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932719+0000","last_change":"2024-09-13T22:39:12.041436+0000","last_active":"2024-09-13T22:39:39.932719+0000","last_peered":"2024-09-13T22:39:39.932719+0000","last_clean":"2024-09-13T22:39:39.932719+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:39:39.932719+0000","last_undegraded":"2024-09-13T22:39:39.932719+0000","last_fullsized":"2024-09-13T22:39:39.932719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:39:12.041252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.041252+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.755767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":110,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931969+0000","last_change":"2024-09-13T22:39:01.656159+0000","last_active":"2024-09-13T22:39:39.931969+0000","last_peered":"2024-09-13T22:39:39.931969+0000","last_clean":"2024-09-13T22:39:39.931969+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:39:39.931969+0000","last_undegraded":"2024-09-13T22:39:39.931969+0000","last_fullsized":"2024-09-13T22:39:39.931969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.655949+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.600034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931977+0000","last_change":"2024-09-13T22:39:25.685706+0000","last_active":"2024-09-13T22:39:39.931977+0000","last_peered":"2024-09-13T22:39:39.931977+0000","last_clean":"2024-09-13T22:39:39.931977+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:39:39.931977+0000","last_undegraded":"2024-09-13T22:39:39.931977+0000","last_fullsized":"2024-09-13T22:39:39.931977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.685432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.685432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.180606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":114,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828157+0000","last_change":"2024-09-13T22:39:10.618742+0000","last_active":"2024-09-13T22:39:39.828157+0000","last_peered":"2024-09-13T22:39:39.828157+0000","last_clean":"2024-09-13T22:39:39.828157+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:39:39.828157+0000","last_undegraded":"2024-09-13T22:39:39.828157+0000","last_fullsized":"2024-09-13T22:39:39.828157+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:10.618592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.618592+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.736269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936938+0000","last_change":"2024-09-13T22:38:56.500284+0000","last_active":"2024-09-13T22:39:39.936938+0000","last_peered":"2024-09-13T22:39:39.936938+0000","last_clean":"2024-09-13T22:39:39.936938+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:39:39.936938+0000","last_undegraded":"2024-09-13T22:39:39.936938+0000","last_fullsized":"2024-09-13T22:39:39.936938+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.500054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:00.349788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932462+0000","last_change":"2024-09-13T22:39:02.682368+0000","last_active":"2024-09-13T22:39:39.932462+0000","last_peered":"2024-09-13T22:39:39.932462+0000","last_clean":"2024-09-13T22:39:39.932462+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:39:39.932462+0000","last_undegraded":"2024-09-13T22:39:39.932462+0000","last_fullsized":"2024-09-13T22:39:39.932462+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.682170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.682170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.706353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828250+0000","last_change":"2024-09-13T22:37:46.919226+0000","last_active":"2024-09-13T22:39:39.828250+0000","last_peered":"2024-09-13T22:39:39.828250+0000","last_clean":"2024-09-13T22:39:39.828250+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:39:39.828250+0000","last_undegraded":"2024-09-13T22:39:39.828250+0000","last_fullsized":"2024-09-13T22:39:39.828250+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"82'18","reported_seq":112,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931981+0000","last_change":"2024-09-13T22:39:19.966260+0000","last_active":"2024-09-13T22:39:39.931981+0000","last_peered":"2024-09-13T22:39:39.931981+0000","last_clean":"2024-09-13T22:39:39.931981+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:39:39.931981+0000","last_undegraded":"2024-09-13T22:39:39.931981+0000","last_fullsized":"2024-09-13T22:39:39.931981+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-13T22:39:19.966043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.966043+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.842672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":90,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933980+0000","last_change":"2024-09-13T22:39:15.472999+0000","last_active":"2024-09-13T22:39:39.933980+0000","last_peered":"2024-09-13T22:39:39.933980+0000","last_clean":"2024-09-13T22:39:39.933980+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:39:39.933980+0000","last_undegraded":"2024-09-13T22:39:39.933980+0000","last_fullsized":"2024-09-13T22:39:39.933980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:15.472777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.472777+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:41.403750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934717+0000","last_change":"2024-09-13T22:38:54.478611+0000","last_active":"2024-09-13T22:39:39.934717+0000","last_peered":"2024-09-13T22:39:39.934717+0000","last_clean":"2024-09-13T22:39:39.934717+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:39:39.934717+0000","last_undegraded":"2024-09-13T22:39:39.934717+0000","last_fullsized":"2024-09-13T22:39:39.934717+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:54.478424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.478424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.387064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934401+0000","last_change":"2024-09-13T22:39:20.748359+0000","last_active":"2024-09-13T22:39:39.934401+0000","last_peered":"2024-09-13T22:39:39.934401+0000","last_clean":"2024-09-13T22:39:39.934401+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:39:39.934401+0000","last_undegraded":"2024-09-13T22:39:39.934401+0000","last_fullsized":"2024-09-13T22:39:39.934401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.748082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.748082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:30.962778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931905+0000","last_change":"2024-09-13T22:39:05.725791+0000","last_active":"2024-09-13T22:39:39.931905+0000","last_peered":"2024-09-13T22:39:39.931905+0000","last_clean":"2024-09-13T22:39:39.931905+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:39:39.931905+0000","last_undegraded":"2024-09-13T22:39:39.931905+0000","last_fullsized":"2024-09-13T22:39:39.931905+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:39:05.725568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.725568+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.221576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933615+0000","last_change":"2024-09-13T22:39:34.703390+0000","last_active":"2024-09-13T22:39:39.933615+0000","last_peered":"2024-09-13T22:39:39.933615+0000","last_clean":"2024-09-13T22:39:39.933615+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:39:39.933615+0000","last_undegraded":"2024-09-13T22:39:39.933615+0000","last_fullsized":"2024-09-13T22:39:39.933615+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:26.772349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.772349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.725362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936839+0000","last_change":"2024-09-13T22:39:12.302084+0000","last_active":"2024-09-13T22:39:39.936839+0000","last_peered":"2024-09-13T22:39:39.936839+0000","last_clean":"2024-09-13T22:39:39.936839+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:39:39.936839+0000","last_undegraded":"2024-09-13T22:39:39.936839+0000","last_fullsized":"2024-09-13T22:39:39.936839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.301909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.301909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.703405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":71,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931979+0000","last_change":"2024-09-13T22:38:57.971090+0000","last_active":"2024-09-13T22:39:39.931979+0000","last_peered":"2024-09-13T22:39:39.931979+0000","last_clean":"2024-09-13T22:39:39.931979+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:39:39.931979+0000","last_undegraded":"2024-09-13T22:39:39.931979+0000","last_fullsized":"2024-09-13T22:39:39.931979+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.970850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.970850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:09.411139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827733+0000","last_change":"2024-09-13T22:39:04.843005+0000","last_active":"2024-09-13T22:39:39.827733+0000","last_peered":"2024-09-13T22:39:39.827733+0000","last_clean":"2024-09-13T22:39:39.827733+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:39:39.827733+0000","last_undegraded":"2024-09-13T22:39:39.827733+0000","last_fullsized":"2024-09-13T22:39:39.827733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:04.842757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.842757+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.409317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936730+0000","last_change":"2024-09-13T22:38:53.554534+0000","last_active":"2024-09-13T22:39:39.936730+0000","last_peered":"2024-09-13T22:39:39.936730+0000","last_clean":"2024-09-13T22:39:39.936730+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:39:39.936730+0000","last_undegraded":"2024-09-13T22:39:39.936730+0000","last_fullsized":"2024-09-13T22:39:39.936730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:53.554288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:53.554288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:07.563224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":336,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933944+0000","last_change":"2024-09-13T22:38:14.773746+0000","last_active":"2024-09-13T22:39:39.933944+0000","last_peered":"2024-09-13T22:39:39.933944+0000","last_clean":"2024-09-13T22:39:39.933944+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:39:39.933944+0000","last_undegraded":"2024-09-13T22:39:39.933944+0000","last_fullsized":"2024-09-13T22:39:39.933944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'119","last_scrub_stamp":"2024-09-13T22:38:14.773565+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:38:14.773565+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934314+0000","last_change":"2024-09-13T22:38:56.676371+0000","last_active":"2024-09-13T22:39:39.934314+0000","last_peered":"2024-09-13T22:39:39.934314+0000","last_clean":"2024-09-13T22:39:39.934314+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:39:39.934314+0000","last_undegraded":"2024-09-13T22:39:39.934314+0000","last_fullsized":"2024-09-13T22:39:39.934314+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.676172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.058738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936566+0000","last_change":"2024-09-13T22:39:00.431121+0000","last_active":"2024-09-13T22:39:39.936566+0000","last_peered":"2024-09-13T22:39:39.936566+0000","last_clean":"2024-09-13T22:39:39.936566+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:39:39.936566+0000","last_undegraded":"2024-09-13T22:39:39.936566+0000","last_fullsized":"2024-09-13T22:39:39.936566+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:00.430982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.430982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:04.949172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":106,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:42.666640+0000","last_change":"2024-09-13T22:39:42.666727+0000","last_active":"2024-09-13T22:39:42.666640+0000","last_peered":"2024-09-13T22:39:42.666640+0000","last_clean":"2024-09-13T22:39:42.666640+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:39:42.666640+0000","last_undegraded":"2024-09-13T22:39:42.666640+0000","last_fullsized":"2024-09-13T22:39:42.666640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:39:42.666504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:42.666504+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.765205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934541+0000","last_change":"2024-09-13T22:38:44.582475+0000","last_active":"2024-09-13T22:39:39.934541+0000","last_peered":"2024-09-13T22:39:39.934541+0000","last_clean":"2024-09-13T22:39:39.934541+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:39:39.934541+0000","last_undegraded":"2024-09-13T22:39:39.934541+0000","last_fullsized":"2024-09-13T22:39:39.934541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:44.582283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:44.582283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:47.447502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934210+0000","last_change":"2024-09-13T22:39:12.437612+0000","last_active":"2024-09-13T22:39:39.934210+0000","last_peered":"2024-09-13T22:39:39.934210+0000","last_clean":"2024-09-13T22:39:39.934210+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:39:39.934210+0000","last_undegraded":"2024-09-13T22:39:39.934210+0000","last_fullsized":"2024-09-13T22:39:39.934210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.437298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.437298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.840182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933721+0000","last_change":"2024-09-13T22:37:46.924543+0000","last_active":"2024-09-13T22:39:39.933721+0000","last_peered":"2024-09-13T22:39:39.933721+0000","last_clean":"2024-09-13T22:39:39.933721+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:39:39.933721+0000","last_undegraded":"2024-09-13T22:39:39.933721+0000","last_fullsized":"2024-09-13T22:39:39.933721+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.937107+0000","last_change":"2024-09-13T22:39:09.333581+0000","last_active":"2024-09-13T22:39:39.937107+0000","last_peered":"2024-09-13T22:39:39.937107+0000","last_clean":"2024-09-13T22:39:39.937107+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:39:39.937107+0000","last_undegraded":"2024-09-13T22:39:39.937107+0000","last_fullsized":"2024-09-13T22:39:39.937107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:09.333359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.333359+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.270070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933515+0000","last_change":"2024-09-13T22:38:52.100566+0000","last_active":"2024-09-13T22:39:39.933515+0000","last_peered":"2024-09-13T22:39:39.933515+0000","last_clean":"2024-09-13T22:39:39.933515+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:39:39.933515+0000","last_undegraded":"2024-09-13T22:39:39.933515+0000","last_fullsized":"2024-09-13T22:39:39.933515+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:52.100377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:52.100377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:09.993651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934076+0000","last_change":"2024-09-13T22:39:00.541876+0000","last_active":"2024-09-13T22:39:39.934076+0000","last_peered":"2024-09-13T22:39:39.934076+0000","last_clean":"2024-09-13T22:39:39.934076+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:39:39.934076+0000","last_undegraded":"2024-09-13T22:39:39.934076+0000","last_fullsized":"2024-09-13T22:39:39.934076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:00.541555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.541555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:04.820100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931811+0000","last_change":"2024-09-13T22:38:51.576528+0000","last_active":"2024-09-13T22:39:39.931811+0000","last_peered":"2024-09-13T22:39:39.931811+0000","last_clean":"2024-09-13T22:39:39.931811+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:39:39.931811+0000","last_undegraded":"2024-09-13T22:39:39.931811+0000","last_fullsized":"2024-09-13T22:39:39.931811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:51.576344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:51.576344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:09.615053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"82'15","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935586+0000","last_change":"2024-09-13T22:37:42.764960+0000","last_active":"2024-09-13T22:39:39.935586+0000","last_peered":"2024-09-13T22:39:39.935586+0000","last_clean":"2024-09-13T22:39:39.935586+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:39:39.935586+0000","last_undegraded":"2024-09-13T22:39:39.935586+0000","last_fullsized":"2024-09-13T22:39:39.935586+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:37:41.731441+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935055+0000","last_change":"2024-09-13T22:39:04.517758+0000","last_active":"2024-09-13T22:39:39.935055+0000","last_peered":"2024-09-13T22:39:39.935055+0000","last_clean":"2024-09-13T22:39:39.935055+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:39:39.935055+0000","last_undegraded":"2024-09-13T22:39:39.935055+0000","last_fullsized":"2024-09-13T22:39:39.935055+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:04.517658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.517658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.659731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934821+0000","last_change":"2024-09-13T22:39:08.790837+0000","last_active":"2024-09-13T22:39:39.934821+0000","last_peered":"2024-09-13T22:39:39.934821+0000","last_clean":"2024-09-13T22:39:39.934821+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:39:39.934821+0000","last_undegraded":"2024-09-13T22:39:39.934821+0000","last_fullsized":"2024-09-13T22:39:39.934821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:08.790729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.790729+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.315232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935669+0000","last_change":"2024-09-13T22:39:10.297437+0000","last_active":"2024-09-13T22:39:39.935669+0000","last_peered":"2024-09-13T22:39:39.935669+0000","last_clean":"2024-09-13T22:39:39.935669+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:39:39.935669+0000","last_undegraded":"2024-09-13T22:39:39.935669+0000","last_fullsized":"2024-09-13T22:39:39.935669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:10.297219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.297219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.668374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":97,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934911+0000","last_change":"2024-09-13T22:38:53.711228+0000","last_active":"2024-09-13T22:39:39.934911+0000","last_peered":"2024-09-13T22:39:39.934911+0000","last_clean":"2024-09-13T22:39:39.934911+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:39:39.934911+0000","last_undegraded":"2024-09-13T22:39:39.934911+0000","last_fullsized":"2024-09-13T22:39:39.934911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:53.711126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:53.711126+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:01.034551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":98,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828844+0000","last_change":"2024-09-13T22:39:00.654413+0000","last_active":"2024-09-13T22:39:39.828844+0000","last_peered":"2024-09-13T22:39:39.828844+0000","last_clean":"2024-09-13T22:39:39.828844+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:39:39.828844+0000","last_undegraded":"2024-09-13T22:39:39.828844+0000","last_fullsized":"2024-09-13T22:39:39.828844+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'2","last_scrub_stamp":"2024-09-13T22:39:00.654142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.654142+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.020250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933787+0000","last_change":"2024-09-13T22:39:21.523815+0000","last_active":"2024-09-13T22:39:39.933787+0000","last_peered":"2024-09-13T22:39:39.933787+0000","last_clean":"2024-09-13T22:39:39.933787+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:39:39.933787+0000","last_undegraded":"2024-09-13T22:39:39.933787+0000","last_fullsized":"2024-09-13T22:39:39.933787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.523625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.523625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:51.184833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827733+0000","last_change":"2024-09-13T22:39:18.998697+0000","last_active":"2024-09-13T22:39:39.827733+0000","last_peered":"2024-09-13T22:39:39.827733+0000","last_clean":"2024-09-13T22:39:39.827733+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:39:39.827733+0000","last_undegraded":"2024-09-13T22:39:39.827733+0000","last_fullsized":"2024-09-13T22:39:39.827733+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.998416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.890380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"82'11","reported_seq":97,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935198+0000","last_change":"2024-09-13T22:39:10.823267+0000","last_active":"2024-09-13T22:39:39.935198+0000","last_peered":"2024-09-13T22:39:39.935198+0000","last_clean":"2024-09-13T22:39:39.935198+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:39:39.935198+0000","last_undegraded":"2024-09-13T22:39:39.935198+0000","last_fullsized":"2024-09-13T22:39:39.935198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:10.823176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.823176+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.690199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933865+0000","last_change":"2024-09-13T22:38:56.494481+0000","last_active":"2024-09-13T22:39:39.933865+0000","last_peered":"2024-09-13T22:39:39.933865+0000","last_clean":"2024-09-13T22:39:39.933865+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:39:39.933865+0000","last_undegraded":"2024-09-13T22:39:39.933865+0000","last_fullsized":"2024-09-13T22:39:39.933865+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.494413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.494413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.935415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828486+0000","last_change":"2024-09-13T22:39:11.903829+0000","last_active":"2024-09-13T22:39:39.828486+0000","last_peered":"2024-09-13T22:39:39.828486+0000","last_clean":"2024-09-13T22:39:39.828486+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:39:39.828486+0000","last_undegraded":"2024-09-13T22:39:39.828486+0000","last_fullsized":"2024-09-13T22:39:39.828486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.903758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.903758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:21.502704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934272+0000","last_change":"2024-09-13T22:38:59.523926+0000","last_active":"2024-09-13T22:39:39.934272+0000","last_peered":"2024-09-13T22:39:39.934272+0000","last_clean":"2024-09-13T22:39:39.934272+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:39:39.934272+0000","last_undegraded":"2024-09-13T22:39:39.934272+0000","last_fullsized":"2024-09-13T22:39:39.934272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:59.523812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.523812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:02.812639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":84,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.894592+0000","last_change":"2024-09-13T22:39:04.876157+0000","last_active":"2024-09-13T22:39:39.894592+0000","last_peered":"2024-09-13T22:39:39.894592+0000","last_clean":"2024-09-13T22:39:39.894592+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:39:39.894592+0000","last_undegraded":"2024-09-13T22:39:39.894592+0000","last_fullsized":"2024-09-13T22:39:39.894592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:39:04.875957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.875957+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.926851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.895068+0000","last_change":"2024-09-13T22:39:18.864519+0000","last_active":"2024-09-13T22:39:39.895068+0000","last_peered":"2024-09-13T22:39:39.895068+0000","last_clean":"2024-09-13T22:39:39.895068+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:39:39.895068+0000","last_undegraded":"2024-09-13T22:39:39.895068+0000","last_fullsized":"2024-09-13T22:39:39.895068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.864331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.864331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.702214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936814+0000","last_change":"2024-09-13T22:39:31.439089+0000","last_active":"2024-09-13T22:39:39.936814+0000","last_peered":"2024-09-13T22:39:39.936814+0000","last_clean":"2024-09-13T22:39:39.936814+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:39:39.936814+0000","last_undegraded":"2024-09-13T22:39:39.936814+0000","last_fullsized":"2024-09-13T22:39:39.936814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.438868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.438868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.289969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934085+0000","last_change":"2024-09-13T22:39:11.834762+0000","last_active":"2024-09-13T22:39:39.934085+0000","last_peered":"2024-09-13T22:39:39.934085+0000","last_clean":"2024-09-13T22:39:39.934085+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:39:39.934085+0000","last_undegraded":"2024-09-13T22:39:39.934085+0000","last_fullsized":"2024-09-13T22:39:39.934085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.834629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.834629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:37.419658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":97,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934451+0000","last_change":"2024-09-13T22:39:30.707102+0000","last_active":"2024-09-13T22:39:39.934451+0000","last_peered":"2024-09-13T22:39:39.934451+0000","last_clean":"2024-09-13T22:39:39.934451+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:39:39.934451+0000","last_undegraded":"2024-09-13T22:39:39.934451+0000","last_fullsized":"2024-09-13T22:39:39.934451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:30.706887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:30.706887+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.189746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936675+0000","last_change":"2024-09-13T22:39:15.307350+0000","last_active":"2024-09-13T22:39:39.936675+0000","last_peered":"2024-09-13T22:39:39.936675+0000","last_clean":"2024-09-13T22:39:39.936675+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:39:39.936675+0000","last_undegraded":"2024-09-13T22:39:39.936675+0000","last_fullsized":"2024-09-13T22:39:39.936675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.307195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.307195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.086503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":112,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.937554+0000","last_change":"2024-09-13T22:38:59.477201+0000","last_active":"2024-09-13T22:39:39.937554+0000","last_peered":"2024-09-13T22:39:39.937554+0000","last_clean":"2024-09-13T22:39:39.937554+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:39:39.937554+0000","last_undegraded":"2024-09-13T22:39:39.937554+0000","last_fullsized":"2024-09-13T22:39:39.937554+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:59.477008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.477008+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:59.917494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932713+0000","last_change":"2024-09-13T22:39:18.688333+0000","last_active":"2024-09-13T22:39:39.932713+0000","last_peered":"2024-09-13T22:39:39.932713+0000","last_clean":"2024-09-13T22:39:39.932713+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:39:39.932713+0000","last_undegraded":"2024-09-13T22:39:39.932713+0000","last_fullsized":"2024-09-13T22:39:39.932713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.688141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.688141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.166952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931700+0000","last_change":"2024-09-13T22:38:54.600790+0000","last_active":"2024-09-13T22:39:39.931700+0000","last_peered":"2024-09-13T22:39:39.931700+0000","last_clean":"2024-09-13T22:39:39.931700+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:39:39.931700+0000","last_undegraded":"2024-09-13T22:39:39.931700+0000","last_fullsized":"2024-09-13T22:39:39.931700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:54.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.600571+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:11.578225+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932342+0000","last_change":"2024-09-13T22:39:21.654486+0000","last_active":"2024-09-13T22:39:39.932342+0000","last_peered":"2024-09-13T22:39:39.932342+0000","last_clean":"2024-09-13T22:39:39.932342+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:39:39.932342+0000","last_undegraded":"2024-09-13T22:39:39.932342+0000","last_fullsized":"2024-09-13T22:39:39.932342+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.654273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.654273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.171794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":109,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828176+0000","last_change":"2024-09-13T22:38:59.864543+0000","last_active":"2024-09-13T22:39:39.828176+0000","last_peered":"2024-09-13T22:39:39.828176+0000","last_clean":"2024-09-13T22:39:39.828176+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:39:39.828176+0000","last_undegraded":"2024-09-13T22:39:39.828176+0000","last_fullsized":"2024-09-13T22:39:39.828176+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:59.864340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.864340+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:03.765599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828671+0000","last_change":"2024-09-13T22:39:17.629172+0000","last_active":"2024-09-13T22:39:39.828671+0000","last_peered":"2024-09-13T22:39:39.828671+0000","last_clean":"2024-09-13T22:39:39.828671+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:39:39.828671+0000","last_undegraded":"2024-09-13T22:39:39.828671+0000","last_fullsized":"2024-09-13T22:39:39.828671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:17.628975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:17.628975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:18.565445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.935145+0000","last_change":"2024-09-13T22:39:25.745828+0000","last_active":"2024-09-13T22:39:39.935145+0000","last_peered":"2024-09-13T22:39:39.935145+0000","last_clean":"2024-09-13T22:39:39.935145+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:39:39.935145+0000","last_undegraded":"2024-09-13T22:39:39.935145+0000","last_fullsized":"2024-09-13T22:39:39.935145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:25.745613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.745613+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:49.997201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.895423+0000","last_change":"2024-09-13T22:39:15.852762+0000","last_active":"2024-09-13T22:39:39.895423+0000","last_peered":"2024-09-13T22:39:39.895423+0000","last_clean":"2024-09-13T22:39:39.895423+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:39:39.895423+0000","last_undegraded":"2024-09-13T22:39:39.895423+0000","last_fullsized":"2024-09-13T22:39:39.895423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.852596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.852596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.570592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827839+0000","last_change":"2024-09-13T22:39:29.136457+0000","last_active":"2024-09-13T22:39:39.827839+0000","last_peered":"2024-09-13T22:39:39.827839+0000","last_clean":"2024-09-13T22:39:39.827839+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:39:39.827839+0000","last_undegraded":"2024-09-13T22:39:39.827839+0000","last_fullsized":"2024-09-13T22:39:39.827839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.136366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.072781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.931499+0000","last_change":"2024-09-13T22:39:27.605785+0000","last_active":"2024-09-13T22:39:39.931499+0000","last_peered":"2024-09-13T22:39:39.931499+0000","last_clean":"2024-09-13T22:39:39.931499+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:39:39.931499+0000","last_undegraded":"2024-09-13T22:39:39.931499+0000","last_fullsized":"2024-09-13T22:39:39.931499+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.605612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.605612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.057619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":93,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828439+0000","last_change":"2024-09-13T22:38:44.652800+0000","last_active":"2024-09-13T22:39:39.828439+0000","last_peered":"2024-09-13T22:39:39.828439+0000","last_clean":"2024-09-13T22:39:39.828439+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:39:39.828439+0000","last_undegraded":"2024-09-13T22:39:39.828439+0000","last_fullsized":"2024-09-13T22:39:39.828439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:38:44.652584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:44.652584+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:44.656183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933530+0000","last_change":"2024-09-13T22:39:13.008954+0000","last_active":"2024-09-13T22:39:39.933530+0000","last_peered":"2024-09-13T22:39:39.933530+0000","last_clean":"2024-09-13T22:39:39.933530+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:39:39.933530+0000","last_undegraded":"2024-09-13T22:39:39.933530+0000","last_fullsized":"2024-09-13T22:39:39.933530+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:13.008764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.008764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.397180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":75,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932130+0000","last_change":"2024-09-13T22:38:57.018120+0000","last_active":"2024-09-13T22:39:39.932130+0000","last_peered":"2024-09-13T22:39:39.932130+0000","last_clean":"2024-09-13T22:39:39.932130+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:39:39.932130+0000","last_undegraded":"2024-09-13T22:39:39.932130+0000","last_fullsized":"2024-09-13T22:39:39.932130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.017923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.017923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:15.532446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827275+0000","last_change":"2024-09-13T22:38:52.760354+0000","last_active":"2024-09-13T22:39:39.827275+0000","last_peered":"2024-09-13T22:39:39.827275+0000","last_clean":"2024-09-13T22:39:39.827275+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:39:39.827275+0000","last_undegraded":"2024-09-13T22:39:39.827275+0000","last_fullsized":"2024-09-13T22:39:39.827275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:52.760268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:52.760268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.301827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":87,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932609+0000","last_change":"2024-09-13T22:39:07.744502+0000","last_active":"2024-09-13T22:39:39.932609+0000","last_peered":"2024-09-13T22:39:39.932609+0000","last_clean":"2024-09-13T22:39:39.932609+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:39:39.932609+0000","last_undegraded":"2024-09-13T22:39:39.932609+0000","last_fullsized":"2024-09-13T22:39:39.932609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:39:07.744124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.744124+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.659429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.937221+0000","last_change":"2024-09-13T22:39:21.378052+0000","last_active":"2024-09-13T22:39:39.937221+0000","last_peered":"2024-09-13T22:39:39.937221+0000","last_clean":"2024-09-13T22:39:39.937221+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:39:39.937221+0000","last_undegraded":"2024-09-13T22:39:39.937221+0000","last_fullsized":"2024-09-13T22:39:39.937221+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.377829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.377829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.746784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.894595+0000","last_change":"2024-09-13T22:39:05.855096+0000","last_active":"2024-09-13T22:39:39.894595+0000","last_peered":"2024-09-13T22:39:39.894595+0000","last_clean":"2024-09-13T22:39:39.894595+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:39:39.894595+0000","last_undegraded":"2024-09-13T22:39:39.894595+0000","last_fullsized":"2024-09-13T22:39:39.894595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.854893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.334889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933649+0000","last_change":"2024-09-13T22:37:46.916035+0000","last_active":"2024-09-13T22:39:39.933649+0000","last_peered":"2024-09-13T22:39:39.933649+0000","last_clean":"2024-09-13T22:39:39.933649+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:39:39.933649+0000","last_undegraded":"2024-09-13T22:39:39.933649+0000","last_fullsized":"2024-09-13T22:39:39.933649+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.936964+0000","last_change":"2024-09-13T22:39:13.339189+0000","last_active":"2024-09-13T22:39:39.936964+0000","last_peered":"2024-09-13T22:39:39.936964+0000","last_clean":"2024-09-13T22:39:39.936964+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:39:39.936964+0000","last_undegraded":"2024-09-13T22:39:39.936964+0000","last_fullsized":"2024-09-13T22:39:39.936964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:13.338970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.338970+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.771039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.895940+0000","last_change":"2024-09-13T22:39:07.882650+0000","last_active":"2024-09-13T22:39:39.895940+0000","last_peered":"2024-09-13T22:39:39.895940+0000","last_clean":"2024-09-13T22:39:39.895940+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:39:39.895940+0000","last_undegraded":"2024-09-13T22:39:39.895940+0000","last_fullsized":"2024-09-13T22:39:39.895940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:07.882460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.882460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.349629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933205+0000","last_change":"2024-09-13T22:39:29.666045+0000","last_active":"2024-09-13T22:39:39.933205+0000","last_peered":"2024-09-13T22:39:39.933205+0000","last_clean":"2024-09-13T22:39:39.933205+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:39:39.933205+0000","last_undegraded":"2024-09-13T22:39:39.933205+0000","last_fullsized":"2024-09-13T22:39:39.933205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.665857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.665857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.766151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827878+0000","last_change":"2024-09-13T22:38:49.749377+0000","last_active":"2024-09-13T22:39:39.827878+0000","last_peered":"2024-09-13T22:39:39.827878+0000","last_clean":"2024-09-13T22:39:39.827878+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:39:39.827878+0000","last_undegraded":"2024-09-13T22:39:39.827878+0000","last_fullsized":"2024-09-13T22:39:39.827878+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:49.749300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.749300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:14.449565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":72,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.934243+0000","last_change":"2024-09-13T22:39:02.697718+0000","last_active":"2024-09-13T22:39:39.934243+0000","last_peered":"2024-09-13T22:39:39.934243+0000","last_clean":"2024-09-13T22:39:39.934243+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:39:39.934243+0000","last_undegraded":"2024-09-13T22:39:39.934243+0000","last_fullsized":"2024-09-13T22:39:39.934243+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:39:02.697574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.697574+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.049416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"82'15","reported_seq":104,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933100+0000","last_change":"2024-09-13T22:39:16.963966+0000","last_active":"2024-09-13T22:39:39.933100+0000","last_peered":"2024-09-13T22:39:39.933100+0000","last_clean":"2024-09-13T22:39:39.933100+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:39:39.933100+0000","last_undegraded":"2024-09-13T22:39:39.933100+0000","last_fullsized":"2024-09-13T22:39:39.933100+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:39:16.963643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.963643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.423517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933508+0000","last_change":"2024-09-13T22:39:19.315930+0000","last_active":"2024-09-13T22:39:39.933508+0000","last_peered":"2024-09-13T22:39:39.933508+0000","last_clean":"2024-09-13T22:39:39.933508+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:39:39.933508+0000","last_undegraded":"2024-09-13T22:39:39.933508+0000","last_fullsized":"2024-09-13T22:39:39.933508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.315707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.315707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.650430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":112,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.895993+0000","last_change":"2024-09-13T22:39:19.870202+0000","last_active":"2024-09-13T22:39:39.895993+0000","last_peered":"2024-09-13T22:39:39.895993+0000","last_clean":"2024-09-13T22:39:39.895993+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:39:39.895993+0000","last_undegraded":"2024-09-13T22:39:39.895993+0000","last_fullsized":"2024-09-13T22:39:39.895993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.869975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.869975+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.843605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":71,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933245+0000","last_change":"2024-09-13T22:39:18.975664+0000","last_active":"2024-09-13T22:39:39.933245+0000","last_peered":"2024-09-13T22:39:39.933245+0000","last_clean":"2024-09-13T22:39:39.933245+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:39:39.933245+0000","last_undegraded":"2024-09-13T22:39:39.933245+0000","last_fullsized":"2024-09-13T22:39:39.933245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.975461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.975461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.889969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827374+0000","last_change":"2024-09-13T22:39:28.141412+0000","last_active":"2024-09-13T22:39:39.827374+0000","last_peered":"2024-09-13T22:39:39.827374+0000","last_clean":"2024-09-13T22:39:39.827374+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:39:39.827374+0000","last_undegraded":"2024-09-13T22:39:39.827374+0000","last_fullsized":"2024-09-13T22:39:39.827374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:28.141327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:28.141327+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.373914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":83,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.827476+0000","last_change":"2024-09-13T22:39:09.960753+0000","last_active":"2024-09-13T22:39:39.827476+0000","last_peered":"2024-09-13T22:39:39.827476+0000","last_clean":"2024-09-13T22:39:39.827476+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:39:39.827476+0000","last_undegraded":"2024-09-13T22:39:39.827476+0000","last_fullsized":"2024-09-13T22:39:39.827476+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:09.960550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.960550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:38.827394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932803+0000","last_change":"2024-09-13T22:39:03.677235+0000","last_active":"2024-09-13T22:39:39.932803+0000","last_peered":"2024-09-13T22:39:39.932803+0000","last_clean":"2024-09-13T22:39:39.932803+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:39:39.932803+0000","last_undegraded":"2024-09-13T22:39:39.932803+0000","last_fullsized":"2024-09-13T22:39:39.932803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.677120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.433725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":70,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828727+0000","last_change":"2024-09-13T22:39:02.572852+0000","last_active":"2024-09-13T22:39:39.828727+0000","last_peered":"2024-09-13T22:39:39.828727+0000","last_clean":"2024-09-13T22:39:39.828727+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:39:39.828727+0000","last_undegraded":"2024-09-13T22:39:39.828727+0000","last_fullsized":"2024-09-13T22:39:39.828727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.572698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.572698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.965367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.895170+0000","last_change":"2024-09-13T22:38:56.897025+0000","last_active":"2024-09-13T22:39:39.895170+0000","last_peered":"2024-09-13T22:39:39.895170+0000","last_clean":"2024-09-13T22:39:39.895170+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:39:39.895170+0000","last_undegraded":"2024-09-13T22:39:39.895170+0000","last_fullsized":"2024-09-13T22:39:39.895170+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.896829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.896829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:58.985623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":90,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.932197+0000","last_change":"2024-09-13T22:38:58.572985+0000","last_active":"2024-09-13T22:39:39.932197+0000","last_peered":"2024-09-13T22:39:39.932197+0000","last_clean":"2024-09-13T22:39:39.932197+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:39:39.932197+0000","last_undegraded":"2024-09-13T22:39:39.932197+0000","last_fullsized":"2024-09-13T22:39:39.932197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.572643+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:00.431556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":75,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.933417+0000","last_change":"2024-09-13T22:39:23.072821+0000","last_active":"2024-09-13T22:39:39.933417+0000","last_peered":"2024-09-13T22:39:39.933417+0000","last_clean":"2024-09-13T22:39:39.933417+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:39:39.933417+0000","last_undegraded":"2024-09-13T22:39:39.933417+0000","last_fullsized":"2024-09-13T22:39:39.933417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:39:23.072590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.247551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":71,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828474+0000","last_change":"2024-09-13T22:38:57.829670+0000","last_active":"2024-09-13T22:39:39.828474+0000","last_peered":"2024-09-13T22:39:39.828474+0000","last_clean":"2024-09-13T22:39:39.828474+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:39:39.828474+0000","last_undegraded":"2024-09-13T22:39:39.828474+0000","last_fullsized":"2024-09-13T22:39:39.828474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:38:57.829468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.829468+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.502396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":82,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828014+0000","last_change":"2024-09-13T22:39:03.881680+0000","last_active":"2024-09-13T22:39:39.828014+0000","last_peered":"2024-09-13T22:39:39.828014+0000","last_clean":"2024-09-13T22:39:39.828014+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:39:39.828014+0000","last_undegraded":"2024-09-13T22:39:39.828014+0000","last_fullsized":"2024-09-13T22:39:39.828014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.881493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.881493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.403295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":94,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828393+0000","last_change":"2024-09-13T22:39:19.638335+0000","last_active":"2024-09-13T22:39:39.828393+0000","last_peered":"2024-09-13T22:39:39.828393+0000","last_clean":"2024-09-13T22:39:39.828393+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:39:39.828393+0000","last_undegraded":"2024-09-13T22:39:39.828393+0000","last_fullsized":"2024-09-13T22:39:39.828393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:19.638128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.638128+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.918763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":74,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-13T22:39:39.828312+0000","last_change":"2024-09-13T22:38:57.663485+0000","last_active":"2024-09-13T22:39:39.828312+0000","last_peered":"2024-09-13T22:39:39.828312+0000","last_clean":"2024-09-13T22:39:39.828312+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:39:39.828312+0000","last_undegraded":"2024-09-13T22:39:39.828312+0000","last_fullsized":"2024-09-13T22:39:39.828312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.663262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.663262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:07.199420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":99,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808865,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32436,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712716,"statfs":{"total":95995035648,"available":95961821184,"internally_reserved":0,"allocated":1282048,"data_stored":902468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135916,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32412,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712740,"statfs":{"total":95995035648,"available":95961845760,"internally_reserved":0,"allocated":1253376,"data_stored":900146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528375,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31836,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713316,"statfs":{"total":95995035648,"available":95962435584,"internally_reserved":0,"allocated":663552,"data_stored":310180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888129,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":696320,"data_stored":310710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084363,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31836,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713316,"statfs":{"total":95995035648,"available":95962435584,"internally_reserved":0,"allocated":663552,"data_stored":310601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378711,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":663552,"data_stored":310694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27688,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":675840,"data_stored":311089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1273856,"data_stored":902217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:39:48.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:39:48.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:48 smithi192 bash[24608]: audit 2024-09-13T22:39:47.311193+0000 mgr.y (mgr.14613) 80 : audit [DBG] from='client.24668 v1:172.21.15.178:0/742348729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:39:48.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:48 smithi192 bash[24608]: audit 2024-09-13T22:39:47.634897+0000 mon.b (mon.1) 83 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:48.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:48 smithi192 bash[24608]: cluster 2024-09-13T22:39:47.705819+0000 osd.7 (osd.7) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:48.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:48 smithi192 bash[24608]: cluster 2024-09-13T22:39:47.705931+0000 osd.7 (osd.7) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:48.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:48 smithi192 bash[24608]: cluster 2024-09-13T22:39:47.815820+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.c scrub starts 2024-09-13T22:39:48.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:48 smithi192 bash[24608]: cluster 2024-09-13T22:39:47.817283+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.c scrub ok 2024-09-13T22:39:48.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[22840]: audit 2024-09-13T22:39:47.311193+0000 mgr.y (mgr.14613) 80 : audit [DBG] from='client.24668 v1:172.21.15.178:0/742348729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:39:48.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[22840]: audit 2024-09-13T22:39:47.634897+0000 mon.b (mon.1) 83 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:48.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[22840]: cluster 2024-09-13T22:39:47.705819+0000 osd.7 (osd.7) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[22840]: cluster 2024-09-13T22:39:47.705931+0000 osd.7 (osd.7) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[22840]: cluster 2024-09-13T22:39:47.815820+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.c scrub starts 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[22840]: cluster 2024-09-13T22:39:47.817283+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.c scrub ok 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[29889]: audit 2024-09-13T22:39:47.311193+0000 mgr.y (mgr.14613) 80 : audit [DBG] from='client.24668 v1:172.21.15.178:0/742348729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[29889]: audit 2024-09-13T22:39:47.634897+0000 mon.b (mon.1) 83 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[29889]: cluster 2024-09-13T22:39:47.705819+0000 osd.7 (osd.7) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:39:48.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[29889]: cluster 2024-09-13T22:39:47.705931+0000 osd.7 (osd.7) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:39:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[29889]: cluster 2024-09-13T22:39:47.815820+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.c scrub starts 2024-09-13T22:39:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:48 smithi178 bash[29889]: cluster 2024-09-13T22:39:47.817283+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.c scrub ok 2024-09-13T22:39:49.351 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[60120]: ts=2024-09-13T22:39:49.040Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001373134s 2024-09-13T22:39:49.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:49 smithi192 bash[24608]: cluster 2024-09-13T22:39:48.348302+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-13T22:39:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:49 smithi192 bash[24608]: cluster 2024-09-13T22:39:48.446960+0000 mgr.y (mgr.14613) 81 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-09-13T22:39:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:49 smithi192 bash[24608]: cluster 2024-09-13T22:39:48.724457+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:39:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:49 smithi192 bash[24608]: cluster 2024-09-13T22:39:48.728487+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:39:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:49 smithi192 bash[24608]: cluster 2024-09-13T22:39:48.733744+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a deep-scrub starts 2024-09-13T22:39:49.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:49 smithi192 bash[24608]: cluster 2024-09-13T22:39:48.735976+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a deep-scrub ok 2024-09-13T22:39:49.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[22840]: cluster 2024-09-13T22:39:48.348302+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-13T22:39:49.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[22840]: cluster 2024-09-13T22:39:48.446960+0000 mgr.y (mgr.14613) 81 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-09-13T22:39:49.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[22840]: cluster 2024-09-13T22:39:48.724457+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:39:49.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[22840]: cluster 2024-09-13T22:39:48.728487+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:39:49.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[22840]: cluster 2024-09-13T22:39:48.733744+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a deep-scrub starts 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[22840]: cluster 2024-09-13T22:39:48.735976+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a deep-scrub ok 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[29889]: cluster 2024-09-13T22:39:48.348302+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[29889]: cluster 2024-09-13T22:39:48.446960+0000 mgr.y (mgr.14613) 81 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[29889]: cluster 2024-09-13T22:39:48.724457+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[29889]: cluster 2024-09-13T22:39:48.728487+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[29889]: cluster 2024-09-13T22:39:48.733744+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a deep-scrub starts 2024-09-13T22:39:49.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[29889]: cluster 2024-09-13T22:39:48.735976+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a deep-scrub ok 2024-09-13T22:39:50.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:39:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:39:50.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:50 smithi192 bash[24608]: cluster 2024-09-13T22:39:49.736358+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:39:50.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:50 smithi192 bash[24608]: cluster 2024-09-13T22:39:49.737987+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:39:50.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:50 smithi178 bash[29889]: cluster 2024-09-13T22:39:49.736358+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:39:50.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:50 smithi178 bash[29889]: cluster 2024-09-13T22:39:49.737987+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:39:50.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:50 smithi178 bash[22840]: cluster 2024-09-13T22:39:49.736358+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:39:50.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:50 smithi178 bash[22840]: cluster 2024-09-13T22:39:49.737987+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:39:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:51 smithi192 bash[24608]: cluster 2024-09-13T22:39:50.448378+0000 mgr.y (mgr.14613) 82 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:51.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:51 smithi178 bash[29889]: cluster 2024-09-13T22:39:50.448378+0000 mgr.y (mgr.14613) 82 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:51.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:51 smithi178 bash[22840]: cluster 2024-09-13T22:39:50.448378+0000 mgr.y (mgr.14613) 82 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:52.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:52 smithi192 bash[24608]: cluster 2024-09-13T22:39:51.639192+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.e scrub starts 2024-09-13T22:39:52.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:52 smithi192 bash[24608]: cluster 2024-09-13T22:39:51.642055+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.e scrub ok 2024-09-13T22:39:52.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:52 smithi178 bash[22840]: cluster 2024-09-13T22:39:51.639192+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.e scrub starts 2024-09-13T22:39:52.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:52 smithi178 bash[22840]: cluster 2024-09-13T22:39:51.642055+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.e scrub ok 2024-09-13T22:39:52.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:52 smithi178 bash[29889]: cluster 2024-09-13T22:39:51.639192+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.e scrub starts 2024-09-13T22:39:52.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:52 smithi178 bash[29889]: cluster 2024-09-13T22:39:51.642055+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.e scrub ok 2024-09-13T22:39:52.868 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:39:53.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:53 smithi192 bash[24608]: cluster 2024-09-13T22:39:52.231222+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:39:53.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:53 smithi192 bash[24608]: cluster 2024-09-13T22:39:52.233672+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:39:53.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:53 smithi192 bash[24608]: cluster 2024-09-13T22:39:52.449049+0000 mgr.y (mgr.14613) 83 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:53.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:53 smithi178 bash[29889]: cluster 2024-09-13T22:39:52.231222+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:39:53.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:53 smithi178 bash[29889]: cluster 2024-09-13T22:39:52.233672+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:39:53.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:53 smithi178 bash[29889]: cluster 2024-09-13T22:39:52.449049+0000 mgr.y (mgr.14613) 83 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:53.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:53 smithi178 bash[22840]: cluster 2024-09-13T22:39:52.231222+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:39:53.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:53 smithi178 bash[22840]: cluster 2024-09-13T22:39:52.233672+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:39:53.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:53 smithi178 bash[22840]: cluster 2024-09-13T22:39:52.449049+0000 mgr.y (mgr.14613) 83 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:54.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:54 smithi178 bash[22840]: cluster 2024-09-13T22:39:53.211736+0000 osd.3 (osd.3) 51 : cluster [DBG] 6.c scrub starts 2024-09-13T22:39:54.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:54 smithi178 bash[22840]: cluster 2024-09-13T22:39:53.214306+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.c scrub ok 2024-09-13T22:39:54.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:54 smithi178 bash[29889]: cluster 2024-09-13T22:39:53.211736+0000 osd.3 (osd.3) 51 : cluster [DBG] 6.c scrub starts 2024-09-13T22:39:54.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:54 smithi178 bash[29889]: cluster 2024-09-13T22:39:53.214306+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.c scrub ok 2024-09-13T22:39:54.503 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:39:54.505 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:39:54.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:54 smithi192 bash[24608]: cluster 2024-09-13T22:39:53.211736+0000 osd.3 (osd.3) 51 : cluster [DBG] 6.c scrub starts 2024-09-13T22:39:54.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:54 smithi192 bash[24608]: cluster 2024-09-13T22:39:53.214306+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.c scrub ok 2024-09-13T22:39:55.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:39:54 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:39:55.168 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-13T22:39:54.449334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1064,"num_read_kb":910,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":670,"ondisk_log_size":670,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244108,"kb_used_data":7100,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749717108,"statfs":{"total":767960285184,"available":767710318592,"internally_reserved":0,"allocated":7270400,"data_stored":4309713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004890"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:52.233819+0000","last_change":"2024-09-13T22:39:52.233876+0000","last_active":"2024-09-13T22:39:52.233819+0000","last_peered":"2024-09-13T22:39:52.233819+0000","last_clean":"2024-09-13T22:39:52.233819+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:39:52.233819+0000","last_undegraded":"2024-09-13T22:39:52.233819+0000","last_fullsized":"2024-09-13T22:39:52.233819+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:52.233682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:52.233682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:15.212128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.808500+0000","last_change":"2024-09-13T22:38:46.613749+0000","last_active":"2024-09-13T22:39:48.808500+0000","last_peered":"2024-09-13T22:39:48.808500+0000","last_clean":"2024-09-13T22:39:48.808500+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:39:48.808500+0000","last_undegraded":"2024-09-13T22:39:48.808500+0000","last_fullsized":"2024-09-13T22:39:48.808500+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.613466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:15.391727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":101,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349344+0000","last_change":"2024-09-13T22:38:54.776466+0000","last_active":"2024-09-13T22:39:48.349344+0000","last_peered":"2024-09-13T22:39:48.349344+0000","last_clean":"2024-09-13T22:39:48.349344+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:39:48.349344+0000","last_undegraded":"2024-09-13T22:39:48.349344+0000","last_fullsized":"2024-09-13T22:39:48.349344+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.776238+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.515503+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349648+0000","last_change":"2024-09-13T22:39:20.600784+0000","last_active":"2024-09-13T22:39:48.349648+0000","last_peered":"2024-09-13T22:39:48.349648+0000","last_clean":"2024-09-13T22:39:48.349648+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:39:48.349648+0000","last_undegraded":"2024-09-13T22:39:48.349648+0000","last_fullsized":"2024-09-13T22:39:48.349648+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.600571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.907326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349433+0000","last_change":"2024-09-13T22:39:27.118459+0000","last_active":"2024-09-13T22:39:48.349433+0000","last_peered":"2024-09-13T22:39:48.349433+0000","last_clean":"2024-09-13T22:39:48.349433+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:39:48.349433+0000","last_undegraded":"2024-09-13T22:39:48.349433+0000","last_fullsized":"2024-09-13T22:39:48.349433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.118162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.118162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.139079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":105,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810329+0000","last_change":"2024-09-13T22:39:08.754354+0000","last_active":"2024-09-13T22:39:48.810329+0000","last_peered":"2024-09-13T22:39:48.810329+0000","last_clean":"2024-09-13T22:39:48.810329+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:39:48.810329+0000","last_undegraded":"2024-09-13T22:39:48.810329+0000","last_fullsized":"2024-09-13T22:39:48.810329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.754231+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:14.661781+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.730406+0000","last_change":"2024-09-13T22:39:24.033368+0000","last_active":"2024-09-13T22:39:48.730406+0000","last_peered":"2024-09-13T22:39:48.730406+0000","last_clean":"2024-09-13T22:39:48.730406+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:39:48.730406+0000","last_undegraded":"2024-09-13T22:39:48.730406+0000","last_fullsized":"2024-09-13T22:39:48.730406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.033224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.033224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.108639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350860+0000","last_change":"2024-09-13T22:38:59.657956+0000","last_active":"2024-09-13T22:39:48.350860+0000","last_peered":"2024-09-13T22:39:48.350860+0000","last_clean":"2024-09-13T22:39:48.350860+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:39:48.350860+0000","last_undegraded":"2024-09-13T22:39:48.350860+0000","last_fullsized":"2024-09-13T22:39:48.350860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:59.657646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.657646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.987419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.724141+0000","last_change":"2024-09-13T22:39:19.778773+0000","last_active":"2024-09-13T22:39:48.724141+0000","last_peered":"2024-09-13T22:39:48.724141+0000","last_clean":"2024-09-13T22:39:48.724141+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:39:48.724141+0000","last_undegraded":"2024-09-13T22:39:48.724141+0000","last_fullsized":"2024-09-13T22:39:48.724141+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.778590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.778590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.572565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":112,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.008629+0000","last_change":"2024-09-13T22:39:26.318996+0000","last_active":"2024-09-13T22:39:49.008629+0000","last_peered":"2024-09-13T22:39:49.008629+0000","last_clean":"2024-09-13T22:39:49.008629+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:39:49.008629+0000","last_undegraded":"2024-09-13T22:39:49.008629+0000","last_fullsized":"2024-09-13T22:39:49.008629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.318753+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.307721+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.736123+0000","last_change":"2024-09-13T22:39:48.736178+0000","last_active":"2024-09-13T22:39:48.736123+0000","last_peered":"2024-09-13T22:39:48.736123+0000","last_clean":"2024-09-13T22:39:48.736123+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:39:48.736123+0000","last_undegraded":"2024-09-13T22:39:48.736123+0000","last_fullsized":"2024-09-13T22:39:48.736123+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:39:48.735986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:14.424901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.007584+0000","last_change":"2024-09-13T22:39:37.426328+0000","last_active":"2024-09-13T22:39:49.007584+0000","last_peered":"2024-09-13T22:39:49.007584+0000","last_clean":"2024-09-13T22:39:49.007584+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:39:49.007584+0000","last_undegraded":"2024-09-13T22:39:49.007584+0000","last_fullsized":"2024-09-13T22:39:49.007584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:37.426096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:37.426096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.076012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.724753+0000","last_change":"2024-09-13T22:38:54.689196+0000","last_active":"2024-09-13T22:39:48.724753+0000","last_peered":"2024-09-13T22:39:48.724753+0000","last_clean":"2024-09-13T22:39:48.724753+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:39:48.724753+0000","last_undegraded":"2024-09-13T22:39:48.724753+0000","last_fullsized":"2024-09-13T22:39:48.724753+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:54.689114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.689114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:59.621133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":103,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.009136+0000","last_change":"2024-09-13T22:38:49.543240+0000","last_active":"2024-09-13T22:39:49.009136+0000","last_peered":"2024-09-13T22:39:49.009136+0000","last_clean":"2024-09-13T22:39:49.009136+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:39:49.009136+0000","last_undegraded":"2024-09-13T22:39:49.009136+0000","last_fullsized":"2024-09-13T22:39:49.009136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.543010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.543010+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.890250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.008164+0000","last_change":"2024-09-13T22:38:50.497874+0000","last_active":"2024-09-13T22:39:49.008164+0000","last_peered":"2024-09-13T22:39:49.008164+0000","last_clean":"2024-09-13T22:39:49.008164+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:39:49.008164+0000","last_undegraded":"2024-09-13T22:39:49.008164+0000","last_fullsized":"2024-09-13T22:39:49.008164+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:50.497567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:50.497567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.010369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.808898+0000","last_change":"2024-09-13T22:39:14.752874+0000","last_active":"2024-09-13T22:39:48.808898+0000","last_peered":"2024-09-13T22:39:48.808898+0000","last_clean":"2024-09-13T22:39:48.808898+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:39:48.808898+0000","last_undegraded":"2024-09-13T22:39:48.808898+0000","last_fullsized":"2024-09-13T22:39:48.808898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.752683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.752683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.453233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"82'19","reported_seq":111,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.718589+0000","last_change":"2024-09-13T22:39:21.873986+0000","last_active":"2024-09-13T22:39:48.718589+0000","last_peered":"2024-09-13T22:39:48.718589+0000","last_clean":"2024-09-13T22:39:48.718589+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:39:48.718589+0000","last_undegraded":"2024-09-13T22:39:48.718589+0000","last_fullsized":"2024-09-13T22:39:48.718589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:21.873747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.873747+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.726394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.724581+0000","last_change":"2024-09-13T22:39:35.662591+0000","last_active":"2024-09-13T22:39:48.724581+0000","last_peered":"2024-09-13T22:39:48.724581+0000","last_clean":"2024-09-13T22:39:48.724581+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:39:48.724581+0000","last_undegraded":"2024-09-13T22:39:48.724581+0000","last_fullsized":"2024-09-13T22:39:48.724581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:35.662396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:35.662396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:48.563351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731135+0000","last_change":"2024-09-13T22:39:15.972091+0000","last_active":"2024-09-13T22:39:48.731135+0000","last_peered":"2024-09-13T22:39:48.731135+0000","last_clean":"2024-09-13T22:39:48.731135+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:39:48.731135+0000","last_undegraded":"2024-09-13T22:39:48.731135+0000","last_fullsized":"2024-09-13T22:39:48.731135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.971904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.971904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:40.478614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.857368+0000","last_change":"2024-09-13T22:39:14.436930+0000","last_active":"2024-09-13T22:39:48.857368+0000","last_peered":"2024-09-13T22:39:48.857368+0000","last_clean":"2024-09-13T22:39:48.857368+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:39:48.857368+0000","last_undegraded":"2024-09-13T22:39:48.857368+0000","last_fullsized":"2024-09-13T22:39:48.857368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.436746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.436746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.039947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'9","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349356+0000","last_change":"2024-09-13T22:38:43.752187+0000","last_active":"2024-09-13T22:39:48.349356+0000","last_peered":"2024-09-13T22:39:48.349356+0000","last_clean":"2024-09-13T22:39:48.349356+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:39:48.349356+0000","last_undegraded":"2024-09-13T22:39:48.349356+0000","last_fullsized":"2024-09-13T22:39:48.349356+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:43.751981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:43.751981+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.845059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":86,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731740+0000","last_change":"2024-09-13T22:39:01.945920+0000","last_active":"2024-09-13T22:39:48.731740+0000","last_peered":"2024-09-13T22:39:48.731740+0000","last_clean":"2024-09-13T22:39:48.731740+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:39:48.731740+0000","last_undegraded":"2024-09-13T22:39:48.731740+0000","last_fullsized":"2024-09-13T22:39:48.731740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:01.945733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.945733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.448893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":113,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.857309+0000","last_change":"2024-09-13T22:39:19.489746+0000","last_active":"2024-09-13T22:39:48.857309+0000","last_peered":"2024-09-13T22:39:48.857309+0000","last_clean":"2024-09-13T22:39:48.857309+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:39:48.857309+0000","last_undegraded":"2024-09-13T22:39:48.857309+0000","last_fullsized":"2024-09-13T22:39:48.857309+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.489549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.489549+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:47.360174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":80,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:51.642210+0000","last_change":"2024-09-13T22:39:51.642269+0000","last_active":"2024-09-13T22:39:51.642210+0000","last_peered":"2024-09-13T22:39:51.642210+0000","last_clean":"2024-09-13T22:39:51.642210+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:39:51.642210+0000","last_undegraded":"2024-09-13T22:39:51.642210+0000","last_fullsized":"2024-09-13T22:39:51.642210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:51.642066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:51.642066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:03.230502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":106,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.348426+0000","last_change":"2024-09-13T22:38:58.835483+0000","last_active":"2024-09-13T22:39:48.348426+0000","last_peered":"2024-09-13T22:39:48.348426+0000","last_clean":"2024-09-13T22:39:48.348426+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:39:48.348426+0000","last_undegraded":"2024-09-13T22:39:48.348426+0000","last_fullsized":"2024-09-13T22:39:48.348426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:38:58.835275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.835275+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.927373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":86,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731017+0000","last_change":"2024-09-13T22:38:46.006268+0000","last_active":"2024-09-13T22:39:48.731017+0000","last_peered":"2024-09-13T22:39:48.731017+0000","last_clean":"2024-09-13T22:39:48.731017+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:39:48.731017+0000","last_undegraded":"2024-09-13T22:39:48.731017+0000","last_fullsized":"2024-09-13T22:39:48.731017+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.006072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.006072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:13.250082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":114,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350980+0000","last_change":"2024-09-13T22:39:12.592344+0000","last_active":"2024-09-13T22:39:48.350980+0000","last_peered":"2024-09-13T22:39:48.350980+0000","last_clean":"2024-09-13T22:39:48.350980+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:39:48.350980+0000","last_undegraded":"2024-09-13T22:39:48.350980+0000","last_fullsized":"2024-09-13T22:39:48.350980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.592134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.802063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.008547+0000","last_change":"2024-09-13T22:39:33.392634+0000","last_active":"2024-09-13T22:39:49.008547+0000","last_peered":"2024-09-13T22:39:49.008547+0000","last_clean":"2024-09-13T22:39:49.008547+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:39:49.008547+0000","last_undegraded":"2024-09-13T22:39:49.008547+0000","last_fullsized":"2024-09-13T22:39:49.008547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_clean_scrub_stamp":"2024-09-13T22:39:33.392318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.641551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":104,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.351085+0000","last_change":"2024-09-13T22:38:49.656312+0000","last_active":"2024-09-13T22:39:48.351085+0000","last_peered":"2024-09-13T22:39:48.351085+0000","last_clean":"2024-09-13T22:39:48.351085+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:39:48.351085+0000","last_undegraded":"2024-09-13T22:39:48.351085+0000","last_fullsized":"2024-09-13T22:39:48.351085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.656093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.656093+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.977202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722704+0000","last_change":"2024-09-13T22:39:40.612195+0000","last_active":"2024-09-13T22:39:48.722704+0000","last_peered":"2024-09-13T22:39:48.722704+0000","last_clean":"2024-09-13T22:39:48.722704+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:39:48.722704+0000","last_undegraded":"2024-09-13T22:39:48.722704+0000","last_fullsized":"2024-09-13T22:39:48.722704+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:40.611997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.611997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:10.478412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.009592+0000","last_change":"2024-09-13T22:38:51.542882+0000","last_active":"2024-09-13T22:39:49.009592+0000","last_peered":"2024-09-13T22:39:49.009592+0000","last_clean":"2024-09-13T22:39:49.009592+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:39:49.009592+0000","last_undegraded":"2024-09-13T22:39:49.009592+0000","last_fullsized":"2024-09-13T22:39:49.009592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:51.542757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:51.542757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:05.178866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:53.214484+0000","last_change":"2024-09-13T22:39:53.214599+0000","last_active":"2024-09-13T22:39:53.214484+0000","last_peered":"2024-09-13T22:39:53.214484+0000","last_clean":"2024-09-13T22:39:53.214484+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:39:53.214484+0000","last_undegraded":"2024-09-13T22:39:53.214484+0000","last_fullsized":"2024-09-13T22:39:53.214484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:53.214316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:53.214316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:03.248657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":99,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810658+0000","last_change":"2024-09-13T22:39:04.726321+0000","last_active":"2024-09-13T22:39:48.810658+0000","last_peered":"2024-09-13T22:39:48.810658+0000","last_clean":"2024-09-13T22:39:48.810658+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:39:48.810658+0000","last_undegraded":"2024-09-13T22:39:48.810658+0000","last_fullsized":"2024-09-13T22:39:48.810658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:39:04.726105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.726105+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.304382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.719495+0000","last_change":"2024-09-13T22:39:08.858599+0000","last_active":"2024-09-13T22:39:48.719495+0000","last_peered":"2024-09-13T22:39:48.719495+0000","last_clean":"2024-09-13T22:39:48.719495+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:39:48.719495+0000","last_undegraded":"2024-09-13T22:39:48.719495+0000","last_fullsized":"2024-09-13T22:39:48.719495+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:08.858481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.858481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.756681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350451+0000","last_change":"2024-09-13T22:39:15.627485+0000","last_active":"2024-09-13T22:39:48.350451+0000","last_peered":"2024-09-13T22:39:48.350451+0000","last_clean":"2024-09-13T22:39:48.350451+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:39:48.350451+0000","last_undegraded":"2024-09-13T22:39:48.350451+0000","last_fullsized":"2024-09-13T22:39:48.350451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:39:15.627264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.627264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.016937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":114,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.461473+0000","last_change":"2024-09-13T22:39:40.265509+0000","last_active":"2024-09-13T22:39:48.461473+0000","last_peered":"2024-09-13T22:39:48.461473+0000","last_clean":"2024-09-13T22:39:48.461473+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:39:48.461473+0000","last_undegraded":"2024-09-13T22:39:48.461473+0000","last_fullsized":"2024-09-13T22:39:48.461473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:40.265350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.265350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:00.451629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":73,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.738117+0000","last_change":"2024-09-13T22:39:49.738117+0000","last_active":"2024-09-13T22:39:49.738117+0000","last_peered":"2024-09-13T22:39:49.738117+0000","last_clean":"2024-09-13T22:39:49.738117+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:39:49.738117+0000","last_undegraded":"2024-09-13T22:39:49.738117+0000","last_fullsized":"2024-09-13T22:39:49.738117+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:49.737992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:49.737992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:55.782646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":113,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349655+0000","last_change":"2024-09-13T22:39:16.975346+0000","last_active":"2024-09-13T22:39:48.349655+0000","last_peered":"2024-09-13T22:39:48.349655+0000","last_clean":"2024-09-13T22:39:48.349655+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:39:48.349655+0000","last_undegraded":"2024-09-13T22:39:48.349655+0000","last_fullsized":"2024-09-13T22:39:48.349655+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.975244+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.689091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":87,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731985+0000","last_change":"2024-09-13T22:39:07.049286+0000","last_active":"2024-09-13T22:39:48.731985+0000","last_peered":"2024-09-13T22:39:48.731985+0000","last_clean":"2024-09-13T22:39:48.731985+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:39:48.731985+0000","last_undegraded":"2024-09-13T22:39:48.731985+0000","last_fullsized":"2024-09-13T22:39:48.731985+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:07.049084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.049084+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:35.995865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":205,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:52.005087+0000","last_change":"2024-09-13T22:39:38.224453+0000","last_active":"2024-09-13T22:39:52.005087+0000","last_peered":"2024-09-13T22:39:52.005087+0000","last_clean":"2024-09-13T22:39:52.005087+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:39:52.005087+0000","last_undegraded":"2024-09-13T22:39:52.005087+0000","last_fullsized":"2024-09-13T22:39:52.005087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:39:38.224366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:38.224366+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.543802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":114,"num_read_kb":109,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350052+0000","last_change":"2024-09-13T22:38:58.704351+0000","last_active":"2024-09-13T22:39:48.350052+0000","last_peered":"2024-09-13T22:39:48.350052+0000","last_clean":"2024-09-13T22:39:48.350052+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:39:48.350052+0000","last_undegraded":"2024-09-13T22:39:48.350052+0000","last_fullsized":"2024-09-13T22:39:48.350052+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:58.704193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.704193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:04.200792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":73,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350048+0000","last_change":"2024-09-13T22:39:05.561714+0000","last_active":"2024-09-13T22:39:48.350048+0000","last_peered":"2024-09-13T22:39:48.350048+0000","last_clean":"2024-09-13T22:39:48.350048+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:39:48.350048+0000","last_undegraded":"2024-09-13T22:39:48.350048+0000","last_fullsized":"2024-09-13T22:39:48.350048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.561489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.561489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.439424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":159,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731482+0000","last_change":"2024-09-13T22:39:11.036017+0000","last_active":"2024-09-13T22:39:48.731482+0000","last_peered":"2024-09-13T22:39:48.731482+0000","last_clean":"2024-09-13T22:39:48.731482+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:39:48.731482+0000","last_undegraded":"2024-09-13T22:39:48.731482+0000","last_fullsized":"2024-09-13T22:39:48.731482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:39:11.035787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.035787+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.606987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.724149+0000","last_change":"2024-09-13T22:39:31.671011+0000","last_active":"2024-09-13T22:39:48.724149+0000","last_peered":"2024-09-13T22:39:48.724149+0000","last_clean":"2024-09-13T22:39:48.724149+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:39:48.724149+0000","last_undegraded":"2024-09-13T22:39:48.724149+0000","last_fullsized":"2024-09-13T22:39:48.724149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.670821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.670821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.397166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.718242+0000","last_change":"2024-09-13T22:39:24.762341+0000","last_active":"2024-09-13T22:39:48.718242+0000","last_peered":"2024-09-13T22:39:48.718242+0000","last_clean":"2024-09-13T22:39:48.718242+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:39:48.718242+0000","last_undegraded":"2024-09-13T22:39:48.718242+0000","last_fullsized":"2024-09-13T22:39:48.718242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.762123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.762123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.895897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":74,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.730408+0000","last_change":"2024-09-13T22:39:25.019347+0000","last_active":"2024-09-13T22:39:48.730408+0000","last_peered":"2024-09-13T22:39:48.730408+0000","last_clean":"2024-09-13T22:39:48.730408+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:39:48.730408+0000","last_undegraded":"2024-09-13T22:39:48.730408+0000","last_fullsized":"2024-09-13T22:39:48.730408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.019168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.019168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.669620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":138,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:50.904485+0000","last_change":"2024-09-13T22:39:24.368074+0000","last_active":"2024-09-13T22:39:50.904485+0000","last_peered":"2024-09-13T22:39:50.904485+0000","last_clean":"2024-09-13T22:39:50.904485+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:39:50.904485+0000","last_undegraded":"2024-09-13T22:39:50.904485+0000","last_fullsized":"2024-09-13T22:39:50.904485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.367843+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.291892+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":94,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731536+0000","last_change":"2024-09-13T22:39:06.025706+0000","last_active":"2024-09-13T22:39:48.731536+0000","last_peered":"2024-09-13T22:39:48.731536+0000","last_clean":"2024-09-13T22:39:48.731536+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:39:48.731536+0000","last_undegraded":"2024-09-13T22:39:48.731536+0000","last_fullsized":"2024-09-13T22:39:48.731536+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:06.025488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:06.025488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.356278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":91,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731136+0000","last_change":"2024-09-13T22:39:12.041436+0000","last_active":"2024-09-13T22:39:48.731136+0000","last_peered":"2024-09-13T22:39:48.731136+0000","last_clean":"2024-09-13T22:39:48.731136+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:39:48.731136+0000","last_undegraded":"2024-09-13T22:39:48.731136+0000","last_fullsized":"2024-09-13T22:39:48.731136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:39:12.041252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.041252+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.755767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":115,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.809817+0000","last_change":"2024-09-13T22:39:01.656159+0000","last_active":"2024-09-13T22:39:48.809817+0000","last_peered":"2024-09-13T22:39:48.809817+0000","last_clean":"2024-09-13T22:39:48.809817+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:39:48.809817+0000","last_undegraded":"2024-09-13T22:39:48.809817+0000","last_fullsized":"2024-09-13T22:39:48.809817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.655949+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.600034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":73,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.809699+0000","last_change":"2024-09-13T22:39:25.685706+0000","last_active":"2024-09-13T22:39:48.809699+0000","last_peered":"2024-09-13T22:39:48.809699+0000","last_clean":"2024-09-13T22:39:48.809699+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:39:48.809699+0000","last_undegraded":"2024-09-13T22:39:48.809699+0000","last_fullsized":"2024-09-13T22:39:48.809699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.685432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.685432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.180606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350439+0000","last_change":"2024-09-13T22:39:10.618742+0000","last_active":"2024-09-13T22:39:48.350439+0000","last_peered":"2024-09-13T22:39:48.350439+0000","last_clean":"2024-09-13T22:39:48.350439+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:39:48.350439+0000","last_undegraded":"2024-09-13T22:39:48.350439+0000","last_fullsized":"2024-09-13T22:39:48.350439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:10.618592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.618592+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.736269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.006602+0000","last_change":"2024-09-13T22:38:56.500284+0000","last_active":"2024-09-13T22:39:49.006602+0000","last_peered":"2024-09-13T22:39:49.006602+0000","last_clean":"2024-09-13T22:39:49.006602+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:39:49.006602+0000","last_undegraded":"2024-09-13T22:39:49.006602+0000","last_fullsized":"2024-09-13T22:39:49.006602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.500054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:00.349788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810339+0000","last_change":"2024-09-13T22:39:02.682368+0000","last_active":"2024-09-13T22:39:48.810339+0000","last_peered":"2024-09-13T22:39:48.810339+0000","last_clean":"2024-09-13T22:39:48.810339+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:39:48.810339+0000","last_undegraded":"2024-09-13T22:39:48.810339+0000","last_fullsized":"2024-09-13T22:39:48.810339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.682170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.682170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.706353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349170+0000","last_change":"2024-09-13T22:39:45.281595+0000","last_active":"2024-09-13T22:39:48.349170+0000","last_peered":"2024-09-13T22:39:48.349170+0000","last_clean":"2024-09-13T22:39:48.349170+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:39:48.349170+0000","last_undegraded":"2024-09-13T22:39:48.349170+0000","last_fullsized":"2024-09-13T22:39:48.349170+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:45.281512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:45.281512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:55.823803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"82'18","reported_seq":114,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.730595+0000","last_change":"2024-09-13T22:39:19.966260+0000","last_active":"2024-09-13T22:39:48.730595+0000","last_peered":"2024-09-13T22:39:48.730595+0000","last_clean":"2024-09-13T22:39:48.730595+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:39:48.730595+0000","last_undegraded":"2024-09-13T22:39:48.730595+0000","last_fullsized":"2024-09-13T22:39:48.730595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-13T22:39:19.966043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.966043+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.842672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.855878+0000","last_change":"2024-09-13T22:39:15.472999+0000","last_active":"2024-09-13T22:39:48.855878+0000","last_peered":"2024-09-13T22:39:48.855878+0000","last_clean":"2024-09-13T22:39:48.855878+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:39:48.855878+0000","last_undegraded":"2024-09-13T22:39:48.855878+0000","last_fullsized":"2024-09-13T22:39:48.855878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:15.472777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.472777+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:41.403750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.857337+0000","last_change":"2024-09-13T22:38:54.478611+0000","last_active":"2024-09-13T22:39:48.857337+0000","last_peered":"2024-09-13T22:39:48.857337+0000","last_clean":"2024-09-13T22:39:48.857337+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:39:48.857337+0000","last_undegraded":"2024-09-13T22:39:48.857337+0000","last_fullsized":"2024-09-13T22:39:48.857337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:54.478424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.478424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.387064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":73,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723596+0000","last_change":"2024-09-13T22:39:20.748359+0000","last_active":"2024-09-13T22:39:48.723596+0000","last_peered":"2024-09-13T22:39:48.723596+0000","last_clean":"2024-09-13T22:39:48.723596+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:39:48.723596+0000","last_undegraded":"2024-09-13T22:39:48.723596+0000","last_fullsized":"2024-09-13T22:39:48.723596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.748082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.748082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:30.962778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":102,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.809178+0000","last_change":"2024-09-13T22:39:05.725791+0000","last_active":"2024-09-13T22:39:48.809178+0000","last_peered":"2024-09-13T22:39:48.809178+0000","last_clean":"2024-09-13T22:39:48.809178+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:39:48.809178+0000","last_undegraded":"2024-09-13T22:39:48.809178+0000","last_fullsized":"2024-09-13T22:39:48.809178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:39:05.725568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.725568+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.221576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.856185+0000","last_change":"2024-09-13T22:39:34.703390+0000","last_active":"2024-09-13T22:39:48.856185+0000","last_peered":"2024-09-13T22:39:48.856185+0000","last_clean":"2024-09-13T22:39:48.856185+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:39:48.856185+0000","last_undegraded":"2024-09-13T22:39:48.856185+0000","last_fullsized":"2024-09-13T22:39:48.856185+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:26.772349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.772349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.725362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.009787+0000","last_change":"2024-09-13T22:39:12.302084+0000","last_active":"2024-09-13T22:39:49.009787+0000","last_peered":"2024-09-13T22:39:49.009787+0000","last_clean":"2024-09-13T22:39:49.009787+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:39:49.009787+0000","last_undegraded":"2024-09-13T22:39:49.009787+0000","last_fullsized":"2024-09-13T22:39:49.009787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.301909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.301909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.703405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":74,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731000+0000","last_change":"2024-09-13T22:38:57.971090+0000","last_active":"2024-09-13T22:39:48.731000+0000","last_peered":"2024-09-13T22:39:48.731000+0000","last_clean":"2024-09-13T22:39:48.731000+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:39:48.731000+0000","last_undegraded":"2024-09-13T22:39:48.731000+0000","last_fullsized":"2024-09-13T22:39:48.731000+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.970850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.970850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:09.411139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349204+0000","last_change":"2024-09-13T22:39:04.843005+0000","last_active":"2024-09-13T22:39:48.349204+0000","last_peered":"2024-09-13T22:39:48.349204+0000","last_clean":"2024-09-13T22:39:48.349204+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:39:48.349204+0000","last_undegraded":"2024-09-13T22:39:48.349204+0000","last_fullsized":"2024-09-13T22:39:48.349204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:04.842757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.842757+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.409317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.006641+0000","last_change":"2024-09-13T22:38:53.554534+0000","last_active":"2024-09-13T22:39:49.006641+0000","last_peered":"2024-09-13T22:39:49.006641+0000","last_clean":"2024-09-13T22:39:49.006641+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:39:49.006641+0000","last_undegraded":"2024-09-13T22:39:49.006641+0000","last_fullsized":"2024-09-13T22:39:49.006641+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:53.554288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:53.554288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:07.563224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":338,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722856+0000","last_change":"2024-09-13T22:38:14.773746+0000","last_active":"2024-09-13T22:39:48.722856+0000","last_peered":"2024-09-13T22:39:48.722856+0000","last_clean":"2024-09-13T22:39:48.722856+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:39:48.722856+0000","last_undegraded":"2024-09-13T22:39:48.722856+0000","last_fullsized":"2024-09-13T22:39:48.722856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'119","last_scrub_stamp":"2024-09-13T22:38:14.773565+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:38:14.773565+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723140+0000","last_change":"2024-09-13T22:38:56.676371+0000","last_active":"2024-09-13T22:39:48.723140+0000","last_peered":"2024-09-13T22:39:48.723140+0000","last_clean":"2024-09-13T22:39:48.723140+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:39:48.723140+0000","last_undegraded":"2024-09-13T22:39:48.723140+0000","last_fullsized":"2024-09-13T22:39:48.723140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.676172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.058738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.009341+0000","last_change":"2024-09-13T22:39:00.431121+0000","last_active":"2024-09-13T22:39:49.009341+0000","last_peered":"2024-09-13T22:39:49.009341+0000","last_clean":"2024-09-13T22:39:49.009341+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:39:49.009341+0000","last_undegraded":"2024-09-13T22:39:49.009341+0000","last_fullsized":"2024-09-13T22:39:49.009341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:00.430982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.430982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:04.949172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":108,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723832+0000","last_change":"2024-09-13T22:39:42.666727+0000","last_active":"2024-09-13T22:39:48.723832+0000","last_peered":"2024-09-13T22:39:48.723832+0000","last_clean":"2024-09-13T22:39:48.723832+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:39:48.723832+0000","last_undegraded":"2024-09-13T22:39:48.723832+0000","last_fullsized":"2024-09-13T22:39:48.723832+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:39:42.666504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:42.666504+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.765205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":92,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.857404+0000","last_change":"2024-09-13T22:39:47.817369+0000","last_active":"2024-09-13T22:39:48.857404+0000","last_peered":"2024-09-13T22:39:48.857404+0000","last_clean":"2024-09-13T22:39:48.857404+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:39:48.857404+0000","last_undegraded":"2024-09-13T22:39:48.857404+0000","last_fullsized":"2024-09-13T22:39:48.857404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:47.817287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:47.817287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.198399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.856621+0000","last_change":"2024-09-13T22:39:12.437612+0000","last_active":"2024-09-13T22:39:48.856621+0000","last_peered":"2024-09-13T22:39:48.856621+0000","last_clean":"2024-09-13T22:39:48.856621+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:39:48.856621+0000","last_undegraded":"2024-09-13T22:39:48.856621+0000","last_fullsized":"2024-09-13T22:39:48.856621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.437298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.437298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.840182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722637+0000","last_change":"2024-09-13T22:37:46.924543+0000","last_active":"2024-09-13T22:39:48.722637+0000","last_peered":"2024-09-13T22:39:48.722637+0000","last_clean":"2024-09-13T22:39:48.722637+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:39:48.722637+0000","last_undegraded":"2024-09-13T22:39:48.722637+0000","last_fullsized":"2024-09-13T22:39:48.722637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":95,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.008800+0000","last_change":"2024-09-13T22:39:09.333581+0000","last_active":"2024-09-13T22:39:49.008800+0000","last_peered":"2024-09-13T22:39:49.008800+0000","last_clean":"2024-09-13T22:39:49.008800+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:39:49.008800+0000","last_undegraded":"2024-09-13T22:39:49.008800+0000","last_fullsized":"2024-09-13T22:39:49.008800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:09.333359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.333359+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.270070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731821+0000","last_change":"2024-09-13T22:38:52.100566+0000","last_active":"2024-09-13T22:39:48.731821+0000","last_peered":"2024-09-13T22:39:48.731821+0000","last_clean":"2024-09-13T22:39:48.731821+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:39:48.731821+0000","last_undegraded":"2024-09-13T22:39:48.731821+0000","last_fullsized":"2024-09-13T22:39:48.731821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:52.100377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:52.100377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:09.993651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.856580+0000","last_change":"2024-09-13T22:39:00.541876+0000","last_active":"2024-09-13T22:39:48.856580+0000","last_peered":"2024-09-13T22:39:48.856580+0000","last_clean":"2024-09-13T22:39:48.856580+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:39:48.856580+0000","last_undegraded":"2024-09-13T22:39:48.856580+0000","last_fullsized":"2024-09-13T22:39:48.856580+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:00.541555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.541555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:04.820100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.808930+0000","last_change":"2024-09-13T22:38:51.576528+0000","last_active":"2024-09-13T22:39:48.808930+0000","last_peered":"2024-09-13T22:39:48.808930+0000","last_clean":"2024-09-13T22:39:48.808930+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:39:48.808930+0000","last_undegraded":"2024-09-13T22:39:48.808930+0000","last_fullsized":"2024-09-13T22:39:48.808930+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:51.576344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:51.576344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:09.615053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":112,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723810+0000","last_change":"2024-09-13T22:39:46.684767+0000","last_active":"2024-09-13T22:39:48.723810+0000","last_peered":"2024-09-13T22:39:48.723810+0000","last_clean":"2024-09-13T22:39:48.723810+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:39:48.723810+0000","last_undegraded":"2024-09-13T22:39:48.723810+0000","last_fullsized":"2024-09-13T22:39:48.723810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:39:46.684514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.368244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.855911+0000","last_change":"2024-09-13T22:39:04.517758+0000","last_active":"2024-09-13T22:39:48.855911+0000","last_peered":"2024-09-13T22:39:48.855911+0000","last_clean":"2024-09-13T22:39:48.855911+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:39:48.855911+0000","last_undegraded":"2024-09-13T22:39:48.855911+0000","last_fullsized":"2024-09-13T22:39:48.855911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:04.517658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.517658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.659731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":113,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722866+0000","last_change":"2024-09-13T22:39:08.790837+0000","last_active":"2024-09-13T22:39:48.722866+0000","last_peered":"2024-09-13T22:39:48.722866+0000","last_clean":"2024-09-13T22:39:48.722866+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:39:48.722866+0000","last_undegraded":"2024-09-13T22:39:48.722866+0000","last_fullsized":"2024-09-13T22:39:48.722866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:08.790729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.790729+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.315232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.007769+0000","last_change":"2024-09-13T22:39:10.297437+0000","last_active":"2024-09-13T22:39:49.007769+0000","last_peered":"2024-09-13T22:39:49.007769+0000","last_clean":"2024-09-13T22:39:49.007769+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:39:49.007769+0000","last_undegraded":"2024-09-13T22:39:49.007769+0000","last_fullsized":"2024-09-13T22:39:49.007769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:10.297219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.297219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.668374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":99,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.724400+0000","last_change":"2024-09-13T22:38:53.711228+0000","last_active":"2024-09-13T22:39:48.724400+0000","last_peered":"2024-09-13T22:39:48.724400+0000","last_clean":"2024-09-13T22:39:48.724400+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:39:48.724400+0000","last_undegraded":"2024-09-13T22:39:48.724400+0000","last_fullsized":"2024-09-13T22:39:48.724400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:53.711126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:53.711126+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:01.034551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":100,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.351977+0000","last_change":"2024-09-13T22:39:00.654413+0000","last_active":"2024-09-13T22:39:48.351977+0000","last_peered":"2024-09-13T22:39:48.351977+0000","last_clean":"2024-09-13T22:39:48.351977+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:39:48.351977+0000","last_undegraded":"2024-09-13T22:39:48.351977+0000","last_fullsized":"2024-09-13T22:39:48.351977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'2","last_scrub_stamp":"2024-09-13T22:39:00.654142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.654142+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.020250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.856520+0000","last_change":"2024-09-13T22:39:21.523815+0000","last_active":"2024-09-13T22:39:48.856520+0000","last_peered":"2024-09-13T22:39:48.856520+0000","last_clean":"2024-09-13T22:39:48.856520+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:39:48.856520+0000","last_undegraded":"2024-09-13T22:39:48.856520+0000","last_fullsized":"2024-09-13T22:39:48.856520+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.523625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.523625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:51.184833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.348949+0000","last_change":"2024-09-13T22:39:18.998697+0000","last_active":"2024-09-13T22:39:48.348949+0000","last_peered":"2024-09-13T22:39:48.348949+0000","last_clean":"2024-09-13T22:39:48.348949+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:39:48.348949+0000","last_undegraded":"2024-09-13T22:39:48.348949+0000","last_fullsized":"2024-09-13T22:39:48.348949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.998416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.890380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"82'11","reported_seq":99,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722943+0000","last_change":"2024-09-13T22:39:10.823267+0000","last_active":"2024-09-13T22:39:48.722943+0000","last_peered":"2024-09-13T22:39:48.722943+0000","last_clean":"2024-09-13T22:39:48.722943+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:39:48.722943+0000","last_undegraded":"2024-09-13T22:39:48.722943+0000","last_fullsized":"2024-09-13T22:39:48.722943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:10.823176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.823176+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.690199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.855857+0000","last_change":"2024-09-13T22:38:56.494481+0000","last_active":"2024-09-13T22:39:48.855857+0000","last_peered":"2024-09-13T22:39:48.855857+0000","last_clean":"2024-09-13T22:39:48.855857+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:39:48.855857+0000","last_undegraded":"2024-09-13T22:39:48.855857+0000","last_fullsized":"2024-09-13T22:39:48.855857+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.494413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.494413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.935415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349493+0000","last_change":"2024-09-13T22:39:11.903829+0000","last_active":"2024-09-13T22:39:48.349493+0000","last_peered":"2024-09-13T22:39:48.349493+0000","last_clean":"2024-09-13T22:39:48.349493+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:39:48.349493+0000","last_undegraded":"2024-09-13T22:39:48.349493+0000","last_fullsized":"2024-09-13T22:39:48.349493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.903758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.903758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:21.502704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.856909+0000","last_change":"2024-09-13T22:38:59.523926+0000","last_active":"2024-09-13T22:39:48.856909+0000","last_peered":"2024-09-13T22:39:48.856909+0000","last_clean":"2024-09-13T22:39:48.856909+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:39:48.856909+0000","last_undegraded":"2024-09-13T22:39:48.856909+0000","last_fullsized":"2024-09-13T22:39:48.856909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:59.523812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.523812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:02.812639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":86,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.717997+0000","last_change":"2024-09-13T22:39:04.876157+0000","last_active":"2024-09-13T22:39:48.717997+0000","last_peered":"2024-09-13T22:39:48.717997+0000","last_clean":"2024-09-13T22:39:48.717997+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:39:48.717997+0000","last_undegraded":"2024-09-13T22:39:48.717997+0000","last_fullsized":"2024-09-13T22:39:48.717997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:39:04.875957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.875957+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.926851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.718625+0000","last_change":"2024-09-13T22:39:18.864519+0000","last_active":"2024-09-13T22:39:48.718625+0000","last_peered":"2024-09-13T22:39:48.718625+0000","last_clean":"2024-09-13T22:39:48.718625+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:39:48.718625+0000","last_undegraded":"2024-09-13T22:39:48.718625+0000","last_fullsized":"2024-09-13T22:39:48.718625+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.864331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.864331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.702214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.008946+0000","last_change":"2024-09-13T22:39:31.439089+0000","last_active":"2024-09-13T22:39:49.008946+0000","last_peered":"2024-09-13T22:39:49.008946+0000","last_clean":"2024-09-13T22:39:49.008946+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:39:49.008946+0000","last_undegraded":"2024-09-13T22:39:49.008946+0000","last_fullsized":"2024-09-13T22:39:49.008946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.438868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.438868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.289969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723406+0000","last_change":"2024-09-13T22:39:11.834762+0000","last_active":"2024-09-13T22:39:48.723406+0000","last_peered":"2024-09-13T22:39:48.723406+0000","last_clean":"2024-09-13T22:39:48.723406+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:39:48.723406+0000","last_undegraded":"2024-09-13T22:39:48.723406+0000","last_fullsized":"2024-09-13T22:39:48.723406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.834629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.834629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:37.419658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":99,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723884+0000","last_change":"2024-09-13T22:39:30.707102+0000","last_active":"2024-09-13T22:39:48.723884+0000","last_peered":"2024-09-13T22:39:48.723884+0000","last_clean":"2024-09-13T22:39:48.723884+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:39:48.723884+0000","last_undegraded":"2024-09-13T22:39:48.723884+0000","last_fullsized":"2024-09-13T22:39:48.723884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:30.706887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:30.706887+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.189746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.006898+0000","last_change":"2024-09-13T22:39:15.307350+0000","last_active":"2024-09-13T22:39:49.006898+0000","last_peered":"2024-09-13T22:39:49.006898+0000","last_clean":"2024-09-13T22:39:49.006898+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:39:49.006898+0000","last_undegraded":"2024-09-13T22:39:49.006898+0000","last_fullsized":"2024-09-13T22:39:49.006898+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.307195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.307195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.086503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":116,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.008976+0000","last_change":"2024-09-13T22:38:59.477201+0000","last_active":"2024-09-13T22:39:49.008976+0000","last_peered":"2024-09-13T22:39:49.008976+0000","last_clean":"2024-09-13T22:39:49.008976+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:39:49.008976+0000","last_undegraded":"2024-09-13T22:39:49.008976+0000","last_fullsized":"2024-09-13T22:39:49.008976+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:59.477008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.477008+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:59.917494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810207+0000","last_change":"2024-09-13T22:39:18.688333+0000","last_active":"2024-09-13T22:39:48.810207+0000","last_peered":"2024-09-13T22:39:48.810207+0000","last_clean":"2024-09-13T22:39:48.810207+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:39:48.810207+0000","last_undegraded":"2024-09-13T22:39:48.810207+0000","last_fullsized":"2024-09-13T22:39:48.810207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.688141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.688141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.166952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.809101+0000","last_change":"2024-09-13T22:38:54.600790+0000","last_active":"2024-09-13T22:39:48.809101+0000","last_peered":"2024-09-13T22:39:48.809101+0000","last_clean":"2024-09-13T22:39:48.809101+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:39:48.809101+0000","last_undegraded":"2024-09-13T22:39:48.809101+0000","last_fullsized":"2024-09-13T22:39:48.809101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:54.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.600571+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:11.578225+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.808106+0000","last_change":"2024-09-13T22:39:21.654486+0000","last_active":"2024-09-13T22:39:48.808106+0000","last_peered":"2024-09-13T22:39:48.808106+0000","last_clean":"2024-09-13T22:39:48.808106+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:39:48.808106+0000","last_undegraded":"2024-09-13T22:39:48.808106+0000","last_fullsized":"2024-09-13T22:39:48.808106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.654273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.654273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.171794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":113,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.348909+0000","last_change":"2024-09-13T22:38:59.864543+0000","last_active":"2024-09-13T22:39:48.348909+0000","last_peered":"2024-09-13T22:39:48.348909+0000","last_clean":"2024-09-13T22:39:48.348909+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:39:48.348909+0000","last_undegraded":"2024-09-13T22:39:48.348909+0000","last_fullsized":"2024-09-13T22:39:48.348909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:59.864340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.864340+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:03.765599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.351354+0000","last_change":"2024-09-13T22:39:17.629172+0000","last_active":"2024-09-13T22:39:48.351354+0000","last_peered":"2024-09-13T22:39:48.351354+0000","last_clean":"2024-09-13T22:39:48.351354+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:39:48.351354+0000","last_undegraded":"2024-09-13T22:39:48.351354+0000","last_fullsized":"2024-09-13T22:39:48.351354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:17.628975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:17.628975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:18.565445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.723809+0000","last_change":"2024-09-13T22:39:25.745828+0000","last_active":"2024-09-13T22:39:48.723809+0000","last_peered":"2024-09-13T22:39:48.723809+0000","last_clean":"2024-09-13T22:39:48.723809+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:39:48.723809+0000","last_undegraded":"2024-09-13T22:39:48.723809+0000","last_fullsized":"2024-09-13T22:39:48.723809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:25.745613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.745613+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:49.997201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.719024+0000","last_change":"2024-09-13T22:39:15.852762+0000","last_active":"2024-09-13T22:39:48.719024+0000","last_peered":"2024-09-13T22:39:48.719024+0000","last_clean":"2024-09-13T22:39:48.719024+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:39:48.719024+0000","last_undegraded":"2024-09-13T22:39:48.719024+0000","last_fullsized":"2024-09-13T22:39:48.719024+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.852596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.852596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.570592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349069+0000","last_change":"2024-09-13T22:39:29.136457+0000","last_active":"2024-09-13T22:39:48.349069+0000","last_peered":"2024-09-13T22:39:48.349069+0000","last_clean":"2024-09-13T22:39:48.349069+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:39:48.349069+0000","last_undegraded":"2024-09-13T22:39:48.349069+0000","last_fullsized":"2024-09-13T22:39:48.349069+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.136366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.072781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.808441+0000","last_change":"2024-09-13T22:39:27.605785+0000","last_active":"2024-09-13T22:39:48.808441+0000","last_peered":"2024-09-13T22:39:48.808441+0000","last_clean":"2024-09-13T22:39:48.808441+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:39:48.808441+0000","last_undegraded":"2024-09-13T22:39:48.808441+0000","last_fullsized":"2024-09-13T22:39:48.808441+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.605612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.605612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.057619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":103,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.728638+0000","last_change":"2024-09-13T22:39:48.728707+0000","last_active":"2024-09-13T22:39:48.728638+0000","last_peered":"2024-09-13T22:39:48.728638+0000","last_clean":"2024-09-13T22:39:48.728638+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:39:48.728638+0000","last_undegraded":"2024-09-13T22:39:48.728638+0000","last_fullsized":"2024-09-13T22:39:48.728638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:48.728497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:48.728497+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.833414+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731639+0000","last_change":"2024-09-13T22:39:13.008954+0000","last_active":"2024-09-13T22:39:48.731639+0000","last_peered":"2024-09-13T22:39:48.731639+0000","last_clean":"2024-09-13T22:39:48.731639+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:39:48.731639+0000","last_undegraded":"2024-09-13T22:39:48.731639+0000","last_fullsized":"2024-09-13T22:39:48.731639+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:13.008764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.008764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.397180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.730595+0000","last_change":"2024-09-13T22:38:57.018120+0000","last_active":"2024-09-13T22:39:48.730595+0000","last_peered":"2024-09-13T22:39:48.730595+0000","last_clean":"2024-09-13T22:39:48.730595+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:39:48.730595+0000","last_undegraded":"2024-09-13T22:39:48.730595+0000","last_fullsized":"2024-09-13T22:39:48.730595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.017923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.017923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:15.532446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.348182+0000","last_change":"2024-09-13T22:38:52.760354+0000","last_active":"2024-09-13T22:39:48.348182+0000","last_peered":"2024-09-13T22:39:48.348182+0000","last_clean":"2024-09-13T22:39:48.348182+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:39:48.348182+0000","last_undegraded":"2024-09-13T22:39:48.348182+0000","last_fullsized":"2024-09-13T22:39:48.348182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:52.760268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:52.760268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.301827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810413+0000","last_change":"2024-09-13T22:39:07.744502+0000","last_active":"2024-09-13T22:39:48.810413+0000","last_peered":"2024-09-13T22:39:48.810413+0000","last_clean":"2024-09-13T22:39:48.810413+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:39:48.810413+0000","last_undegraded":"2024-09-13T22:39:48.810413+0000","last_fullsized":"2024-09-13T22:39:48.810413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:39:07.744124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.744124+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.659429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.006911+0000","last_change":"2024-09-13T22:39:21.378052+0000","last_active":"2024-09-13T22:39:49.006911+0000","last_peered":"2024-09-13T22:39:49.006911+0000","last_clean":"2024-09-13T22:39:49.006911+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:39:49.006911+0000","last_undegraded":"2024-09-13T22:39:49.006911+0000","last_fullsized":"2024-09-13T22:39:49.006911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.377829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.377829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.746784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.718188+0000","last_change":"2024-09-13T22:39:05.855096+0000","last_active":"2024-09-13T22:39:48.718188+0000","last_peered":"2024-09-13T22:39:48.718188+0000","last_clean":"2024-09-13T22:39:48.718188+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:39:48.718188+0000","last_undegraded":"2024-09-13T22:39:48.718188+0000","last_fullsized":"2024-09-13T22:39:48.718188+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.854893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.334889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722551+0000","last_change":"2024-09-13T22:37:46.916035+0000","last_active":"2024-09-13T22:39:48.722551+0000","last_peered":"2024-09-13T22:39:48.722551+0000","last_clean":"2024-09-13T22:39:48.722551+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:39:48.722551+0000","last_undegraded":"2024-09-13T22:39:48.722551+0000","last_fullsized":"2024-09-13T22:39:48.722551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:37:45.861252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.009108+0000","last_change":"2024-09-13T22:39:13.339189+0000","last_active":"2024-09-13T22:39:49.009108+0000","last_peered":"2024-09-13T22:39:49.009108+0000","last_clean":"2024-09-13T22:39:49.009108+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:39:49.009108+0000","last_undegraded":"2024-09-13T22:39:49.009108+0000","last_fullsized":"2024-09-13T22:39:49.009108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:13.338970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.338970+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.771039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.719495+0000","last_change":"2024-09-13T22:39:07.882650+0000","last_active":"2024-09-13T22:39:48.719495+0000","last_peered":"2024-09-13T22:39:48.719495+0000","last_clean":"2024-09-13T22:39:48.719495+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:39:48.719495+0000","last_undegraded":"2024-09-13T22:39:48.719495+0000","last_fullsized":"2024-09-13T22:39:48.719495+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:07.882460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.882460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.349629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722129+0000","last_change":"2024-09-13T22:39:29.666045+0000","last_active":"2024-09-13T22:39:48.722129+0000","last_peered":"2024-09-13T22:39:48.722129+0000","last_clean":"2024-09-13T22:39:48.722129+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:39:48.722129+0000","last_undegraded":"2024-09-13T22:39:48.722129+0000","last_fullsized":"2024-09-13T22:39:48.722129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.665857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.665857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.766151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349083+0000","last_change":"2024-09-13T22:38:49.749377+0000","last_active":"2024-09-13T22:39:48.349083+0000","last_peered":"2024-09-13T22:39:48.349083+0000","last_clean":"2024-09-13T22:39:48.349083+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:39:48.349083+0000","last_undegraded":"2024-09-13T22:39:48.349083+0000","last_fullsized":"2024-09-13T22:39:48.349083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:49.749300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.749300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:14.449565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":74,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.722928+0000","last_change":"2024-09-13T22:39:02.697718+0000","last_active":"2024-09-13T22:39:48.722928+0000","last_peered":"2024-09-13T22:39:48.722928+0000","last_clean":"2024-09-13T22:39:48.722928+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:39:48.722928+0000","last_undegraded":"2024-09-13T22:39:48.722928+0000","last_fullsized":"2024-09-13T22:39:48.722928+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:39:02.697574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.697574+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.049416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"82'15","reported_seq":106,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.730561+0000","last_change":"2024-09-13T22:39:16.963966+0000","last_active":"2024-09-13T22:39:48.730561+0000","last_peered":"2024-09-13T22:39:48.730561+0000","last_clean":"2024-09-13T22:39:48.730561+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:39:48.730561+0000","last_undegraded":"2024-09-13T22:39:48.730561+0000","last_fullsized":"2024-09-13T22:39:48.730561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:39:16.963643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.963643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.423517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:49.006520+0000","last_change":"2024-09-13T22:39:19.315930+0000","last_active":"2024-09-13T22:39:49.006520+0000","last_peered":"2024-09-13T22:39:49.006520+0000","last_clean":"2024-09-13T22:39:49.006520+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:39:49.006520+0000","last_undegraded":"2024-09-13T22:39:49.006520+0000","last_fullsized":"2024-09-13T22:39:49.006520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.315707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.315707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.650430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.719342+0000","last_change":"2024-09-13T22:39:19.870202+0000","last_active":"2024-09-13T22:39:48.719342+0000","last_peered":"2024-09-13T22:39:48.719342+0000","last_clean":"2024-09-13T22:39:48.719342+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:39:48.719342+0000","last_undegraded":"2024-09-13T22:39:48.719342+0000","last_fullsized":"2024-09-13T22:39:48.719342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.869975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.869975+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.843605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":73,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.730883+0000","last_change":"2024-09-13T22:39:18.975664+0000","last_active":"2024-09-13T22:39:48.730883+0000","last_peered":"2024-09-13T22:39:48.730883+0000","last_clean":"2024-09-13T22:39:48.730883+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:39:48.730883+0000","last_undegraded":"2024-09-13T22:39:48.730883+0000","last_fullsized":"2024-09-13T22:39:48.730883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.975461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.975461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.889969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.348255+0000","last_change":"2024-09-13T22:39:28.141412+0000","last_active":"2024-09-13T22:39:48.348255+0000","last_peered":"2024-09-13T22:39:48.348255+0000","last_clean":"2024-09-13T22:39:48.348255+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:39:48.348255+0000","last_undegraded":"2024-09-13T22:39:48.348255+0000","last_fullsized":"2024-09-13T22:39:48.348255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:28.141327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:28.141327+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.373914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.348598+0000","last_change":"2024-09-13T22:39:09.960753+0000","last_active":"2024-09-13T22:39:48.348598+0000","last_peered":"2024-09-13T22:39:48.348598+0000","last_clean":"2024-09-13T22:39:48.348598+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:39:48.348598+0000","last_undegraded":"2024-09-13T22:39:48.348598+0000","last_fullsized":"2024-09-13T22:39:48.348598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:09.960550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.960550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:38.827394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810065+0000","last_change":"2024-09-13T22:39:03.677235+0000","last_active":"2024-09-13T22:39:48.810065+0000","last_peered":"2024-09-13T22:39:48.810065+0000","last_clean":"2024-09-13T22:39:48.810065+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:39:48.810065+0000","last_undegraded":"2024-09-13T22:39:48.810065+0000","last_fullsized":"2024-09-13T22:39:48.810065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.677120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.433725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":73,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350972+0000","last_change":"2024-09-13T22:39:02.572852+0000","last_active":"2024-09-13T22:39:48.350972+0000","last_peered":"2024-09-13T22:39:48.350972+0000","last_clean":"2024-09-13T22:39:48.350972+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:39:48.350972+0000","last_undegraded":"2024-09-13T22:39:48.350972+0000","last_fullsized":"2024-09-13T22:39:48.350972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.572698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.572698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.965367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.718738+0000","last_change":"2024-09-13T22:38:56.897025+0000","last_active":"2024-09-13T22:39:48.718738+0000","last_peered":"2024-09-13T22:39:48.718738+0000","last_clean":"2024-09-13T22:39:48.718738+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:39:48.718738+0000","last_undegraded":"2024-09-13T22:39:48.718738+0000","last_fullsized":"2024-09-13T22:39:48.718738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.896829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.896829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:39:58.985623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":92,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.810562+0000","last_change":"2024-09-13T22:38:58.572985+0000","last_active":"2024-09-13T22:39:48.810562+0000","last_peered":"2024-09-13T22:39:48.810562+0000","last_clean":"2024-09-13T22:39:48.810562+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:39:48.810562+0000","last_undegraded":"2024-09-13T22:39:48.810562+0000","last_fullsized":"2024-09-13T22:39:48.810562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.572643+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:00.431556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":77,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.731469+0000","last_change":"2024-09-13T22:39:23.072821+0000","last_active":"2024-09-13T22:39:48.731469+0000","last_peered":"2024-09-13T22:39:48.731469+0000","last_clean":"2024-09-13T22:39:48.731469+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:39:48.731469+0000","last_undegraded":"2024-09-13T22:39:48.731469+0000","last_fullsized":"2024-09-13T22:39:48.731469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:39:23.072590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.247551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":74,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349512+0000","last_change":"2024-09-13T22:38:57.829670+0000","last_active":"2024-09-13T22:39:48.349512+0000","last_peered":"2024-09-13T22:39:48.349512+0000","last_clean":"2024-09-13T22:39:48.349512+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:39:48.349512+0000","last_undegraded":"2024-09-13T22:39:48.349512+0000","last_fullsized":"2024-09-13T22:39:48.349512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:38:57.829468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.829468+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.502396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.349769+0000","last_change":"2024-09-13T22:39:03.881680+0000","last_active":"2024-09-13T22:39:48.349769+0000","last_peered":"2024-09-13T22:39:48.349769+0000","last_clean":"2024-09-13T22:39:48.349769+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:39:48.349769+0000","last_undegraded":"2024-09-13T22:39:48.349769+0000","last_fullsized":"2024-09-13T22:39:48.349769+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.881493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.881493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.403295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.351382+0000","last_change":"2024-09-13T22:39:19.638335+0000","last_active":"2024-09-13T22:39:48.351382+0000","last_peered":"2024-09-13T22:39:48.351382+0000","last_clean":"2024-09-13T22:39:48.351382+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:39:48.351382+0000","last_undegraded":"2024-09-13T22:39:48.351382+0000","last_fullsized":"2024-09-13T22:39:48.351382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:19.638128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.638128+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.918763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T22:39:48.350371+0000","last_change":"2024-09-13T22:38:57.663485+0000","last_active":"2024-09-13T22:39:48.350371+0000","last_peered":"2024-09-13T22:39:48.350371+0000","last_clean":"2024-09-13T22:39:48.350371+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:39:48.350371+0000","last_undegraded":"2024-09-13T22:39:48.350371+0000","last_fullsized":"2024-09-13T22:39:48.350371+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.663262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.663262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:07.199420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":109,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808867,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712704,"statfs":{"total":95995035648,"available":95961808896,"internally_reserved":0,"allocated":1294336,"data_stored":908919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135918,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32424,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712728,"statfs":{"total":95995035648,"available":95961833472,"internally_reserved":0,"allocated":1265664,"data_stored":906597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528377,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31848,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":675840,"data_stored":316631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888131,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":708608,"data_stored":317161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084365,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":675840,"data_stored":317052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378713,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":675840,"data_stored":317145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27700,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":688128,"data_stored":317540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1286144,"data_stored":908668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:39:55.173 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T22:39:55.173 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T22:39:55.173 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T22:39:55.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T22:39:55.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[29889]: cluster 2024-09-13T22:39:54.450432+0000 mgr.y (mgr.14613) 84 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:55.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[29889]: audit 2024-09-13T22:39:54.476889+0000 mgr.y (mgr.14613) 85 : audit [DBG] from='client.24674 v1:172.21.15.178:0/3176187958' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:39:55.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[29889]: cluster 2024-09-13T22:39:54.769994+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:39:55.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[29889]: cluster 2024-09-13T22:39:54.772396+0000 osd.7 (osd.7) 50 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:39:55.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[22840]: cluster 2024-09-13T22:39:54.450432+0000 mgr.y (mgr.14613) 84 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:55.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[22840]: audit 2024-09-13T22:39:54.476889+0000 mgr.y (mgr.14613) 85 : audit [DBG] from='client.24674 v1:172.21.15.178:0/3176187958' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:39:55.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[22840]: cluster 2024-09-13T22:39:54.769994+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:39:55.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:55 smithi178 bash[22840]: cluster 2024-09-13T22:39:54.772396+0000 osd.7 (osd.7) 50 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:39:55.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:55 smithi192 bash[24608]: cluster 2024-09-13T22:39:54.450432+0000 mgr.y (mgr.14613) 84 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:55.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:55 smithi192 bash[24608]: audit 2024-09-13T22:39:54.476889+0000 mgr.y (mgr.14613) 85 : audit [DBG] from='client.24674 v1:172.21.15.178:0/3176187958' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:39:55.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:55 smithi192 bash[24608]: cluster 2024-09-13T22:39:54.769994+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:39:55.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:55 smithi192 bash[24608]: cluster 2024-09-13T22:39:54.772396+0000 osd.7 (osd.7) 50 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:39:56.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:56 smithi192 bash[24608]: audit 2024-09-13T22:39:54.881161+0000 mgr.y (mgr.14613) 86 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:56.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:56 smithi178 bash[29889]: audit 2024-09-13T22:39:54.881161+0000 mgr.y (mgr.14613) 86 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:56.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:56 smithi178 bash[22840]: audit 2024-09-13T22:39:54.881161+0000 mgr.y (mgr.14613) 86 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:39:57.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:57 smithi192 bash[24608]: cluster 2024-09-13T22:39:56.451116+0000 mgr.y (mgr.14613) 87 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:57.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:57 smithi178 bash[22840]: cluster 2024-09-13T22:39:56.451116+0000 mgr.y (mgr.14613) 87 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:57.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:57 smithi178 bash[29889]: cluster 2024-09-13T22:39:56.451116+0000 mgr.y (mgr.14613) 87 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:39:59.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:39:59 smithi178 bash[22840]: cluster 2024-09-13T22:39:58.451755+0000 mgr.y (mgr.14613) 88 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s 2024-09-13T22:39:59.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:39:59 smithi178 bash[29889]: cluster 2024-09-13T22:39:58.451755+0000 mgr.y (mgr.14613) 88 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s 2024-09-13T22:39:59.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:39:59 smithi192 bash[24608]: cluster 2024-09-13T22:39:58.451755+0000 mgr.y (mgr.14613) 88 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s 2024-09-13T22:39:59.971 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:00.064 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:39:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:39:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:00 smithi192 bash[24608]: cluster 2024-09-13T22:39:59.317027+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:00 smithi192 bash[24608]: cluster 2024-09-13T22:39:59.317130+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:00 smithi192 bash[24608]: cluster 2024-09-13T22:39:59.521485+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-13T22:40:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:00 smithi192 bash[24608]: cluster 2024-09-13T22:39:59.760265+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:40:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:00 smithi192 bash[24608]: cluster 2024-09-13T22:39:59.762374+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:40:00.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:00 smithi192 bash[24608]: cluster 2024-09-13T22:40:00.000110+0000 mon.a (mon.0) 899 : cluster [INF] overall HEALTH_OK 2024-09-13T22:40:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[29889]: cluster 2024-09-13T22:39:59.317027+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[29889]: cluster 2024-09-13T22:39:59.317130+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[29889]: cluster 2024-09-13T22:39:59.521485+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-13T22:40:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[29889]: cluster 2024-09-13T22:39:59.760265+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:40:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[29889]: cluster 2024-09-13T22:39:59.762374+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:40:00.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[29889]: cluster 2024-09-13T22:40:00.000110+0000 mon.a (mon.0) 899 : cluster [INF] overall HEALTH_OK 2024-09-13T22:40:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[22840]: cluster 2024-09-13T22:39:59.317027+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[22840]: cluster 2024-09-13T22:39:59.317130+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[22840]: cluster 2024-09-13T22:39:59.521485+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-13T22:40:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[22840]: cluster 2024-09-13T22:39:59.760265+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:40:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[22840]: cluster 2024-09-13T22:39:59.762374+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:40:00.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:00 smithi178 bash[22840]: cluster 2024-09-13T22:40:00.000110+0000 mon.a (mon.0) 899 : cluster [INF] overall HEALTH_OK 2024-09-13T22:40:01.664 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:40:01.665 INFO:teuthology.orchestra.run.smithi178.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T22:40:01.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[22840]: cluster 2024-09-13T22:40:00.452464+0000 mgr.y (mgr.14613) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:01.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[22840]: cluster 2024-09-13T22:40:00.584033+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:01.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[22840]: cluster 2024-09-13T22:40:00.584167+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:01.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[22840]: cluster 2024-09-13T22:40:00.795769+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:40:01.886 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[22840]: cluster 2024-09-13T22:40:00.799419+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:40:01.886 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[29889]: cluster 2024-09-13T22:40:00.452464+0000 mgr.y (mgr.14613) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:01.886 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[29889]: cluster 2024-09-13T22:40:00.584033+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:01.886 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[29889]: cluster 2024-09-13T22:40:00.584167+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:01.886 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[29889]: cluster 2024-09-13T22:40:00.795769+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:40:01.886 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:01 smithi178 bash[29889]: cluster 2024-09-13T22:40:00.799419+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:40:01.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:01 smithi192 bash[24608]: cluster 2024-09-13T22:40:00.452464+0000 mgr.y (mgr.14613) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:01 smithi192 bash[24608]: cluster 2024-09-13T22:40:00.584033+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:01 smithi192 bash[24608]: cluster 2024-09-13T22:40:00.584167+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:01 smithi192 bash[24608]: cluster 2024-09-13T22:40:00.795769+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:40:01.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:01 smithi192 bash[24608]: cluster 2024-09-13T22:40:00.799419+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:40:02.346 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T22:40:02.347 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T22:40:02.347 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T22:40:02.358 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T22:40:02.358 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}}, 'workunit': {'branch': 'squid', 'sha1': 'ecab7be58963a5bd03910cd73eef66405d7fdd50'}} 2024-09-13T22:40:02.358 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T22:40:02.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T22:40:02.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:02 smithi178 bash[29889]: cluster 2024-09-13T22:40:01.531593+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-13T22:40:02.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:02 smithi178 bash[29889]: audit 2024-09-13T22:40:01.668292+0000 mon.a (mon.0) 901 : audit [DBG] from='client.? v1:172.21.15.178:0/887852833' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T22:40:02.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:02 smithi178 bash[22840]: cluster 2024-09-13T22:40:01.531593+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-13T22:40:02.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:02 smithi178 bash[22840]: audit 2024-09-13T22:40:01.668292+0000 mon.a (mon.0) 901 : audit [DBG] from='client.? v1:172.21.15.178:0/887852833' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T22:40:02.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:02 smithi192 bash[24608]: cluster 2024-09-13T22:40:01.531593+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-13T22:40:02.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:02 smithi192 bash[24608]: audit 2024-09-13T22:40:01.668292+0000 mon.a (mon.0) 901 : audit [DBG] from='client.? v1:172.21.15.178:0/887852833' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T22:40:03.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:03 smithi192 bash[24608]: cluster 2024-09-13T22:40:02.452946+0000 mgr.y (mgr.14613) 90 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:40:03.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:03 smithi192 bash[24608]: audit 2024-09-13T22:40:02.661744+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:03.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:03 smithi178 bash[22840]: cluster 2024-09-13T22:40:02.452946+0000 mgr.y (mgr.14613) 90 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:40:03.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:03 smithi178 bash[22840]: audit 2024-09-13T22:40:02.661744+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:03.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:03 smithi178 bash[29889]: cluster 2024-09-13T22:40:02.452946+0000 mgr.y (mgr.14613) 90 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:40:03.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:03 smithi178 bash[29889]: audit 2024-09-13T22:40:02.661744+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:04.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:04 smithi192 bash[24608]: cluster 2024-09-13T22:40:03.439704+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:40:04.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:04 smithi192 bash[24608]: cluster 2024-09-13T22:40:03.442912+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:40:04.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:04 smithi178 bash[29889]: cluster 2024-09-13T22:40:03.439704+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:40:04.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:04 smithi178 bash[29889]: cluster 2024-09-13T22:40:03.442912+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:40:04.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:04 smithi178 bash[22840]: cluster 2024-09-13T22:40:03.439704+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:40:04.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:04 smithi178 bash[22840]: cluster 2024-09-13T22:40:03.442912+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:40:05.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:40:04 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.233050+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.235840+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.454336+0000 mgr.y (mgr.14613) 91 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.565775+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.568109+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.826106+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:40:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:05 smithi192 bash[24608]: cluster 2024-09-13T22:40:04.828532+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.233050+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.235840+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.454336+0000 mgr.y (mgr.14613) 91 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.565775+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.568109+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.826106+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:40:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[22840]: cluster 2024-09-13T22:40:04.828532+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.233050+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.235840+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.454336+0000 mgr.y (mgr.14613) 91 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.565775+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.568109+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.826106+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:40:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:05 smithi178 bash[29889]: cluster 2024-09-13T22:40:04.828532+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:40:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:06 smithi192 bash[24608]: audit 2024-09-13T22:40:04.885502+0000 mgr.y (mgr.14613) 92 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:06 smithi192 bash[24608]: cluster 2024-09-13T22:40:05.599905+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:40:06.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:06 smithi192 bash[24608]: cluster 2024-09-13T22:40:05.603126+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:40:06.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:06 smithi178 bash[29889]: audit 2024-09-13T22:40:04.885502+0000 mgr.y (mgr.14613) 92 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:06.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:06 smithi178 bash[29889]: cluster 2024-09-13T22:40:05.599905+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:40:06.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:06 smithi178 bash[29889]: cluster 2024-09-13T22:40:05.603126+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:40:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:06 smithi178 bash[22840]: audit 2024-09-13T22:40:04.885502+0000 mgr.y (mgr.14613) 92 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:06 smithi178 bash[22840]: cluster 2024-09-13T22:40:05.599905+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:40:06.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:06 smithi178 bash[22840]: cluster 2024-09-13T22:40:05.603126+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:40:07.162 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:07 smithi192 bash[24608]: cluster 2024-09-13T22:40:06.455051+0000 mgr.y (mgr.14613) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:07.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:07 smithi178 bash[29889]: cluster 2024-09-13T22:40:06.455051+0000 mgr.y (mgr.14613) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:07.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:07 smithi178 bash[22840]: cluster 2024-09-13T22:40:06.455051+0000 mgr.y (mgr.14613) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:08.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[29889]: cluster 2024-09-13T22:40:07.463642+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:40:08.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[29889]: cluster 2024-09-13T22:40:07.465676+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:40:08.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[29889]: cluster 2024-09-13T22:40:07.585520+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:40:08.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[29889]: cluster 2024-09-13T22:40:07.587899+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:40:08.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[29889]: cluster 2024-09-13T22:40:08.455434+0000 mgr.y (mgr.14613) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-09-13T22:40:08.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[22840]: cluster 2024-09-13T22:40:07.463642+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:40:08.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[22840]: cluster 2024-09-13T22:40:07.465676+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:40:08.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[22840]: cluster 2024-09-13T22:40:07.585520+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:40:08.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[22840]: cluster 2024-09-13T22:40:07.587899+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:40:08.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[22840]: cluster 2024-09-13T22:40:08.455434+0000 mgr.y (mgr.14613) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-09-13T22:40:08.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:08 smithi192 bash[24608]: cluster 2024-09-13T22:40:07.463642+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:40:08.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:08 smithi192 bash[24608]: cluster 2024-09-13T22:40:07.465676+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:40:08.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:08 smithi192 bash[24608]: cluster 2024-09-13T22:40:07.585520+0000 osd.4 (osd.4) 45 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:40:08.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:08 smithi192 bash[24608]: cluster 2024-09-13T22:40:07.587899+0000 osd.4 (osd.4) 46 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:40:08.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:08 smithi192 bash[24608]: cluster 2024-09-13T22:40:08.455434+0000 mgr.y (mgr.14613) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-09-13T22:40:09.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[23095]: [13/Sep/2024:22:40:08] ENGINE Bus STOPPING 2024-09-13T22:40:09.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[23095]: [13/Sep/2024:22:40:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T22:40:09.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[23095]: [13/Sep/2024:22:40:08] ENGINE Bus STOPPED 2024-09-13T22:40:09.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[23095]: [13/Sep/2024:22:40:08] ENGINE Bus STARTING 2024-09-13T22:40:09.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[23095]: [13/Sep/2024:22:40:08] ENGINE Serving on http://:::9283 2024-09-13T22:40:09.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:08 smithi178 bash[23095]: [13/Sep/2024:22:40:08] ENGINE Bus STARTED 2024-09-13T22:40:09.391 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T22:40:09.392 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T22:40:09.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:40:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:09 smithi192 bash[24608]: cluster 2024-09-13T22:40:08.494482+0000 osd.5 (osd.5) 53 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:40:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:09 smithi192 bash[24608]: cluster 2024-09-13T22:40:08.496733+0000 osd.5 (osd.5) 54 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:40:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:09 smithi192 bash[24608]: audit 2024-09-13T22:40:08.700095+0000 mon.a (mon.0) 902 : audit [INF] from='client.24686 ' entity='client.admin' 2024-09-13T22:40:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:09 smithi192 bash[24608]: audit 2024-09-13T22:40:08.715608+0000 mon.b (mon.1) 85 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:40:10.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:40:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:10.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[29889]: cluster 2024-09-13T22:40:08.494482+0000 osd.5 (osd.5) 53 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:40:10.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[29889]: cluster 2024-09-13T22:40:08.496733+0000 osd.5 (osd.5) 54 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:40:10.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[29889]: audit 2024-09-13T22:40:08.700095+0000 mon.a (mon.0) 902 : audit [INF] from='client.24686 ' entity='client.admin' 2024-09-13T22:40:10.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[29889]: audit 2024-09-13T22:40:08.715608+0000 mon.b (mon.1) 85 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:40:10.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[22840]: cluster 2024-09-13T22:40:08.494482+0000 osd.5 (osd.5) 53 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:40:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[22840]: cluster 2024-09-13T22:40:08.496733+0000 osd.5 (osd.5) 54 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:40:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[22840]: audit 2024-09-13T22:40:08.700095+0000 mon.a (mon.0) 902 : audit [INF] from='client.24686 ' entity='client.admin' 2024-09-13T22:40:10.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:09 smithi178 bash[22840]: audit 2024-09-13T22:40:08.715608+0000 mon.b (mon.1) 85 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:40:11.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:10 smithi192 bash[24608]: cluster 2024-09-13T22:40:09.492498+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub starts 2024-09-13T22:40:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:10 smithi192 bash[24608]: cluster 2024-09-13T22:40:09.494838+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.f scrub ok 2024-09-13T22:40:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:10 smithi192 bash[24608]: cluster 2024-09-13T22:40:09.773564+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:40:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:10 smithi192 bash[24608]: cluster 2024-09-13T22:40:09.776760+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:40:11.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:10 smithi192 bash[24608]: cluster 2024-09-13T22:40:10.456905+0000 mgr.y (mgr.14613) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[22840]: cluster 2024-09-13T22:40:09.492498+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub starts 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[22840]: cluster 2024-09-13T22:40:09.494838+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.f scrub ok 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[22840]: cluster 2024-09-13T22:40:09.773564+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[22840]: cluster 2024-09-13T22:40:09.776760+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[22840]: cluster 2024-09-13T22:40:10.456905+0000 mgr.y (mgr.14613) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[29889]: cluster 2024-09-13T22:40:09.492498+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub starts 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[29889]: cluster 2024-09-13T22:40:09.494838+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.f scrub ok 2024-09-13T22:40:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[29889]: cluster 2024-09-13T22:40:09.773564+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:40:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[29889]: cluster 2024-09-13T22:40:09.776760+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:40:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:10 smithi178 bash[29889]: cluster 2024-09-13T22:40:10.456905+0000 mgr.y (mgr.14613) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:11 smithi192 bash[24608]: cluster 2024-09-13T22:40:10.562826+0000 osd.7 (osd.7) 55 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:40:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:11 smithi192 bash[24608]: cluster 2024-09-13T22:40:10.565238+0000 osd.7 (osd.7) 56 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:40:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:11 smithi192 bash[24608]: cluster 2024-09-13T22:40:11.073077+0000 osd.1 (osd.1) 51 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T22:40:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:11 smithi192 bash[24608]: cluster 2024-09-13T22:40:11.075290+0000 osd.1 (osd.1) 52 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[29889]: cluster 2024-09-13T22:40:10.562826+0000 osd.7 (osd.7) 55 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[29889]: cluster 2024-09-13T22:40:10.565238+0000 osd.7 (osd.7) 56 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[29889]: cluster 2024-09-13T22:40:11.073077+0000 osd.1 (osd.1) 51 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[29889]: cluster 2024-09-13T22:40:11.075290+0000 osd.1 (osd.1) 52 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[22840]: cluster 2024-09-13T22:40:10.562826+0000 osd.7 (osd.7) 55 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[22840]: cluster 2024-09-13T22:40:10.565238+0000 osd.7 (osd.7) 56 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[22840]: cluster 2024-09-13T22:40:11.073077+0000 osd.1 (osd.1) 51 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T22:40:12.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:11 smithi178 bash[22840]: cluster 2024-09-13T22:40:11.075290+0000 osd.1 (osd.1) 52 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T22:40:13.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:12 smithi192 bash[24608]: cluster 2024-09-13T22:40:11.877491+0000 osd.2 (osd.2) 39 : cluster [DBG] 5.a scrub starts 2024-09-13T22:40:13.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:12 smithi192 bash[24608]: cluster 2024-09-13T22:40:11.879634+0000 osd.2 (osd.2) 40 : cluster [DBG] 5.a scrub ok 2024-09-13T22:40:13.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:12 smithi192 bash[24608]: cluster 2024-09-13T22:40:12.457621+0000 mgr.y (mgr.14613) 96 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-13T22:40:13.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:12 smithi178 bash[29889]: cluster 2024-09-13T22:40:11.877491+0000 osd.2 (osd.2) 39 : cluster [DBG] 5.a scrub starts 2024-09-13T22:40:13.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:12 smithi178 bash[29889]: cluster 2024-09-13T22:40:11.879634+0000 osd.2 (osd.2) 40 : cluster [DBG] 5.a scrub ok 2024-09-13T22:40:13.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:12 smithi178 bash[29889]: cluster 2024-09-13T22:40:12.457621+0000 mgr.y (mgr.14613) 96 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-13T22:40:13.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:12 smithi178 bash[22840]: cluster 2024-09-13T22:40:11.877491+0000 osd.2 (osd.2) 39 : cluster [DBG] 5.a scrub starts 2024-09-13T22:40:13.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:12 smithi178 bash[22840]: cluster 2024-09-13T22:40:11.879634+0000 osd.2 (osd.2) 40 : cluster [DBG] 5.a scrub ok 2024-09-13T22:40:13.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:12 smithi178 bash[22840]: cluster 2024-09-13T22:40:12.457621+0000 mgr.y (mgr.14613) 96 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-13T22:40:13.789 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:14.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:12.622186+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.1c deep-scrub starts 2024-09-13T22:40:14.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:12.625109+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.1c deep-scrub ok 2024-09-13T22:40:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:12.710567+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:40:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:12.713304+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:40:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:13.055356+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.d scrub starts 2024-09-13T22:40:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:13.057599+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.d scrub ok 2024-09-13T22:40:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:13.601200+0000 osd.5 (osd.5) 57 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:40:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:13 smithi192 bash[24608]: cluster 2024-09-13T22:40:13.605478+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:40:14.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:12.622186+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.1c deep-scrub starts 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:12.625109+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.1c deep-scrub ok 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:12.710567+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:12.713304+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:13.055356+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.d scrub starts 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:13.057599+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.d scrub ok 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:13.601200+0000 osd.5 (osd.5) 57 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[22840]: cluster 2024-09-13T22:40:13.605478+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:12.622186+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.1c deep-scrub starts 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:12.625109+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.1c deep-scrub ok 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:12.710567+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:12.713304+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:40:14.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:13.055356+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.d scrub starts 2024-09-13T22:40:14.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:13.057599+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.d scrub ok 2024-09-13T22:40:14.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:13.601200+0000 osd.5 (osd.5) 57 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:40:14.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:13 smithi178 bash[29889]: cluster 2024-09-13T22:40:13.605478+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:40:15.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:40:14 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:40:15.216 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:40:15.218 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.321750+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.326853+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: cluster 2024-09-13T22:40:14.459105+0000 mgr.y (mgr.14613) 97 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: cluster 2024-09-13T22:40:14.561095+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.e scrub starts 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: cluster 2024-09-13T22:40:14.564279+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.e scrub ok 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.704566+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.715950+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.719508+0000 mon.b (mon.1) 86 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.721362+0000 mon.b (mon.1) 87 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: audit 2024-09-13T22:40:14.731459+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: cluster 2024-09-13T22:40:14.843938+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T22:40:15.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:15 smithi192 bash[24608]: cluster 2024-09-13T22:40:14.845498+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T22:40:15.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.321750+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.326853+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: cluster 2024-09-13T22:40:14.459105+0000 mgr.y (mgr.14613) 97 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: cluster 2024-09-13T22:40:14.561095+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.e scrub starts 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: cluster 2024-09-13T22:40:14.564279+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.e scrub ok 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.704566+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.715950+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.719508+0000 mon.b (mon.1) 86 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.721362+0000 mon.b (mon.1) 87 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: audit 2024-09-13T22:40:14.731459+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: cluster 2024-09-13T22:40:14.843938+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[29889]: cluster 2024-09-13T22:40:14.845498+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T22:40:15.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.321750+0000 mon.a (mon.0) 903 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.326853+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: cluster 2024-09-13T22:40:14.459105+0000 mgr.y (mgr.14613) 97 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: cluster 2024-09-13T22:40:14.561095+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.e scrub starts 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: cluster 2024-09-13T22:40:14.564279+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.e scrub ok 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.704566+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.715950+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.719508+0000 mon.b (mon.1) 86 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.721362+0000 mon.b (mon.1) 87 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:40:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: audit 2024-09-13T22:40:14.731459+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:40:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: cluster 2024-09-13T22:40:14.843938+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T22:40:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:15 smithi178 bash[22840]: cluster 2024-09-13T22:40:14.845498+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T22:40:15.947 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-13T22:40:14.457899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1084,"num_read_kb":930,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":670,"ondisk_log_size":670,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244284,"kb_used_data":7292,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716932,"statfs":{"total":767960285184,"available":767710138368,"internally_reserved":0,"allocated":7467008,"data_stored":4412929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005169"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101153+0000","last_change":"2024-09-13T22:39:52.233876+0000","last_active":"2024-09-13T22:40:03.101153+0000","last_peered":"2024-09-13T22:40:03.101153+0000","last_clean":"2024-09-13T22:40:03.101153+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:40:03.101153+0000","last_undegraded":"2024-09-13T22:40:03.101153+0000","last_fullsized":"2024-09-13T22:40:03.101153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:52.233682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:52.233682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:15.212128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.610103+0000","last_change":"2024-09-13T22:38:46.613749+0000","last_active":"2024-09-13T22:40:01.610103+0000","last_peered":"2024-09-13T22:40:01.610103+0000","last_clean":"2024-09-13T22:40:01.610103+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:40:01.610103+0000","last_undegraded":"2024-09-13T22:40:01.610103+0000","last_fullsized":"2024-09-13T22:40:01.610103+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.613466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:15.391727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103792+0000","last_change":"2024-09-13T22:38:54.776466+0000","last_active":"2024-09-13T22:40:03.103792+0000","last_peered":"2024-09-13T22:40:03.103792+0000","last_clean":"2024-09-13T22:40:03.103792+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:40:03.103792+0000","last_undegraded":"2024-09-13T22:40:03.103792+0000","last_fullsized":"2024-09-13T22:40:03.103792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.776238+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.515503+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100804+0000","last_change":"2024-09-13T22:39:20.600784+0000","last_active":"2024-09-13T22:40:03.100804+0000","last_peered":"2024-09-13T22:40:03.100804+0000","last_clean":"2024-09-13T22:40:03.100804+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:40:03.100804+0000","last_undegraded":"2024-09-13T22:40:03.100804+0000","last_fullsized":"2024-09-13T22:40:03.100804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.600571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.907326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104067+0000","last_change":"2024-09-13T22:39:27.118459+0000","last_active":"2024-09-13T22:40:03.104067+0000","last_peered":"2024-09-13T22:40:03.104067+0000","last_clean":"2024-09-13T22:40:03.104067+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:40:03.104067+0000","last_undegraded":"2024-09-13T22:40:03.104067+0000","last_fullsized":"2024-09-13T22:40:03.104067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.118162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.118162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.139079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609923+0000","last_change":"2024-09-13T22:39:08.754354+0000","last_active":"2024-09-13T22:40:01.609923+0000","last_peered":"2024-09-13T22:40:01.609923+0000","last_clean":"2024-09-13T22:40:01.609923+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:40:01.609923+0000","last_undegraded":"2024-09-13T22:40:01.609923+0000","last_fullsized":"2024-09-13T22:40:01.609923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.754231+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:14.661781+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100433+0000","last_change":"2024-09-13T22:39:24.033368+0000","last_active":"2024-09-13T22:40:03.100433+0000","last_peered":"2024-09-13T22:40:03.100433+0000","last_clean":"2024-09-13T22:40:03.100433+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:40:03.100433+0000","last_undegraded":"2024-09-13T22:40:03.100433+0000","last_fullsized":"2024-09-13T22:40:03.100433+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.033224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.033224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.108639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100108+0000","last_change":"2024-09-13T22:38:59.657956+0000","last_active":"2024-09-13T22:40:03.100108+0000","last_peered":"2024-09-13T22:40:03.100108+0000","last_clean":"2024-09-13T22:40:03.100108+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:40:03.100108+0000","last_undegraded":"2024-09-13T22:40:03.100108+0000","last_fullsized":"2024-09-13T22:40:03.100108+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:59.657646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.657646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.987419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106448+0000","last_change":"2024-09-13T22:39:19.778773+0000","last_active":"2024-09-13T22:40:03.106448+0000","last_peered":"2024-09-13T22:40:03.106448+0000","last_clean":"2024-09-13T22:40:03.106448+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:40:03.106448+0000","last_undegraded":"2024-09-13T22:40:03.106448+0000","last_fullsized":"2024-09-13T22:40:03.106448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.778590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.778590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.572565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":117,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101880+0000","last_change":"2024-09-13T22:39:26.318996+0000","last_active":"2024-09-13T22:40:03.101880+0000","last_peered":"2024-09-13T22:40:03.101880+0000","last_clean":"2024-09-13T22:40:03.101880+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:40:03.101880+0000","last_undegraded":"2024-09-13T22:40:03.101880+0000","last_fullsized":"2024-09-13T22:40:03.101880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.318753+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.307721+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104412+0000","last_change":"2024-09-13T22:39:48.736178+0000","last_active":"2024-09-13T22:40:03.104412+0000","last_peered":"2024-09-13T22:40:03.104412+0000","last_clean":"2024-09-13T22:40:03.104412+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:40:03.104412+0000","last_undegraded":"2024-09-13T22:40:03.104412+0000","last_fullsized":"2024-09-13T22:40:03.104412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:39:48.735986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:14.424901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100233+0000","last_change":"2024-09-13T22:39:37.426328+0000","last_active":"2024-09-13T22:40:03.100233+0000","last_peered":"2024-09-13T22:40:03.100233+0000","last_clean":"2024-09-13T22:40:03.100233+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:40:03.100233+0000","last_undegraded":"2024-09-13T22:40:03.100233+0000","last_fullsized":"2024-09-13T22:40:03.100233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:37.426096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:37.426096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.076012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:10.565444+0000","last_change":"2024-09-13T22:40:10.565444+0000","last_active":"2024-09-13T22:40:10.565444+0000","last_peered":"2024-09-13T22:40:10.565444+0000","last_clean":"2024-09-13T22:40:10.565444+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:40:10.565444+0000","last_undegraded":"2024-09-13T22:40:10.565444+0000","last_fullsized":"2024-09-13T22:40:10.565444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:10.565251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:10.565251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:37.474394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":108,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106783+0000","last_change":"2024-09-13T22:38:49.543240+0000","last_active":"2024-09-13T22:40:03.106783+0000","last_peered":"2024-09-13T22:40:03.106783+0000","last_clean":"2024-09-13T22:40:03.106783+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:40:03.106783+0000","last_undegraded":"2024-09-13T22:40:03.106783+0000","last_fullsized":"2024-09-13T22:40:03.106783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.543010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.543010+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102478+0000","last_change":"2024-09-13T22:38:50.497874+0000","last_active":"2024-09-13T22:40:03.102478+0000","last_peered":"2024-09-13T22:40:03.102478+0000","last_clean":"2024-09-13T22:40:03.102478+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:40:03.102478+0000","last_undegraded":"2024-09-13T22:40:03.102478+0000","last_fullsized":"2024-09-13T22:40:03.102478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:50.497567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:50.497567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.010369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.608673+0000","last_change":"2024-09-13T22:39:14.752874+0000","last_active":"2024-09-13T22:40:01.608673+0000","last_peered":"2024-09-13T22:40:01.608673+0000","last_clean":"2024-09-13T22:40:01.608673+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:40:01.608673+0000","last_undegraded":"2024-09-13T22:40:01.608673+0000","last_fullsized":"2024-09-13T22:40:01.608673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.752683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.752683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.453233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"82'19","reported_seq":116,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529488+0000","last_change":"2024-09-13T22:39:21.873986+0000","last_active":"2024-09-13T22:40:01.529488+0000","last_peered":"2024-09-13T22:40:01.529488+0000","last_clean":"2024-09-13T22:40:01.529488+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:40:01.529488+0000","last_undegraded":"2024-09-13T22:40:01.529488+0000","last_fullsized":"2024-09-13T22:40:01.529488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:21.873747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.873747+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.726394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105483+0000","last_change":"2024-09-13T22:39:35.662591+0000","last_active":"2024-09-13T22:40:03.105483+0000","last_peered":"2024-09-13T22:40:03.105483+0000","last_clean":"2024-09-13T22:40:03.105483+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:40:03.105483+0000","last_undegraded":"2024-09-13T22:40:03.105483+0000","last_fullsized":"2024-09-13T22:40:03.105483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:35.662396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:35.662396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:48.563351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102577+0000","last_change":"2024-09-13T22:39:15.972091+0000","last_active":"2024-09-13T22:40:03.102577+0000","last_peered":"2024-09-13T22:40:03.102577+0000","last_clean":"2024-09-13T22:40:03.102577+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:40:03.102577+0000","last_undegraded":"2024-09-13T22:40:03.102577+0000","last_fullsized":"2024-09-13T22:40:03.102577+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.971904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.971904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:40.478614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529132+0000","last_change":"2024-09-13T22:39:14.436930+0000","last_active":"2024-09-13T22:40:01.529132+0000","last_peered":"2024-09-13T22:40:01.529132+0000","last_clean":"2024-09-13T22:40:01.529132+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:40:01.529132+0000","last_undegraded":"2024-09-13T22:40:01.529132+0000","last_fullsized":"2024-09-13T22:40:01.529132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.436746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.436746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.039947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'9","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103228+0000","last_change":"2024-09-13T22:38:43.752187+0000","last_active":"2024-09-13T22:40:03.103228+0000","last_peered":"2024-09-13T22:40:03.103228+0000","last_clean":"2024-09-13T22:40:03.103228+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:40:03.103228+0000","last_undegraded":"2024-09-13T22:40:03.103228+0000","last_fullsized":"2024-09-13T22:40:03.103228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:43.751981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:43.751981+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.845059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101728+0000","last_change":"2024-09-13T22:39:01.945920+0000","last_active":"2024-09-13T22:40:03.101728+0000","last_peered":"2024-09-13T22:40:03.101728+0000","last_clean":"2024-09-13T22:40:03.101728+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:40:03.101728+0000","last_undegraded":"2024-09-13T22:40:03.101728+0000","last_fullsized":"2024-09-13T22:40:03.101728+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:01.945733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.945733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.448893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.916096+0000","last_change":"2024-09-13T22:39:19.489746+0000","last_active":"2024-09-13T22:40:05.916096+0000","last_peered":"2024-09-13T22:40:05.916096+0000","last_clean":"2024-09-13T22:40:05.916096+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:40:05.916096+0000","last_undegraded":"2024-09-13T22:40:05.916096+0000","last_fullsized":"2024-09-13T22:40:05.916096+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.489549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.489549+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:47.360174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.099630+0000","last_change":"2024-09-13T22:39:51.642269+0000","last_active":"2024-09-13T22:40:03.099630+0000","last_peered":"2024-09-13T22:40:03.099630+0000","last_clean":"2024-09-13T22:40:03.099630+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:40:03.099630+0000","last_undegraded":"2024-09-13T22:40:03.099630+0000","last_fullsized":"2024-09-13T22:40:03.099630+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:51.642066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:51.642066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:03.230502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104612+0000","last_change":"2024-09-13T22:38:58.835483+0000","last_active":"2024-09-13T22:40:03.104612+0000","last_peered":"2024-09-13T22:40:03.104612+0000","last_clean":"2024-09-13T22:40:03.104612+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:40:03.104612+0000","last_undegraded":"2024-09-13T22:40:03.104612+0000","last_fullsized":"2024-09-13T22:40:03.104612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:38:58.835275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.835275+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.927373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101043+0000","last_change":"2024-09-13T22:38:46.006268+0000","last_active":"2024-09-13T22:40:03.101043+0000","last_peered":"2024-09-13T22:40:03.101043+0000","last_clean":"2024-09-13T22:40:03.101043+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:40:03.101043+0000","last_undegraded":"2024-09-13T22:40:03.101043+0000","last_fullsized":"2024-09-13T22:40:03.101043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.006072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.006072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:13.250082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":123,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.916370+0000","last_change":"2024-09-13T22:39:12.592344+0000","last_active":"2024-09-13T22:40:05.916370+0000","last_peered":"2024-09-13T22:40:05.916370+0000","last_clean":"2024-09-13T22:40:05.916370+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:40:05.916370+0000","last_undegraded":"2024-09-13T22:40:05.916370+0000","last_fullsized":"2024-09-13T22:40:05.916370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.592134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.802063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105814+0000","last_change":"2024-09-13T22:39:33.392634+0000","last_active":"2024-09-13T22:40:03.105814+0000","last_peered":"2024-09-13T22:40:03.105814+0000","last_clean":"2024-09-13T22:40:03.105814+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:40:03.105814+0000","last_undegraded":"2024-09-13T22:40:03.105814+0000","last_fullsized":"2024-09-13T22:40:03.105814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_clean_scrub_stamp":"2024-09-13T22:39:33.392318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.641551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":108,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101999+0000","last_change":"2024-09-13T22:38:49.656312+0000","last_active":"2024-09-13T22:40:03.101999+0000","last_peered":"2024-09-13T22:40:03.101999+0000","last_clean":"2024-09-13T22:40:03.101999+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:40:03.101999+0000","last_undegraded":"2024-09-13T22:40:03.101999+0000","last_fullsized":"2024-09-13T22:40:03.101999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.656093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.656093+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.977202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104170+0000","last_change":"2024-09-13T22:39:40.612195+0000","last_active":"2024-09-13T22:40:03.104170+0000","last_peered":"2024-09-13T22:40:03.104170+0000","last_clean":"2024-09-13T22:40:03.104170+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:40:03.104170+0000","last_undegraded":"2024-09-13T22:40:03.104170+0000","last_fullsized":"2024-09-13T22:40:03.104170+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:40.611997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.611997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:10.478412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:09.494980+0000","last_change":"2024-09-13T22:40:09.495032+0000","last_active":"2024-09-13T22:40:09.494980+0000","last_peered":"2024-09-13T22:40:09.494980+0000","last_clean":"2024-09-13T22:40:09.494980+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:40:09.494980+0000","last_undegraded":"2024-09-13T22:40:09.494980+0000","last_fullsized":"2024-09-13T22:40:09.494980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:09.494846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:09.494846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:15.238012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103372+0000","last_change":"2024-09-13T22:39:53.214599+0000","last_active":"2024-09-13T22:40:03.103372+0000","last_peered":"2024-09-13T22:40:03.103372+0000","last_clean":"2024-09-13T22:40:03.103372+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:40:03.103372+0000","last_undegraded":"2024-09-13T22:40:03.103372+0000","last_fullsized":"2024-09-13T22:40:03.103372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:53.214316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:53.214316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:03.248657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.610170+0000","last_change":"2024-09-13T22:39:04.726321+0000","last_active":"2024-09-13T22:40:01.610170+0000","last_peered":"2024-09-13T22:40:01.610170+0000","last_clean":"2024-09-13T22:40:01.610170+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:40:01.610170+0000","last_undegraded":"2024-09-13T22:40:01.610170+0000","last_fullsized":"2024-09-13T22:40:01.610170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:39:04.726105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.726105+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.304382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.530695+0000","last_change":"2024-09-13T22:39:08.858599+0000","last_active":"2024-09-13T22:40:01.530695+0000","last_peered":"2024-09-13T22:40:01.530695+0000","last_clean":"2024-09-13T22:40:01.530695+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:40:01.530695+0000","last_undegraded":"2024-09-13T22:40:01.530695+0000","last_fullsized":"2024-09-13T22:40:01.530695+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:08.858481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.858481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.756681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101722+0000","last_change":"2024-09-13T22:39:15.627485+0000","last_active":"2024-09-13T22:40:03.101722+0000","last_peered":"2024-09-13T22:40:03.101722+0000","last_clean":"2024-09-13T22:40:03.101722+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:40:03.101722+0000","last_undegraded":"2024-09-13T22:40:03.101722+0000","last_fullsized":"2024-09-13T22:40:03.101722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:39:15.627264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.627264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.016937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.915907+0000","last_change":"2024-09-13T22:39:40.265509+0000","last_active":"2024-09-13T22:40:05.915907+0000","last_peered":"2024-09-13T22:40:05.915907+0000","last_clean":"2024-09-13T22:40:05.915907+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:40:05.915907+0000","last_undegraded":"2024-09-13T22:40:05.915907+0000","last_fullsized":"2024-09-13T22:40:05.915907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:40.265350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.265350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:00.451629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104854+0000","last_change":"2024-09-13T22:39:49.738117+0000","last_active":"2024-09-13T22:40:03.104854+0000","last_peered":"2024-09-13T22:40:03.104854+0000","last_clean":"2024-09-13T22:40:03.104854+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:40:03.104854+0000","last_undegraded":"2024-09-13T22:40:03.104854+0000","last_fullsized":"2024-09-13T22:40:03.104854+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:49.737992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:49.737992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:55.782646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":117,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104498+0000","last_change":"2024-09-13T22:39:16.975346+0000","last_active":"2024-09-13T22:40:03.104498+0000","last_peered":"2024-09-13T22:40:03.104498+0000","last_clean":"2024-09-13T22:40:03.104498+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:40:03.104498+0000","last_undegraded":"2024-09-13T22:40:03.104498+0000","last_fullsized":"2024-09-13T22:40:03.104498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.975244+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.689091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102329+0000","last_change":"2024-09-13T22:39:07.049286+0000","last_active":"2024-09-13T22:40:03.102329+0000","last_peered":"2024-09-13T22:40:03.102329+0000","last_clean":"2024-09-13T22:40:03.102329+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:40:03.102329+0000","last_undegraded":"2024-09-13T22:40:03.102329+0000","last_fullsized":"2024-09-13T22:40:03.102329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:07.049084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.049084+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:35.995865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":229,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:12.050594+0000","last_change":"2024-09-13T22:39:38.224453+0000","last_active":"2024-09-13T22:40:12.050594+0000","last_peered":"2024-09-13T22:40:12.050594+0000","last_clean":"2024-09-13T22:40:12.050594+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:40:12.050594+0000","last_undegraded":"2024-09-13T22:40:12.050594+0000","last_fullsized":"2024-09-13T22:40:12.050594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:39:38.224366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:38.224366+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.543802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":134,"num_read_kb":129,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:07.588076+0000","last_change":"2024-09-13T22:40:07.588135+0000","last_active":"2024-09-13T22:40:07.588076+0000","last_peered":"2024-09-13T22:40:07.588076+0000","last_clean":"2024-09-13T22:40:07.588076+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:40:07.588076+0000","last_undegraded":"2024-09-13T22:40:07.588076+0000","last_fullsized":"2024-09-13T22:40:07.588076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:07.587908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:07.587908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:17.098329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.099420+0000","last_change":"2024-09-13T22:39:05.561714+0000","last_active":"2024-09-13T22:40:03.099420+0000","last_peered":"2024-09-13T22:40:03.099420+0000","last_clean":"2024-09-13T22:40:03.099420+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:40:03.099420+0000","last_undegraded":"2024-09-13T22:40:03.099420+0000","last_fullsized":"2024-09-13T22:40:03.099420+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.561489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.561489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.439424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":167,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.915954+0000","last_change":"2024-09-13T22:39:11.036017+0000","last_active":"2024-09-13T22:40:05.915954+0000","last_peered":"2024-09-13T22:40:05.915954+0000","last_clean":"2024-09-13T22:40:05.915954+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:40:05.915954+0000","last_undegraded":"2024-09-13T22:40:05.915954+0000","last_fullsized":"2024-09-13T22:40:05.915954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:39:11.035787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.035787+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.606987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106516+0000","last_change":"2024-09-13T22:39:31.671011+0000","last_active":"2024-09-13T22:40:03.106516+0000","last_peered":"2024-09-13T22:40:03.106516+0000","last_clean":"2024-09-13T22:40:03.106516+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:40:03.106516+0000","last_undegraded":"2024-09-13T22:40:03.106516+0000","last_fullsized":"2024-09-13T22:40:03.106516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.670821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.670821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.397166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529242+0000","last_change":"2024-09-13T22:39:24.762341+0000","last_active":"2024-09-13T22:40:01.529242+0000","last_peered":"2024-09-13T22:40:01.529242+0000","last_clean":"2024-09-13T22:40:01.529242+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:40:01.529242+0000","last_undegraded":"2024-09-13T22:40:01.529242+0000","last_fullsized":"2024-09-13T22:40:01.529242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.762123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.762123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.895897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100022+0000","last_change":"2024-09-13T22:39:25.019347+0000","last_active":"2024-09-13T22:40:03.100022+0000","last_peered":"2024-09-13T22:40:03.100022+0000","last_clean":"2024-09-13T22:40:03.100022+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:40:03.100022+0000","last_undegraded":"2024-09-13T22:40:03.100022+0000","last_fullsized":"2024-09-13T22:40:03.100022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.019168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.019168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.669620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":146,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.916529+0000","last_change":"2024-09-13T22:39:24.368074+0000","last_active":"2024-09-13T22:40:05.916529+0000","last_peered":"2024-09-13T22:40:05.916529+0000","last_clean":"2024-09-13T22:40:05.916529+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:40:05.916529+0000","last_undegraded":"2024-09-13T22:40:05.916529+0000","last_fullsized":"2024-09-13T22:40:05.916529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.367843+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.291892+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102981+0000","last_change":"2024-09-13T22:39:06.025706+0000","last_active":"2024-09-13T22:40:03.102981+0000","last_peered":"2024-09-13T22:40:03.102981+0000","last_clean":"2024-09-13T22:40:03.102981+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:40:03.102981+0000","last_undegraded":"2024-09-13T22:40:03.102981+0000","last_fullsized":"2024-09-13T22:40:03.102981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:06.025488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:06.025488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.356278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101295+0000","last_change":"2024-09-13T22:39:12.041436+0000","last_active":"2024-09-13T22:40:03.101295+0000","last_peered":"2024-09-13T22:40:03.101295+0000","last_clean":"2024-09-13T22:40:03.101295+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:40:03.101295+0000","last_undegraded":"2024-09-13T22:40:03.101295+0000","last_fullsized":"2024-09-13T22:40:03.101295+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:39:12.041252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.041252+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.755767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":123,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.916210+0000","last_change":"2024-09-13T22:39:01.656159+0000","last_active":"2024-09-13T22:40:05.916210+0000","last_peered":"2024-09-13T22:40:05.916210+0000","last_clean":"2024-09-13T22:40:05.916210+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:40:05.916210+0000","last_undegraded":"2024-09-13T22:40:05.916210+0000","last_fullsized":"2024-09-13T22:40:05.916210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.655949+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.600034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609737+0000","last_change":"2024-09-13T22:39:25.685706+0000","last_active":"2024-09-13T22:40:01.609737+0000","last_peered":"2024-09-13T22:40:01.609737+0000","last_clean":"2024-09-13T22:40:01.609737+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:40:01.609737+0000","last_undegraded":"2024-09-13T22:40:01.609737+0000","last_fullsized":"2024-09-13T22:40:01.609737+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.685432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.685432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.180606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100546+0000","last_change":"2024-09-13T22:39:10.618742+0000","last_active":"2024-09-13T22:40:03.100546+0000","last_peered":"2024-09-13T22:40:03.100546+0000","last_clean":"2024-09-13T22:40:03.100546+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:40:03.100546+0000","last_undegraded":"2024-09-13T22:40:03.100546+0000","last_fullsized":"2024-09-13T22:40:03.100546+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:10.618592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.618592+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.736269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:07.465823+0000","last_change":"2024-09-13T22:40:07.465921+0000","last_active":"2024-09-13T22:40:07.465823+0000","last_peered":"2024-09-13T22:40:07.465823+0000","last_clean":"2024-09-13T22:40:07.465823+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:40:07.465823+0000","last_undegraded":"2024-09-13T22:40:07.465823+0000","last_fullsized":"2024-09-13T22:40:07.465823+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:07.465685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:40:07.465685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:33.865225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609322+0000","last_change":"2024-09-13T22:39:02.682368+0000","last_active":"2024-09-13T22:40:01.609322+0000","last_peered":"2024-09-13T22:40:01.609322+0000","last_clean":"2024-09-13T22:40:01.609322+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:40:01.609322+0000","last_undegraded":"2024-09-13T22:40:01.609322+0000","last_fullsized":"2024-09-13T22:40:01.609322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.682170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.682170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.706353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103848+0000","last_change":"2024-09-13T22:39:45.281595+0000","last_active":"2024-09-13T22:40:03.103848+0000","last_peered":"2024-09-13T22:40:03.103848+0000","last_clean":"2024-09-13T22:40:03.103848+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:40:03.103848+0000","last_undegraded":"2024-09-13T22:40:03.103848+0000","last_fullsized":"2024-09-13T22:40:03.103848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:45.281512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:45.281512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:55.823803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"82'18","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101113+0000","last_change":"2024-09-13T22:39:19.966260+0000","last_active":"2024-09-13T22:40:03.101113+0000","last_peered":"2024-09-13T22:40:03.101113+0000","last_clean":"2024-09-13T22:40:03.101113+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:40:03.101113+0000","last_undegraded":"2024-09-13T22:40:03.101113+0000","last_fullsized":"2024-09-13T22:40:03.101113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-13T22:39:19.966043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.966043+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.842672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.528702+0000","last_change":"2024-09-13T22:39:15.472999+0000","last_active":"2024-09-13T22:40:01.528702+0000","last_peered":"2024-09-13T22:40:01.528702+0000","last_clean":"2024-09-13T22:40:01.528702+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:40:01.528702+0000","last_undegraded":"2024-09-13T22:40:01.528702+0000","last_fullsized":"2024-09-13T22:40:01.528702+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:15.472777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.472777+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:41.403750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529353+0000","last_change":"2024-09-13T22:38:54.478611+0000","last_active":"2024-09-13T22:40:01.529353+0000","last_peered":"2024-09-13T22:40:01.529353+0000","last_clean":"2024-09-13T22:40:01.529353+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:40:01.529353+0000","last_undegraded":"2024-09-13T22:40:01.529353+0000","last_fullsized":"2024-09-13T22:40:01.529353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:54.478424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.478424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.387064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105696+0000","last_change":"2024-09-13T22:39:20.748359+0000","last_active":"2024-09-13T22:40:03.105696+0000","last_peered":"2024-09-13T22:40:03.105696+0000","last_clean":"2024-09-13T22:40:03.105696+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:40:03.105696+0000","last_undegraded":"2024-09-13T22:40:03.105696+0000","last_fullsized":"2024-09-13T22:40:03.105696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.748082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.748082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:30.962778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609036+0000","last_change":"2024-09-13T22:39:05.725791+0000","last_active":"2024-09-13T22:40:01.609036+0000","last_peered":"2024-09-13T22:40:01.609036+0000","last_clean":"2024-09-13T22:40:01.609036+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:40:01.609036+0000","last_undegraded":"2024-09-13T22:40:01.609036+0000","last_fullsized":"2024-09-13T22:40:01.609036+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:39:05.725568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.725568+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.221576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.528014+0000","last_change":"2024-09-13T22:39:34.703390+0000","last_active":"2024-09-13T22:40:01.528014+0000","last_peered":"2024-09-13T22:40:01.528014+0000","last_clean":"2024-09-13T22:40:01.528014+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:40:01.528014+0000","last_undegraded":"2024-09-13T22:40:01.528014+0000","last_fullsized":"2024-09-13T22:40:01.528014+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:26.772349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.772349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.725362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105000+0000","last_change":"2024-09-13T22:39:12.302084+0000","last_active":"2024-09-13T22:40:03.105000+0000","last_peered":"2024-09-13T22:40:03.105000+0000","last_clean":"2024-09-13T22:40:03.105000+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:40:03.105000+0000","last_undegraded":"2024-09-13T22:40:03.105000+0000","last_fullsized":"2024-09-13T22:40:03.105000+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.301909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.301909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.703405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:11.075470+0000","last_change":"2024-09-13T22:40:11.075524+0000","last_active":"2024-09-13T22:40:11.075470+0000","last_peered":"2024-09-13T22:40:11.075470+0000","last_clean":"2024-09-13T22:40:11.075470+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:40:11.075470+0000","last_undegraded":"2024-09-13T22:40:11.075470+0000","last_fullsized":"2024-09-13T22:40:11.075470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:40:11.075324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:33.797037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102424+0000","last_change":"2024-09-13T22:39:04.843005+0000","last_active":"2024-09-13T22:40:03.102424+0000","last_peered":"2024-09-13T22:40:03.102424+0000","last_clean":"2024-09-13T22:40:03.102424+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:40:03.102424+0000","last_undegraded":"2024-09-13T22:40:03.102424+0000","last_fullsized":"2024-09-13T22:40:03.102424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:04.842757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.842757+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.409317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102878+0000","last_change":"2024-09-13T22:38:53.554534+0000","last_active":"2024-09-13T22:40:03.102878+0000","last_peered":"2024-09-13T22:40:03.102878+0000","last_clean":"2024-09-13T22:40:03.102878+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:40:03.102878+0000","last_undegraded":"2024-09-13T22:40:03.102878+0000","last_fullsized":"2024-09-13T22:40:03.102878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:53.554288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:53.554288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:07.563224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":350,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.603277+0000","last_change":"2024-09-13T22:40:05.603351+0000","last_active":"2024-09-13T22:40:05.603277+0000","last_peered":"2024-09-13T22:40:05.603277+0000","last_clean":"2024-09-13T22:40:05.603277+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:40:05.603277+0000","last_undegraded":"2024-09-13T22:40:05.603277+0000","last_fullsized":"2024-09-13T22:40:05.603277+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:40:05.603139+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:40:05.603139+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:07.244070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105136+0000","last_change":"2024-09-13T22:38:56.676371+0000","last_active":"2024-09-13T22:40:03.105136+0000","last_peered":"2024-09-13T22:40:03.105136+0000","last_clean":"2024-09-13T22:40:03.105136+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:40:03.105136+0000","last_undegraded":"2024-09-13T22:40:03.105136+0000","last_fullsized":"2024-09-13T22:40:03.105136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.676172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.058738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:08.496918+0000","last_change":"2024-09-13T22:40:08.497019+0000","last_active":"2024-09-13T22:40:08.496918+0000","last_peered":"2024-09-13T22:40:08.496918+0000","last_clean":"2024-09-13T22:40:08.496918+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:40:08.496918+0000","last_undegraded":"2024-09-13T22:40:08.496918+0000","last_fullsized":"2024-09-13T22:40:08.496918+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:08.496742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:08.496742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:20.536460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106189+0000","last_change":"2024-09-13T22:39:42.666727+0000","last_active":"2024-09-13T22:40:03.106189+0000","last_peered":"2024-09-13T22:40:03.106189+0000","last_clean":"2024-09-13T22:40:03.106189+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:40:03.106189+0000","last_undegraded":"2024-09-13T22:40:03.106189+0000","last_fullsized":"2024-09-13T22:40:03.106189+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:39:42.666504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:42.666504+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.765205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529920+0000","last_change":"2024-09-13T22:39:47.817369+0000","last_active":"2024-09-13T22:40:01.529920+0000","last_peered":"2024-09-13T22:40:01.529920+0000","last_clean":"2024-09-13T22:40:01.529920+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:40:01.529920+0000","last_undegraded":"2024-09-13T22:40:01.529920+0000","last_fullsized":"2024-09-13T22:40:01.529920+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:47.817287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:47.817287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.198399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529949+0000","last_change":"2024-09-13T22:39:12.437612+0000","last_active":"2024-09-13T22:40:01.529949+0000","last_peered":"2024-09-13T22:40:01.529949+0000","last_clean":"2024-09-13T22:40:01.529949+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:40:01.529949+0000","last_undegraded":"2024-09-13T22:40:01.529949+0000","last_fullsized":"2024-09-13T22:40:01.529949+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.437298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.437298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.840182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103094+0000","last_change":"2024-09-13T22:39:54.772601+0000","last_active":"2024-09-13T22:40:03.103094+0000","last_peered":"2024-09-13T22:40:03.103094+0000","last_clean":"2024-09-13T22:40:03.103094+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:40:03.103094+0000","last_undegraded":"2024-09-13T22:40:03.103094+0000","last_fullsized":"2024-09-13T22:40:03.103094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:54.772409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:54.772409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:01.661911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102190+0000","last_change":"2024-09-13T22:39:09.333581+0000","last_active":"2024-09-13T22:40:03.102190+0000","last_peered":"2024-09-13T22:40:03.102190+0000","last_clean":"2024-09-13T22:40:03.102190+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:40:03.102190+0000","last_undegraded":"2024-09-13T22:40:03.102190+0000","last_fullsized":"2024-09-13T22:40:03.102190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:09.333359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.333359+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.270070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:13.057751+0000","last_change":"2024-09-13T22:40:13.057805+0000","last_active":"2024-09-13T22:40:13.057751+0000","last_peered":"2024-09-13T22:40:13.057751+0000","last_clean":"2024-09-13T22:40:13.057751+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:40:13.057751+0000","last_undegraded":"2024-09-13T22:40:13.057751+0000","last_fullsized":"2024-09-13T22:40:13.057751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:13.057611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:13.057611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:20.204584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:11.879788+0000","last_change":"2024-09-13T22:40:11.879840+0000","last_active":"2024-09-13T22:40:11.879788+0000","last_peered":"2024-09-13T22:40:11.879788+0000","last_clean":"2024-09-13T22:40:11.879788+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:40:11.879788+0000","last_undegraded":"2024-09-13T22:40:11.879788+0000","last_fullsized":"2024-09-13T22:40:11.879788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:11.879646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:11.879646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:41.651249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:12.713538+0000","last_change":"2024-09-13T22:40:12.713650+0000","last_active":"2024-09-13T22:40:12.713538+0000","last_peered":"2024-09-13T22:40:12.713538+0000","last_clean":"2024-09-13T22:40:12.713538+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:40:12.713538+0000","last_undegraded":"2024-09-13T22:40:12.713538+0000","last_fullsized":"2024-09-13T22:40:12.713538+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:12.713324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:12.713324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:19.316363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":117,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106595+0000","last_change":"2024-09-13T22:39:46.684767+0000","last_active":"2024-09-13T22:40:03.106595+0000","last_peered":"2024-09-13T22:40:03.106595+0000","last_clean":"2024-09-13T22:40:03.106595+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:40:03.106595+0000","last_undegraded":"2024-09-13T22:40:03.106595+0000","last_fullsized":"2024-09-13T22:40:03.106595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:39:46.684514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.368244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529901+0000","last_change":"2024-09-13T22:39:04.517758+0000","last_active":"2024-09-13T22:40:01.529901+0000","last_peered":"2024-09-13T22:40:01.529901+0000","last_clean":"2024-09-13T22:40:01.529901+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:40:01.529901+0000","last_undegraded":"2024-09-13T22:40:01.529901+0000","last_fullsized":"2024-09-13T22:40:01.529901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:04.517658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.517658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.659731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.916323+0000","last_change":"2024-09-13T22:39:08.790837+0000","last_active":"2024-09-13T22:40:05.916323+0000","last_peered":"2024-09-13T22:40:05.916323+0000","last_clean":"2024-09-13T22:40:05.916323+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:40:05.916323+0000","last_undegraded":"2024-09-13T22:40:05.916323+0000","last_fullsized":"2024-09-13T22:40:05.916323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:08.790729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.790729+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.315232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100924+0000","last_change":"2024-09-13T22:39:10.297437+0000","last_active":"2024-09-13T22:40:03.100924+0000","last_peered":"2024-09-13T22:40:03.100924+0000","last_clean":"2024-09-13T22:40:03.100924+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:40:03.100924+0000","last_undegraded":"2024-09-13T22:40:03.100924+0000","last_fullsized":"2024-09-13T22:40:03.100924+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:10.297219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.297219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.668374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106402+0000","last_change":"2024-09-13T22:38:53.711228+0000","last_active":"2024-09-13T22:40:03.106402+0000","last_peered":"2024-09-13T22:40:03.106402+0000","last_clean":"2024-09-13T22:40:03.106402+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:40:03.106402+0000","last_undegraded":"2024-09-13T22:40:03.106402+0000","last_fullsized":"2024-09-13T22:40:03.106402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:38:53.711126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:53.711126+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104318+0000","last_change":"2024-09-13T22:39:00.654413+0000","last_active":"2024-09-13T22:40:03.104318+0000","last_peered":"2024-09-13T22:40:03.104318+0000","last_clean":"2024-09-13T22:40:03.104318+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:40:03.104318+0000","last_undegraded":"2024-09-13T22:40:03.104318+0000","last_fullsized":"2024-09-13T22:40:03.104318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'2","last_scrub_stamp":"2024-09-13T22:39:00.654142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.654142+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.020250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.528302+0000","last_change":"2024-09-13T22:39:21.523815+0000","last_active":"2024-09-13T22:40:01.528302+0000","last_peered":"2024-09-13T22:40:01.528302+0000","last_clean":"2024-09-13T22:40:01.528302+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:40:01.528302+0000","last_undegraded":"2024-09-13T22:40:01.528302+0000","last_fullsized":"2024-09-13T22:40:01.528302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.523625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.523625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:51.184833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101554+0000","last_change":"2024-09-13T22:39:18.998697+0000","last_active":"2024-09-13T22:40:03.101554+0000","last_peered":"2024-09-13T22:40:03.101554+0000","last_clean":"2024-09-13T22:40:03.101554+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:40:03.101554+0000","last_undegraded":"2024-09-13T22:40:03.101554+0000","last_fullsized":"2024-09-13T22:40:03.101554+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.998416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.890380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"82'11","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102146+0000","last_change":"2024-09-13T22:39:10.823267+0000","last_active":"2024-09-13T22:40:03.102146+0000","last_peered":"2024-09-13T22:40:03.102146+0000","last_clean":"2024-09-13T22:40:03.102146+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:40:03.102146+0000","last_undegraded":"2024-09-13T22:40:03.102146+0000","last_fullsized":"2024-09-13T22:40:03.102146+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:10.823176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.823176+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.690199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.528633+0000","last_change":"2024-09-13T22:38:56.494481+0000","last_active":"2024-09-13T22:40:01.528633+0000","last_peered":"2024-09-13T22:40:01.528633+0000","last_clean":"2024-09-13T22:40:01.528633+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:40:01.528633+0000","last_undegraded":"2024-09-13T22:40:01.528633+0000","last_fullsized":"2024-09-13T22:40:01.528633+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.494413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.494413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:06.935415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104421+0000","last_change":"2024-09-13T22:39:11.903829+0000","last_active":"2024-09-13T22:40:03.104421+0000","last_peered":"2024-09-13T22:40:03.104421+0000","last_clean":"2024-09-13T22:40:03.104421+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:40:03.104421+0000","last_undegraded":"2024-09-13T22:40:03.104421+0000","last_fullsized":"2024-09-13T22:40:03.104421+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.903758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.903758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:21.502704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:04.828734+0000","last_change":"2024-09-13T22:40:04.828734+0000","last_active":"2024-09-13T22:40:04.828734+0000","last_peered":"2024-09-13T22:40:04.828734+0000","last_clean":"2024-09-13T22:40:04.828734+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:40:04.828734+0000","last_undegraded":"2024-09-13T22:40:04.828734+0000","last_fullsized":"2024-09-13T22:40:04.828734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:04.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:04.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:17.849951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":99,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:09.776914+0000","last_change":"2024-09-13T22:40:09.776987+0000","last_active":"2024-09-13T22:40:09.776914+0000","last_peered":"2024-09-13T22:40:09.776914+0000","last_clean":"2024-09-13T22:40:09.776914+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:40:09.776914+0000","last_undegraded":"2024-09-13T22:40:09.776914+0000","last_fullsized":"2024-09-13T22:40:09.776914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:40:09.776771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:40:09.776771+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:31.771703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529984+0000","last_change":"2024-09-13T22:39:18.864519+0000","last_active":"2024-09-13T22:40:01.529984+0000","last_peered":"2024-09-13T22:40:01.529984+0000","last_clean":"2024-09-13T22:40:01.529984+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:40:01.529984+0000","last_undegraded":"2024-09-13T22:40:01.529984+0000","last_fullsized":"2024-09-13T22:40:01.529984+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.864331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.864331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.702214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103851+0000","last_change":"2024-09-13T22:39:31.439089+0000","last_active":"2024-09-13T22:40:03.103851+0000","last_peered":"2024-09-13T22:40:03.103851+0000","last_clean":"2024-09-13T22:40:03.103851+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:40:03.103851+0000","last_undegraded":"2024-09-13T22:40:03.103851+0000","last_fullsized":"2024-09-13T22:40:03.103851+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.438868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.438868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.289969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105551+0000","last_change":"2024-09-13T22:39:11.834762+0000","last_active":"2024-09-13T22:40:03.105551+0000","last_peered":"2024-09-13T22:40:03.105551+0000","last_clean":"2024-09-13T22:40:03.105551+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:40:03.105551+0000","last_undegraded":"2024-09-13T22:40:03.105551+0000","last_fullsized":"2024-09-13T22:40:03.105551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.834629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.834629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:37.419658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.105460+0000","last_change":"2024-09-13T22:39:30.707102+0000","last_active":"2024-09-13T22:40:03.105460+0000","last_peered":"2024-09-13T22:40:03.105460+0000","last_clean":"2024-09-13T22:40:03.105460+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:40:03.105460+0000","last_undegraded":"2024-09-13T22:40:03.105460+0000","last_fullsized":"2024-09-13T22:40:03.105460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:30.706887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:30.706887+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.189746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101774+0000","last_change":"2024-09-13T22:39:15.307350+0000","last_active":"2024-09-13T22:40:03.101774+0000","last_peered":"2024-09-13T22:40:03.101774+0000","last_clean":"2024-09-13T22:40:03.101774+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:40:03.101774+0000","last_undegraded":"2024-09-13T22:40:03.101774+0000","last_fullsized":"2024-09-13T22:40:03.101774+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.307195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.307195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.086503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":133,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:08.463222+0000","last_change":"2024-09-13T22:40:03.443131+0000","last_active":"2024-09-13T22:40:08.463222+0000","last_peered":"2024-09-13T22:40:08.463222+0000","last_clean":"2024-09-13T22:40:08.463222+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:40:08.463222+0000","last_undegraded":"2024-09-13T22:40:08.463222+0000","last_fullsized":"2024-09-13T22:40:08.463222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:40:03.442922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:03.442922+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.024934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609841+0000","last_change":"2024-09-13T22:39:18.688333+0000","last_active":"2024-09-13T22:40:01.609841+0000","last_peered":"2024-09-13T22:40:01.609841+0000","last_clean":"2024-09-13T22:40:01.609841+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:40:01.609841+0000","last_undegraded":"2024-09-13T22:40:01.609841+0000","last_fullsized":"2024-09-13T22:40:01.609841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.688141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.688141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.166952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609014+0000","last_change":"2024-09-13T22:38:54.600790+0000","last_active":"2024-09-13T22:40:01.609014+0000","last_peered":"2024-09-13T22:40:01.609014+0000","last_clean":"2024-09-13T22:40:01.609014+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:40:01.609014+0000","last_undegraded":"2024-09-13T22:40:01.609014+0000","last_fullsized":"2024-09-13T22:40:01.609014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:54.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.600571+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:11.578225+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609926+0000","last_change":"2024-09-13T22:39:21.654486+0000","last_active":"2024-09-13T22:40:01.609926+0000","last_peered":"2024-09-13T22:40:01.609926+0000","last_clean":"2024-09-13T22:40:01.609926+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:40:01.609926+0000","last_undegraded":"2024-09-13T22:40:01.609926+0000","last_fullsized":"2024-09-13T22:40:01.609926+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.654273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.654273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.171794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":130,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:08.464861+0000","last_change":"2024-09-13T22:40:04.236057+0000","last_active":"2024-09-13T22:40:08.464861+0000","last_peered":"2024-09-13T22:40:08.464861+0000","last_clean":"2024-09-13T22:40:08.464861+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:40:08.464861+0000","last_undegraded":"2024-09-13T22:40:08.464861+0000","last_fullsized":"2024-09-13T22:40:08.464861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:40:04.235854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:04.235854+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:10.334937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.099941+0000","last_change":"2024-09-13T22:39:17.629172+0000","last_active":"2024-09-13T22:40:03.099941+0000","last_peered":"2024-09-13T22:40:03.099941+0000","last_clean":"2024-09-13T22:40:03.099941+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:40:03.099941+0000","last_undegraded":"2024-09-13T22:40:03.099941+0000","last_fullsized":"2024-09-13T22:40:03.099941+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:17.628975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:17.628975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:18.565445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106171+0000","last_change":"2024-09-13T22:39:25.745828+0000","last_active":"2024-09-13T22:40:03.106171+0000","last_peered":"2024-09-13T22:40:03.106171+0000","last_clean":"2024-09-13T22:40:03.106171+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:40:03.106171+0000","last_undegraded":"2024-09-13T22:40:03.106171+0000","last_fullsized":"2024-09-13T22:40:03.106171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:25.745613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.745613+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:49.997201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.530166+0000","last_change":"2024-09-13T22:39:15.852762+0000","last_active":"2024-09-13T22:40:01.530166+0000","last_peered":"2024-09-13T22:40:01.530166+0000","last_clean":"2024-09-13T22:40:01.530166+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:40:01.530166+0000","last_undegraded":"2024-09-13T22:40:01.530166+0000","last_fullsized":"2024-09-13T22:40:01.530166+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.852596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.852596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.570592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103063+0000","last_change":"2024-09-13T22:39:29.136457+0000","last_active":"2024-09-13T22:40:03.103063+0000","last_peered":"2024-09-13T22:40:03.103063+0000","last_clean":"2024-09-13T22:40:03.103063+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:40:03.103063+0000","last_undegraded":"2024-09-13T22:40:03.103063+0000","last_fullsized":"2024-09-13T22:40:03.103063+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.136366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.072781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.608624+0000","last_change":"2024-09-13T22:39:27.605785+0000","last_active":"2024-09-13T22:40:01.608624+0000","last_peered":"2024-09-13T22:40:01.608624+0000","last_clean":"2024-09-13T22:40:01.608624+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:40:01.608624+0000","last_undegraded":"2024-09-13T22:40:01.608624+0000","last_fullsized":"2024-09-13T22:40:01.608624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.605612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.605612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.057619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":108,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102167+0000","last_change":"2024-09-13T22:39:48.728707+0000","last_active":"2024-09-13T22:40:03.102167+0000","last_peered":"2024-09-13T22:40:03.102167+0000","last_clean":"2024-09-13T22:40:03.102167+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:40:03.102167+0000","last_undegraded":"2024-09-13T22:40:03.102167+0000","last_fullsized":"2024-09-13T22:40:03.102167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:48.728497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:48.728497+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.833414+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101982+0000","last_change":"2024-09-13T22:39:13.008954+0000","last_active":"2024-09-13T22:40:03.101982+0000","last_peered":"2024-09-13T22:40:03.101982+0000","last_clean":"2024-09-13T22:40:03.101982+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:40:03.101982+0000","last_undegraded":"2024-09-13T22:40:03.101982+0000","last_fullsized":"2024-09-13T22:40:03.101982+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:13.008764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.008764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:17.397180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100927+0000","last_change":"2024-09-13T22:38:57.018120+0000","last_active":"2024-09-13T22:40:03.100927+0000","last_peered":"2024-09-13T22:40:03.100927+0000","last_clean":"2024-09-13T22:40:03.100927+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:40:03.100927+0000","last_undegraded":"2024-09-13T22:40:03.100927+0000","last_fullsized":"2024-09-13T22:40:03.100927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:57.017923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.017923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:15.532446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.099849+0000","last_change":"2024-09-13T22:38:52.760354+0000","last_active":"2024-09-13T22:40:03.099849+0000","last_peered":"2024-09-13T22:40:03.099849+0000","last_clean":"2024-09-13T22:40:03.099849+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:40:03.099849+0000","last_undegraded":"2024-09-13T22:40:03.099849+0000","last_fullsized":"2024-09-13T22:40:03.099849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:52.760268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:52.760268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:12.301827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609963+0000","last_change":"2024-09-13T22:39:07.744502+0000","last_active":"2024-09-13T22:40:01.609963+0000","last_peered":"2024-09-13T22:40:01.609963+0000","last_clean":"2024-09-13T22:40:01.609963+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:40:01.609963+0000","last_undegraded":"2024-09-13T22:40:01.609963+0000","last_fullsized":"2024-09-13T22:40:01.609963+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:39:07.744124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.744124+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.659429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104324+0000","last_change":"2024-09-13T22:39:21.378052+0000","last_active":"2024-09-13T22:40:03.104324+0000","last_peered":"2024-09-13T22:40:03.104324+0000","last_clean":"2024-09-13T22:40:03.104324+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:40:03.104324+0000","last_undegraded":"2024-09-13T22:40:03.104324+0000","last_fullsized":"2024-09-13T22:40:03.104324+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.377829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.377829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.746784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529333+0000","last_change":"2024-09-13T22:39:05.855096+0000","last_active":"2024-09-13T22:40:01.529333+0000","last_peered":"2024-09-13T22:40:01.529333+0000","last_clean":"2024-09-13T22:40:01.529333+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:40:01.529333+0000","last_undegraded":"2024-09-13T22:40:01.529333+0000","last_fullsized":"2024-09-13T22:40:01.529333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.854893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.334889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:04.568330+0000","last_change":"2024-09-13T22:40:04.568330+0000","last_active":"2024-09-13T22:40:04.568330+0000","last_peered":"2024-09-13T22:40:04.568330+0000","last_clean":"2024-09-13T22:40:04.568330+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:40:04.568330+0000","last_undegraded":"2024-09-13T22:40:04.568330+0000","last_fullsized":"2024-09-13T22:40:04.568330+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:04.568121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:04.568121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:30.948205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104116+0000","last_change":"2024-09-13T22:39:13.339189+0000","last_active":"2024-09-13T22:40:03.104116+0000","last_peered":"2024-09-13T22:40:03.104116+0000","last_clean":"2024-09-13T22:40:03.104116+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:40:03.104116+0000","last_undegraded":"2024-09-13T22:40:03.104116+0000","last_fullsized":"2024-09-13T22:40:03.104116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:13.338970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.338970+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.771039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.530683+0000","last_change":"2024-09-13T22:39:07.882650+0000","last_active":"2024-09-13T22:40:01.530683+0000","last_peered":"2024-09-13T22:40:01.530683+0000","last_clean":"2024-09-13T22:40:01.530683+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:40:01.530683+0000","last_undegraded":"2024-09-13T22:40:01.530683+0000","last_fullsized":"2024-09-13T22:40:01.530683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:07.882460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.882460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.349629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103496+0000","last_change":"2024-09-13T22:39:29.666045+0000","last_active":"2024-09-13T22:40:03.103496+0000","last_peered":"2024-09-13T22:40:03.103496+0000","last_clean":"2024-09-13T22:40:03.103496+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:40:03.103496+0000","last_undegraded":"2024-09-13T22:40:03.103496+0000","last_fullsized":"2024-09-13T22:40:03.103496+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.665857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.665857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.766151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103248+0000","last_change":"2024-09-13T22:38:49.749377+0000","last_active":"2024-09-13T22:40:03.103248+0000","last_peered":"2024-09-13T22:40:03.103248+0000","last_clean":"2024-09-13T22:40:03.103248+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:40:03.103248+0000","last_undegraded":"2024-09-13T22:40:03.103248+0000","last_fullsized":"2024-09-13T22:40:03.103248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:49.749300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.749300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:14.449565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.102458+0000","last_change":"2024-09-13T22:39:02.697718+0000","last_active":"2024-09-13T22:40:03.102458+0000","last_peered":"2024-09-13T22:40:03.102458+0000","last_clean":"2024-09-13T22:40:03.102458+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:40:03.102458+0000","last_undegraded":"2024-09-13T22:40:03.102458+0000","last_fullsized":"2024-09-13T22:40:03.102458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:39:02.697574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.697574+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.049416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"82'15","reported_seq":111,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100707+0000","last_change":"2024-09-13T22:39:16.963966+0000","last_active":"2024-09-13T22:40:03.100707+0000","last_peered":"2024-09-13T22:40:03.100707+0000","last_clean":"2024-09-13T22:40:03.100707+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:40:03.100707+0000","last_undegraded":"2024-09-13T22:40:03.100707+0000","last_fullsized":"2024-09-13T22:40:03.100707+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:39:16.963643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.963643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.423517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.106741+0000","last_change":"2024-09-13T22:39:19.315930+0000","last_active":"2024-09-13T22:40:03.106741+0000","last_peered":"2024-09-13T22:40:03.106741+0000","last_clean":"2024-09-13T22:40:03.106741+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:40:03.106741+0000","last_undegraded":"2024-09-13T22:40:03.106741+0000","last_fullsized":"2024-09-13T22:40:03.106741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.315707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.315707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.650430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":125,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:05.916323+0000","last_change":"2024-09-13T22:39:19.870202+0000","last_active":"2024-09-13T22:40:05.916323+0000","last_peered":"2024-09-13T22:40:05.916323+0000","last_clean":"2024-09-13T22:40:05.916323+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:40:05.916323+0000","last_undegraded":"2024-09-13T22:40:05.916323+0000","last_fullsized":"2024-09-13T22:40:05.916323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.869975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.869975+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.843605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100849+0000","last_change":"2024-09-13T22:39:18.975664+0000","last_active":"2024-09-13T22:40:03.100849+0000","last_peered":"2024-09-13T22:40:03.100849+0000","last_clean":"2024-09-13T22:40:03.100849+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:40:03.100849+0000","last_undegraded":"2024-09-13T22:40:03.100849+0000","last_fullsized":"2024-09-13T22:40:03.100849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.975461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.975461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.889969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104452+0000","last_change":"2024-09-13T22:39:28.141412+0000","last_active":"2024-09-13T22:40:03.104452+0000","last_peered":"2024-09-13T22:40:03.104452+0000","last_clean":"2024-09-13T22:40:03.104452+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:40:03.104452+0000","last_undegraded":"2024-09-13T22:40:03.104452+0000","last_fullsized":"2024-09-13T22:40:03.104452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:28.141327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:28.141327+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.373914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101808+0000","last_change":"2024-09-13T22:39:09.960753+0000","last_active":"2024-09-13T22:40:03.101808+0000","last_peered":"2024-09-13T22:40:03.101808+0000","last_clean":"2024-09-13T22:40:03.101808+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:40:03.101808+0000","last_undegraded":"2024-09-13T22:40:03.101808+0000","last_fullsized":"2024-09-13T22:40:03.101808+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:09.960550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.960550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:38.827394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609866+0000","last_change":"2024-09-13T22:39:03.677235+0000","last_active":"2024-09-13T22:40:01.609866+0000","last_peered":"2024-09-13T22:40:01.609866+0000","last_clean":"2024-09-13T22:40:01.609866+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:40:01.609866+0000","last_undegraded":"2024-09-13T22:40:01.609866+0000","last_fullsized":"2024-09-13T22:40:01.609866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.677120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.433725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103232+0000","last_change":"2024-09-13T22:39:02.572852+0000","last_active":"2024-09-13T22:40:03.103232+0000","last_peered":"2024-09-13T22:40:03.103232+0000","last_clean":"2024-09-13T22:40:03.103232+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:40:03.103232+0000","last_undegraded":"2024-09-13T22:40:03.103232+0000","last_fullsized":"2024-09-13T22:40:03.103232+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.572698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.572698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:10.965367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.529800+0000","last_change":"2024-09-13T22:39:59.762653+0000","last_active":"2024-09-13T22:40:01.529800+0000","last_peered":"2024-09-13T22:40:01.529800+0000","last_clean":"2024-09-13T22:40:01.529800+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:40:01.529800+0000","last_undegraded":"2024-09-13T22:40:01.529800+0000","last_fullsized":"2024-09-13T22:40:01.529800+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:59.762383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:59.762383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:13.508489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:01.609686+0000","last_change":"2024-09-13T22:40:00.799746+0000","last_active":"2024-09-13T22:40:01.609686+0000","last_peered":"2024-09-13T22:40:01.609686+0000","last_clean":"2024-09-13T22:40:01.609686+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:40:01.609686+0000","last_undegraded":"2024-09-13T22:40:01.609686+0000","last_fullsized":"2024-09-13T22:40:01.609686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:40:00.799429+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:40:00.799429+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.815402+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.101631+0000","last_change":"2024-09-13T22:39:23.072821+0000","last_active":"2024-09-13T22:40:03.101631+0000","last_peered":"2024-09-13T22:40:03.101631+0000","last_clean":"2024-09-13T22:40:03.101631+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:40:03.101631+0000","last_undegraded":"2024-09-13T22:40:03.101631+0000","last_fullsized":"2024-09-13T22:40:03.101631+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:39:23.072590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.247551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.103946+0000","last_change":"2024-09-13T22:38:57.829670+0000","last_active":"2024-09-13T22:40:03.103946+0000","last_peered":"2024-09-13T22:40:03.103946+0000","last_clean":"2024-09-13T22:40:03.103946+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:40:03.103946+0000","last_undegraded":"2024-09-13T22:40:03.103946+0000","last_fullsized":"2024-09-13T22:40:03.103946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:38:57.829468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.829468+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:16.502396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.104021+0000","last_change":"2024-09-13T22:39:03.881680+0000","last_active":"2024-09-13T22:40:03.104021+0000","last_peered":"2024-09-13T22:40:03.104021+0000","last_clean":"2024-09-13T22:40:03.104021+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:40:03.104021+0000","last_undegraded":"2024-09-13T22:40:03.104021+0000","last_fullsized":"2024-09-13T22:40:03.104021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.881493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.881493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.403295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:03.100982+0000","last_change":"2024-09-13T22:39:19.638335+0000","last_active":"2024-09-13T22:40:03.100982+0000","last_peered":"2024-09-13T22:40:03.100982+0000","last_clean":"2024-09-13T22:40:03.100982+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:40:03.100982+0000","last_undegraded":"2024-09-13T22:40:03.100982+0000","last_fullsized":"2024-09-13T22:40:03.100982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:19.638128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.638128+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.918763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T22:40:12.625324+0000","last_change":"2024-09-13T22:40:12.625324+0000","last_active":"2024-09-13T22:40:12.625324+0000","last_peered":"2024-09-13T22:40:12.625324+0000","last_clean":"2024-09-13T22:40:12.625324+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:40:12.625324+0000","last_undegraded":"2024-09-13T22:40:12.625324+0000","last_fullsized":"2024-09-13T22:40:12.625324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:40:12.625120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:25.466463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":134,"num_read_kb":129,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808871,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32472,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712680,"statfs":{"total":95995035648,"available":95961784320,"internally_reserved":0,"allocated":1318912,"data_stored":921821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135922,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32444,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712708,"statfs":{"total":95995035648,"available":95961812992,"internally_reserved":0,"allocated":1290240,"data_stored":919499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528381,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":700416,"data_stored":329533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":733184,"data_stored":330063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084369,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":700416,"data_stored":329954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378717,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":700416,"data_stored":330047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":712704,"data_stored":330442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1310720,"data_stored":921570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:40:15.953 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: audit 2024-09-13T22:40:14.896752+0000 mgr.y (mgr.14613) 98 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.059222+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.061451+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: audit 2024-09-13T22:40:15.187924+0000 mgr.y (mgr.14613) 99 : audit [DBG] from='client.14796 v1:172.21.15.178:0/2859925886' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.281296+0000 osd.3 (osd.3) 57 : cluster [DBG] 3.b deep-scrub starts 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.285968+0000 osd.3 (osd.3) 58 : cluster [DBG] 3.b deep-scrub ok 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.590586+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.592898+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.730449+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:40:16.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:16 smithi192 bash[24608]: cluster 2024-09-13T22:40:15.732742+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:40:16.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: audit 2024-09-13T22:40:14.896752+0000 mgr.y (mgr.14613) 98 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:16.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.059222+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:40:16.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.061451+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:40:16.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: audit 2024-09-13T22:40:15.187924+0000 mgr.y (mgr.14613) 99 : audit [DBG] from='client.14796 v1:172.21.15.178:0/2859925886' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:40:16.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.281296+0000 osd.3 (osd.3) 57 : cluster [DBG] 3.b deep-scrub starts 2024-09-13T22:40:16.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.285968+0000 osd.3 (osd.3) 58 : cluster [DBG] 3.b deep-scrub ok 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.590586+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.592898+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.730449+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[22840]: cluster 2024-09-13T22:40:15.732742+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: audit 2024-09-13T22:40:14.896752+0000 mgr.y (mgr.14613) 98 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.059222+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.061451+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: audit 2024-09-13T22:40:15.187924+0000 mgr.y (mgr.14613) 99 : audit [DBG] from='client.14796 v1:172.21.15.178:0/2859925886' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.281296+0000 osd.3 (osd.3) 57 : cluster [DBG] 3.b deep-scrub starts 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.285968+0000 osd.3 (osd.3) 58 : cluster [DBG] 3.b deep-scrub ok 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.590586+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:40:16.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.592898+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:40:16.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.730449+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:40:16.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:16 smithi178 bash[29889]: cluster 2024-09-13T22:40:15.732742+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:40:17.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:17 smithi192 bash[24608]: cluster 2024-09-13T22:40:16.459803+0000 mgr.y (mgr.14613) 100 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:40:17.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:17 smithi178 bash[29889]: cluster 2024-09-13T22:40:16.459803+0000 mgr.y (mgr.14613) 100 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:40:17.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:17 smithi178 bash[22840]: cluster 2024-09-13T22:40:16.459803+0000 mgr.y (mgr.14613) 100 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.149406+0000 osd.1 (osd.1) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.149545+0000 osd.1 (osd.1) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.347865+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.486219+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.489314+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: audit 2024-09-13T22:40:17.689247+0000 mon.b (mon.1) 88 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.816910+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:18 smithi192 bash[24608]: cluster 2024-09-13T22:40:17.819152+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.149406+0000 osd.1 (osd.1) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.149545+0000 osd.1 (osd.1) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.347865+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.486219+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.489314+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: audit 2024-09-13T22:40:17.689247+0000 mon.b (mon.1) 88 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.816910+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:40:18.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[29889]: cluster 2024-09-13T22:40:17.819152+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.149406+0000 osd.1 (osd.1) 57 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.149545+0000 osd.1 (osd.1) 58 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.347865+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.486219+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.489314+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: audit 2024-09-13T22:40:17.689247+0000 mon.b (mon.1) 88 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.816910+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:40:18.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:18 smithi178 bash[22840]: cluster 2024-09-13T22:40:17.819152+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.060425+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.062667+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.460544+0000 mgr.y (mgr.14613) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.510959+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.513326+0000 osd.7 (osd.7) 62 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.570425+0000 osd.5 (osd.5) 61 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:40:19.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:19 smithi192 bash[24608]: cluster 2024-09-13T22:40:18.572417+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.060425+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.062667+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.460544+0000 mgr.y (mgr.14613) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.510959+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.513326+0000 osd.7 (osd.7) 62 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.570425+0000 osd.5 (osd.5) 61 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:40:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[29889]: cluster 2024-09-13T22:40:18.572417+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.060425+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.062667+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.460544+0000 mgr.y (mgr.14613) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.510959+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.513326+0000 osd.7 (osd.7) 62 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.570425+0000 osd.5 (osd.5) 61 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:40:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[22840]: cluster 2024-09-13T22:40:18.572417+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:40:20.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:40:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:20.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.032890+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.035095+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.325040+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.11 deep-scrub starts 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.327254+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.11 deep-scrub ok 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.571950+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.574294+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.624971+0000 osd.4 (osd.4) 49 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:40:20.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:20 smithi192 bash[24608]: cluster 2024-09-13T22:40:19.627801+0000 osd.4 (osd.4) 50 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:40:20.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.032890+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:40:20.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.035095+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:40:20.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.325040+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.11 deep-scrub starts 2024-09-13T22:40:20.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.327254+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.11 deep-scrub ok 2024-09-13T22:40:20.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.571950+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.574294+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.624971+0000 osd.4 (osd.4) 49 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[29889]: cluster 2024-09-13T22:40:19.627801+0000 osd.4 (osd.4) 50 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.032890+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.035095+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.325040+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.11 deep-scrub starts 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.327254+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.11 deep-scrub ok 2024-09-13T22:40:20.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.571950+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:40:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.574294+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:40:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.624971+0000 osd.4 (osd.4) 49 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:40:20.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:20 smithi178 bash[22840]: cluster 2024-09-13T22:40:19.627801+0000 osd.4 (osd.4) 50 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:40:20.755 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:21.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:21 smithi192 bash[24608]: cluster 2024-09-13T22:40:20.462123+0000 mgr.y (mgr.14613) 102 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:21 smithi192 bash[24608]: cluster 2024-09-13T22:40:20.477533+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:21 smithi192 bash[24608]: cluster 2024-09-13T22:40:20.477649+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:21 smithi192 bash[24608]: cluster 2024-09-13T22:40:20.636144+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:40:21.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:21 smithi192 bash[24608]: cluster 2024-09-13T22:40:20.638359+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[29889]: cluster 2024-09-13T22:40:20.462123+0000 mgr.y (mgr.14613) 102 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[29889]: cluster 2024-09-13T22:40:20.477533+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[29889]: cluster 2024-09-13T22:40:20.477649+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[29889]: cluster 2024-09-13T22:40:20.636144+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[29889]: cluster 2024-09-13T22:40:20.638359+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[22840]: cluster 2024-09-13T22:40:20.462123+0000 mgr.y (mgr.14613) 102 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:21.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[22840]: cluster 2024-09-13T22:40:20.477533+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[22840]: cluster 2024-09-13T22:40:20.477649+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[22840]: cluster 2024-09-13T22:40:20.636144+0000 osd.4 (osd.4) 51 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:40:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:21 smithi178 bash[22840]: cluster 2024-09-13T22:40:20.638359+0000 osd.4 (osd.4) 52 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:40:22.228 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:40:22.230 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.057957+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.070281+0000 osd.1 (osd.1) 64 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.364888+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.733381+0000 osd.0 (osd.0) 55 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.738016+0000 osd.0 (osd.0) 56 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.871888+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 deep-scrub starts 2024-09-13T22:40:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:22 smithi192 bash[24608]: cluster 2024-09-13T22:40:21.874085+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 deep-scrub ok 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.057957+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.070281+0000 osd.1 (osd.1) 64 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.364888+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.733381+0000 osd.0 (osd.0) 55 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.738016+0000 osd.0 (osd.0) 56 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.871888+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 deep-scrub starts 2024-09-13T22:40:22.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[29889]: cluster 2024-09-13T22:40:21.874085+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 deep-scrub ok 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.057957+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.070281+0000 osd.1 (osd.1) 64 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.364888+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.733381+0000 osd.0 (osd.0) 55 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.738016+0000 osd.0 (osd.0) 56 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.871888+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.17 deep-scrub starts 2024-09-13T22:40:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:22 smithi178 bash[22840]: cluster 2024-09-13T22:40:21.874085+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.17 deep-scrub ok 2024-09-13T22:40:22.919 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-13T22:40:21.363155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1089,"num_read_kb":935,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":674,"ondisk_log_size":674,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244396,"kb_used_data":7376,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716820,"statfs":{"total":767960285184,"available":767710023680,"internally_reserved":0,"allocated":7553024,"data_stored":4458086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.905917"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.705938+0000","last_change":"2024-09-13T22:39:52.233876+0000","last_active":"2024-09-13T22:40:17.705938+0000","last_peered":"2024-09-13T22:40:17.705938+0000","last_clean":"2024-09-13T22:40:17.705938+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:40:17.705938+0000","last_undegraded":"2024-09-13T22:40:17.705938+0000","last_fullsized":"2024-09-13T22:40:17.705938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:52.233682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:52.233682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:15.212128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.705639+0000","last_change":"2024-09-13T22:38:46.613749+0000","last_active":"2024-09-13T22:40:17.705639+0000","last_peered":"2024-09-13T22:40:17.705639+0000","last_clean":"2024-09-13T22:40:17.705639+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:40:17.705639+0000","last_undegraded":"2024-09-13T22:40:17.705639+0000","last_fullsized":"2024-09-13T22:40:17.705639+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:38:46.613466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707000+0000","last_change":"2024-09-13T22:38:54.776466+0000","last_active":"2024-09-13T22:40:17.707000+0000","last_peered":"2024-09-13T22:40:17.707000+0000","last_clean":"2024-09-13T22:40:17.707000+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:40:17.707000+0000","last_undegraded":"2024-09-13T22:40:17.707000+0000","last_fullsized":"2024-09-13T22:40:17.707000+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:38:54.776238+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.776238+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.480650+0000","last_change":"2024-09-13T22:39:20.600784+0000","last_active":"2024-09-13T22:40:17.480650+0000","last_peered":"2024-09-13T22:40:17.480650+0000","last_clean":"2024-09-13T22:40:17.480650+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:40:17.480650+0000","last_undegraded":"2024-09-13T22:40:17.480650+0000","last_fullsized":"2024-09-13T22:40:17.480650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:20.600571+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.600571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.907326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707268+0000","last_change":"2024-09-13T22:39:27.118459+0000","last_active":"2024-09-13T22:40:17.707268+0000","last_peered":"2024-09-13T22:40:17.707268+0000","last_clean":"2024-09-13T22:40:17.707268+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:40:17.707268+0000","last_undegraded":"2024-09-13T22:40:17.707268+0000","last_fullsized":"2024-09-13T22:40:17.707268+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.118162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.118162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.139079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":111,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704993+0000","last_change":"2024-09-13T22:39:08.754354+0000","last_active":"2024-09-13T22:40:17.704993+0000","last_peered":"2024-09-13T22:40:17.704993+0000","last_clean":"2024-09-13T22:40:17.704993+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:40:17.704993+0000","last_undegraded":"2024-09-13T22:40:17.704993+0000","last_fullsized":"2024-09-13T22:40:17.704993+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:39:08.754231+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707994+0000","last_change":"2024-09-13T22:39:24.033368+0000","last_active":"2024-09-13T22:40:17.707994+0000","last_peered":"2024-09-13T22:40:17.707994+0000","last_clean":"2024-09-13T22:40:17.707994+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:40:17.707994+0000","last_undegraded":"2024-09-13T22:40:17.707994+0000","last_fullsized":"2024-09-13T22:40:17.707994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.033224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.033224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.108639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481872+0000","last_change":"2024-09-13T22:38:59.657956+0000","last_active":"2024-09-13T22:40:17.481872+0000","last_peered":"2024-09-13T22:40:17.481872+0000","last_clean":"2024-09-13T22:40:17.481872+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:40:17.481872+0000","last_undegraded":"2024-09-13T22:40:17.481872+0000","last_fullsized":"2024-09-13T22:40:17.481872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:59.657646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:59.657646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345929+0000","last_change":"2024-09-13T22:39:19.778773+0000","last_active":"2024-09-13T22:40:17.345929+0000","last_peered":"2024-09-13T22:40:17.345929+0000","last_clean":"2024-09-13T22:40:17.345929+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:40:17.345929+0000","last_undegraded":"2024-09-13T22:40:17.345929+0000","last_fullsized":"2024-09-13T22:40:17.345929+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.778590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.778590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.572565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":119,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703255+0000","last_change":"2024-09-13T22:39:26.318996+0000","last_active":"2024-09-13T22:40:17.703255+0000","last_peered":"2024-09-13T22:40:17.703255+0000","last_clean":"2024-09-13T22:40:17.703255+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:40:17.703255+0000","last_undegraded":"2024-09-13T22:40:17.703255+0000","last_fullsized":"2024-09-13T22:40:17.703255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.318753+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.307721+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344233+0000","last_change":"2024-09-13T22:39:48.736178+0000","last_active":"2024-09-13T22:40:17.344233+0000","last_peered":"2024-09-13T22:40:17.344233+0000","last_clean":"2024-09-13T22:40:17.344233+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:40:17.344233+0000","last_undegraded":"2024-09-13T22:40:17.344233+0000","last_fullsized":"2024-09-13T22:40:17.344233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:39:48.735986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:14.424901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702716+0000","last_change":"2024-09-13T22:39:37.426328+0000","last_active":"2024-09-13T22:40:17.702716+0000","last_peered":"2024-09-13T22:40:17.702716+0000","last_clean":"2024-09-13T22:40:17.702716+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:40:17.702716+0000","last_undegraded":"2024-09-13T22:40:17.702716+0000","last_fullsized":"2024-09-13T22:40:17.702716+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:37.426096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:37.426096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.076012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345679+0000","last_change":"2024-09-13T22:40:10.565444+0000","last_active":"2024-09-13T22:40:17.345679+0000","last_peered":"2024-09-13T22:40:17.345679+0000","last_clean":"2024-09-13T22:40:17.345679+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:40:17.345679+0000","last_undegraded":"2024-09-13T22:40:17.345679+0000","last_fullsized":"2024-09-13T22:40:17.345679+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:10.565251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:10.565251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:37.474394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":118,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704326+0000","last_change":"2024-09-13T22:40:13.605708+0000","last_active":"2024-09-13T22:40:17.704326+0000","last_peered":"2024-09-13T22:40:17.704326+0000","last_clean":"2024-09-13T22:40:17.704326+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:40:17.704326+0000","last_undegraded":"2024-09-13T22:40:17.704326+0000","last_fullsized":"2024-09-13T22:40:17.704326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:40:13.605492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:40:13.605492+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:32.337625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702847+0000","last_change":"2024-09-13T22:38:50.497874+0000","last_active":"2024-09-13T22:40:17.702847+0000","last_peered":"2024-09-13T22:40:17.702847+0000","last_clean":"2024-09-13T22:40:17.702847+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:40:17.702847+0000","last_undegraded":"2024-09-13T22:40:17.702847+0000","last_fullsized":"2024-09-13T22:40:17.702847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:50.497567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:50.497567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702639+0000","last_change":"2024-09-13T22:39:14.752874+0000","last_active":"2024-09-13T22:40:17.702639+0000","last_peered":"2024-09-13T22:40:17.702639+0000","last_clean":"2024-09-13T22:40:17.702639+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:40:17.702639+0000","last_undegraded":"2024-09-13T22:40:17.702639+0000","last_fullsized":"2024-09-13T22:40:17.702639+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.752683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.752683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.453233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"82'19","reported_seq":118,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.485466+0000","last_change":"2024-09-13T22:39:21.873986+0000","last_active":"2024-09-13T22:40:17.485466+0000","last_peered":"2024-09-13T22:40:17.485466+0000","last_clean":"2024-09-13T22:40:17.485466+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:40:17.485466+0000","last_undegraded":"2024-09-13T22:40:17.485466+0000","last_fullsized":"2024-09-13T22:40:17.485466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:21.873747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.873747+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.726394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345506+0000","last_change":"2024-09-13T22:39:35.662591+0000","last_active":"2024-09-13T22:40:17.345506+0000","last_peered":"2024-09-13T22:40:17.345506+0000","last_clean":"2024-09-13T22:40:17.345506+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:40:17.345506+0000","last_undegraded":"2024-09-13T22:40:17.345506+0000","last_fullsized":"2024-09-13T22:40:17.345506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:35.662396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:35.662396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:48.563351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.708896+0000","last_change":"2024-09-13T22:39:15.972091+0000","last_active":"2024-09-13T22:40:17.708896+0000","last_peered":"2024-09-13T22:40:17.708896+0000","last_clean":"2024-09-13T22:40:17.708896+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:40:17.708896+0000","last_undegraded":"2024-09-13T22:40:17.708896+0000","last_fullsized":"2024-09-13T22:40:17.708896+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.971904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.971904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:40.478614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345798+0000","last_change":"2024-09-13T22:39:14.436930+0000","last_active":"2024-09-13T22:40:17.345798+0000","last_peered":"2024-09-13T22:40:17.345798+0000","last_clean":"2024-09-13T22:40:17.345798+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:40:17.345798+0000","last_undegraded":"2024-09-13T22:40:17.345798+0000","last_fullsized":"2024-09-13T22:40:17.345798+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:14.436746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:14.436746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.039947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"110'13","reported_seq":115,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706475+0000","last_change":"2024-09-13T22:40:15.286078+0000","last_active":"2024-09-13T22:40:17.706475+0000","last_peered":"2024-09-13T22:40:17.706475+0000","last_clean":"2024-09-13T22:40:17.706475+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:40:17.706475+0000","last_undegraded":"2024-09-13T22:40:17.706475+0000","last_fullsized":"2024-09-13T22:40:17.706475+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:40:15.285976+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:37.913805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.709503+0000","last_change":"2024-09-13T22:39:01.945920+0000","last_active":"2024-09-13T22:40:17.709503+0000","last_peered":"2024-09-13T22:40:17.709503+0000","last_clean":"2024-09-13T22:40:17.709503+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:40:17.709503+0000","last_undegraded":"2024-09-13T22:40:17.709503+0000","last_fullsized":"2024-09-13T22:40:17.709503+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:01.945733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.945733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.448893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":125,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345277+0000","last_change":"2024-09-13T22:39:19.489746+0000","last_active":"2024-09-13T22:40:17.345277+0000","last_peered":"2024-09-13T22:40:17.345277+0000","last_clean":"2024-09-13T22:40:17.345277+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:40:17.345277+0000","last_undegraded":"2024-09-13T22:40:17.345277+0000","last_fullsized":"2024-09-13T22:40:17.345277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.489549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.489549+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:47.360174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481338+0000","last_change":"2024-09-13T22:39:51.642269+0000","last_active":"2024-09-13T22:40:17.481338+0000","last_peered":"2024-09-13T22:40:17.481338+0000","last_clean":"2024-09-13T22:40:17.481338+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:40:17.481338+0000","last_undegraded":"2024-09-13T22:40:17.481338+0000","last_fullsized":"2024-09-13T22:40:17.481338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:51.642066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:51.642066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:03.230502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":112,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707493+0000","last_change":"2024-09-13T22:38:58.835483+0000","last_active":"2024-09-13T22:40:17.707493+0000","last_peered":"2024-09-13T22:40:17.707493+0000","last_clean":"2024-09-13T22:40:17.707493+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:40:17.707493+0000","last_undegraded":"2024-09-13T22:40:17.707493+0000","last_fullsized":"2024-09-13T22:40:17.707493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:38:58.835275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:58.835275+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.927373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.708785+0000","last_change":"2024-09-13T22:40:15.061637+0000","last_active":"2024-09-13T22:40:17.708785+0000","last_peered":"2024-09-13T22:40:17.708785+0000","last_clean":"2024-09-13T22:40:17.708785+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:40:17.708785+0000","last_undegraded":"2024-09-13T22:40:17.708785+0000","last_fullsized":"2024-09-13T22:40:17.708785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:15.061459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:15.061459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:34.608416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":126,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.482081+0000","last_change":"2024-09-13T22:39:12.592344+0000","last_active":"2024-09-13T22:40:17.482081+0000","last_peered":"2024-09-13T22:40:17.482081+0000","last_clean":"2024-09-13T22:40:17.482081+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:40:17.482081+0000","last_undegraded":"2024-09-13T22:40:17.482081+0000","last_fullsized":"2024-09-13T22:40:17.482081+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.592134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.802063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703579+0000","last_change":"2024-09-13T22:39:33.392634+0000","last_active":"2024-09-13T22:40:17.703579+0000","last_peered":"2024-09-13T22:40:17.703579+0000","last_clean":"2024-09-13T22:40:17.703579+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:40:17.703579+0000","last_undegraded":"2024-09-13T22:40:17.703579+0000","last_fullsized":"2024-09-13T22:40:17.703579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_clean_scrub_stamp":"2024-09-13T22:39:33.392318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.641551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":110,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.482434+0000","last_change":"2024-09-13T22:38:49.656312+0000","last_active":"2024-09-13T22:40:17.482434+0000","last_peered":"2024-09-13T22:40:17.482434+0000","last_clean":"2024-09-13T22:40:17.482434+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:40:17.482434+0000","last_undegraded":"2024-09-13T22:40:17.482434+0000","last_fullsized":"2024-09-13T22:40:17.482434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:38:49.656093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.656093+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344281+0000","last_change":"2024-09-13T22:39:40.612195+0000","last_active":"2024-09-13T22:40:17.344281+0000","last_peered":"2024-09-13T22:40:17.344281+0000","last_clean":"2024-09-13T22:40:17.344281+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:40:17.344281+0000","last_undegraded":"2024-09-13T22:40:17.344281+0000","last_fullsized":"2024-09-13T22:40:17.344281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:40.611997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.611997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:10.478412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704112+0000","last_change":"2024-09-13T22:40:09.495032+0000","last_active":"2024-09-13T22:40:17.704112+0000","last_peered":"2024-09-13T22:40:17.704112+0000","last_clean":"2024-09-13T22:40:17.704112+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:40:17.704112+0000","last_undegraded":"2024-09-13T22:40:17.704112+0000","last_fullsized":"2024-09-13T22:40:17.704112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:09.494846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:09.494846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:15.238012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704414+0000","last_change":"2024-09-13T22:39:53.214599+0000","last_active":"2024-09-13T22:40:17.704414+0000","last_peered":"2024-09-13T22:40:17.704414+0000","last_clean":"2024-09-13T22:40:17.704414+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:40:17.704414+0000","last_undegraded":"2024-09-13T22:40:17.704414+0000","last_fullsized":"2024-09-13T22:40:17.704414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:53.214316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:53.214316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:03.248657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":105,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706645+0000","last_change":"2024-09-13T22:39:04.726321+0000","last_active":"2024-09-13T22:40:17.706645+0000","last_peered":"2024-09-13T22:40:17.706645+0000","last_clean":"2024-09-13T22:40:17.706645+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:40:17.706645+0000","last_undegraded":"2024-09-13T22:40:17.706645+0000","last_fullsized":"2024-09-13T22:40:17.706645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:39:04.726105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.726105+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.304382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.819300+0000","last_change":"2024-09-13T22:40:17.819352+0000","last_active":"2024-09-13T22:40:17.819300+0000","last_peered":"2024-09-13T22:40:17.819300+0000","last_clean":"2024-09-13T22:40:17.819300+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:40:17.819300+0000","last_undegraded":"2024-09-13T22:40:17.819300+0000","last_fullsized":"2024-09-13T22:40:17.819300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:17.819161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:17.819161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:28.879407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":95,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.482269+0000","last_change":"2024-09-13T22:39:15.627485+0000","last_active":"2024-09-13T22:40:17.482269+0000","last_peered":"2024-09-13T22:40:17.482269+0000","last_clean":"2024-09-13T22:40:17.482269+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:40:17.482269+0000","last_undegraded":"2024-09-13T22:40:17.482269+0000","last_fullsized":"2024-09-13T22:40:17.482269+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:39:15.627264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.627264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.016937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":125,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707040+0000","last_change":"2024-09-13T22:39:40.265509+0000","last_active":"2024-09-13T22:40:17.707040+0000","last_peered":"2024-09-13T22:40:17.707040+0000","last_clean":"2024-09-13T22:40:17.707040+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:40:17.707040+0000","last_undegraded":"2024-09-13T22:40:17.707040+0000","last_fullsized":"2024-09-13T22:40:17.707040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:40.265350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:40.265350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:00.451629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344401+0000","last_change":"2024-09-13T22:39:49.738117+0000","last_active":"2024-09-13T22:40:17.344401+0000","last_peered":"2024-09-13T22:40:17.344401+0000","last_clean":"2024-09-13T22:40:17.344401+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:40:17.344401+0000","last_undegraded":"2024-09-13T22:40:17.344401+0000","last_fullsized":"2024-09-13T22:40:17.344401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:49.737992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:49.737992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:55.782646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":119,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707580+0000","last_change":"2024-09-13T22:39:16.975346+0000","last_active":"2024-09-13T22:40:17.707580+0000","last_peered":"2024-09-13T22:40:17.707580+0000","last_clean":"2024-09-13T22:40:17.707580+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:40:17.707580+0000","last_undegraded":"2024-09-13T22:40:17.707580+0000","last_fullsized":"2024-09-13T22:40:17.707580+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.975244+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.689091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":93,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.709760+0000","last_change":"2024-09-13T22:39:07.049286+0000","last_active":"2024-09-13T22:40:17.709760+0000","last_peered":"2024-09-13T22:40:17.709760+0000","last_clean":"2024-09-13T22:40:17.709760+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:40:17.709760+0000","last_undegraded":"2024-09-13T22:40:17.709760+0000","last_fullsized":"2024-09-13T22:40:17.709760+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:07.049084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.049084+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:35.995865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":236,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707294+0000","last_change":"2024-09-13T22:39:38.224453+0000","last_active":"2024-09-13T22:40:17.707294+0000","last_peered":"2024-09-13T22:40:17.707294+0000","last_clean":"2024-09-13T22:40:17.707294+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:40:17.707294+0000","last_undegraded":"2024-09-13T22:40:17.707294+0000","last_fullsized":"2024-09-13T22:40:17.707294+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:39:38.224366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:38.224366+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.543802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.480953+0000","last_change":"2024-09-13T22:40:07.588135+0000","last_active":"2024-09-13T22:40:17.480953+0000","last_peered":"2024-09-13T22:40:17.480953+0000","last_clean":"2024-09-13T22:40:17.480953+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:40:17.480953+0000","last_undegraded":"2024-09-13T22:40:17.480953+0000","last_fullsized":"2024-09-13T22:40:17.480953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:07.587908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:07.587908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:17.098329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.480702+0000","last_change":"2024-09-13T22:39:05.561714+0000","last_active":"2024-09-13T22:40:17.480702+0000","last_peered":"2024-09-13T22:40:17.480702+0000","last_clean":"2024-09-13T22:40:17.480702+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:40:17.480702+0000","last_undegraded":"2024-09-13T22:40:17.480702+0000","last_fullsized":"2024-09-13T22:40:17.480702+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.561489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.561489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.439424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":170,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.709300+0000","last_change":"2024-09-13T22:39:11.036017+0000","last_active":"2024-09-13T22:40:17.709300+0000","last_peered":"2024-09-13T22:40:17.709300+0000","last_clean":"2024-09-13T22:40:17.709300+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:40:17.709300+0000","last_undegraded":"2024-09-13T22:40:17.709300+0000","last_fullsized":"2024-09-13T22:40:17.709300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:39:11.035787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.035787+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.606987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.346132+0000","last_change":"2024-09-13T22:39:31.671011+0000","last_active":"2024-09-13T22:40:17.346132+0000","last_peered":"2024-09-13T22:40:17.346132+0000","last_clean":"2024-09-13T22:40:17.346132+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:40:17.346132+0000","last_undegraded":"2024-09-13T22:40:17.346132+0000","last_fullsized":"2024-09-13T22:40:17.346132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.670821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.670821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.397166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.485180+0000","last_change":"2024-09-13T22:39:24.762341+0000","last_active":"2024-09-13T22:40:17.485180+0000","last_peered":"2024-09-13T22:40:17.485180+0000","last_clean":"2024-09-13T22:40:17.485180+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:40:17.485180+0000","last_undegraded":"2024-09-13T22:40:17.485180+0000","last_fullsized":"2024-09-13T22:40:17.485180+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:24.762123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.762123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.895897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707933+0000","last_change":"2024-09-13T22:39:25.019347+0000","last_active":"2024-09-13T22:40:17.707933+0000","last_peered":"2024-09-13T22:40:17.707933+0000","last_clean":"2024-09-13T22:40:17.707933+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:40:17.707933+0000","last_undegraded":"2024-09-13T22:40:17.707933+0000","last_fullsized":"2024-09-13T22:40:17.707933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.019168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.019168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.669620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":149,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704358+0000","last_change":"2024-09-13T22:39:24.368074+0000","last_active":"2024-09-13T22:40:17.704358+0000","last_peered":"2024-09-13T22:40:17.704358+0000","last_clean":"2024-09-13T22:40:17.704358+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:40:17.704358+0000","last_undegraded":"2024-09-13T22:40:17.704358+0000","last_fullsized":"2024-09-13T22:40:17.704358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:39:24.367843+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:50.291892+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.709564+0000","last_change":"2024-09-13T22:39:06.025706+0000","last_active":"2024-09-13T22:40:17.709564+0000","last_peered":"2024-09-13T22:40:17.709564+0000","last_clean":"2024-09-13T22:40:17.709564+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:40:17.709564+0000","last_undegraded":"2024-09-13T22:40:17.709564+0000","last_fullsized":"2024-09-13T22:40:17.709564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:06.025488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:06.025488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.356278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":97,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.709084+0000","last_change":"2024-09-13T22:39:12.041436+0000","last_active":"2024-09-13T22:40:17.709084+0000","last_peered":"2024-09-13T22:40:17.709084+0000","last_clean":"2024-09-13T22:40:17.709084+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:40:17.709084+0000","last_undegraded":"2024-09-13T22:40:17.709084+0000","last_fullsized":"2024-09-13T22:40:17.709084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:39:12.041252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.041252+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.755767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":126,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702555+0000","last_change":"2024-09-13T22:39:01.656159+0000","last_active":"2024-09-13T22:40:17.702555+0000","last_peered":"2024-09-13T22:40:17.702555+0000","last_clean":"2024-09-13T22:40:17.702555+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:40:17.702555+0000","last_undegraded":"2024-09-13T22:40:17.702555+0000","last_fullsized":"2024-09-13T22:40:17.702555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:39:01.655949+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.600034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702976+0000","last_change":"2024-09-13T22:39:25.685706+0000","last_active":"2024-09-13T22:40:17.702976+0000","last_peered":"2024-09-13T22:40:17.702976+0000","last_clean":"2024-09-13T22:40:17.702976+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:40:17.702976+0000","last_undegraded":"2024-09-13T22:40:17.702976+0000","last_fullsized":"2024-09-13T22:40:17.702976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:25.685432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.685432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:46.180606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":123,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481180+0000","last_change":"2024-09-13T22:39:10.618742+0000","last_active":"2024-09-13T22:40:17.481180+0000","last_peered":"2024-09-13T22:40:17.481180+0000","last_clean":"2024-09-13T22:40:17.481180+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:40:17.481180+0000","last_undegraded":"2024-09-13T22:40:17.481180+0000","last_fullsized":"2024-09-13T22:40:17.481180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:39:10.618592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.618592+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.736269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703244+0000","last_change":"2024-09-13T22:40:07.465921+0000","last_active":"2024-09-13T22:40:17.703244+0000","last_peered":"2024-09-13T22:40:17.703244+0000","last_clean":"2024-09-13T22:40:17.703244+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:40:17.703244+0000","last_undegraded":"2024-09-13T22:40:17.703244+0000","last_fullsized":"2024-09-13T22:40:17.703244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:07.465685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:40:07.465685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:33.865225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703867+0000","last_change":"2024-09-13T22:39:02.682368+0000","last_active":"2024-09-13T22:40:17.703867+0000","last_peered":"2024-09-13T22:40:17.703867+0000","last_clean":"2024-09-13T22:40:17.703867+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:40:17.703867+0000","last_undegraded":"2024-09-13T22:40:17.703867+0000","last_fullsized":"2024-09-13T22:40:17.703867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.682170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.682170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:26.706353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706377+0000","last_change":"2024-09-13T22:39:45.281595+0000","last_active":"2024-09-13T22:40:17.706377+0000","last_peered":"2024-09-13T22:40:17.706377+0000","last_clean":"2024-09-13T22:40:17.706377+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:40:17.706377+0000","last_undegraded":"2024-09-13T22:40:17.706377+0000","last_fullsized":"2024-09-13T22:40:17.706377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:45.281512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:45.281512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:55.823803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"82'18","reported_seq":120,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.708142+0000","last_change":"2024-09-13T22:39:19.966260+0000","last_active":"2024-09-13T22:40:17.708142+0000","last_peered":"2024-09-13T22:40:17.708142+0000","last_clean":"2024-09-13T22:40:17.708142+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:40:17.708142+0000","last_undegraded":"2024-09-13T22:40:17.708142+0000","last_fullsized":"2024-09-13T22:40:17.708142+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-13T22:39:19.966043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.966043+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.842672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344045+0000","last_change":"2024-09-13T22:39:15.472999+0000","last_active":"2024-09-13T22:40:17.344045+0000","last_peered":"2024-09-13T22:40:17.344045+0000","last_clean":"2024-09-13T22:40:17.344045+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:40:17.344045+0000","last_undegraded":"2024-09-13T22:40:17.344045+0000","last_fullsized":"2024-09-13T22:40:17.344045+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:15.472777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.472777+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:41.403750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345509+0000","last_change":"2024-09-13T22:38:54.478611+0000","last_active":"2024-09-13T22:40:17.345509+0000","last_peered":"2024-09-13T22:40:17.345509+0000","last_clean":"2024-09-13T22:40:17.345509+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:40:17.345509+0000","last_undegraded":"2024-09-13T22:40:17.345509+0000","last_fullsized":"2024-09-13T22:40:17.345509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:54.478424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.478424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.387064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345034+0000","last_change":"2024-09-13T22:39:20.748359+0000","last_active":"2024-09-13T22:40:17.345034+0000","last_peered":"2024-09-13T22:40:17.345034+0000","last_clean":"2024-09-13T22:40:17.345034+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:40:17.345034+0000","last_undegraded":"2024-09-13T22:40:17.345034+0000","last_fullsized":"2024-09-13T22:40:17.345034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:20.748082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:20.748082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:30.962778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":108,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702979+0000","last_change":"2024-09-13T22:39:05.725791+0000","last_active":"2024-09-13T22:40:17.702979+0000","last_peered":"2024-09-13T22:40:17.702979+0000","last_clean":"2024-09-13T22:40:17.702979+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:40:17.702979+0000","last_undegraded":"2024-09-13T22:40:17.702979+0000","last_fullsized":"2024-09-13T22:40:17.702979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:39:05.725568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.725568+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.343610+0000","last_change":"2024-09-13T22:39:34.703390+0000","last_active":"2024-09-13T22:40:17.343610+0000","last_peered":"2024-09-13T22:40:17.343610+0000","last_clean":"2024-09-13T22:40:17.343610+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:40:17.343610+0000","last_undegraded":"2024-09-13T22:40:17.343610+0000","last_fullsized":"2024-09-13T22:40:17.343610+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:26.772349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:26.772349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.725362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704200+0000","last_change":"2024-09-13T22:39:12.302084+0000","last_active":"2024-09-13T22:40:17.704200+0000","last_peered":"2024-09-13T22:40:17.704200+0000","last_clean":"2024-09-13T22:40:17.704200+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:40:17.704200+0000","last_undegraded":"2024-09-13T22:40:17.704200+0000","last_fullsized":"2024-09-13T22:40:17.704200+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.301909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.301909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.703405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.708641+0000","last_change":"2024-09-13T22:40:11.075524+0000","last_active":"2024-09-13T22:40:17.708641+0000","last_peered":"2024-09-13T22:40:17.708641+0000","last_clean":"2024-09-13T22:40:17.708641+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:40:17.708641+0000","last_undegraded":"2024-09-13T22:40:17.708641+0000","last_fullsized":"2024-09-13T22:40:17.708641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:40:11.075324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:33.797037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":102,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.705573+0000","last_change":"2024-09-13T22:39:04.843005+0000","last_active":"2024-09-13T22:40:17.705573+0000","last_peered":"2024-09-13T22:40:17.705573+0000","last_clean":"2024-09-13T22:40:17.705573+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:40:17.705573+0000","last_undegraded":"2024-09-13T22:40:17.705573+0000","last_fullsized":"2024-09-13T22:40:17.705573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:04.842757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.842757+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.409317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703188+0000","last_change":"2024-09-13T22:40:15.593108+0000","last_active":"2024-09-13T22:40:17.703188+0000","last_peered":"2024-09-13T22:40:17.703188+0000","last_clean":"2024-09-13T22:40:17.703188+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:40:17.703188+0000","last_undegraded":"2024-09-13T22:40:17.703188+0000","last_fullsized":"2024-09-13T22:40:17.703188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:15.592907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:15.592907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:36.118376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":352,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344559+0000","last_change":"2024-09-13T22:40:05.603351+0000","last_active":"2024-09-13T22:40:17.344559+0000","last_peered":"2024-09-13T22:40:17.344559+0000","last_clean":"2024-09-13T22:40:17.344559+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:40:17.344559+0000","last_undegraded":"2024-09-13T22:40:17.344559+0000","last_fullsized":"2024-09-13T22:40:17.344559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:40:05.603139+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:40:05.603139+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:07.244070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345021+0000","last_change":"2024-09-13T22:38:56.676371+0000","last_active":"2024-09-13T22:40:17.345021+0000","last_peered":"2024-09-13T22:40:17.345021+0000","last_clean":"2024-09-13T22:40:17.345021+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:40:17.345021+0000","last_undegraded":"2024-09-13T22:40:17.345021+0000","last_fullsized":"2024-09-13T22:40:17.345021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:38:56.676172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703913+0000","last_change":"2024-09-13T22:40:08.497019+0000","last_active":"2024-09-13T22:40:17.703913+0000","last_peered":"2024-09-13T22:40:17.703913+0000","last_clean":"2024-09-13T22:40:17.703913+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:40:17.703913+0000","last_undegraded":"2024-09-13T22:40:17.703913+0000","last_fullsized":"2024-09-13T22:40:17.703913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:08.496742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:08.496742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:20.536460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":115,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345621+0000","last_change":"2024-09-13T22:39:42.666727+0000","last_active":"2024-09-13T22:40:17.345621+0000","last_peered":"2024-09-13T22:40:17.345621+0000","last_clean":"2024-09-13T22:40:17.345621+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:40:17.345621+0000","last_undegraded":"2024-09-13T22:40:17.345621+0000","last_fullsized":"2024-09-13T22:40:17.345621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:39:42.666504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:42.666504+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.765205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345615+0000","last_change":"2024-09-13T22:39:47.817369+0000","last_active":"2024-09-13T22:40:17.345615+0000","last_peered":"2024-09-13T22:40:17.345615+0000","last_clean":"2024-09-13T22:40:17.345615+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:40:17.345615+0000","last_undegraded":"2024-09-13T22:40:17.345615+0000","last_fullsized":"2024-09-13T22:40:17.345615+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:47.817287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:47.817287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.198399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344634+0000","last_change":"2024-09-13T22:39:12.437612+0000","last_active":"2024-09-13T22:40:17.344634+0000","last_peered":"2024-09-13T22:40:17.344634+0000","last_clean":"2024-09-13T22:40:17.344634+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:40:17.344634+0000","last_undegraded":"2024-09-13T22:40:17.344634+0000","last_fullsized":"2024-09-13T22:40:17.344634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:12.437298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:12.437298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.840182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344389+0000","last_change":"2024-09-13T22:39:54.772601+0000","last_active":"2024-09-13T22:40:17.344389+0000","last_peered":"2024-09-13T22:40:17.344389+0000","last_clean":"2024-09-13T22:40:17.344389+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:40:17.344389+0000","last_undegraded":"2024-09-13T22:40:17.344389+0000","last_fullsized":"2024-09-13T22:40:17.344389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:54.772409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:54.772409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:01.661911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":102,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703507+0000","last_change":"2024-09-13T22:39:09.333581+0000","last_active":"2024-09-13T22:40:17.703507+0000","last_peered":"2024-09-13T22:40:17.703507+0000","last_clean":"2024-09-13T22:40:17.703507+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:40:17.703507+0000","last_undegraded":"2024-09-13T22:40:17.703507+0000","last_fullsized":"2024-09-13T22:40:17.703507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:09.333359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.333359+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.270070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.710232+0000","last_change":"2024-09-13T22:40:13.057805+0000","last_active":"2024-09-13T22:40:17.710232+0000","last_peered":"2024-09-13T22:40:17.710232+0000","last_clean":"2024-09-13T22:40:17.710232+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:40:17.710232+0000","last_undegraded":"2024-09-13T22:40:17.710232+0000","last_fullsized":"2024-09-13T22:40:17.710232+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:13.057611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:13.057611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:20.204584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344181+0000","last_change":"2024-09-13T22:40:11.879840+0000","last_active":"2024-09-13T22:40:17.344181+0000","last_peered":"2024-09-13T22:40:17.344181+0000","last_clean":"2024-09-13T22:40:17.344181+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:40:17.344181+0000","last_undegraded":"2024-09-13T22:40:17.344181+0000","last_fullsized":"2024-09-13T22:40:17.344181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:11.879646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:11.879646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:41.651249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702699+0000","last_change":"2024-09-13T22:40:12.713650+0000","last_active":"2024-09-13T22:40:17.702699+0000","last_peered":"2024-09-13T22:40:17.702699+0000","last_clean":"2024-09-13T22:40:17.702699+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:40:17.702699+0000","last_undegraded":"2024-09-13T22:40:17.702699+0000","last_fullsized":"2024-09-13T22:40:17.702699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:12.713324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:12.713324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:19.316363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":119,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345587+0000","last_change":"2024-09-13T22:39:46.684767+0000","last_active":"2024-09-13T22:40:17.345587+0000","last_peered":"2024-09-13T22:40:17.345587+0000","last_clean":"2024-09-13T22:40:17.345587+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:40:17.345587+0000","last_undegraded":"2024-09-13T22:40:17.345587+0000","last_fullsized":"2024-09-13T22:40:17.345587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:39:46.684514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:52.368244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345898+0000","last_change":"2024-09-13T22:39:04.517758+0000","last_active":"2024-09-13T22:40:17.345898+0000","last_peered":"2024-09-13T22:40:17.345898+0000","last_clean":"2024-09-13T22:40:17.345898+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:40:17.345898+0000","last_undegraded":"2024-09-13T22:40:17.345898+0000","last_fullsized":"2024-09-13T22:40:17.345898+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:04.517658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:04.517658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.659731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":133,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.489467+0000","last_change":"2024-09-13T22:40:17.489534+0000","last_active":"2024-09-13T22:40:17.489467+0000","last_peered":"2024-09-13T22:40:17.489467+0000","last_clean":"2024-09-13T22:40:17.489467+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:40:17.489467+0000","last_undegraded":"2024-09-13T22:40:17.489467+0000","last_fullsized":"2024-09-13T22:40:17.489467+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:40:17.489324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:17.489324+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:21.623552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702627+0000","last_change":"2024-09-13T22:39:10.297437+0000","last_active":"2024-09-13T22:40:17.702627+0000","last_peered":"2024-09-13T22:40:17.702627+0000","last_clean":"2024-09-13T22:40:17.702627+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:40:17.702627+0000","last_undegraded":"2024-09-13T22:40:17.702627+0000","last_fullsized":"2024-09-13T22:40:17.702627+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:10.297219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.297219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.668374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":114,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.346269+0000","last_change":"2024-09-13T22:40:14.564516+0000","last_active":"2024-09-13T22:40:17.346269+0000","last_peered":"2024-09-13T22:40:17.346269+0000","last_clean":"2024-09-13T22:40:17.346269+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:40:17.346269+0000","last_undegraded":"2024-09-13T22:40:17.346269+0000","last_fullsized":"2024-09-13T22:40:17.346269+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:40:14.564287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:40:14.564287+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:32.887951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.482768+0000","last_change":"2024-09-13T22:39:00.654413+0000","last_active":"2024-09-13T22:40:17.482768+0000","last_peered":"2024-09-13T22:40:17.482768+0000","last_clean":"2024-09-13T22:40:17.482768+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:40:17.482768+0000","last_undegraded":"2024-09-13T22:40:17.482768+0000","last_fullsized":"2024-09-13T22:40:17.482768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'2","last_scrub_stamp":"2024-09-13T22:39:00.654142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:00.654142+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344558+0000","last_change":"2024-09-13T22:39:21.523815+0000","last_active":"2024-09-13T22:40:17.344558+0000","last_peered":"2024-09-13T22:40:17.344558+0000","last_clean":"2024-09-13T22:40:17.344558+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:40:17.344558+0000","last_undegraded":"2024-09-13T22:40:17.344558+0000","last_fullsized":"2024-09-13T22:40:17.344558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.523625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.523625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:51.184833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706003+0000","last_change":"2024-09-13T22:39:18.998697+0000","last_active":"2024-09-13T22:40:17.706003+0000","last_peered":"2024-09-13T22:40:17.706003+0000","last_clean":"2024-09-13T22:40:17.706003+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:40:17.706003+0000","last_undegraded":"2024-09-13T22:40:17.706003+0000","last_fullsized":"2024-09-13T22:40:17.706003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.998416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:33.890380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"82'11","reported_seq":106,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344763+0000","last_change":"2024-09-13T22:39:10.823267+0000","last_active":"2024-09-13T22:40:17.344763+0000","last_peered":"2024-09-13T22:40:17.344763+0000","last_clean":"2024-09-13T22:40:17.344763+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:40:17.344763+0000","last_undegraded":"2024-09-13T22:40:17.344763+0000","last_fullsized":"2024-09-13T22:40:17.344763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:10.823176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:10.823176+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:23.690199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344351+0000","last_change":"2024-09-13T22:40:14.845637+0000","last_active":"2024-09-13T22:40:17.344351+0000","last_peered":"2024-09-13T22:40:17.344351+0000","last_clean":"2024-09-13T22:40:17.344351+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:40:17.344351+0000","last_undegraded":"2024-09-13T22:40:17.344351+0000","last_fullsized":"2024-09-13T22:40:17.344351+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:14.845505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:14.845505+0000","last_clean_scrub_stamp":"2024-09-13T22:40:14.845505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:43.600974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707152+0000","last_change":"2024-09-13T22:39:11.903829+0000","last_active":"2024-09-13T22:40:17.707152+0000","last_peered":"2024-09-13T22:40:17.707152+0000","last_clean":"2024-09-13T22:40:17.707152+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:40:17.707152+0000","last_undegraded":"2024-09-13T22:40:17.707152+0000","last_fullsized":"2024-09-13T22:40:17.707152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.903758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.903758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:21.502704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345059+0000","last_change":"2024-09-13T22:40:04.828734+0000","last_active":"2024-09-13T22:40:17.345059+0000","last_peered":"2024-09-13T22:40:17.345059+0000","last_clean":"2024-09-13T22:40:17.345059+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:40:17.345059+0000","last_undegraded":"2024-09-13T22:40:17.345059+0000","last_fullsized":"2024-09-13T22:40:17.345059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:04.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:04.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:17.849951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.485071+0000","last_change":"2024-09-13T22:40:09.776987+0000","last_active":"2024-09-13T22:40:17.485071+0000","last_peered":"2024-09-13T22:40:17.485071+0000","last_clean":"2024-09-13T22:40:17.485071+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:40:17.485071+0000","last_undegraded":"2024-09-13T22:40:17.485071+0000","last_fullsized":"2024-09-13T22:40:17.485071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:40:09.776771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:40:09.776771+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:31.771703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.485583+0000","last_change":"2024-09-13T22:39:18.864519+0000","last_active":"2024-09-13T22:40:17.485583+0000","last_peered":"2024-09-13T22:40:17.485583+0000","last_clean":"2024-09-13T22:40:17.485583+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:40:17.485583+0000","last_undegraded":"2024-09-13T22:40:17.485583+0000","last_fullsized":"2024-09-13T22:40:17.485583+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.864331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.864331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.702214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703649+0000","last_change":"2024-09-13T22:39:31.439089+0000","last_active":"2024-09-13T22:40:17.703649+0000","last_peered":"2024-09-13T22:40:17.703649+0000","last_clean":"2024-09-13T22:40:17.703649+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:40:17.703649+0000","last_undegraded":"2024-09-13T22:40:17.703649+0000","last_fullsized":"2024-09-13T22:40:17.703649+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:31.438868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:31.438868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:45.289969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345011+0000","last_change":"2024-09-13T22:39:11.834762+0000","last_active":"2024-09-13T22:40:17.345011+0000","last_peered":"2024-09-13T22:40:17.345011+0000","last_clean":"2024-09-13T22:40:17.345011+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:40:17.345011+0000","last_undegraded":"2024-09-13T22:40:17.345011+0000","last_fullsized":"2024-09-13T22:40:17.345011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:11.834629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:11.834629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:37.419658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":106,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345141+0000","last_change":"2024-09-13T22:39:30.707102+0000","last_active":"2024-09-13T22:40:17.345141+0000","last_peered":"2024-09-13T22:40:17.345141+0000","last_clean":"2024-09-13T22:40:17.345141+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:40:17.345141+0000","last_undegraded":"2024-09-13T22:40:17.345141+0000","last_fullsized":"2024-09-13T22:40:17.345141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:30.706887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:30.706887+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.189746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703159+0000","last_change":"2024-09-13T22:39:15.307350+0000","last_active":"2024-09-13T22:40:17.703159+0000","last_peered":"2024-09-13T22:40:17.703159+0000","last_clean":"2024-09-13T22:40:17.703159+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:40:17.703159+0000","last_undegraded":"2024-09-13T22:40:17.703159+0000","last_fullsized":"2024-09-13T22:40:17.703159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.307195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.307195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.086503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":136,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703823+0000","last_change":"2024-09-13T22:40:03.443131+0000","last_active":"2024-09-13T22:40:17.703823+0000","last_peered":"2024-09-13T22:40:17.703823+0000","last_clean":"2024-09-13T22:40:17.703823+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:40:17.703823+0000","last_undegraded":"2024-09-13T22:40:17.703823+0000","last_fullsized":"2024-09-13T22:40:17.703823+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:40:03.442922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:03.442922+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.024934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706022+0000","last_change":"2024-09-13T22:39:18.688333+0000","last_active":"2024-09-13T22:40:17.706022+0000","last_peered":"2024-09-13T22:40:17.706022+0000","last_clean":"2024-09-13T22:40:17.706022+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:40:17.706022+0000","last_undegraded":"2024-09-13T22:40:17.706022+0000","last_fullsized":"2024-09-13T22:40:17.706022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.688141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.688141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.166952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703422+0000","last_change":"2024-09-13T22:38:54.600790+0000","last_active":"2024-09-13T22:40:17.703422+0000","last_peered":"2024-09-13T22:40:17.703422+0000","last_clean":"2024-09-13T22:40:17.703422+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:40:17.703422+0000","last_undegraded":"2024-09-13T22:40:17.703422+0000","last_fullsized":"2024-09-13T22:40:17.703422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:38:54.600571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:38:54.600571+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703762+0000","last_change":"2024-09-13T22:39:21.654486+0000","last_active":"2024-09-13T22:40:17.703762+0000","last_peered":"2024-09-13T22:40:17.703762+0000","last_clean":"2024-09-13T22:40:17.703762+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:40:17.703762+0000","last_undegraded":"2024-09-13T22:40:17.703762+0000","last_fullsized":"2024-09-13T22:40:17.703762+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.654273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.654273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.171794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":133,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704681+0000","last_change":"2024-09-13T22:40:04.236057+0000","last_active":"2024-09-13T22:40:17.704681+0000","last_peered":"2024-09-13T22:40:17.704681+0000","last_clean":"2024-09-13T22:40:17.704681+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:40:17.704681+0000","last_undegraded":"2024-09-13T22:40:17.704681+0000","last_fullsized":"2024-09-13T22:40:17.704681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:40:04.235854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:04.235854+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:10.334937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481934+0000","last_change":"2024-09-13T22:39:17.629172+0000","last_active":"2024-09-13T22:40:17.481934+0000","last_peered":"2024-09-13T22:40:17.481934+0000","last_clean":"2024-09-13T22:40:17.481934+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:40:17.481934+0000","last_undegraded":"2024-09-13T22:40:17.481934+0000","last_fullsized":"2024-09-13T22:40:17.481934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:17.628975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:17.628975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:18.565445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345339+0000","last_change":"2024-09-13T22:39:25.745828+0000","last_active":"2024-09-13T22:40:17.345339+0000","last_peered":"2024-09-13T22:40:17.345339+0000","last_clean":"2024-09-13T22:40:17.345339+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:40:17.345339+0000","last_undegraded":"2024-09-13T22:40:17.345339+0000","last_fullsized":"2024-09-13T22:40:17.345339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:25.745613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:25.745613+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:49.997201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.486086+0000","last_change":"2024-09-13T22:39:15.852762+0000","last_active":"2024-09-13T22:40:17.486086+0000","last_peered":"2024-09-13T22:40:17.486086+0000","last_clean":"2024-09-13T22:40:17.486086+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:40:17.486086+0000","last_undegraded":"2024-09-13T22:40:17.486086+0000","last_fullsized":"2024-09-13T22:40:17.486086+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:15.852596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:15.852596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.570592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706126+0000","last_change":"2024-09-13T22:39:29.136457+0000","last_active":"2024-09-13T22:40:17.706126+0000","last_peered":"2024-09-13T22:40:17.706126+0000","last_clean":"2024-09-13T22:40:17.706126+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:40:17.706126+0000","last_undegraded":"2024-09-13T22:40:17.706126+0000","last_fullsized":"2024-09-13T22:40:17.706126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.136366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:32.072781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702048+0000","last_change":"2024-09-13T22:39:27.605785+0000","last_active":"2024-09-13T22:40:17.702048+0000","last_peered":"2024-09-13T22:40:17.702048+0000","last_clean":"2024-09-13T22:40:17.702048+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:40:17.702048+0000","last_undegraded":"2024-09-13T22:40:17.702048+0000","last_fullsized":"2024-09-13T22:40:17.702048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:27.605612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:27.605612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:57.057619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":110,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481909+0000","last_change":"2024-09-13T22:39:48.728707+0000","last_active":"2024-09-13T22:40:17.481909+0000","last_peered":"2024-09-13T22:40:17.481909+0000","last_clean":"2024-09-13T22:40:17.481909+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:40:17.481909+0000","last_undegraded":"2024-09-13T22:40:17.481909+0000","last_fullsized":"2024-09-13T22:40:17.481909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:39:48.728497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:48.728497+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:59.833414+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.710099+0000","last_change":"2024-09-13T22:39:13.008954+0000","last_active":"2024-09-13T22:40:17.710099+0000","last_peered":"2024-09-13T22:40:17.710099+0000","last_clean":"2024-09-13T22:40:17.710099+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:40:17.710099+0000","last_undegraded":"2024-09-13T22:40:17.710099+0000","last_fullsized":"2024-09-13T22:40:17.710099+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:13.008764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.008764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:18.062805+0000","last_change":"2024-09-13T22:40:18.062857+0000","last_active":"2024-09-13T22:40:18.062805+0000","last_peered":"2024-09-13T22:40:18.062805+0000","last_clean":"2024-09-13T22:40:18.062805+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:40:18.062805+0000","last_undegraded":"2024-09-13T22:40:18.062805+0000","last_fullsized":"2024-09-13T22:40:18.062805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:18.062677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:18.062677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:38.524142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703963+0000","last_change":"2024-09-13T22:38:52.760354+0000","last_active":"2024-09-13T22:40:17.703963+0000","last_peered":"2024-09-13T22:40:17.703963+0000","last_clean":"2024-09-13T22:40:17.703963+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:40:17.703963+0000","last_undegraded":"2024-09-13T22:40:17.703963+0000","last_fullsized":"2024-09-13T22:40:17.703963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:52.760268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:52.760268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.705247+0000","last_change":"2024-09-13T22:39:07.744502+0000","last_active":"2024-09-13T22:40:17.705247+0000","last_peered":"2024-09-13T22:40:17.705247+0000","last_clean":"2024-09-13T22:40:17.705247+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:40:17.705247+0000","last_undegraded":"2024-09-13T22:40:17.705247+0000","last_fullsized":"2024-09-13T22:40:17.705247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:39:07.744124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.744124+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:24.659429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.704072+0000","last_change":"2024-09-13T22:39:21.378052+0000","last_active":"2024-09-13T22:40:17.704072+0000","last_peered":"2024-09-13T22:40:17.704072+0000","last_clean":"2024-09-13T22:40:17.704072+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:40:17.704072+0000","last_undegraded":"2024-09-13T22:40:17.704072+0000","last_fullsized":"2024-09-13T22:40:17.704072+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:21.377829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:21.377829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:36.746784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.485219+0000","last_change":"2024-09-13T22:39:05.855096+0000","last_active":"2024-09-13T22:40:17.485219+0000","last_peered":"2024-09-13T22:40:17.485219+0000","last_clean":"2024-09-13T22:40:17.485219+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:40:17.485219+0000","last_undegraded":"2024-09-13T22:40:17.485219+0000","last_fullsized":"2024-09-13T22:40:17.485219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:05.854893+0000","last_clean_scrub_stamp":"2024-09-13T22:39:05.854893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.334889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.344110+0000","last_change":"2024-09-13T22:40:04.568330+0000","last_active":"2024-09-13T22:40:17.344110+0000","last_peered":"2024-09-13T22:40:17.344110+0000","last_clean":"2024-09-13T22:40:17.344110+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:40:17.344110+0000","last_undegraded":"2024-09-13T22:40:17.344110+0000","last_fullsized":"2024-09-13T22:40:17.344110+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:04.568121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:04.568121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:30.948205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703811+0000","last_change":"2024-09-13T22:39:13.339189+0000","last_active":"2024-09-13T22:40:17.703811+0000","last_peered":"2024-09-13T22:40:17.703811+0000","last_clean":"2024-09-13T22:40:17.703811+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:40:17.703811+0000","last_undegraded":"2024-09-13T22:40:17.703811+0000","last_fullsized":"2024-09-13T22:40:17.703811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:13.338970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:13.338970+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:29.771039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.486583+0000","last_change":"2024-09-13T22:39:07.882650+0000","last_active":"2024-09-13T22:40:17.486583+0000","last_peered":"2024-09-13T22:40:17.486583+0000","last_clean":"2024-09-13T22:40:17.486583+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:40:17.486583+0000","last_undegraded":"2024-09-13T22:40:17.486583+0000","last_fullsized":"2024-09-13T22:40:17.486583+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:07.882460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:07.882460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:20.349629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.343618+0000","last_change":"2024-09-13T22:39:29.666045+0000","last_active":"2024-09-13T22:40:17.343618+0000","last_peered":"2024-09-13T22:40:17.343618+0000","last_clean":"2024-09-13T22:40:17.343618+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:40:17.343618+0000","last_undegraded":"2024-09-13T22:40:17.343618+0000","last_fullsized":"2024-09-13T22:40:17.343618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:29.665857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:29.665857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:42.766151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706248+0000","last_change":"2024-09-13T22:38:49.749377+0000","last_active":"2024-09-13T22:40:17.706248+0000","last_peered":"2024-09-13T22:40:17.706248+0000","last_clean":"2024-09-13T22:40:17.706248+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:40:17.706248+0000","last_undegraded":"2024-09-13T22:40:17.706248+0000","last_fullsized":"2024-09-13T22:40:17.706248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:38:49.749300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:49.749300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":81,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.345093+0000","last_change":"2024-09-13T22:39:02.697718+0000","last_active":"2024-09-13T22:40:17.345093+0000","last_peered":"2024-09-13T22:40:17.345093+0000","last_clean":"2024-09-13T22:40:17.345093+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:40:17.345093+0000","last_undegraded":"2024-09-13T22:40:17.345093+0000","last_fullsized":"2024-09-13T22:40:17.345093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:39:02.697574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.697574+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:25.049416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"82'15","reported_seq":113,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.708248+0000","last_change":"2024-09-13T22:39:16.963966+0000","last_active":"2024-09-13T22:40:17.708248+0000","last_peered":"2024-09-13T22:40:17.708248+0000","last_clean":"2024-09-13T22:40:17.708248+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:40:17.708248+0000","last_undegraded":"2024-09-13T22:40:17.708248+0000","last_fullsized":"2024-09-13T22:40:17.708248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:39:16.963643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:16.963643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.423517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.702438+0000","last_change":"2024-09-13T22:39:19.315930+0000","last_active":"2024-09-13T22:40:17.702438+0000","last_peered":"2024-09-13T22:40:17.702438+0000","last_clean":"2024-09-13T22:40:17.702438+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:40:17.702438+0000","last_undegraded":"2024-09-13T22:40:17.702438+0000","last_fullsized":"2024-09-13T22:40:17.702438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:19.315707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.315707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:39.650430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":128,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.486589+0000","last_change":"2024-09-13T22:39:19.870202+0000","last_active":"2024-09-13T22:40:17.486589+0000","last_peered":"2024-09-13T22:40:17.486589+0000","last_clean":"2024-09-13T22:40:17.486589+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:40:17.486589+0000","last_undegraded":"2024-09-13T22:40:17.486589+0000","last_fullsized":"2024-09-13T22:40:17.486589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:39:19.869975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.869975+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:27.843605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.708232+0000","last_change":"2024-09-13T22:39:18.975664+0000","last_active":"2024-09-13T22:40:17.708232+0000","last_peered":"2024-09-13T22:40:17.708232+0000","last_clean":"2024-09-13T22:40:17.708232+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:40:17.708232+0000","last_undegraded":"2024-09-13T22:40:17.708232+0000","last_fullsized":"2024-09-13T22:40:17.708232+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:18.975461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:18.975461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:19.889969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.707437+0000","last_change":"2024-09-13T22:39:28.141412+0000","last_active":"2024-09-13T22:40:17.707437+0000","last_peered":"2024-09-13T22:40:17.707437+0000","last_clean":"2024-09-13T22:40:17.707437+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:40:17.707437+0000","last_undegraded":"2024-09-13T22:40:17.707437+0000","last_fullsized":"2024-09-13T22:40:17.707437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:28.141327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:28.141327+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:28.373914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.705012+0000","last_change":"2024-09-13T22:39:09.960753+0000","last_active":"2024-09-13T22:40:17.705012+0000","last_peered":"2024-09-13T22:40:17.705012+0000","last_clean":"2024-09-13T22:40:17.705012+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:40:17.705012+0000","last_undegraded":"2024-09-13T22:40:17.705012+0000","last_fullsized":"2024-09-13T22:40:17.705012+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:39:09.960550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:09.960550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:38.827394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706260+0000","last_change":"2024-09-13T22:40:15.732935+0000","last_active":"2024-09-13T22:40:17.706260+0000","last_peered":"2024-09-13T22:40:17.706260+0000","last_clean":"2024-09-13T22:40:17.706260+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:40:17.706260+0000","last_undegraded":"2024-09-13T22:40:17.706260+0000","last_fullsized":"2024-09-13T22:40:17.706260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:15.732754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:40:15.732754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:20.696049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481444+0000","last_change":"2024-09-13T22:39:02.572852+0000","last_active":"2024-09-13T22:40:17.481444+0000","last_peered":"2024-09-13T22:40:17.481444+0000","last_clean":"2024-09-13T22:40:17.481444+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:40:17.481444+0000","last_undegraded":"2024-09-13T22:40:17.481444+0000","last_fullsized":"2024-09-13T22:40:17.481444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:02.572698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:39:02.572698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.486325+0000","last_change":"2024-09-13T22:39:59.762653+0000","last_active":"2024-09-13T22:40:17.486325+0000","last_peered":"2024-09-13T22:40:17.486325+0000","last_clean":"2024-09-13T22:40:17.486325+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:40:17.486325+0000","last_undegraded":"2024-09-13T22:40:17.486325+0000","last_fullsized":"2024-09-13T22:40:17.486325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:59.762383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:59.762383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:13.508489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.703363+0000","last_change":"2024-09-13T22:40:00.799746+0000","last_active":"2024-09-13T22:40:17.703363+0000","last_peered":"2024-09-13T22:40:17.703363+0000","last_clean":"2024-09-13T22:40:17.703363+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:40:17.703363+0000","last_undegraded":"2024-09-13T22:40:17.703363+0000","last_fullsized":"2024-09-13T22:40:17.703363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:40:00.799429+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:40:00.799429+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:04.815402+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.710112+0000","last_change":"2024-09-13T22:39:23.072821+0000","last_active":"2024-09-13T22:40:17.710112+0000","last_peered":"2024-09-13T22:40:17.710112+0000","last_clean":"2024-09-13T22:40:17.710112+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:40:17.710112+0000","last_undegraded":"2024-09-13T22:40:17.710112+0000","last_fullsized":"2024-09-13T22:40:17.710112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:39:23.072590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:34.247551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706842+0000","last_change":"2024-09-13T22:38:57.829670+0000","last_active":"2024-09-13T22:40:17.706842+0000","last_peered":"2024-09-13T22:40:17.706842+0000","last_clean":"2024-09-13T22:40:17.706842+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:40:17.706842+0000","last_undegraded":"2024-09-13T22:40:17.706842+0000","last_fullsized":"2024-09-13T22:40:17.706842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:38:57.829468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:38:57.829468+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.706748+0000","last_change":"2024-09-13T22:39:03.881680+0000","last_active":"2024-09-13T22:40:17.706748+0000","last_peered":"2024-09-13T22:40:17.706748+0000","last_clean":"2024-09-13T22:40:17.706748+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:40:17.706748+0000","last_undegraded":"2024-09-13T22:40:17.706748+0000","last_fullsized":"2024-09-13T22:40:17.706748+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:39:03.881493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:39:03.881493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:31.403295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481541+0000","last_change":"2024-09-13T22:39:19.638335+0000","last_active":"2024-09-13T22:40:17.481541+0000","last_peered":"2024-09-13T22:40:17.481541+0000","last_clean":"2024-09-13T22:40:17.481541+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:40:17.481541+0000","last_undegraded":"2024-09-13T22:40:17.481541+0000","last_fullsized":"2024-09-13T22:40:17.481541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:39:19.638128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:39:19.638128+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:40:22.918763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":91,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T22:40:17.481259+0000","last_change":"2024-09-13T22:40:12.625324+0000","last_active":"2024-09-13T22:40:17.481259+0000","last_peered":"2024-09-13T22:40:17.481259+0000","last_clean":"2024-09-13T22:40:17.481259+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:40:17.481259+0000","last_undegraded":"2024-09-13T22:40:17.481259+0000","last_fullsized":"2024-09-13T22:40:17.481259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:40:12.625120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:41:25.466463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":438,"ondisk_log_size":438,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808872,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32488,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712664,"statfs":{"total":95995035648,"available":95961767936,"internally_reserved":0,"allocated":1331200,"data_stored":928272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135923,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32460,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712692,"statfs":{"total":95995035648,"available":95961796608,"internally_reserved":0,"allocated":1302528,"data_stored":925950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528382,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":700416,"data_stored":329533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888136,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":745472,"data_stored":336514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084370,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31884,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713268,"statfs":{"total":95995035648,"available":95962386432,"internally_reserved":0,"allocated":712704,"data_stored":336405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378718,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":712704,"data_stored":336498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":724992,"data_stored":336893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28320,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716832,"statfs":{"total":95995035648,"available":95966035968,"internally_reserved":0,"allocated":1323008,"data_stored":928021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:40:22.925 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T22:40:22.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-13T22:40:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:23 smithi192 bash[24608]: cluster 2024-09-13T22:40:22.009477+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:40:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:23 smithi192 bash[24608]: cluster 2024-09-13T22:40:22.012493+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:40:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:23 smithi192 bash[24608]: audit 2024-09-13T22:40:22.218636+0000 mgr.y (mgr.14613) 103 : audit [DBG] from='client.14802 v1:172.21.15.178:0/2883233033' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:40:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:23 smithi192 bash[24608]: cluster 2024-09-13T22:40:22.444257+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:40:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:23 smithi192 bash[24608]: cluster 2024-09-13T22:40:22.446537+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:40:23.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:23 smithi192 bash[24608]: cluster 2024-09-13T22:40:22.462967+0000 mgr.y (mgr.14613) 104 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:40:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[29889]: cluster 2024-09-13T22:40:22.009477+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:40:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[29889]: cluster 2024-09-13T22:40:22.012493+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:40:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[29889]: audit 2024-09-13T22:40:22.218636+0000 mgr.y (mgr.14613) 103 : audit [DBG] from='client.14802 v1:172.21.15.178:0/2883233033' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:40:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[29889]: cluster 2024-09-13T22:40:22.444257+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:40:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[29889]: cluster 2024-09-13T22:40:22.446537+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:40:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[29889]: cluster 2024-09-13T22:40:22.462967+0000 mgr.y (mgr.14613) 104 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:40:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[22840]: cluster 2024-09-13T22:40:22.009477+0000 osd.1 (osd.1) 65 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:40:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[22840]: cluster 2024-09-13T22:40:22.012493+0000 osd.1 (osd.1) 66 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:40:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[22840]: audit 2024-09-13T22:40:22.218636+0000 mgr.y (mgr.14613) 103 : audit [DBG] from='client.14802 v1:172.21.15.178:0/2883233033' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:40:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[22840]: cluster 2024-09-13T22:40:22.444257+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:40:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[22840]: cluster 2024-09-13T22:40:22.446537+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:40:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:23 smithi178 bash[22840]: cluster 2024-09-13T22:40:22.462967+0000 mgr.y (mgr.14613) 104 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:40:25.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:40:24 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:40:25.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:25 smithi192 bash[24608]: cluster 2024-09-13T22:40:24.464496+0000 mgr.y (mgr.14613) 105 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:25.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:25 smithi178 bash[29889]: cluster 2024-09-13T22:40:24.464496+0000 mgr.y (mgr.14613) 105 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:25.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:25 smithi178 bash[22840]: cluster 2024-09-13T22:40:24.464496+0000 mgr.y (mgr.14613) 105 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:26 smithi192 bash[24608]: audit 2024-09-13T22:40:24.906279+0000 mgr.y (mgr.14613) 106 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:26.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:26 smithi192 bash[24608]: cluster 2024-09-13T22:40:25.415851+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:40:26.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:26 smithi192 bash[24608]: cluster 2024-09-13T22:40:25.419259+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:40:26.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:26 smithi178 bash[22840]: audit 2024-09-13T22:40:24.906279+0000 mgr.y (mgr.14613) 106 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:26.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:26 smithi178 bash[22840]: cluster 2024-09-13T22:40:25.415851+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:40:26.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:26 smithi178 bash[22840]: cluster 2024-09-13T22:40:25.419259+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:40:26.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:26 smithi178 bash[29889]: audit 2024-09-13T22:40:24.906279+0000 mgr.y (mgr.14613) 106 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:26.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:26 smithi178 bash[29889]: cluster 2024-09-13T22:40:25.415851+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:40:26.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:26 smithi178 bash[29889]: cluster 2024-09-13T22:40:25.419259+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:40:27.732 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.465309+0000 mgr.y (mgr.14613) 107 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.492882+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.503348+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.707103+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.710957+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.732079+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 deep-scrub starts 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.734286+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 deep-scrub ok 2024-09-13T22:40:27.883 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.827288+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[29889]: cluster 2024-09-13T22:40:26.829621+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.465309+0000 mgr.y (mgr.14613) 107 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.492882+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.503348+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.707103+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.710957+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.732079+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 deep-scrub starts 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.734286+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 deep-scrub ok 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.827288+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T22:40:27.884 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:27 smithi178 bash[22840]: cluster 2024-09-13T22:40:26.829621+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.465309+0000 mgr.y (mgr.14613) 107 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.492882+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.503348+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.707103+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.710957+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.732079+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 deep-scrub starts 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.734286+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 deep-scrub ok 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.827288+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T22:40:27.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:27 smithi192 bash[24608]: cluster 2024-09-13T22:40:26.829621+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:27.404949+0000 osd.3 (osd.3) 65 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:27.408844+0000 osd.3 (osd.3) 66 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:27.678224+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:27.680571+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.009033+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.022222+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.466264+0000 mgr.y (mgr.14613) 108 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.645056+0000 osd.5 (osd.5) 67 : cluster [DBG] 6.a scrub starts 2024-09-13T22:40:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:28 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.647301+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.a scrub ok 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:27.404949+0000 osd.3 (osd.3) 65 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:27.408844+0000 osd.3 (osd.3) 66 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:27.678224+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:27.680571+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.009033+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.022222+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.466264+0000 mgr.y (mgr.14613) 108 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.645056+0000 osd.5 (osd.5) 67 : cluster [DBG] 6.a scrub starts 2024-09-13T22:40:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.647301+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.a scrub ok 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:27.404949+0000 osd.3 (osd.3) 65 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:27.408844+0000 osd.3 (osd.3) 66 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:27.678224+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:27.680571+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.009033+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.022222+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.466264+0000 mgr.y (mgr.14613) 108 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.645056+0000 osd.5 (osd.5) 67 : cluster [DBG] 6.a scrub starts 2024-09-13T22:40:29.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:28 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.647301+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.a scrub ok 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:epoch 112 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:fsid 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:created 2024-09-13T22:27:31.500176+0000 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:modified 2024-09-13T22:40:21.353444+0000 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:crush_version 18 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:full_ratio 0.95 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:backfillfull_ratio 0.9 2024-09-13T22:40:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:nearfull_ratio 0.85 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:require_min_compat_client luminous 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:min_compat_client luminous 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:require_osd_release squid 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:stretch_mode_enabled false 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:pool 4 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:max_osd 8 2024-09-13T22:40:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0 up in weight 1 up_from 8 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6801/1489719354 v1:172.21.15.178:6802/1489719354 exists,up 394f2664-bba1-4b15-ace8-78f70f970cac 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1 up in weight 1 up_from 13 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6805/4187084287 v1:172.21.15.178:6806/4187084287 exists,up 8e65c08b-6431-4849-a6af-2da9cb1efdfa 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2 up in weight 1 up_from 19 up_thru 105 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6809/2303802730 v1:172.21.15.178:6810/2303802730 exists,up 40a79075-5102-4f6c-b41a-255102064624 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3 up in weight 1 up_from 28 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6813/320675660 v1:172.21.15.178:6814/320675660 exists,up 3962c796-6f97-4cc6-a186-4b6ef58d67ad 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4 up in weight 1 up_from 34 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6800/3593670527 v1:172.21.15.192:6801/3593670527 exists,up 1b89db2f-9e81-40b1-89bd-8de1dbff133b 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5 up in weight 1 up_from 45 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6804/3271425338 v1:172.21.15.192:6805/3271425338 exists,up efa849a2-d789-4fdd-acc7-04f2c1bebfa8 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6 up in weight 1 up_from 57 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6808/684205162 v1:172.21.15.192:6809/684205162 exists,up 54a26ae9-bf31-49cf-a924-901eed4a0c98 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7 up in weight 1 up_from 67 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6812/1303971738 v1:172.21.15.192:6813/1303971738 exists,up 13e146db-61ac-4fe6-8f26-cbf8d1623a52 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:pg_upmap_items 2.0 [7,2] 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/268210503 expires 2024-09-14T22:27:58.436396+0000 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:6800/2039160385 expires 2024-09-14T22:27:58.436396+0000 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/669877409 expires 2024-09-14T22:27:58.436396+0000 2024-09-13T22:40:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/129655740 expires 2024-09-14T22:38:32.396576+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/3632788039 expires 2024-09-14T22:28:30.168687+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:6800/388635774 expires 2024-09-14T22:28:30.168687+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/3265105147 expires 2024-09-14T22:38:32.396576+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/2959920382 expires 2024-09-14T22:27:58.436396+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/2494442689 expires 2024-09-14T22:38:32.396576+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:6800/2676020773 expires 2024-09-14T22:38:32.396576+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/2675812306 expires 2024-09-14T22:28:30.168687+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/1834442338 expires 2024-09-14T22:38:32.396576+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/2835351236 expires 2024-09-14T22:28:30.168687+0000 2024-09-13T22:40:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:blocklist 172.21.15.178:0/3836928220 expires 2024-09-14T22:38:32.396576+0000 2024-09-13T22:40:29.954 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6801/1489719354 v1:172.21.15.178:6802/1489719354 exists,up 394f2664-bba1-4b15-ace8-78f70f970cac', 'osd.1 up in weight 1 up_from 13 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6805/4187084287 v1:172.21.15.178:6806/4187084287 exists,up 8e65c08b-6431-4849-a6af-2da9cb1efdfa', 'osd.2 up in weight 1 up_from 19 up_thru 105 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6809/2303802730 v1:172.21.15.178:6810/2303802730 exists,up 40a79075-5102-4f6c-b41a-255102064624', 'osd.3 up in weight 1 up_from 28 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.178:6813/320675660 v1:172.21.15.178:6814/320675660 exists,up 3962c796-6f97-4cc6-a186-4b6ef58d67ad', 'osd.4 up in weight 1 up_from 34 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6800/3593670527 v1:172.21.15.192:6801/3593670527 exists,up 1b89db2f-9e81-40b1-89bd-8de1dbff133b', 'osd.5 up in weight 1 up_from 45 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6804/3271425338 v1:172.21.15.192:6805/3271425338 exists,up efa849a2-d789-4fdd-acc7-04f2c1bebfa8', 'osd.6 up in weight 1 up_from 57 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6808/684205162 v1:172.21.15.192:6809/684205162 exists,up 54a26ae9-bf31-49cf-a924-901eed4a0c98', 'osd.7 up in weight 1 up_from 67 up_thru 80 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6812/1303971738 v1:172.21.15.192:6813/1303971738 exists,up 13e146db-61ac-4fe6-8f26-cbf8d1623a52'] 2024-09-13T22:40:29.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.699937+0000 osd.4 (osd.4) 53 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: cluster 2024-09-13T22:40:28.704062+0000 osd.4 (osd.4) 54 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.033055+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.036066+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.207066+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.207161+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:30.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:29 smithi192 bash[24608]: audit 2024-09-13T22:40:29.246644+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.178:0/1607181754' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T22:40:30.177 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:40:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.699937+0000 osd.4 (osd.4) 53 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: cluster 2024-09-13T22:40:28.704062+0000 osd.4 (osd.4) 54 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.033055+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.036066+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.207066+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.207161+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[22840]: audit 2024-09-13T22:40:29.246644+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.178:0/1607181754' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.699937+0000 osd.4 (osd.4) 53 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: cluster 2024-09-13T22:40:28.704062+0000 osd.4 (osd.4) 54 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.033055+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.036066+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.207066+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.207161+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:30.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:29 smithi178 bash[29889]: audit 2024-09-13T22:40:29.246644+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.178:0/1607181754' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.411091+0000 osd.3 (osd.3) 67 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.415314+0000 osd.3 (osd.3) 68 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.501339+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.504498+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.669611+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.676491+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.757269+0000 osd.0 (osd.0) 61 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.761668+0000 osd.0 (osd.0) 62 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.783622+0000 osd.2 (osd.2) 47 : cluster [DBG] 6.f scrub starts 2024-09-13T22:40:31.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.786155+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.f scrub ok 2024-09-13T22:40:31.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:29.792953+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-13T22:40:31.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:30 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.466954+0000 mgr.y (mgr.14613) 109 : cluster [DBG] pgmap v80: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:40:31.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.411091+0000 osd.3 (osd.3) 67 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:40:31.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.415314+0000 osd.3 (osd.3) 68 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:40:31.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.501339+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:40:31.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.504498+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:40:31.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.669611+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T22:40:31.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.676491+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.757269+0000 osd.0 (osd.0) 61 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.761668+0000 osd.0 (osd.0) 62 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.783622+0000 osd.2 (osd.2) 47 : cluster [DBG] 6.f scrub starts 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.786155+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.f scrub ok 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:29.792953+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.466954+0000 mgr.y (mgr.14613) 109 : cluster [DBG] pgmap v80: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.411091+0000 osd.3 (osd.3) 67 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.415314+0000 osd.3 (osd.3) 68 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.501339+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.504498+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:40:31.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.669611+0000 osd.4 (osd.4) 55 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.676491+0000 osd.4 (osd.4) 56 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.757269+0000 osd.0 (osd.0) 61 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.761668+0000 osd.0 (osd.0) 62 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.783622+0000 osd.2 (osd.2) 47 : cluster [DBG] 6.f scrub starts 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.786155+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.f scrub ok 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:29.792953+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-13T22:40:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:30 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.466954+0000 mgr.y (mgr.14613) 109 : cluster [DBG] pgmap v80: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:40:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:31 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.393322+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:40:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:31 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.396769+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:40:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:31 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.788533+0000 osd.0 (osd.0) 63 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:40:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:31 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.791647+0000 osd.0 (osd.0) 64 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:40:32.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.393322+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:40:32.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.396769+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:40:32.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.788533+0000 osd.0 (osd.0) 63 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:40:32.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.791647+0000 osd.0 (osd.0) 64 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:40:32.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.393322+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:40:32.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.396769+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:40:32.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.788533+0000 osd.0 (osd.0) 63 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:40:32.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:31 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.791647+0000 osd.0 (osd.0) 64 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.897978+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:30.900254+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:31.378744+0000 osd.3 (osd.3) 71 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:31.383041+0000 osd.3 (osd.3) 72 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:31.528000+0000 osd.7 (osd.7) 67 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:31.530153+0000 osd.7 (osd.7) 68 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:31.833349+0000 osd.0 (osd.0) 65 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:31.836770+0000 osd.0 (osd.0) 66 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:40:33.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: cluster 2024-09-13T22:40:32.467651+0000 mgr.y (mgr.14613) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:33.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:32 smithi192 bash[24608]: audit 2024-09-13T22:40:32.720163+0000 mon.b (mon.1) 89 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.897978+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:30.900254+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:31.378744+0000 osd.3 (osd.3) 71 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:31.383041+0000 osd.3 (osd.3) 72 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:31.528000+0000 osd.7 (osd.7) 67 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:31.530153+0000 osd.7 (osd.7) 68 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:31.833349+0000 osd.0 (osd.0) 65 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:31.836770+0000 osd.0 (osd.0) 66 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:40:33.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: cluster 2024-09-13T22:40:32.467651+0000 mgr.y (mgr.14613) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[29889]: audit 2024-09-13T22:40:32.720163+0000 mon.b (mon.1) 89 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.897978+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:30.900254+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:31.378744+0000 osd.3 (osd.3) 71 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:31.383041+0000 osd.3 (osd.3) 72 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:31.528000+0000 osd.7 (osd.7) 67 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:31.530153+0000 osd.7 (osd.7) 68 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:31.833349+0000 osd.0 (osd.0) 65 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:31.836770+0000 osd.0 (osd.0) 66 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: cluster 2024-09-13T22:40:32.467651+0000 mgr.y (mgr.14613) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:33.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:32 smithi178 bash[22840]: audit 2024-09-13T22:40:32.720163+0000 mon.b (mon.1) 89 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:33 smithi192 bash[24608]: cluster 2024-09-13T22:40:32.639515+0000 osd.4 (osd.4) 57 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:40:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:33 smithi192 bash[24608]: cluster 2024-09-13T22:40:32.641938+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:40:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:33 smithi178 bash[29889]: cluster 2024-09-13T22:40:32.639515+0000 osd.4 (osd.4) 57 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:40:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:33 smithi178 bash[29889]: cluster 2024-09-13T22:40:32.641938+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:40:34.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:33 smithi178 bash[22840]: cluster 2024-09-13T22:40:32.639515+0000 osd.4 (osd.4) 57 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:40:34.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:33 smithi178 bash[22840]: cluster 2024-09-13T22:40:32.641938+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:40:34.759 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:35.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:40:34 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:40:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:34 smithi192 bash[24608]: cluster 2024-09-13T22:40:33.777656+0000 osd.2 (osd.2) 49 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:40:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:34 smithi192 bash[24608]: cluster 2024-09-13T22:40:33.779877+0000 osd.2 (osd.2) 50 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:40:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:34 smithi192 bash[24608]: cluster 2024-09-13T22:40:34.469182+0000 mgr.y (mgr.14613) 111 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:34 smithi178 bash[29889]: cluster 2024-09-13T22:40:33.777656+0000 osd.2 (osd.2) 49 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:40:35.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:34 smithi178 bash[29889]: cluster 2024-09-13T22:40:33.779877+0000 osd.2 (osd.2) 50 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:40:35.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:34 smithi178 bash[29889]: cluster 2024-09-13T22:40:34.469182+0000 mgr.y (mgr.14613) 111 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:34 smithi178 bash[22840]: cluster 2024-09-13T22:40:33.777656+0000 osd.2 (osd.2) 49 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:40:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:34 smithi178 bash[22840]: cluster 2024-09-13T22:40:33.779877+0000 osd.2 (osd.2) 50 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:40:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:34 smithi178 bash[22840]: cluster 2024-09-13T22:40:34.469182+0000 mgr.y (mgr.14613) 111 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:36.145 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:40:36.145 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "mon.a", 2024-09-13T22:40:36.145 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster": "ceph", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "admin_socket_mode": "", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_client_required": "cephx, none", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_cluster_required": "cephx", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_debug": "true", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_service_required": "cephx", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "auth_supported": "", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_aio": "true", 2024-09-13T22:40:36.146 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_aio_reap_max": "16", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_async_discard": "false", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_block_size": "4096", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_debug_aio": "false", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_enable_discard": "false", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_flock_retry": "3", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_inject_crash": "0", 2024-09-13T22:40:36.147 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_ioring": "false", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_ioring_hipri": "false", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bdev_type": "", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_allocator": "hybrid", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_buffered_io": "true", 2024-09-13T22:40:36.148 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_replay_recovery": "false", 2024-09-13T22:40:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_sync_write": "false", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_allocator": "hybrid", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T22:40:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_create": "true", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_db_create": "false", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_db_path": "", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_db_size": "0", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_path": "", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_size": "107374182400", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_wal_create": "false", 2024-09-13T22:40:36.151 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_wal_path": "", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bluefs": "true", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_autotune": "true", 2024-09-13T22:40:36.152 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_size": "0", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_type": "2q", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_clone_cow": "true", 2024-09-13T22:40:36.153 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_mode": "none", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_freelist": "false", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T22:40:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T22:40:36.155 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fail_eio": "false", 2024-09-13T22:40:36.156 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T22:40:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_max_blob_size": "0", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T22:40:36.158 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T22:40:36.159 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_spdk_mem": "512", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T22:40:36.160 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_tracing": "false", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T22:40:36.161 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T22:40:36.162 INFO:teuthology.orchestra.run.smithi178.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_cluster_require_version": "2", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_require_signatures": "false", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_require_version": "2", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_service_require_signatures": "false", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_service_require_version": "2", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cephx_sign_messages": "true", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "chdir": "", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "client_acl_type": "", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "client_asio_thread_count": "2", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "client_cache_mid": "0.750000", 2024-09-13T22:40:36.163 INFO:teuthology.orchestra.run.smithi178.stdout: "client_cache_size": "16384", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_caps_release_delay": "5", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_check_pool_perm": "true", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_debug_force_sync_read": "false", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_debug_getattr_caps": "false", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_debug_inject_features": "", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_die_on_failed_remount": "false", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_dirsize_rbytes": "true", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_force_lazyio": "false", 2024-09-13T22:40:36.164 INFO:teuthology.orchestra.run.smithi178.stdout: "client_fs": "", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_inject_release_failure": "false", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_max_inline_size": "4096", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_mds_namespace": "", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_metadata": "", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_mount_gid": "-1", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_mount_timeout": "300", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_mount_uid": "-1", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_mountpoint": "/", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_notify_timeout": "10", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_oc": "true", 2024-09-13T22:40:36.165 INFO:teuthology.orchestra.run.smithi178.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_oc_max_objects": "1000", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_oc_size": "209715200", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_permissions": "true", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_quota": "true", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_quota_df": "true", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_readahead_max_bytes": "0", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_readahead_max_periods": "4", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_readahead_min": "131072", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_reconnect_stale": "false", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_shutdown_timeout": "30", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_snapdir": ".snap", 2024-09-13T22:40:36.166 INFO:teuthology.orchestra.run.smithi178.stdout: "client_tick_interval": "1", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "client_trace": "", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "client_use_faked_inos": "false", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "client_use_random_mds": "false", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_graylog": "false", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_graylog_port": "12201", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_monitors": "default=true", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_syslog": "false", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "clog_to_syslog_level": "info", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "-", 2024-09-13T22:40:36.167 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_network": "", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_network_interface": "", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "compressor_zlib_isal": "false", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "compressor_zlib_level": "5", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "compressor_zstd_level": "1", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b893710f6933dd3672a2a633ce5ac92081563259207d21cd52290129179ca808", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T22:40:36.168 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_location": "", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_location_hook": "", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_location_hook_timeout": "10", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "daemonize": "false", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "daos_pool": "tank", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T22:40:36.169 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_alienstore": "0/5", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_asok": "1/5", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_asok_assert_abort": "false", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_auth": "1/5", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_bdev": "1/3", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_bluefs": "1/5", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_bluestore": "1/5", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_buffer": "0/1", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T22:40:36.170 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_cephsqlite": "0/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_client": "0/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_compressor": "1/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_context": "0/1", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_crush": "1/1", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_crypto": "1/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_cyanstore": "0/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_dpdk": "1/5", 2024-09-13T22:40:36.171 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_eventtrace": "1/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_filer": "0/1", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_filestore": "1/3", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_finisher": "1/1", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_fuse": "1/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_javaclient": "1/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_journal": "1/3", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_journaler": "0/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_kstore": "1/5", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_lockdep": "0/1", 2024-09-13T22:40:36.172 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mclock": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds_balancer": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds_locker": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds_log": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds_migrator": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_memstore": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mgr": "2/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mgrc": "1/5", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_mon": "20/20", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_monc": "0/10", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_ms": "1/1", 2024-09-13T22:40:36.173 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_none": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_objclass": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_objectcacher": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_objecter": "0/1", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_optracker": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_osd": "1/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_paxos": "20/20", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_perfcounter": "1/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_prioritycache": "1/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rados": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rbd": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T22:40:36.174 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rbd_replay": "0/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_refs": "0/0", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_reserver": "1/1", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw_access": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw_flight": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rgw_sync": "1/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_rocksdb": "4/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore": "0/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_backref": "0/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_cache": "0/5", 2024-09-13T22:40:36.175 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_device": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_epm": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_journal": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_lba": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_odata": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_omap": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_onode": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_t": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_seastore_tm": "0/5", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_striper": "0/1", 2024-09-13T22:40:36.176 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_test": "0/5", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_throttle": "1/1", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_timer": "0/1", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_tp": "0/5", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "debug_trace": "1/5", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "device_failure_prediction_mode": "none", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "err_to_graylog": "false", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "err_to_journald": "false", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "err_to_stderr": "true", 2024-09-13T22:40:36.177 INFO:teuthology.orchestra.run.smithi178.stdout: "err_to_syslog": "false", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "event_tracing": "false", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "exporter_http_port": "9926", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "exporter_prio_limit": "5", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "exporter_sort_metrics": "true", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "exporter_stats_period": "5", 2024-09-13T22:40:36.178 INFO:teuthology.orchestra.run.smithi178.stdout: "fake_statfs_for_testing": "0", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "fatal_signal_handlers": "true", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filer_max_purge_ops": "10", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filer_max_truncate_ops": "128", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_blackhole": "false", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_btrfs_snap": "true", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_caller_concurrency": "10", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T22:40:36.179 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_debug_omap_check": "false", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_debug_verify_split": "false", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_dump_file": "", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fadvise": "true", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fail_eio": "true", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fd_cache_size": "128", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fiemap": "false", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T22:40:36.180 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_inject_stall": "0", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_journal_parallel": "false", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_journal_trailing": "false", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_journal_writeahead": "false", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_kill_at": "0", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T22:40:36.181 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_merge_threshold": "-10", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_odsync_write": "false", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_omap_backend_path": "", 2024-09-13T22:40:36.182 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_op_threads": "2", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_punch_hole": "false", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T22:40:36.183 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_queue_max_ops": "50", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_seek_data_hole": "false", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_sloppy_crc": "false", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_splice": "false", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_split_multiple": "2", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_split_rand_factor": "20", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_update_to": "1000", 2024-09-13T22:40:36.184 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T22:40:36.185 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_xfs_extsize": "false", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "filestore_zfs_snap": "false", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fio_dir": "/tmp/fio", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_allow_other": "true", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_big_writes": "true", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_debug": "false", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_default_permissions": "false", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_disable_pagecache": "false", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_max_write": "0", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_multithreaded": "true", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_require_active_mds": "true", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_set_user_groups": "true", 2024-09-13T22:40:36.186 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_splice_move": "true", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_splice_read": "true", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_splice_write": "true", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "gss_target_name": "ceph", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_file": "", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_inject_failure": "0", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_interval": "5", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "host": "smithi178", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T22:40:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "inject_early_sigterm": "false", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "jaeger_agent_port": "6799", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "jaeger_tracing_enable": "false", 2024-09-13T22:40:36.188 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_aio": "true", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_align_min_size": "65536", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_block_align": "true", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_block_size": "4096", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_dio": "true", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_discard": "false", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_force_aio": "false", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_ignore_corruption": "false", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_max_write_entries": "100", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_replay_from": "0", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T22:40:36.189 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_write_header_frequency": "0", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "journal_zero_on_create": "false", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "journaler_prefetch_periods": "10", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "journaler_prezero_periods": "5", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "journaler_write_head_interval": "15", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "keyfile": "", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_backend": "rocksdb", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_max_bytes": "67108864", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_max_ops": "512", 2024-09-13T22:40:36.190 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_onode_map_size": "1024", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "kstore_sync_transaction": "false", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "librados_thread_count": "2", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "lockdep": "false", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "lockdep_force_backtrace": "false", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_coarse_timestamps": "true", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_flush_on_exit": "false", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_graylog_port": "12201", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_max_new": "1000", 2024-09-13T22:40:36.191 INFO:teuthology.orchestra.run.smithi178.stdout: "log_max_recent": "10000", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_stderr_prefix": "debug ", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_to_file": "true", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_to_graylog": "false", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_to_journald": "false", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_to_stderr": "false", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "log_to_syslog": "false", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_action_on_write_error": "1", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_alternate_name_max": "8192", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_asio_thread_count": "2", 2024-09-13T22:40:36.192 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_export_pin": "true", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_interval": "10", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_max": "-1", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_max_until": "-1", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_merge_size": "50", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T22:40:36.193 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_mode": "0", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_split_bits": "3", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_split_size": "10000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T22:40:36.194 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_mid": "0.700000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_trim_interval": "1", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T22:40:36.195 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_debug_auth_pins": "false", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_debug_frag": "false", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_debug_scatterstat": "false", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_debug_subtrees": "false", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_default_dir_hash": "2", 2024-09-13T22:40:36.196 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_defer_session_stale": "true", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dir_max_entries": "0", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dir_prefetch": "true", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_early_reply": "true", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_enable_op_tracker": "true", 2024-09-13T22:40:36.197 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_enforce_unique_name": "true", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T22:40:36.198 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_inject_health_dummy": "false", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_join_fs": "", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_journal_format": "1", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_create_at": "0", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_export_at": "0", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_import_at": "0", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_journal_at": "0", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T22:40:36.199 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_link_at": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_openc_at": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_rename_at": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_max_events": "-1", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_max_segments": "128", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_pause": "false", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_segment_size": "0", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T22:40:36.200 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_trim_threshold": "128", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_completed_requests": "100000", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_export_size": "20971520", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_file_recover": "32", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_purge_files": "64", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_purge_ops": "8192", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T22:40:36.201 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_metrics_update_interval": "2", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_min_caps_per_client": "100", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_numa_node": "-1", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_op_history_duration": "600", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_op_history_size": "20", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_op_log_threshold": "5", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_ping_grace": "15", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_ping_interval": "5", 2024-09-13T22:40:36.202 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_recall_max_caps": "30000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_replay_interval": "1.000000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_root_ino_gid": "0", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_root_ino_uid": "0", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T22:40:36.203 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_shutdown_check": "0", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_skip_ino": "0", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_snap_min_uid": "0", 2024-09-13T22:40:36.204 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_snap_rstat": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_symlink_recovery": "true", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_thrash_exports": "0", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_thrash_fragments": "0", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_tick_interval": "5.000000", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_valgrind_exit": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_verify_backtrace": "1", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_verify_scatter": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_wipe_sessions": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool_debug": "false", 2024-09-13T22:40:36.205 INFO:teuthology.orchestra.run.smithi178.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "memstore_page_set": "false", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "memstore_page_size": "65536", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_client_bytes": "134217728", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_client_messages": "512", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_disabled_modules": "", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T22:40:36.206 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_mds_messages": "128", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_mon_messages": "128", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_op_history_duration": "600", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_op_history_size": "20", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_op_log_threshold": "5", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T22:40:36.207 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_osd_messages": "8192", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_pool": "true", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_standby_modules": "true", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_stats_period": "5", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_stats_threshold": "5", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_tick_period": "2", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_allow_pool_delete": "true", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_allow_pool_size_one": "false", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T22:40:36.208 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_bytes": "104857600", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_directed_command_retry": "2", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_client_target_rank": "-1", 2024-09-13T22:40:36.209 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_file_level": "debug", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_journald": "false", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_stderr": "true", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-13T22:40:36.210 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_compact_on_start": "false", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_compact_on_trim": "true", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_cpu_threads": "4", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_data_avail_crit": "5", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_data_avail_warn": "5", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T22:40:36.211 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_dump_json": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_extra_checks": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_down_mkfs_grace": "60", 2024-09-13T22:40:36.212 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_down_uptime_grace": "60", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_election_default_strategy": "1", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_election_timeout": "5.000000", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_enable_op_tracker": "true", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_fake_pool_delete": "false", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_force_quorum_join": "false", 2024-09-13T22:40:36.213 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_health_detail_to_clog": "true", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_health_log_update_period": "5", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_health_max_detail": "50", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_health_to_clog": "true", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_host": "172.21.15.178:6789/0", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_host_override": "", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_initial_members": "", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T22:40:36.214 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_lease": "5.000000", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_log_full_interval": "50", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_log_max": "10000", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_log_max_summary": "50", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_log_epochs": "500", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T22:40:36.215 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_osd": "10000", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_memory_autotune": "true", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_memory_target": "2147483648", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mgr_digest_period": "5", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T22:40:36.216 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_min_osdmap_epochs": "50", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_op_complaint_time": "30", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_op_history_duration": "600", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_op_history_size": "20", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_op_log_threshold": "5", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T22:40:36.217 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_cache_size": "500", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T22:40:36.218 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_report_timeout": "900", 2024-09-13T22:40:36.219 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T22:40:36.220 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_reweight_max_osds": "4", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_scrub_interval": "86400", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_scrub_max_keys": "100", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_scrub_timeout": "300", 2024-09-13T22:40:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_session_timeout": "300", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_smart_report_timeout": "5", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_stretch_pool_size": "4", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_sync_debug": "false", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T22:40:36.222 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_tick_interval": "5", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T22:40:36.223 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_older_version": "true", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "monmap": "", 2024-09-13T22:40:36.224 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "motr_tracing_enabled": "false", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_op_threads": "3", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_cm": "false", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_device_name": "", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T22:40:36.225 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_sl": "3", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_rdma_type": "ib", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_async_reap_threshold": "5", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_before_connect": "false", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_ipv4": "true", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_ipv6": "false", 2024-09-13T22:40:36.226 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_msgr1": "true", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_msgr2": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_port_max": "7568", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_port_min": "6800", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_retry_count": "3", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_bind_retry_delay": "5", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_blackhole_client": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_blackhole_mds": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_blackhole_mgr": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_blackhole_mon": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_blackhole_osd": "false", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_client_mode": "crc secure", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T22:40:36.227 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_cluster_type": "", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_compress_secure": "false", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_crc_data": "true", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_crc_header": "true", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_die_on_bug": "true", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_die_on_old_message": "true", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T22:40:36.228 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_hugepages": "", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_lro": "true", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_pmd": "", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_port_id": "0", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T22:40:36.229 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_dump_on_send": "false", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_inject_delay_type": "", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_inject_network_congestion": "0", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_inject_socket_failures": "0", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_max_accept_failures": "4", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_max_backoff": "15.000000", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T22:40:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_osd_compress_mode": "none", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_pq_min_cost": "65536", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_public_type": "", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_service_mode": "crc secure", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_tcp_nodelay": "true", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "ms_type": "async+posix", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "no_config_file": "false", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T22:40:36.231 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_inflight_ops": "1024", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter_timeout": "10.000000", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objectstore_blackhole": "false", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "openssl_engine_opts": "", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T22:40:36.232 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_max_ops": "4", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_agent_slop": "0.020000", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_asio_thread_count": "2", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_backfill_scan_max": "512", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_backfill_scan_min": "64", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T22:40:36.233 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_backoff_on_peering": "false", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_beacon_report_interval": "300", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bench_duration": "30", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_blkin_trace_all": "false", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T22:40:36.234 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_class_update_on_start": "true", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_client_message_cap": "256", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_client_op_priority": "63", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_client_watch_timeout": "30", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_command_max_records": "256", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_compact_on_start": "false", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T22:40:36.235 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_crush_update_on_start": "true", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T22:40:36.236 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_op_order": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_shutdown": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_trim_objects": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T22:40:36.237 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_default_notify_timeout": "30", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T22:40:36.238 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_enable_op_tracker": "true", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_fast_info": "true", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_fast_shutdown": "true", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_function_tracing": "false", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_grace": "20", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_interval": "6", 2024-09-13T22:40:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_stale": "600", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_journal_size": "5120", 2024-09-13T22:40:36.240 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_kill_backfill_at": "0", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_map_cache_size": "50", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_map_dedup": "true", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_map_message_max": "40", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_attr_name_len": "100", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_attr_size": "0", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_backfills": "1", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_markdown_count": "5", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_markdown_period": "600", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_object_name_len": "2048", 2024-09-13T22:40:36.241 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_object_size": "134217728", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_pgls": "1024", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_push_cost": "8388608", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_push_objects": "10", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_scrubs": "3", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T22:40:36.242 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_max_write_size": "90", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_profile": "balanced", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T22:40:36.243 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_base": "805306368", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T22:40:36.244 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_target": "4294967296", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_target_autotune": "false", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_min_recovery_priority": "0", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mon_report_interval": "5", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_num_cache_shards": "32", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T22:40:36.245 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_numa_node": "-1", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_objecter_finishers": "1", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_objectstore": "bluestore", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_objectstore_fuse": "false", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_objectstore_tracing": "false", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_history_duration": "600", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_history_size": "20", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_log_threshold": "5", 2024-09-13T22:40:36.246 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_num_shards": "0", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_op_thread_timeout": "15", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_open_classes_on_start": "true", 2024-09-13T22:40:36.247 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_os_flags": "0", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_peering_op_priority": "255", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_delete_priority": "5", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T22:40:36.248 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_crimson": "false", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T22:40:36.249 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_flags": "0", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_min_size": "0", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_size": "3", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_default_type": "replicated", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T22:40:36.250 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_cost": "20971520", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_max_active": "0", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T22:40:36.251 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_op_priority": "3", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_priority": "5", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_repair_during_recovery": "false", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T22:40:36.252 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_cost": "52428800", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_end_hour": "0", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T22:40:36.253 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_priority": "5", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_skip_data_digest": "false", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_smart_report_timeout": "5", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T22:40:36.254 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_snap_trim_priority": "5", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_target_transaction_size": "30", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T22:40:36.255 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_tracing": "false", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_use_stale_snap": "false", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_kill_at": "0", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_max_join_drift": "10", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_min": "500", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_min_wait": "0.050000", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_service_trim_max": "500", 2024-09-13T22:40:36.256 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_service_trim_min": "10", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_stash_full_interval": "25", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_trim_max": "500", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "paxos_trim_min": "250", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "perf": "true", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "pid_file": "", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "-", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrv": "", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "public_bind_addr": "-", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "public_network": "", 2024-09-13T22:40:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: "public_network_interface": "", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "qat_compressor_enabled": "false", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rados_mon_op_timeout": "0", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rados_osd_op_timeout": "0", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rados_tracing": "false", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_atime_update_interval": "60", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T22:40:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache": "true", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_policy": "writearound", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_size": "33554432", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T22:40:36.259 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_compression_hint": "none", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_clone_format": "auto", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_data_pool": "", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_features": "61", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_format": "2", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_map_options": "", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_order": "22", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_pool": "rbd", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_stripe_count": "0", 2024-09-13T22:40:36.260 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_io_scheduler": "simple", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T22:40:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_order": "24", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_pool": "", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_journal_splay_width": "4", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T22:40:36.262 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T22:40:36.263 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T22:40:36.263 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T22:40:36.264 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T22:40:36.265 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_op_threads": "1", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_plugins": "", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T22:40:36.266 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T22:40:36.267 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T22:40:36.268 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_tracing": "false", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_validate_names": "true", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd_validate_pool": "true", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "restapi_base_url": "", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "restapi_log_level": "", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_admin_entry": "admin", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T22:40:36.269 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_backend_store": "rados", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_barbican_url": "", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_beast_enable_async": "true", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_cache_enabled": "true", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T22:40:36.270 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_config_store": "rados", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_content_length_compat": "false", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_cross_domain_policy": "", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T22:40:36.271 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T22:40:36.272 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T22:40:36.273 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T22:40:36.274 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-13T22:40:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_d4n_port": "6379", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_log_window": "30", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T22:40:36.276 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T22:40:36.277 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dns_name": "", 2024-09-13T22:40:36.278 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dns_s3website_name": "", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_ops_log": "false", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_static_website": "false", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enable_usage_log": "false", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_expose_bucket": "false", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_extended_http_attrs": "", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_filter": "none", 2024-09-13T22:40:36.279 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_max_objs": "32", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T22:40:36.280 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_init_timeout": "300", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_password": "", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T22:40:36.281 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_project": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_token": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_admin_user": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_api_version": "2", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T22:40:36.282 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_url": "", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_max_objs": "32", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_max_worker": "3", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T22:40:36.283 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ldap_uri": "", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_log_http_headers": "", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T22:40:36.284 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_attr_size": "0", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_listing_results": "1000", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_notify_retries": "10", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T22:40:36.285 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T22:40:36.286 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_num_control_oids": "8", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_numa_node": "-1", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T22:40:36.287 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_op_tracing": "false", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_opa_token": "", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_opa_url": "", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T22:40:36.288 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ops_log_rados": "false", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T22:40:36.289 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_print_continue": "true", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_rados_tracing": "false", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_realm": "", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_realm_id": "", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_region": "", 2024-09-13T22:40:36.290 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_request_uri": "", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_resolve_cname": "false", 2024-09-13T22:40:36.291 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_run_sync_thread": "true", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_script_uri": "", 2024-09-13T22:40:36.292 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_service_provider_name": "", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_client_id": "", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_client_secret": "", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_entry": "sts", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_key": "sts", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_auth_url": "", 2024-09-13T22:40:36.293 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_custom_header": "", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_need_stats": "true", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_tenant_name": "", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_url": "", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_lease_period": "120", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T22:40:36.294 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_thread_pool_size": "512", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T22:40:36.295 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_comment": "", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_createby": "", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_encoding": "", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_flag": "false", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_origin": "", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_torrent_tracker": "", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_usage_max_shards": "32", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T22:40:36.296 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_use_opa_authz": "false", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_counters_cache": "false", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T22:40:36.297 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_user_unique_email": "true", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_verify_ssl": "true", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_zone": "", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_zone_id": "", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_zonegroup": "", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_zonegroup_id": "", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T22:40:36.298 INFO:teuthology.orchestra.run.smithi178.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_block_size": "4096", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T22:40:36.299 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_partition_filters": "false", 2024-09-13T22:40:36.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_perf": "false", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "run_dir": "/var/run/ceph", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_block_create": "true", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_device_size": "53687091200", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_full_integrity_check": "false", 2024-09-13T22:40:36.301 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_main_device_type": "SSD", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "seastore_segment_size": "67108864", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "set_keepcaps": "false", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "setgroup": "ceph", 2024-09-13T22:40:36.302 INFO:teuthology.orchestra.run.smithi178.stdout: "setuser": "ceph", 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout: "setuser_match_path": "", 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout: "thp": "false", 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout: "threadpool_default_timeout": "60", 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout: "throttler_perf_counter": "true" 2024-09-13T22:40:36.303 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:35 smithi192 bash[24608]: cluster 2024-09-13T22:40:34.772224+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.c scrub starts 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:35 smithi192 bash[24608]: cluster 2024-09-13T22:40:34.776007+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.c scrub ok 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:35 smithi192 bash[24608]: audit 2024-09-13T22:40:34.914407+0000 mgr.y (mgr.14613) 112 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:35 smithi192 bash[24608]: cluster 2024-09-13T22:40:34.992461+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.a scrub starts 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:35 smithi192 bash[24608]: cluster 2024-09-13T22:40:34.994683+0000 osd.1 (osd.1) 72 : cluster [DBG] 2.a scrub ok 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[29889]: cluster 2024-09-13T22:40:34.772224+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.c scrub starts 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[29889]: cluster 2024-09-13T22:40:34.776007+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.c scrub ok 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[29889]: audit 2024-09-13T22:40:34.914407+0000 mgr.y (mgr.14613) 112 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:36.304 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[29889]: cluster 2024-09-13T22:40:34.992461+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.a scrub starts 2024-09-13T22:40:36.305 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[29889]: cluster 2024-09-13T22:40:34.994683+0000 osd.1 (osd.1) 72 : cluster [DBG] 2.a scrub ok 2024-09-13T22:40:36.305 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[22840]: cluster 2024-09-13T22:40:34.772224+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.c scrub starts 2024-09-13T22:40:36.305 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[22840]: cluster 2024-09-13T22:40:34.776007+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.c scrub ok 2024-09-13T22:40:36.305 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[22840]: audit 2024-09-13T22:40:34.914407+0000 mgr.y (mgr.14613) 112 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:36.305 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[22840]: cluster 2024-09-13T22:40:34.992461+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.a scrub starts 2024-09-13T22:40:36.305 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:35 smithi178 bash[22840]: cluster 2024-09-13T22:40:34.994683+0000 osd.1 (osd.1) 72 : cluster [DBG] 2.a scrub ok 2024-09-13T22:40:36.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T22:40:37.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:35.779136+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-13T22:40:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:35.781535+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-13T22:40:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:35.804277+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:40:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:35.807940+0000 osd.5 (osd.5) 72 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:40:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:35.871925+0000 osd.0 (osd.0) 67 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:40:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:35.874431+0000 osd.0 (osd.0) 68 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:40:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:36 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.470018+0000 mgr.y (mgr.14613) 113 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:35.779136+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:35.781535+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:35.804277+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:35.807940+0000 osd.5 (osd.5) 72 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:35.871925+0000 osd.0 (osd.0) 67 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:35.874431+0000 osd.0 (osd.0) 68 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:40:37.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.470018+0000 mgr.y (mgr.14613) 113 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:35.779136+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.b scrub starts 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:35.781535+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.b scrub ok 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:35.804277+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:35.807940+0000 osd.5 (osd.5) 72 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:35.871925+0000 osd.0 (osd.0) 67 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:35.874431+0000 osd.0 (osd.0) 68 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:40:37.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:36 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.470018+0000 mgr.y (mgr.14613) 113 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:38.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:37 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.433730+0000 osd.3 (osd.3) 73 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:40:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:37 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.437396+0000 osd.3 (osd.3) 74 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:40:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:37 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.552928+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:40:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:37 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.555169+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:40:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:37 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.677075+0000 osd.4 (osd.4) 59 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:40:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:37 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.681812+0000 osd.4 (osd.4) 60 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.433730+0000 osd.3 (osd.3) 73 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.437396+0000 osd.3 (osd.3) 74 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.552928+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.555169+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.677075+0000 osd.4 (osd.4) 59 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.681812+0000 osd.4 (osd.4) 60 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.433730+0000 osd.3 (osd.3) 73 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.437396+0000 osd.3 (osd.3) 74 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:40:38.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.552928+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:40:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.555169+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:40:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.677075+0000 osd.4 (osd.4) 59 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:40:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:37 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.681812+0000 osd.4 (osd.4) 60 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:40:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.895099+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:40:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.897038+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:40:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:36.998623+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:40:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.001833+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:40:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.437175+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:40:39.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.439552+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:40:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.535833+0000 osd.7 (osd.7) 71 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:40:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.539993+0000 osd.7 (osd.7) 72 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:40:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.690963+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:40:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:37.694521+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:40:39.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:38 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.471413+0000 mgr.y (mgr.14613) 114 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:39.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.895099+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:40:39.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.897038+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:40:39.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:36.998623+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:40:39.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.001833+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:40:39.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.437175+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:40:39.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.439552+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.535833+0000 osd.7 (osd.7) 71 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.539993+0000 osd.7 (osd.7) 72 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.690963+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:37.694521+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.471413+0000 mgr.y (mgr.14613) 114 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.895099+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.897038+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:36.998623+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.001833+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:40:39.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.437175+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:40:39.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.439552+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:40:39.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.535833+0000 osd.7 (osd.7) 71 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:40:39.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.539993+0000 osd.7 (osd.7) 72 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:40:39.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.690963+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:40:39.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:37.694521+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:40:39.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:38 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.471413+0000 mgr.y (mgr.14613) 114 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.460767+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.463405+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.579493+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.581789+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.777483+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.779526+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.861450+0000 osd.0 (osd.0) 71 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:40:40.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:39 smithi192 bash[24608]: cluster 2024-09-13T22:40:38.865598+0000 osd.0 (osd.0) 72 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:40:40.177 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:40:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.460767+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.463405+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.579493+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.581789+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.777483+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.779526+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.861450+0000 osd.0 (osd.0) 71 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:40:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[29889]: cluster 2024-09-13T22:40:38.865598+0000 osd.0 (osd.0) 72 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.460767+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.463405+0000 osd.3 (osd.3) 78 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.579493+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.581789+0000 osd.7 (osd.7) 74 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.777483+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.779526+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.861450+0000 osd.0 (osd.0) 71 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:40:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:39 smithi178 bash[22840]: cluster 2024-09-13T22:40:38.865598+0000 osd.0 (osd.0) 72 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:40:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:40 smithi192 bash[24608]: cluster 2024-09-13T22:40:39.048765+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:40:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:40 smithi192 bash[24608]: cluster 2024-09-13T22:40:39.050966+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:40:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:40 smithi192 bash[24608]: cluster 2024-09-13T22:40:39.952036+0000 osd.1 (osd.1) 73 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:40:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:40 smithi192 bash[24608]: cluster 2024-09-13T22:40:39.955289+0000 osd.1 (osd.1) 74 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:40:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:40 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.472237+0000 mgr.y (mgr.14613) 115 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[29889]: cluster 2024-09-13T22:40:39.048765+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[29889]: cluster 2024-09-13T22:40:39.050966+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[29889]: cluster 2024-09-13T22:40:39.952036+0000 osd.1 (osd.1) 73 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[29889]: cluster 2024-09-13T22:40:39.955289+0000 osd.1 (osd.1) 74 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.472237+0000 mgr.y (mgr.14613) 115 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[22840]: cluster 2024-09-13T22:40:39.048765+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[22840]: cluster 2024-09-13T22:40:39.050966+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[22840]: cluster 2024-09-13T22:40:39.952036+0000 osd.1 (osd.1) 73 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[22840]: cluster 2024-09-13T22:40:39.955289+0000 osd.1 (osd.1) 74 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:40:41.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:40 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.472237+0000 mgr.y (mgr.14613) 115 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-09-13T22:40:41.655 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:42.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.353145+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.353287+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.733485+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.736437+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.778186+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.780446+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.859430+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.960583+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:40:42.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:41 smithi192 bash[24608]: cluster 2024-09-13T22:40:40.963632+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.353145+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.353287+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.733485+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.736437+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.778186+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.780446+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.859430+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.960583+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:40:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[29889]: cluster 2024-09-13T22:40:40.963632+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.353145+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.353287+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.733485+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.736437+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.778186+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.780446+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.859430+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.960583+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:40:42.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:41 smithi178 bash[22840]: cluster 2024-09-13T22:40:40.963632+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:40:43.001 INFO:teuthology.orchestra.run.smithi178.stdout:mon.a: {} 2024-09-13T22:40:43.001 INFO:teuthology.orchestra.run.smithi178.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T22:40:43.014 INFO:teuthology.orchestra.run.smithi178.stdout:mon.b: {} 2024-09-13T22:40:43.014 INFO:teuthology.orchestra.run.smithi178.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T22:40:43.023 INFO:teuthology.orchestra.run.smithi178.stdout:mon.c: {} 2024-09-13T22:40:43.024 INFO:teuthology.orchestra.run.smithi178.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T22:40:43.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:42 smithi178 bash[29889]: cluster 2024-09-13T22:40:41.469873+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.b scrub starts 2024-09-13T22:40:43.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:42 smithi178 bash[29889]: cluster 2024-09-13T22:40:41.472277+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.b scrub ok 2024-09-13T22:40:43.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:42 smithi178 bash[29889]: cluster 2024-09-13T22:40:42.472969+0000 mgr.y (mgr.14613) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:43.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:42 smithi178 bash[22840]: cluster 2024-09-13T22:40:41.469873+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.b scrub starts 2024-09-13T22:40:43.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:42 smithi178 bash[22840]: cluster 2024-09-13T22:40:41.472277+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.b scrub ok 2024-09-13T22:40:43.098 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:42 smithi178 bash[22840]: cluster 2024-09-13T22:40:42.472969+0000 mgr.y (mgr.14613) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:43.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:42 smithi192 bash[24608]: cluster 2024-09-13T22:40:41.469873+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.b scrub starts 2024-09-13T22:40:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:42 smithi192 bash[24608]: cluster 2024-09-13T22:40:41.472277+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.b scrub ok 2024-09-13T22:40:43.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:42 smithi192 bash[24608]: cluster 2024-09-13T22:40:42.472969+0000 mgr.y (mgr.14613) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:43.772 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-13T22:40:43.784 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T22:40:43.786 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T22:40:43.786 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T22:40:43.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-13T22:40:43.786 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T22:40:43.787 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:40:43.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:40:43.787 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T22:40:43.787 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T22:40:43.787 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T22:40:43.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T22:40:43.788 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T22:40:43.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:43.788 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T22:40:43.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:40:43.789 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T22:40:43.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T22:40:43.789 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T22:40:43.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:40:43.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:43.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:43.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:43.949 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:43.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:44.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[29889]: cluster 2024-09-13T22:40:42.719571+0000 osd.2 (osd.2) 55 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[29889]: cluster 2024-09-13T22:40:42.722789+0000 osd.2 (osd.2) 56 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[29889]: cluster 2024-09-13T22:40:42.808238+0000 osd.5 (osd.5) 73 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[29889]: cluster 2024-09-13T22:40:42.810478+0000 osd.5 (osd.5) 74 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[29889]: audit 2024-09-13T22:40:42.994236+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? v1:172.21.15.178:0/20835044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[22840]: cluster 2024-09-13T22:40:42.719571+0000 osd.2 (osd.2) 55 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[22840]: cluster 2024-09-13T22:40:42.722789+0000 osd.2 (osd.2) 56 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:40:44.060 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[22840]: cluster 2024-09-13T22:40:42.808238+0000 osd.5 (osd.5) 73 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:40:44.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[22840]: cluster 2024-09-13T22:40:42.810478+0000 osd.5 (osd.5) 74 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:40:44.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:43 smithi178 bash[22840]: audit 2024-09-13T22:40:42.994236+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? v1:172.21.15.178:0/20835044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:40:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:43 smithi192 bash[24608]: cluster 2024-09-13T22:40:42.719571+0000 osd.2 (osd.2) 55 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:40:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:43 smithi192 bash[24608]: cluster 2024-09-13T22:40:42.722789+0000 osd.2 (osd.2) 56 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:40:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:43 smithi192 bash[24608]: cluster 2024-09-13T22:40:42.808238+0000 osd.5 (osd.5) 73 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:40:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:43 smithi192 bash[24608]: cluster 2024-09-13T22:40:42.810478+0000 osd.5 (osd.5) 74 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:40:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:43 smithi192 bash[24608]: audit 2024-09-13T22:40:42.994236+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? v1:172.21.15.178:0/20835044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T22:40:44.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.193 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:44.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.323 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:44.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.452 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:44.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:44.583 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:44.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:44.715 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:44.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:44.845 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:44.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:44.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:44.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:45.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:45.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:40:44 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:40:45.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:44 smithi192 bash[24608]: cluster 2024-09-13T22:40:43.828662+0000 osd.1 (osd.1) 77 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:40:45.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:44 smithi192 bash[24608]: cluster 2024-09-13T22:40:43.830003+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:40:45.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:44 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.474065+0000 mgr.y (mgr.14613) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:45.105 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:45.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:44 smithi178 bash[29889]: cluster 2024-09-13T22:40:43.828662+0000 osd.1 (osd.1) 77 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:40:45.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:44 smithi178 bash[29889]: cluster 2024-09-13T22:40:43.830003+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:40:45.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:44 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.474065+0000 mgr.y (mgr.14613) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:44 smithi178 bash[22840]: cluster 2024-09-13T22:40:43.828662+0000 osd.1 (osd.1) 77 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:40:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:44 smithi178 bash[22840]: cluster 2024-09-13T22:40:43.830003+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:40:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:44 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.474065+0000 mgr.y (mgr.14613) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:45.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:45.235 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:45.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:45.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:45.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:45.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:45.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:45.624 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:45.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:45.754 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:45.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:45.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:45.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:45.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:46.016 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:46.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.146404+0000 osd.6 (osd.6) 47 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.161908+0000 osd.6 (osd.6) 48 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.388665+0000 osd.3 (osd.3) 81 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.391398+0000 osd.3 (osd.3) 82 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.873136+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: cluster 2024-09-13T22:40:44.875462+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:40:46.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:45 smithi192 bash[24608]: audit 2024-09-13T22:40:44.918474+0000 mgr.y (mgr.14613) 118 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:46.146 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.146404+0000 osd.6 (osd.6) 47 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.161908+0000 osd.6 (osd.6) 48 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.388665+0000 osd.3 (osd.3) 81 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.391398+0000 osd.3 (osd.3) 82 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.873136+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: cluster 2024-09-13T22:40:44.875462+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[22840]: audit 2024-09-13T22:40:44.918474+0000 mgr.y (mgr.14613) 118 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:46.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.146404+0000 osd.6 (osd.6) 47 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T22:40:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.161908+0000 osd.6 (osd.6) 48 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T22:40:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.388665+0000 osd.3 (osd.3) 81 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:40:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.391398+0000 osd.3 (osd.3) 82 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:40:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.873136+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:40:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: cluster 2024-09-13T22:40:44.875462+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:40:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:45 smithi178 bash[29889]: audit 2024-09-13T22:40:44.918474+0000 mgr.y (mgr.14613) 118 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:46.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:46.276 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:46.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:46.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:46.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:46.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:46.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:46.665 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:46.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:46.794 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:46.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:46.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.620945+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:40:46.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.622163+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:40:46.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.672187+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:40:46.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.674243+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:40:46.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.793312+0000 osd.5 (osd.5) 75 : cluster [DBG] 2.16 deep-scrub starts 2024-09-13T22:40:46.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.794833+0000 osd.5 (osd.5) 76 : cluster [DBG] 2.16 deep-scrub ok 2024-09-13T22:40:46.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.866446+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:40:46.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:45.868828+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:40:46.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:46 smithi192 bash[24608]: cluster 2024-09-13T22:40:46.474732+0000 mgr.y (mgr.14613) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:46.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:46.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:47.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:47.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:47.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.620945+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.622163+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.672187+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.674243+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.793312+0000 osd.5 (osd.5) 75 : cluster [DBG] 2.16 deep-scrub starts 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.794833+0000 osd.5 (osd.5) 76 : cluster [DBG] 2.16 deep-scrub ok 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.866446+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:45.868828+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:40:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[22840]: cluster 2024-09-13T22:40:46.474732+0000 mgr.y (mgr.14613) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.620945+0000 osd.7 (osd.7) 75 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.622163+0000 osd.7 (osd.7) 76 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.672187+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.674243+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.793312+0000 osd.5 (osd.5) 75 : cluster [DBG] 2.16 deep-scrub starts 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.794833+0000 osd.5 (osd.5) 76 : cluster [DBG] 2.16 deep-scrub ok 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.866446+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:45.868828+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:40:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:46 smithi178 bash[29889]: cluster 2024-09-13T22:40:46.474732+0000 mgr.y (mgr.14613) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:47.185 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:47.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:47.314 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:47.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:47.444 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:47.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:47.574 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:47.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:47.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:47.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:47.830 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:47.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:47.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:47.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:48.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[29889]: cluster 2024-09-13T22:40:46.578006+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[29889]: cluster 2024-09-13T22:40:46.579122+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[29889]: cluster 2024-09-13T22:40:46.658610+0000 osd.4 (osd.4) 65 : cluster [DBG] 6.2 deep-scrub starts 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[29889]: cluster 2024-09-13T22:40:46.659968+0000 osd.4 (osd.4) 66 : cluster [DBG] 6.2 deep-scrub ok 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[29889]: audit 2024-09-13T22:40:47.738203+0000 mon.b (mon.1) 90 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[22840]: cluster 2024-09-13T22:40:46.578006+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:40:48.076 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[22840]: cluster 2024-09-13T22:40:46.579122+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:40:48.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[22840]: cluster 2024-09-13T22:40:46.658610+0000 osd.4 (osd.4) 65 : cluster [DBG] 6.2 deep-scrub starts 2024-09-13T22:40:48.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[22840]: cluster 2024-09-13T22:40:46.659968+0000 osd.4 (osd.4) 66 : cluster [DBG] 6.2 deep-scrub ok 2024-09-13T22:40:48.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:47 smithi178 bash[22840]: audit 2024-09-13T22:40:47.738203+0000 mon.b (mon.1) 90 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:48.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:48.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:47 smithi192 bash[24608]: cluster 2024-09-13T22:40:46.578006+0000 osd.7 (osd.7) 77 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:40:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:47 smithi192 bash[24608]: cluster 2024-09-13T22:40:46.579122+0000 osd.7 (osd.7) 78 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:40:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:47 smithi192 bash[24608]: cluster 2024-09-13T22:40:46.658610+0000 osd.4 (osd.4) 65 : cluster [DBG] 6.2 deep-scrub starts 2024-09-13T22:40:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:47 smithi192 bash[24608]: cluster 2024-09-13T22:40:46.659968+0000 osd.4 (osd.4) 66 : cluster [DBG] 6.2 deep-scrub ok 2024-09-13T22:40:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:47 smithi192 bash[24608]: audit 2024-09-13T22:40:47.738203+0000 mon.b (mon.1) 90 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:40:48.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:48.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:48.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:48.352 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:48.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:48.483 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:48.575 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:40:48.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:48.594 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:48.595 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:48.595 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:48.596 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:48.607 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:48.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:48.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:48.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:48.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:48.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:48.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:48 smithi192 bash[24608]: cluster 2024-09-13T22:40:47.889411+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.c scrub starts 2024-09-13T22:40:48.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:48 smithi192 bash[24608]: cluster 2024-09-13T22:40:47.890941+0000 osd.1 (osd.1) 84 : cluster [DBG] 5.c scrub ok 2024-09-13T22:40:48.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:48 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.476096+0000 mgr.y (mgr.14613) 120 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:40:48.994 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:48.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:49.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:49.108 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:48 smithi178 bash[29889]: cluster 2024-09-13T22:40:47.889411+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.c scrub starts 2024-09-13T22:40:49.108 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:48 smithi178 bash[29889]: cluster 2024-09-13T22:40:47.890941+0000 osd.1 (osd.1) 84 : cluster [DBG] 5.c scrub ok 2024-09-13T22:40:49.108 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:48 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.476096+0000 mgr.y (mgr.14613) 120 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:40:49.108 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:48 smithi178 bash[22840]: cluster 2024-09-13T22:40:47.889411+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.c scrub starts 2024-09-13T22:40:49.108 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:48 smithi178 bash[22840]: cluster 2024-09-13T22:40:47.890941+0000 osd.1 (osd.1) 84 : cluster [DBG] 5.c scrub ok 2024-09-13T22:40:49.109 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:48 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.476096+0000 mgr.y (mgr.14613) 120 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:40:49.124 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:49.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:49.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:49.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:49.385 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:49.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:49.511 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:49.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:49.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:49.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:49.770 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:49.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:49.898 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:49.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:50.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:50.012 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:40:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.547876+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.550116+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.574348+0000 osd.7 (osd.7) 79 : cluster [DBG] 2.b scrub starts 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.576802+0000 osd.7 (osd.7) 80 : cluster [DBG] 2.b scrub ok 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.646265+0000 osd.4 (osd.4) 67 : cluster [DBG] 5.e scrub starts 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.647868+0000 osd.4 (osd.4) 68 : cluster [DBG] 5.e scrub ok 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.760257+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-13T22:40:50.013 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[29889]: cluster 2024-09-13T22:40:48.762516+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.547876+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.550116+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.574348+0000 osd.7 (osd.7) 79 : cluster [DBG] 2.b scrub starts 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.576802+0000 osd.7 (osd.7) 80 : cluster [DBG] 2.b scrub ok 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.646265+0000 osd.4 (osd.4) 67 : cluster [DBG] 5.e scrub starts 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.647868+0000 osd.4 (osd.4) 68 : cluster [DBG] 5.e scrub ok 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.760257+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-13T22:40:50.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:49 smithi178 bash[22840]: cluster 2024-09-13T22:40:48.762516+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-13T22:40:50.015 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:40:50.016 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T22:40:50.016 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:40:50.028 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:50.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.547876+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.550116+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.574348+0000 osd.7 (osd.7) 79 : cluster [DBG] 2.b scrub starts 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.576802+0000 osd.7 (osd.7) 80 : cluster [DBG] 2.b scrub ok 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.646265+0000 osd.4 (osd.4) 67 : cluster [DBG] 5.e scrub starts 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.647868+0000 osd.4 (osd.4) 68 : cluster [DBG] 5.e scrub ok 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.760257+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.d scrub starts 2024-09-13T22:40:50.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:49 smithi192 bash[24608]: cluster 2024-09-13T22:40:48.762516+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.d scrub ok 2024-09-13T22:40:50.150 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:50.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:50.282 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:50.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:50.414 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:50.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:50.543 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:50.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:50.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:50.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:50.807 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:50.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:50.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:50 smithi178 bash[29889]: cluster 2024-09-13T22:40:49.833024+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:40:50.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:50 smithi178 bash[29889]: cluster 2024-09-13T22:40:49.836949+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:40:50.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:50 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.476719+0000 mgr.y (mgr.14613) 121 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:50.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:50 smithi178 bash[22840]: cluster 2024-09-13T22:40:49.833024+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:40:50.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:50 smithi178 bash[22840]: cluster 2024-09-13T22:40:49.836949+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:40:50.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:50 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.476719+0000 mgr.y (mgr.14613) 121 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:50.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:50.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:50 smithi192 bash[24608]: cluster 2024-09-13T22:40:49.833024+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:40:50.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:50 smithi192 bash[24608]: cluster 2024-09-13T22:40:49.836949+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:40:50.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:50 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.476719+0000 mgr.y (mgr.14613) 121 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:40:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:50.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:51.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:51.067 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:51.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:51.196 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:51.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:51.323 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:51.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:51.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:51.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:51.584 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:51.625 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:40:51.625 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:40:51.643 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:40:51.643 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:40:51.660 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:40:51.660 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:40:51.677 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:40:51.677 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:40:51.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:51.699 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:40:51.699 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:40:51.709 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:51.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:51.722 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:40:51.722 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:40:51.743 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:40:51.744 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:40:51.764 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:40:51.764 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:40:51.812 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:51.831 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:51.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:51.959 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:51.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.469977+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.472564+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.521754+0000 osd.7 (osd.7) 81 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.525278+0000 osd.7 (osd.7) 82 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.593830+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:50.595346+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:51.009976+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: cluster 2024-09-13T22:40:51.011775+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[29889]: audit 2024-09-13T22:40:51.611539+0000 mon.a (mon.0) 912 : audit [DBG] from='client.? v1:172.21.15.178:0/2505198621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:40:52.020 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.469977+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.472564+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.521754+0000 osd.7 (osd.7) 81 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.525278+0000 osd.7 (osd.7) 82 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.593830+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:50.595346+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:51.009976+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: cluster 2024-09-13T22:40:51.011775+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:40:52.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:51 smithi178 bash[22840]: audit 2024-09-13T22:40:51.611539+0000 mon.a (mon.0) 912 : audit [DBG] from='client.? v1:172.21.15.178:0/2505198621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:40:52.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:52.090 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.469977+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:40:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.472564+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:40:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.521754+0000 osd.7 (osd.7) 81 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:40:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.525278+0000 osd.7 (osd.7) 82 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:40:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.593830+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:40:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:50.595346+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:40:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:51.009976+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:40:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: cluster 2024-09-13T22:40:51.011775+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:40:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:51 smithi192 bash[24608]: audit 2024-09-13T22:40:51.611539+0000 mon.a (mon.0) 912 : audit [DBG] from='client.? v1:172.21.15.178:0/2505198621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:40:52.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:52.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:52.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:52.344 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:52.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:52.471 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:52.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:52.602 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:52.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:52.729 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:52.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:52.859 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:52.910 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:40:52.962 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:52.981 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:52.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:53.008 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:40:53.009 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:40:53.018 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[22840]: cluster 2024-09-13T22:40:51.787208+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 deep-scrub starts 2024-09-13T22:40:53.018 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[22840]: cluster 2024-09-13T22:40:51.789512+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 deep-scrub ok 2024-09-13T22:40:53.018 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[22840]: cluster 2024-09-13T22:40:51.880850+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:40:53.018 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[22840]: cluster 2024-09-13T22:40:51.882577+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:40:53.018 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[22840]: audit 2024-09-13T22:40:52.358948+0000 mon.a (mon.0) 913 : audit [INF] from='client.? v1:172.21.15.178:0/2475611777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:40:53.018 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.477166+0000 mgr.y (mgr.14613) 122 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-09-13T22:40:53.019 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[29889]: cluster 2024-09-13T22:40:51.787208+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 deep-scrub starts 2024-09-13T22:40:53.019 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[29889]: cluster 2024-09-13T22:40:51.789512+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 deep-scrub ok 2024-09-13T22:40:53.019 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[29889]: cluster 2024-09-13T22:40:51.880850+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:40:53.019 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[29889]: cluster 2024-09-13T22:40:51.882577+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:40:53.019 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[29889]: audit 2024-09-13T22:40:52.358948+0000 mon.a (mon.0) 913 : audit [INF] from='client.? v1:172.21.15.178:0/2475611777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:40:53.019 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:52 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.477166+0000 mgr.y (mgr.14613) 122 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-09-13T22:40:53.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:53.115 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:52 smithi192 bash[24608]: cluster 2024-09-13T22:40:51.787208+0000 osd.5 (osd.5) 79 : cluster [DBG] 2.18 deep-scrub starts 2024-09-13T22:40:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:52 smithi192 bash[24608]: cluster 2024-09-13T22:40:51.789512+0000 osd.5 (osd.5) 80 : cluster [DBG] 2.18 deep-scrub ok 2024-09-13T22:40:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:52 smithi192 bash[24608]: cluster 2024-09-13T22:40:51.880850+0000 osd.1 (osd.1) 85 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:40:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:52 smithi192 bash[24608]: cluster 2024-09-13T22:40:51.882577+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:40:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:52 smithi192 bash[24608]: audit 2024-09-13T22:40:52.358948+0000 mon.a (mon.0) 913 : audit [INF] from='client.? v1:172.21.15.178:0/2475611777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:40:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:52 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.477166+0000 mgr.y (mgr.14613) 122 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-09-13T22:40:53.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:53.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:53.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:53.378 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:53.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:53.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:53.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:53.633 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:53.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:53.763 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:53.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:53.894 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:53.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:53.910 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.659867+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:53.910 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.659996+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.768632+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.769911+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.905548+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: audit 2024-09-13T22:40:52.914171+0000 mon.a (mon.0) 915 : audit [INF] from='client.? v1:172.21.15.178:0/2475611777' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: cluster 2024-09-13T22:40:52.917731+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: audit 2024-09-13T22:40:53.101029+0000 mon.c (mon.2) 19 : audit [INF] from='client.? v1:172.21.15.178:0/3009654261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T22:40:53.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[29889]: audit 2024-09-13T22:40:53.101648+0000 mon.a (mon.0) 917 : audit [INF] from='client.24734 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T22:40:53.929 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_0' created 2024-09-13T22:40:53.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:54.016 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:54.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:54.146 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:54.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.659867+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:54.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.659996+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.768632+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.769911+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.905548+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: audit 2024-09-13T22:40:52.914171+0000 mon.a (mon.0) 915 : audit [INF] from='client.? v1:172.21.15.178:0/2475611777' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: cluster 2024-09-13T22:40:52.917731+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: audit 2024-09-13T22:40:53.101029+0000 mon.c (mon.2) 19 : audit [INF] from='client.? v1:172.21.15.178:0/3009654261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T22:40:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:53 smithi192 bash[24608]: audit 2024-09-13T22:40:53.101648+0000 mon.a (mon.0) 917 : audit [INF] from='client.24734 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T22:40:54.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.659867+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.659996+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.768632+0000 osd.5 (osd.5) 81 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.769911+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.905548+0000 mon.a (mon.0) 914 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: audit 2024-09-13T22:40:52.914171+0000 mon.a (mon.0) 915 : audit [INF] from='client.? v1:172.21.15.178:0/2475611777' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: cluster 2024-09-13T22:40:52.917731+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: audit 2024-09-13T22:40:53.101029+0000 mon.c (mon.2) 19 : audit [INF] from='client.? v1:172.21.15.178:0/3009654261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T22:40:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:53 smithi178 bash[22840]: audit 2024-09-13T22:40:53.101648+0000 mon.a (mon.0) 917 : audit [INF] from='client.24734 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-13T22:40:54.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:54.278 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:54.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:54.406 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:54.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:54.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T22:40:54.536 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:54.639 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:54.664 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:54.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:54.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:54.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:54.924 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:54.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:55.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:55.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:54 smithi178 bash[29889]: audit 2024-09-13T22:40:53.920224+0000 mon.a (mon.0) 918 : audit [INF] from='client.24734 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T22:40:55.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:54 smithi178 bash[29889]: cluster 2024-09-13T22:40:53.928080+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-13T22:40:55.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:54 smithi178 bash[29889]: cluster 2024-09-13T22:40:54.477695+0000 mgr.y (mgr.14613) 123 : cluster [DBG] pgmap v95: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:55.038 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:54 smithi178 bash[22840]: audit 2024-09-13T22:40:53.920224+0000 mon.a (mon.0) 918 : audit [INF] from='client.24734 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T22:40:55.039 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:54 smithi178 bash[22840]: cluster 2024-09-13T22:40:53.928080+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-13T22:40:55.039 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:54 smithi178 bash[22840]: cluster 2024-09-13T22:40:54.477695+0000 mgr.y (mgr.14613) 123 : cluster [DBG] pgmap v95: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:55.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:40:55.055 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:55.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T22:40:55.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:40:54 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:40:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:54 smithi192 bash[24608]: audit 2024-09-13T22:40:53.920224+0000 mon.a (mon.0) 918 : audit [INF] from='client.24734 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T22:40:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:54 smithi192 bash[24608]: cluster 2024-09-13T22:40:53.928080+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-13T22:40:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:54 smithi192 bash[24608]: cluster 2024-09-13T22:40:54.477695+0000 mgr.y (mgr.14613) 123 : cluster [DBG] pgmap v95: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:55.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:55.187 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:55.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:55.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:55.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:55.439 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:55.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:55.570 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:55.673 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:55.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:40:55.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T22:40:55.699 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:55.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:55.831 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:55.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:55.941 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:55 smithi178 bash[22840]: cluster 2024-09-13T22:40:54.920220+0000 mon.a (mon.0) 920 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:40:55.942 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:55 smithi178 bash[22840]: audit 2024-09-13T22:40:54.926437+0000 mgr.y (mgr.14613) 124 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:55.942 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:55 smithi178 bash[22840]: cluster 2024-09-13T22:40:54.937334+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-13T22:40:55.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:55 smithi178 bash[29889]: cluster 2024-09-13T22:40:54.920220+0000 mon.a (mon.0) 920 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:40:55.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:55 smithi178 bash[29889]: audit 2024-09-13T22:40:54.926437+0000 mgr.y (mgr.14613) 124 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:55.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:55 smithi178 bash[29889]: cluster 2024-09-13T22:40:54.937334+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-13T22:40:55.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:55.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:56.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:56.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:55 smithi192 bash[24608]: cluster 2024-09-13T22:40:54.920220+0000 mon.a (mon.0) 920 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:40:56.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:55 smithi192 bash[24608]: audit 2024-09-13T22:40:54.926437+0000 mgr.y (mgr.14613) 124 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:40:56.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:55 smithi192 bash[24608]: cluster 2024-09-13T22:40:54.937334+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-13T22:40:56.087 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:56.190 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:56.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:56.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:56.346 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:56.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:56.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:40:56.476 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:56.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:56.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:56.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:56.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:56.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:56.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:56.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:56.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:56 smithi192 bash[24608]: cluster 2024-09-13T22:40:55.939424+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-13T22:40:56.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:56 smithi192 bash[24608]: cluster 2024-09-13T22:40:56.478194+0000 mgr.y (mgr.14613) 125 : cluster [DBG] pgmap v98: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:56.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:56.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:57.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:57.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:57.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:57.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:57.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:57.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:56 smithi178 bash[29889]: cluster 2024-09-13T22:40:55.939424+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-13T22:40:57.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:56 smithi178 bash[29889]: cluster 2024-09-13T22:40:56.478194+0000 mgr.y (mgr.14613) 125 : cluster [DBG] pgmap v98: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:57.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:56 smithi178 bash[22840]: cluster 2024-09-13T22:40:55.939424+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-13T22:40:57.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:56 smithi178 bash[22840]: cluster 2024-09-13T22:40:56.478194+0000 mgr.y (mgr.14613) 125 : cluster [DBG] pgmap v98: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:40:57.387 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:57.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:57.519 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:57.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:57.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:57.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:57.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:57.911 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:57.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:58.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:58.042 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:40:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:40:58.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:58.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:58.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:58.302 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:58.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:58.431 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:58.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:58.559 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:58.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:58.689 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:40:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:40:58.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:58.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:40:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:40:58.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:58.930 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:40:58 smithi192 bash[24608]: cluster 2024-09-13T22:40:58.479177+0000 mgr.y (mgr.14613) 126 : cluster [DBG] pgmap v99: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-13T22:40:58.946 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:58.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:59.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:59.075 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:59.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:40:58 smithi178 bash[29889]: cluster 2024-09-13T22:40:58.479177+0000 mgr.y (mgr.14613) 126 : cluster [DBG] pgmap v99: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-13T22:40:59.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:40:58 smithi178 bash[22840]: cluster 2024-09-13T22:40:58.479177+0000 mgr.y (mgr.14613) 126 : cluster [DBG] pgmap v99: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-13T22:40:59.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:59.206 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:59.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:59.285 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:59.286 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:59.287 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:40:59.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:59.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:40:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:40:59.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:59.459 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:40:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:40:59.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:59.588 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:40:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:40:59.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:59.719 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:40:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:40:59.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:59.846 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:40:59.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:59.960 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:40:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:40:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:40:59.975 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:40:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:40:59.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:00.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:00.106 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:00.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:00.238 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:00.304 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:00.305 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:00.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:00.366 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:00.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:00.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:00.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:00.624 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:00.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:00.756 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:00.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:00.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:00.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:00.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:01.014 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:01.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:01.142 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:01.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:01.275 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:01.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:01.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:01.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:01.535 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:01.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:01.666 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:01.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:01.782 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:01 smithi192 bash[24608]: cluster 2024-09-13T22:41:00.479799+0000 mgr.y (mgr.14613) 127 : cluster [DBG] pgmap v100: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-09-13T22:41:01.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:01.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:01.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:01 smithi178 bash[29889]: cluster 2024-09-13T22:41:00.479799+0000 mgr.y (mgr.14613) 127 : cluster [DBG] pgmap v100: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-09-13T22:41:01.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:01 smithi178 bash[22840]: cluster 2024-09-13T22:41:00.479799+0000 mgr.y (mgr.14613) 127 : cluster [DBG] pgmap v100: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-09-13T22:41:01.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:01.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:02.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:02.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:02.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:02.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:02.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:02.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:02.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:02.452 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:02.555 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:02.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:02.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:02.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:02.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:02.836 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:02.923 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:41:02.923 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:41:02.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:02.941 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:41:02.941 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:41:02.961 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:41:02.961 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:41:02.962 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:02.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:02.979 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:41:02.980 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:41:03.000 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:41:03.001 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:41:03.029 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:41:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:41:03.054 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:41:03.055 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:41:03.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:03.075 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:41:03.076 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:41:03.090 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:03.153 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:41:03.153 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:41:03.154 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:41:03.154 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:41:03.154 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:41:03.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:03.223 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:03.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:03.355 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:03.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:03.485 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:03.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: cluster 2024-09-13T22:41:02.480538+0000 mgr.y (mgr.14613) 128 : cluster [DBG] pgmap v101: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:02.769335+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:02.771792+0000 mon.b (mon.1) 91 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:02.909435+0000 mon.a (mon.0) 924 : audit [DBG] from='client.? v1:172.21.15.178:0/1944855111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:03.221281+0000 mon.c (mon.2) 20 : audit [INF] from='client.? v1:172.21.15.178:0/1342398767' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:03.221661+0000 mon.a (mon.0) 925 : audit [INF] from='client.24746 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:03.445684+0000 mon.c (mon.2) 21 : audit [INF] from='client.? v1:172.21.15.178:0/3954410379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:03.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[22840]: audit 2024-09-13T22:41:03.445971+0000 mon.a (mon.0) 926 : audit [INF] from='client.24752 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: cluster 2024-09-13T22:41:02.480538+0000 mgr.y (mgr.14613) 128 : cluster [DBG] pgmap v101: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:02.769335+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:02.771792+0000 mon.b (mon.1) 91 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:02.909435+0000 mon.a (mon.0) 924 : audit [DBG] from='client.? v1:172.21.15.178:0/1944855111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:03.221281+0000 mon.c (mon.2) 20 : audit [INF] from='client.? v1:172.21.15.178:0/1342398767' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:03.221661+0000 mon.a (mon.0) 925 : audit [INF] from='client.24746 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:03.445684+0000 mon.c (mon.2) 21 : audit [INF] from='client.? v1:172.21.15.178:0/3954410379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:03.599 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:03 smithi178 bash[29889]: audit 2024-09-13T22:41:03.445971+0000 mon.a (mon.0) 926 : audit [INF] from='client.24752 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:03.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:03.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: cluster 2024-09-13T22:41:02.480538+0000 mgr.y (mgr.14613) 128 : cluster [DBG] pgmap v101: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:02.769335+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:02.771792+0000 mon.b (mon.1) 91 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:02.909435+0000 mon.a (mon.0) 924 : audit [DBG] from='client.? v1:172.21.15.178:0/1944855111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:03.221281+0000 mon.c (mon.2) 20 : audit [INF] from='client.? v1:172.21.15.178:0/1342398767' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:03.221661+0000 mon.a (mon.0) 925 : audit [INF] from='client.24746 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:03.445684+0000 mon.c (mon.2) 21 : audit [INF] from='client.? v1:172.21.15.178:0/3954410379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:03.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:03 smithi192 bash[24608]: audit 2024-09-13T22:41:03.445971+0000 mon.a (mon.0) 926 : audit [INF] from='client.24752 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:03.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:03.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:03.767 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T22:41:03.768 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T22:41:03.768 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:41:03.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:03.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:03.993 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:03.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:04.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.129 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:04.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.258 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:04.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:04.387 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:04.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:04.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.647 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:04.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: audit 2024-09-13T22:41:03.638567+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.178:0/3892385504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: audit 2024-09-13T22:41:03.638861+0000 mon.a (mon.0) 927 : audit [INF] from='client.24758 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: cluster 2024-09-13T22:41:03.767009+0000 mon.a (mon.0) 928 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: audit 2024-09-13T22:41:03.768531+0000 mon.a (mon.0) 929 : audit [INF] from='client.24746 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: audit 2024-09-13T22:41:03.768591+0000 mon.a (mon.0) 930 : audit [INF] from='client.24752 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: audit 2024-09-13T22:41:03.768633+0000 mon.a (mon.0) 931 : audit [INF] from='client.24758 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:41:04.754 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[29889]: cluster 2024-09-13T22:41:03.771753+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: audit 2024-09-13T22:41:03.638567+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.178:0/3892385504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: audit 2024-09-13T22:41:03.638861+0000 mon.a (mon.0) 927 : audit [INF] from='client.24758 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: cluster 2024-09-13T22:41:03.767009+0000 mon.a (mon.0) 928 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: audit 2024-09-13T22:41:03.768531+0000 mon.a (mon.0) 929 : audit [INF] from='client.24746 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: audit 2024-09-13T22:41:03.768591+0000 mon.a (mon.0) 930 : audit [INF] from='client.24752 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: audit 2024-09-13T22:41:03.768633+0000 mon.a (mon.0) 931 : audit [INF] from='client.24758 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:41:04.755 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:04 smithi178 bash[22840]: cluster 2024-09-13T22:41:03.771753+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-13T22:41:04.781 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:04.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: audit 2024-09-13T22:41:03.638567+0000 mon.c (mon.2) 22 : audit [INF] from='client.? v1:172.21.15.178:0/3892385504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: audit 2024-09-13T22:41:03.638861+0000 mon.a (mon.0) 927 : audit [INF] from='client.24758 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: cluster 2024-09-13T22:41:03.767009+0000 mon.a (mon.0) 928 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: audit 2024-09-13T22:41:03.768531+0000 mon.a (mon.0) 929 : audit [INF] from='client.24746 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: audit 2024-09-13T22:41:03.768591+0000 mon.a (mon.0) 930 : audit [INF] from='client.24752 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: audit 2024-09-13T22:41:03.768633+0000 mon.a (mon.0) 931 : audit [INF] from='client.24758 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:41:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[24608]: cluster 2024-09-13T22:41:03.771753+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-13T22:41:04.912 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:04.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:05.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.045 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:05.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:05.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:05.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:41:04 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:41:05.174 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:05.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:05.305 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:05.408 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:05.437 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:05.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:05.566 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:05.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:05.697 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:05.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[22840]: cluster 2024-09-13T22:41:04.358298+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:05.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[22840]: cluster 2024-09-13T22:41:04.358365+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:05.794 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[22840]: cluster 2024-09-13T22:41:04.481031+0000 mgr.y (mgr.14613) 129 : cluster [DBG] pgmap v103: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[22840]: cluster 2024-09-13T22:41:04.768730+0000 mon.a (mon.0) 933 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[22840]: cluster 2024-09-13T22:41:04.782558+0000 mon.a (mon.0) 934 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[29889]: cluster 2024-09-13T22:41:04.358298+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[29889]: cluster 2024-09-13T22:41:04.358365+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[29889]: cluster 2024-09-13T22:41:04.481031+0000 mgr.y (mgr.14613) 129 : cluster [DBG] pgmap v103: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[29889]: cluster 2024-09-13T22:41:04.768730+0000 mon.a (mon.0) 933 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:41:05.795 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:05 smithi178 bash[29889]: cluster 2024-09-13T22:41:04.782558+0000 mon.a (mon.0) 934 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-13T22:41:05.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:05.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:05 smithi192 bash[24608]: cluster 2024-09-13T22:41:04.358298+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:05.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:05 smithi192 bash[24608]: cluster 2024-09-13T22:41:04.358365+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:05.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:05 smithi192 bash[24608]: cluster 2024-09-13T22:41:04.481031+0000 mgr.y (mgr.14613) 129 : cluster [DBG] pgmap v103: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:41:05.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:05 smithi192 bash[24608]: cluster 2024-09-13T22:41:04.768730+0000 mon.a (mon.0) 933 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:41:05.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:05 smithi192 bash[24608]: cluster 2024-09-13T22:41:04.782558+0000 mon.a (mon.0) 934 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-13T22:41:05.826 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:05.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T22:41:05.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:05.997 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:05.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:05.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:06.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:06.125 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:06.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:06.254 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:06.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:06.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:06.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:06.514 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:06.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:06.642 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:06.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:06 smithi178 bash[29889]: audit 2024-09-13T22:41:04.934491+0000 mgr.y (mgr.14613) 130 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:06 smithi178 bash[22840]: audit 2024-09-13T22:41:04.934491+0000 mgr.y (mgr.14613) 130 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:06.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:06.769 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:06.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T22:41:06.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:06.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:06 smithi192 bash[24608]: audit 2024-09-13T22:41:04.934491+0000 mgr.y (mgr.14613) 130 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:06.898 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:06.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:07.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:07.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:07.130 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:07.155 DEBUG:teuthology.orchestra.run.smithi178:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-13T22:41:07.156 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:07.157 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T22:41:07.157 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T22:41:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:07.193 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:hints = 1 2024-09-13T22:41:07.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T22:41:07.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_64772 2024-09-13T22:41:07.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:41:07.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-13T22:41:07.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:07.291 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:07.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:07.426 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:07.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:07.566 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:07.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:07.677 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:07 smithi178 bash[22840]: cluster 2024-09-13T22:41:06.481649+0000 mgr.y (mgr.14613) 131 : cluster [DBG] pgmap v105: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:41:07.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:07 smithi178 bash[22840]: cluster 2024-09-13T22:41:06.765600+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:07.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:07 smithi178 bash[22840]: cluster 2024-09-13T22:41:06.765713+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:07.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:07 smithi178 bash[29889]: cluster 2024-09-13T22:41:06.481649+0000 mgr.y (mgr.14613) 131 : cluster [DBG] pgmap v105: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:41:07.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:07 smithi178 bash[29889]: cluster 2024-09-13T22:41:06.765600+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:07.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:07 smithi178 bash[29889]: cluster 2024-09-13T22:41:06.765713+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:07.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:41:07.701 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:07.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:07.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:07 smithi192 bash[24608]: cluster 2024-09-13T22:41:06.481649+0000 mgr.y (mgr.14613) 131 : cluster [DBG] pgmap v105: 148 pgs: 148 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:41:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:07 smithi192 bash[24608]: cluster 2024-09-13T22:41:06.765600+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:07 smithi192 bash[24608]: cluster 2024-09-13T22:41:06.765713+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:07.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:07.961 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:07.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:08.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:08.091 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:08.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 1 16 2828 2812 175.74 175.75 0.00463567 0.00566874 2024-09-13T22:41:08.194 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:08.219 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:08.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:08.350 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:08.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:08.481 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:08.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:08.614 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:08.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:08.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:08 smithi192 bash[24608]: cluster 2024-09-13T22:41:07.497362+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:08.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:08 smithi192 bash[24608]: cluster 2024-09-13T22:41:07.497430+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:08.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:08 smithi192 bash[24608]: cluster 2024-09-13T22:41:07.555809+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-13T22:41:08.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:08.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:08.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:08.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:41:08.868 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:08.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:08 smithi178 bash[29889]: cluster 2024-09-13T22:41:07.497362+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:08.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:08 smithi178 bash[29889]: cluster 2024-09-13T22:41:07.497430+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:08.873 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:08 smithi178 bash[29889]: cluster 2024-09-13T22:41:07.555809+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-13T22:41:08.873 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:08 smithi178 bash[22840]: cluster 2024-09-13T22:41:07.497362+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:08.873 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:08 smithi178 bash[22840]: cluster 2024-09-13T22:41:07.497430+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:08.874 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:08 smithi178 bash[22840]: cluster 2024-09-13T22:41:07.555809+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-13T22:41:08.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:08.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:08.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:09.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:09.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:09.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 2 16 5597 5581 174.341 173.062 0.0089755 0.00571846 2024-09-13T22:41:09.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.273 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:09.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:09.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:09.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:09.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:09.530 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:09.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:09.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:09.662 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:09 smithi178 bash[29889]: cluster 2024-09-13T22:41:08.483482+0000 mgr.y (mgr.14613) 132 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 38 MiB/s wr, 609 op/s 2024-09-13T22:41:09.662 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:09 smithi178 bash[22840]: cluster 2024-09-13T22:41:08.483482+0000 mgr.y (mgr.14613) 132 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 38 MiB/s wr, 609 op/s 2024-09-13T22:41:09.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:09.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:09.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:09.903 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:41:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:41:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:41:09.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:09 smithi192 bash[24608]: cluster 2024-09-13T22:41:08.483482+0000 mgr.y (mgr.14613) 132 : cluster [DBG] pgmap v107: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 38 MiB/s wr, 609 op/s 2024-09-13T22:41:09.921 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:09.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:10.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:10.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:10.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:10.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:10.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 3 16 8409 8393 174.806 175.75 0.00390531 0.00571139 2024-09-13T22:41:10.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:10.306 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:10.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:10.436 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:10.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:10.573 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:10.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:10.701 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:10.732 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:10.732 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:10.804 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:10.831 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:10.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:10.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:11.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:11.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:11.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:11.194 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 4 16 10855 10839 169.321 152.875 0.00445792 0.00589401 2024-09-13T22:41:11.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:11.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:11.340 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:11.443 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:11.474 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:11.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:11.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:11.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:11 smithi192 bash[24608]: cluster 2024-09-13T22:41:10.483908+0000 mgr.y (mgr.14613) 133 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 34 MiB/s wr, 544 op/s 2024-09-13T22:41:11.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:11.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:11 smithi178 bash[29889]: cluster 2024-09-13T22:41:10.483908+0000 mgr.y (mgr.14613) 133 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 34 MiB/s wr, 544 op/s 2024-09-13T22:41:11.718 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:11 smithi178 bash[22840]: cluster 2024-09-13T22:41:10.483908+0000 mgr.y (mgr.14613) 133 : cluster [DBG] pgmap v108: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 34 MiB/s wr, 544 op/s 2024-09-13T22:41:11.736 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:11.751 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:11.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:11.859 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:11.962 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:11.988 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:11.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:12.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:12.118 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:12.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 5 16 12792 12776 159.668 121.062 0.00648698 0.0062548 2024-09-13T22:41:12.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:12.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:12.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:12.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:12.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:12.502 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:12.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:12.632 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:12.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:12.773 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:12.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:12.895 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:12.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:12.998 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:13.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:13.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:13.147 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:13.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 6 16 14412 14396 149.931 101.25 0.0014848 0.00658608 2024-09-13T22:41:13.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:13.275 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:13.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:13.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:13.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:13.525 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:13.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:13.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:13.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:13.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:13.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:13.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:13 smithi192 bash[24608]: cluster 2024-09-13T22:41:12.484434+0000 mgr.y (mgr.14613) 134 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 456 op/s 2024-09-13T22:41:13.906 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:13.908 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:13 smithi178 bash[22840]: cluster 2024-09-13T22:41:12.484434+0000 mgr.y (mgr.14613) 134 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 456 op/s 2024-09-13T22:41:13.909 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:13 smithi178 bash[29889]: cluster 2024-09-13T22:41:12.484434+0000 mgr.y (mgr.14613) 134 : cluster [DBG] pgmap v109: 148 pgs: 148 active+clean; 229 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 456 op/s 2024-09-13T22:41:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:13.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:14.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:14.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:14.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:14.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:14.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 7 16 15938 15922 142.137 95.375 0.00257559 0.00702847 2024-09-13T22:41:14.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:14.289 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:14.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:14.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:14.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:14.553 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:14.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:14.682 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:14.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:14.812 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:14.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:14.940 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:14.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:15.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:15.066 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:15.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:15.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:41:14 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:41:15.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 8 15 17738 17723 138.439 112.562 0.0130629 0.00721502 2024-09-13T22:41:15.195 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:15.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:15.345 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:15.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:15.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:15.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:15.603 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:15.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:15.717 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:15 smithi178 bash[29889]: cluster 2024-09-13T22:41:14.485321+0000 mgr.y (mgr.14613) 135 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 927 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-13T22:41:15.717 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:15 smithi178 bash[29889]: audit 2024-09-13T22:41:14.876080+0000 mon.b (mon.1) 92 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:41:15.717 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:15 smithi178 bash[22840]: cluster 2024-09-13T22:41:14.485321+0000 mgr.y (mgr.14613) 135 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 927 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-13T22:41:15.717 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:15 smithi178 bash[22840]: audit 2024-09-13T22:41:14.876080+0000 mon.b (mon.1) 92 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:41:15.736 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:15.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:15.876 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:15.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:15 smithi192 bash[24608]: cluster 2024-09-13T22:41:14.485321+0000 mgr.y (mgr.14613) 135 : cluster [DBG] pgmap v110: 148 pgs: 148 active+clean; 927 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-13T22:41:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:15 smithi192 bash[24608]: audit 2024-09-13T22:41:14.876080+0000 mon.b (mon.1) 92 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:41:15.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:16.005 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:16.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:16.130 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:16.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 9 15 19665 19650 136.438 120.438 0.00434016 0.00732199 2024-09-13T22:41:16.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.254 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:16.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:16.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:16.506 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:16.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:16.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[29889]: audit 2024-09-13T22:41:14.944323+0000 mgr.y (mgr.14613) 136 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[29889]: audit 2024-09-13T22:41:15.744549+0000 mon.b (mon.1) 93 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:41:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[29889]: audit 2024-09-13T22:41:15.745604+0000 mon.b (mon.1) 94 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:41:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[29889]: audit 2024-09-13T22:41:15.754383+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:16.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[22840]: audit 2024-09-13T22:41:14.944323+0000 mgr.y (mgr.14613) 136 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:16.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[22840]: audit 2024-09-13T22:41:15.744549+0000 mon.b (mon.1) 93 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:41:16.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[22840]: audit 2024-09-13T22:41:15.745604+0000 mon.b (mon.1) 94 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:41:16.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:16 smithi178 bash[22840]: audit 2024-09-13T22:41:15.754383+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:16.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:16.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:16 smithi192 bash[24608]: audit 2024-09-13T22:41:14.944323+0000 mgr.y (mgr.14613) 136 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:16 smithi192 bash[24608]: audit 2024-09-13T22:41:15.744549+0000 mon.b (mon.1) 93 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:41:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:16 smithi192 bash[24608]: audit 2024-09-13T22:41:15.745604+0000 mon.b (mon.1) 94 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:41:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:16 smithi192 bash[24608]: audit 2024-09-13T22:41:15.754383+0000 mon.a (mon.0) 936 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:17.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:17.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:17.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:17.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:17.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 10 16 21410 21394 133.693 109 0.00440943 0.0074357 2024-09-13T22:41:17.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:17.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:17.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:17.418 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:17.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:17.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:17.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:17.660 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:17 smithi178 bash[29889]: cluster 2024-09-13T22:41:16.485840+0000 mgr.y (mgr.14613) 137 : cluster [DBG] pgmap v111: 148 pgs: 148 active+clean; 927 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-13T22:41:17.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:17 smithi178 bash[22840]: cluster 2024-09-13T22:41:16.485840+0000 mgr.y (mgr.14613) 137 : cluster [DBG] pgmap v111: 148 pgs: 148 active+clean; 927 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-13T22:41:17.674 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:17.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:17.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:17 smithi192 bash[24608]: cluster 2024-09-13T22:41:16.485840+0000 mgr.y (mgr.14613) 137 : cluster [DBG] pgmap v111: 148 pgs: 148 active+clean; 927 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-13T22:41:17.909 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:17.932 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:17.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:18.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:18.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:18.170 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:18.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 11 15 23071 23056 130.982 103.875 0.00836361 0.00762834 2024-09-13T22:41:18.198 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:18.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:18.325 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:18.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:18.451 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:18.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:18.576 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:18.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:18.703 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:18.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:18.840 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:18.943 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:18.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:18.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:19.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.100 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:18 smithi192 bash[24608]: audit 2024-09-13T22:41:17.794540+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:18 smithi192 bash[24608]: audit 2024-09-13T22:41:17.796585+0000 mon.b (mon.1) 95 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:18 smithi192 bash[24608]: cluster 2024-09-13T22:41:18.486408+0000 mgr.y (mgr.14613) 138 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 936 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-13T22:41:19.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 12 16 24213 24197 126.009 71.3125 0.00363669 0.00792887 2024-09-13T22:41:19.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.231 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:19.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[22840]: audit 2024-09-13T22:41:17.794540+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:19.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[22840]: audit 2024-09-13T22:41:17.796585+0000 mon.b (mon.1) 95 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:19.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[22840]: cluster 2024-09-13T22:41:18.486408+0000 mgr.y (mgr.14613) 138 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 936 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-13T22:41:19.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[29889]: audit 2024-09-13T22:41:17.794540+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:19.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[29889]: audit 2024-09-13T22:41:17.796585+0000 mon.b (mon.1) 95 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:19.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[29889]: cluster 2024-09-13T22:41:18.486408+0000 mgr.y (mgr.14613) 138 : cluster [DBG] pgmap v112: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 936 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-13T22:41:19.360 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:19.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:19.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:19.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:19.611 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:19.713 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:19.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:19.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:19.873 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:19.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:19.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:19.985 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:41:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:41:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:41:20.004 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:20.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:20.136 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:20.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 13 16 25994 25978 124.878 111.312 0.0198933 0.00800139 2024-09-13T22:41:20.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:20.265 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:20.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:20.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:20.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:20.524 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:20.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:20.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:20.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:20.778 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:20.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:20.904 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:20.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:21.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:21.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:21.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:21.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:21.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 14 16 27608 27592 123.163 100.875 0.0393937 0.00811053 2024-09-13T22:41:21.234 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:41:21.234 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:41:21.238 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:41:21.238 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:41:21.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:21.284 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:21.286 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:41:21.287 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:41:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:21.315 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:41:21.315 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:41:21.348 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:41:21.349 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:41:21.380 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:41:21.381 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:41:21.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:21.410 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:21.414 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:41:21.414 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:41:21.457 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:41:21.457 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:41:21.493 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:41:21.493 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:41:21.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:21.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:21.552 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T22:41:21.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:21.673 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:21.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:21 smithi192 bash[24608]: cluster 2024-09-13T22:41:20.486856+0000 mgr.y (mgr.14613) 139 : cluster [DBG] pgmap v113: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T22:41:21.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:21 smithi192 bash[24608]: audit 2024-09-13T22:41:21.181611+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? v1:172.21.15.178:0/1987497227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:21.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:21 smithi192 bash[24608]: audit 2024-09-13T22:41:21.268761+0000 mon.c (mon.2) 23 : audit [INF] from='client.? v1:172.21.15.178:0/2324430193' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:41:21.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:21 smithi192 bash[24608]: audit 2024-09-13T22:41:21.269165+0000 mon.a (mon.0) 938 : audit [INF] from='client.24770 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:41:21.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:21.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:21.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:21.912 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[29889]: cluster 2024-09-13T22:41:20.486856+0000 mgr.y (mgr.14613) 139 : cluster [DBG] pgmap v113: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T22:41:21.912 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[29889]: audit 2024-09-13T22:41:21.181611+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? v1:172.21.15.178:0/1987497227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:21.912 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[29889]: audit 2024-09-13T22:41:21.268761+0000 mon.c (mon.2) 23 : audit [INF] from='client.? v1:172.21.15.178:0/2324430193' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:41:21.912 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[29889]: audit 2024-09-13T22:41:21.269165+0000 mon.a (mon.0) 938 : audit [INF] from='client.24770 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:41:21.913 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[22840]: cluster 2024-09-13T22:41:20.486856+0000 mgr.y (mgr.14613) 139 : cluster [DBG] pgmap v113: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T22:41:21.913 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[22840]: audit 2024-09-13T22:41:21.181611+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? v1:172.21.15.178:0/1987497227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:21.913 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[22840]: audit 2024-09-13T22:41:21.268761+0000 mon.c (mon.2) 23 : audit [INF] from='client.? v1:172.21.15.178:0/2324430193' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:41:21.913 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:21 smithi178 bash[22840]: audit 2024-09-13T22:41:21.269165+0000 mon.a (mon.0) 938 : audit [INF] from='client.24770 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:41:21.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:21.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:22.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:22.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:22.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:22.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:22.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 15 16 29271 29255 121.881 103.938 0.0105886 0.00819849 2024-09-13T22:41:22.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:22.321 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:22.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:22.450 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:22.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:22.563 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:41:22.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[29889]: audit 2024-09-13T22:41:21.550380+0000 mon.a (mon.0) 939 : audit [INF] from='client.24770 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:41:22.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[29889]: cluster 2024-09-13T22:41:21.558087+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-13T22:41:22.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[29889]: audit 2024-09-13T22:41:21.637237+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.178:0/2149987544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:41:22.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[29889]: audit 2024-09-13T22:41:21.637824+0000 mon.a (mon.0) 941 : audit [INF] from='client.24778 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:41:22.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[22840]: audit 2024-09-13T22:41:21.550380+0000 mon.a (mon.0) 939 : audit [INF] from='client.24770 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:41:22.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[22840]: cluster 2024-09-13T22:41:21.558087+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-13T22:41:22.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[22840]: audit 2024-09-13T22:41:21.637237+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.178:0/2149987544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:41:22.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:22 smithi178 bash[22840]: audit 2024-09-13T22:41:21.637824+0000 mon.a (mon.0) 941 : audit [INF] from='client.24778 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:41:22.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:22.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:22.713 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:22.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:22.825 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:22 smithi192 bash[24608]: audit 2024-09-13T22:41:21.550380+0000 mon.a (mon.0) 939 : audit [INF] from='client.24770 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:41:22.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:22 smithi192 bash[24608]: cluster 2024-09-13T22:41:21.558087+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-13T22:41:22.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:22 smithi192 bash[24608]: audit 2024-09-13T22:41:21.637237+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.178:0/2149987544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:41:22.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:22 smithi192 bash[24608]: audit 2024-09-13T22:41:21.637824+0000 mon.a (mon.0) 941 : audit [INF] from='client.24778 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:41:22.843 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:22.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:22.969 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:22.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:22.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:23.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:23.100 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:23.195 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 16 16 31076 31060 121.313 112.812 0.00619375 0.00823708 2024-09-13T22:41:23.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:23.230 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:23.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:23.360 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:23.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:23.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:23.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:23.618 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:23.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:21.840551+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:23.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:21.840616+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:23.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.487390+0000 mgr.y (mgr.14613) 140 : cluster [DBG] pgmap v115: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-13T22:41:23.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.550914+0000 mon.a (mon.0) 942 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:41:23.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: audit 2024-09-13T22:41:22.561105+0000 mon.a (mon.0) 943 : audit [INF] from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.570946+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.728783+0000 osd.2 (osd.2) 59 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.739385+0000 osd.2 (osd.2) 60 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:21.840551+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:21.840616+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.487390+0000 mgr.y (mgr.14613) 140 : cluster [DBG] pgmap v115: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.550914+0000 mon.a (mon.0) 942 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: audit 2024-09-13T22:41:22.561105+0000 mon.a (mon.0) 943 : audit [INF] from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.570946+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.728783+0000 osd.2 (osd.2) 59 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:41:23.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:23 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.739385+0000 osd.2 (osd.2) 60 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:41:23.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:21.840551+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:21.840616+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.487390+0000 mgr.y (mgr.14613) 140 : cluster [DBG] pgmap v115: 148 pgs: 148 active+clean; 1.4 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-13T22:41:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.550914+0000 mon.a (mon.0) 942 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:41:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: audit 2024-09-13T22:41:22.561105+0000 mon.a (mon.0) 943 : audit [INF] from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:41:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.570946+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-13T22:41:23.732 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.728783+0000 osd.2 (osd.2) 59 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:41:23.732 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:23 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.739385+0000 osd.2 (osd.2) 60 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:41:23.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:23.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:23.871 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:23.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:23.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:24.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:24.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:24.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 17 16 32527 32511 119.511 90.6875 0.00972613 0.0083414 2024-09-13T22:41:24.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:24.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:24.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:24.386 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:24.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.517 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:24.622 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:24 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.941322+0000 osd.5 (osd.5) 83 : cluster [DBG] 5.f scrub starts 2024-09-13T22:41:24.622 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:24 smithi178 bash[22840]: cluster 2024-09-13T22:41:22.943017+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.f scrub ok 2024-09-13T22:41:24.622 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.648 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:24.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:24 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.941322+0000 osd.5 (osd.5) 83 : cluster [DBG] 5.f scrub starts 2024-09-13T22:41:24.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:24 smithi192 bash[24608]: cluster 2024-09-13T22:41:22.943017+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.f scrub ok 2024-09-13T22:41:24.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:24.862 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:24 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.941322+0000 osd.5 (osd.5) 83 : cluster [DBG] 5.f scrub starts 2024-09-13T22:41:24.863 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:24 smithi178 bash[29889]: cluster 2024-09-13T22:41:22.943017+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.f scrub ok 2024-09-13T22:41:24.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:24.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:24.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:25.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:25.050 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:25.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:25.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:41:24 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:41:25.181 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:25.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 18 16 33573 33557 116.503 65.375 0.0041816 0.00857004 2024-09-13T22:41:25.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T22:41:25.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:25.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:25.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:25.480 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:25.561 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:25 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.488372+0000 mgr.y (mgr.14613) 141 : cluster [DBG] pgmap v117: 148 pgs: 148 active+clean; 1.9 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-09-13T22:41:25.561 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:25 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.715930+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:41:25.561 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:25 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.717959+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:41:25.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:25.614 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:25.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:25.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:25.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:25.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:25.850 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:25 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.488372+0000 mgr.y (mgr.14613) 141 : cluster [DBG] pgmap v117: 148 pgs: 148 active+clean; 1.9 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-09-13T22:41:25.850 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:25 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.715930+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:41:25.851 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:25 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.717959+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:41:25.865 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:25.965 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:25 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.488372+0000 mgr.y (mgr.14613) 141 : cluster [DBG] pgmap v117: 148 pgs: 148 active+clean; 1.9 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-09-13T22:41:25.965 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:25 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.715930+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:41:25.965 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:25 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.717959+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:41:25.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:25.994 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:25.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:26.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:26.130 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:26.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 19 15 34540 34525 113.556 60.5 0.00837533 0.00880053 2024-09-13T22:41:26.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:26.256 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:26.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:26.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:26.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:26.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:26.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.617839+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:41:26.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.620409+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:41:26.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.933970+0000 osd.5 (osd.5) 85 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:41:26.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:24.936269+0000 osd.5 (osd.5) 86 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:41:26.635 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: audit 2024-09-13T22:41:24.954365+0000 mgr.y (mgr.14613) 142 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:26.635 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:25.540513+0000 osd.0 (osd.0) 81 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:41:26.635 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:25.543826+0000 osd.0 (osd.0) 82 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:41:26.635 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:26.070017+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:41:26.635 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[29889]: cluster 2024-09-13T22:41:26.073048+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:41:26.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.617839+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.620409+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.933970+0000 osd.5 (osd.5) 85 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:24.936269+0000 osd.5 (osd.5) 86 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: audit 2024-09-13T22:41:24.954365+0000 mgr.y (mgr.14613) 142 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:25.540513+0000 osd.0 (osd.0) 81 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:25.543826+0000 osd.0 (osd.0) 82 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:26.070017+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:41:26.636 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:26 smithi178 bash[22840]: cluster 2024-09-13T22:41:26.073048+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:41:26.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:26.766 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:26.799 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:26.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.617839+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:41:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.620409+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:41:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.933970+0000 osd.5 (osd.5) 85 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:41:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:24.936269+0000 osd.5 (osd.5) 86 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:41:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: audit 2024-09-13T22:41:24.954365+0000 mgr.y (mgr.14613) 142 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:26.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:25.540513+0000 osd.0 (osd.0) 81 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:41:26.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:25.543826+0000 osd.0 (osd.0) 82 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:41:26.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:26.070017+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:41:26.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:26 smithi192 bash[24608]: cluster 2024-09-13T22:41:26.073048+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:41:26.922 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:26.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:27.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:27.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:27.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:27.178 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:27.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:41:27.200883+0000 min lat: 0.00140062 max lat: 0.407633 avg lat: 0.00891914 2024-09-13T22:41:27.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:41:27.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 20 15 35877 35862 112.056 83.5625 0.0145164 0.00891914 2024-09-13T22:41:27.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T22:41:27.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:27.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:27.485 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:27.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:27.612 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:27.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:27.746 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:27.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:27 smithi178 bash[29889]: cluster 2024-09-13T22:41:26.488835+0000 mgr.y (mgr.14613) 143 : cluster [DBG] pgmap v118: 148 pgs: 148 active+clean; 1.9 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-13T22:41:27.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:27 smithi178 bash[29889]: cluster 2024-09-13T22:41:26.619205+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:41:27.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:27 smithi178 bash[29889]: cluster 2024-09-13T22:41:26.621649+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:41:27.748 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:27 smithi178 bash[22840]: cluster 2024-09-13T22:41:26.488835+0000 mgr.y (mgr.14613) 143 : cluster [DBG] pgmap v118: 148 pgs: 148 active+clean; 1.9 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-13T22:41:27.748 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:27 smithi178 bash[22840]: cluster 2024-09-13T22:41:26.619205+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:41:27.748 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:27 smithi178 bash[22840]: cluster 2024-09-13T22:41:26.621649+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:41:27.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:27.884 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:27.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:27.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:27 smithi192 bash[24608]: cluster 2024-09-13T22:41:26.488835+0000 mgr.y (mgr.14613) 143 : cluster [DBG] pgmap v118: 148 pgs: 148 active+clean; 1.9 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-13T22:41:27.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:27 smithi192 bash[24608]: cluster 2024-09-13T22:41:26.619205+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:41:27.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:27 smithi192 bash[24608]: cluster 2024-09-13T22:41:26.621649+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:41:27.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:28.017 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:28.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:28.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:28.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 21 16 37388 37372 111.213 94.375 0.00900429 0.00898531 2024-09-13T22:41:28.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:28.275 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:28.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:28.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:28.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:28.549 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:28.652 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:28.688 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:28.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:28.820 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:28.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[22840]: cluster 2024-09-13T22:41:27.662044+0000 osd.2 (osd.2) 61 : cluster [DBG] 2.c scrub starts 2024-09-13T22:41:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[22840]: cluster 2024-09-13T22:41:27.668212+0000 osd.2 (osd.2) 62 : cluster [DBG] 2.c scrub ok 2024-09-13T22:41:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[22840]: cluster 2024-09-13T22:41:27.789101+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:41:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[22840]: cluster 2024-09-13T22:41:27.793487+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:41:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[22840]: cluster 2024-09-13T22:41:28.489643+0000 mgr.y (mgr.14613) 144 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 2.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-13T22:41:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[29889]: cluster 2024-09-13T22:41:27.662044+0000 osd.2 (osd.2) 61 : cluster [DBG] 2.c scrub starts 2024-09-13T22:41:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[29889]: cluster 2024-09-13T22:41:27.668212+0000 osd.2 (osd.2) 62 : cluster [DBG] 2.c scrub ok 2024-09-13T22:41:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[29889]: cluster 2024-09-13T22:41:27.789101+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:41:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[29889]: cluster 2024-09-13T22:41:27.793487+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:41:28.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:28 smithi178 bash[29889]: cluster 2024-09-13T22:41:28.489643+0000 mgr.y (mgr.14613) 144 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 2.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-13T22:41:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:28.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:29.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:29.080 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:28 smithi192 bash[24608]: cluster 2024-09-13T22:41:27.662044+0000 osd.2 (osd.2) 61 : cluster [DBG] 2.c scrub starts 2024-09-13T22:41:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:28 smithi192 bash[24608]: cluster 2024-09-13T22:41:27.668212+0000 osd.2 (osd.2) 62 : cluster [DBG] 2.c scrub ok 2024-09-13T22:41:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:28 smithi192 bash[24608]: cluster 2024-09-13T22:41:27.789101+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:41:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:28 smithi192 bash[24608]: cluster 2024-09-13T22:41:27.793487+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:41:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:28 smithi192 bash[24608]: cluster 2024-09-13T22:41:28.489643+0000 mgr.y (mgr.14613) 144 : cluster [DBG] pgmap v119: 148 pgs: 148 active+clean; 2.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-13T22:41:29.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:29.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 22 16 39056 39040 110.896 104.25 0.0188822 0.00901174 2024-09-13T22:41:29.205 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:29.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:29.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:29.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:41:29.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:29.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:41:29.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:29.620 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:29.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:29.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:29.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:29.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:29.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[22840]: cluster 2024-09-13T22:41:28.612549+0000 osd.0 (osd.0) 83 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:41:29.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[22840]: cluster 2024-09-13T22:41:28.619559+0000 osd.0 (osd.0) 84 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:41:29.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[22840]: cluster 2024-09-13T22:41:28.656968+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:41:29.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[22840]: cluster 2024-09-13T22:41:28.659115+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:41:29.919 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:41:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:41:29.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[29889]: cluster 2024-09-13T22:41:28.612549+0000 osd.0 (osd.0) 83 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:41:29.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[29889]: cluster 2024-09-13T22:41:28.619559+0000 osd.0 (osd.0) 84 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:41:29.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[29889]: cluster 2024-09-13T22:41:28.656968+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:41:29.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:29 smithi178 bash[29889]: cluster 2024-09-13T22:41:28.659115+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:41:29.932 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:29.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:30.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.069 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:30.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:30.114 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:30.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:29 smithi192 bash[24608]: cluster 2024-09-13T22:41:28.612549+0000 osd.0 (osd.0) 83 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:41:30.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:29 smithi192 bash[24608]: cluster 2024-09-13T22:41:28.619559+0000 osd.0 (osd.0) 84 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:41:30.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:29 smithi192 bash[24608]: cluster 2024-09-13T22:41:28.656968+0000 osd.2 (osd.2) 63 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:41:30.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:29 smithi192 bash[24608]: cluster 2024-09-13T22:41:28.659115+0000 osd.2 (osd.2) 64 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:41:30.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 23 16 40530 40514 110.08 92.125 0.00616117 0.00907825 2024-09-13T22:41:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:30.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:30.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.448 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:30.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:30.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:30.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:30.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:30.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.841 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:30.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:30.970 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:30.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:31.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:31.082 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[22840]: cluster 2024-09-13T22:41:29.611723+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:41:31.083 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[22840]: cluster 2024-09-13T22:41:29.616659+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:41:31.083 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[22840]: cluster 2024-09-13T22:41:29.669396+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.e scrub starts 2024-09-13T22:41:31.083 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[22840]: cluster 2024-09-13T22:41:29.671356+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.e scrub ok 2024-09-13T22:41:31.083 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[22840]: cluster 2024-09-13T22:41:30.490393+0000 mgr.y (mgr.14613) 145 : cluster [DBG] pgmap v120: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-13T22:41:31.083 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[29889]: cluster 2024-09-13T22:41:29.611723+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:41:31.083 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[29889]: cluster 2024-09-13T22:41:29.616659+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:41:31.084 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[29889]: cluster 2024-09-13T22:41:29.669396+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.e scrub starts 2024-09-13T22:41:31.084 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[29889]: cluster 2024-09-13T22:41:29.671356+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.e scrub ok 2024-09-13T22:41:31.084 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:30 smithi178 bash[29889]: cluster 2024-09-13T22:41:30.490393+0000 mgr.y (mgr.14613) 145 : cluster [DBG] pgmap v120: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-13T22:41:31.104 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:30 smithi192 bash[24608]: cluster 2024-09-13T22:41:29.611723+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:41:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:30 smithi192 bash[24608]: cluster 2024-09-13T22:41:29.616659+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:41:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:30 smithi192 bash[24608]: cluster 2024-09-13T22:41:29.669396+0000 osd.4 (osd.4) 75 : cluster [DBG] 6.e scrub starts 2024-09-13T22:41:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:30 smithi192 bash[24608]: cluster 2024-09-13T22:41:29.671356+0000 osd.4 (osd.4) 76 : cluster [DBG] 6.e scrub ok 2024-09-13T22:41:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:30 smithi192 bash[24608]: cluster 2024-09-13T22:41:30.490393+0000 mgr.y (mgr.14613) 145 : cluster [DBG] pgmap v120: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-13T22:41:31.196 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 24 15 42115 42100 109.623 99.125 0.0167112 0.0091169 2024-09-13T22:41:31.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:31.228 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:31.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:31.355 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:31.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:31.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:31.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:31.623 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:31.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:31.749 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:31.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:31.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:31 smithi192 bash[24608]: cluster 2024-09-13T22:41:30.693753+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:41:31.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:31 smithi192 bash[24608]: cluster 2024-09-13T22:41:30.696283+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:41:31.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:31 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.540502+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.d scrub starts 2024-09-13T22:41:31.861 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:31 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.544516+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.d scrub ok 2024-09-13T22:41:31.873 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:31.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:31.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:32.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:32.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:32.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[22840]: cluster 2024-09-13T22:41:30.693753+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:41:32.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[22840]: cluster 2024-09-13T22:41:30.696283+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:41:32.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.540502+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.d scrub starts 2024-09-13T22:41:32.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.544516+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.d scrub ok 2024-09-13T22:41:32.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[29889]: cluster 2024-09-13T22:41:30.693753+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:41:32.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[29889]: cluster 2024-09-13T22:41:30.696283+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:41:32.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.540502+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.d scrub starts 2024-09-13T22:41:32.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:31 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.544516+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.d scrub ok 2024-09-13T22:41:32.140 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:32.155 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:32.198 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 25 16 43638 43622 109.038 95.125 0.0148115 0.00916548 2024-09-13T22:41:32.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:32.273 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:32.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:32.402 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:32.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:32.537 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:32.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:32.665 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:32.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:32.805 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:32.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.646961+0000 osd.0 (osd.0) 87 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.650791+0000 osd.0 (osd.0) 88 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.739958+0000 osd.6 (osd.6) 57 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.742948+0000 osd.6 (osd.6) 58 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.807050+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:31.810807+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.490787+0000 mgr.y (mgr.14613) 146 : cluster [DBG] pgmap v121: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.680007+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.680065+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:32.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:32 smithi192 bash[24608]: audit 2024-09-13T22:41:32.817820+0000 mon.b (mon.1) 98 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:32.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:32.931 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:32.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:33.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:33.060 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:33.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:33.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.646961+0000 osd.0 (osd.0) 87 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.650791+0000 osd.0 (osd.0) 88 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.739958+0000 osd.6 (osd.6) 57 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.742948+0000 osd.6 (osd.6) 58 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.807050+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:31.810807+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.490787+0000 mgr.y (mgr.14613) 146 : cluster [DBG] pgmap v121: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.680007+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.680065+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: audit 2024-09-13T22:41:32.817820+0000 mon.b (mon.1) 98 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.646961+0000 osd.0 (osd.0) 87 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:41:33.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.650791+0000 osd.0 (osd.0) 88 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.739958+0000 osd.6 (osd.6) 57 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.742948+0000 osd.6 (osd.6) 58 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.807050+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:31.810807+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.490787+0000 mgr.y (mgr.14613) 146 : cluster [DBG] pgmap v121: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.680007+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.680065+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:33.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: audit 2024-09-13T22:41:32.817820+0000 mon.b (mon.1) 98 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:33.191 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:33.197 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 26 16 44827 44811 107.702 74.3125 0.00596578 0.00927835 2024-09-13T22:41:33.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:33.313 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:33.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:33.444 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:33.547 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:33.579 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:33.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:33.712 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:33.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:33.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:33.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.629520+0000 osd.7 (osd.7) 85 : cluster [DBG] 3.f scrub starts 2024-09-13T22:41:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.631807+0000 osd.7 (osd.7) 86 : cluster [DBG] 3.f scrub ok 2024-09-13T22:41:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.992713+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.d scrub starts 2024-09-13T22:41:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.994974+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-13T22:41:33.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[29889]: cluster 2024-09-13T22:41:32.995353+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.d scrub ok 2024-09-13T22:41:33.957 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.629520+0000 osd.7 (osd.7) 85 : cluster [DBG] 3.f scrub starts 2024-09-13T22:41:33.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.631807+0000 osd.7 (osd.7) 86 : cluster [DBG] 3.f scrub ok 2024-09-13T22:41:33.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.992713+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.d scrub starts 2024-09-13T22:41:33.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.994974+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-13T22:41:33.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:33 smithi178 bash[22840]: cluster 2024-09-13T22:41:32.995353+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.d scrub ok 2024-09-13T22:41:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:33.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:34.079 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:34.102 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:33 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.629520+0000 osd.7 (osd.7) 85 : cluster [DBG] 3.f scrub starts 2024-09-13T22:41:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:33 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.631807+0000 osd.7 (osd.7) 86 : cluster [DBG] 3.f scrub ok 2024-09-13T22:41:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:33 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.992713+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.d scrub starts 2024-09-13T22:41:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:33 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.994974+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-13T22:41:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:33 smithi192 bash[24608]: cluster 2024-09-13T22:41:32.995353+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.d scrub ok 2024-09-13T22:41:34.193 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:34.195 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:34.197 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 27 16 46088 46072 106.632 78.8125 0.0263352 0.0093728 2024-09-13T22:41:34.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:34.230 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:34.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:34.357 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:34.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:34.487 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:34.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:34.624 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:34.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:34.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:34.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:34.898 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:34.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:35.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:35.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:35.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:35.136 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.633209+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.637513+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.650450+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.652581+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.703516+0000 osd.6 (osd.6) 59 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.704722+0000 osd.6 (osd.6) 60 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.852053+0000 osd.3 (osd.3) 93 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:33.857355+0000 osd.3 (osd.3) 94 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:41:35.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:34 smithi192 bash[24608]: cluster 2024-09-13T22:41:34.492062+0000 mgr.y (mgr.14613) 147 : cluster [DBG] pgmap v123: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:41:35.146 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:35.188 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.633209+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:41:35.188 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.637513+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.650450+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.652581+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.703516+0000 osd.6 (osd.6) 59 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.704722+0000 osd.6 (osd.6) 60 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.852053+0000 osd.3 (osd.3) 93 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:33.857355+0000 osd.3 (osd.3) 94 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[29889]: cluster 2024-09-13T22:41:34.492062+0000 mgr.y (mgr.14613) 147 : cluster [DBG] pgmap v123: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:41:35.189 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.633209+0000 osd.0 (osd.0) 89 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.637513+0000 osd.0 (osd.0) 90 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.650450+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.652581+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.703516+0000 osd.6 (osd.6) 59 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.704722+0000 osd.6 (osd.6) 60 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.852053+0000 osd.3 (osd.3) 93 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:33.857355+0000 osd.3 (osd.3) 94 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:41:35.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:34 smithi178 bash[22840]: cluster 2024-09-13T22:41:34.492062+0000 mgr.y (mgr.14613) 147 : cluster [DBG] pgmap v123: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:41:35.197 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 28 16 47243 47227 105.402 72.1875 0.0081362 0.00948095 2024-09-13T22:41:35.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:35.276 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:35.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:35.402 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:35.506 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:35.534 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:35.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:35.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:35.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:35.798 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:35.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:35.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:35.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:36.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:36.040 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:35 smithi178 bash[29889]: audit 2024-09-13T22:41:34.962445+0000 mgr.y (mgr.14613) 148 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:36.040 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:35 smithi178 bash[22840]: audit 2024-09-13T22:41:34.962445+0000 mgr.y (mgr.14613) 148 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:36.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:35 smithi192 bash[24608]: audit 2024-09-13T22:41:34.962445+0000 mgr.y (mgr.14613) 148 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:36.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:36.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:36.198 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 29 16 48674 48658 104.851 89.4375 0.00385979 0.00952925 2024-09-13T22:41:36.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:36.307 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:36.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:36.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:36.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:36.558 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:36.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:36.691 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:36.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:36.840 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:36.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[29889]: cluster 2024-09-13T22:41:35.692397+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:41:36.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[29889]: cluster 2024-09-13T22:41:35.694057+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:41:36.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[29889]: cluster 2024-09-13T22:41:35.872840+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.c scrub starts 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[29889]: cluster 2024-09-13T22:41:35.878340+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.c scrub ok 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[29889]: cluster 2024-09-13T22:41:36.492604+0000 mgr.y (mgr.14613) 149 : cluster [DBG] pgmap v124: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[22840]: cluster 2024-09-13T22:41:35.692397+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[22840]: cluster 2024-09-13T22:41:35.694057+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[22840]: cluster 2024-09-13T22:41:35.872840+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.c scrub starts 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[22840]: cluster 2024-09-13T22:41:35.878340+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.c scrub ok 2024-09-13T22:41:36.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:36 smithi178 bash[22840]: cluster 2024-09-13T22:41:36.492604+0000 mgr.y (mgr.14613) 149 : cluster [DBG] pgmap v124: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:41:36.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:36.971 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:36.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:37.059 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:41:37.059 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:41:37.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:37.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:37.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:36 smithi192 bash[24608]: cluster 2024-09-13T22:41:35.692397+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:41:37.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:36 smithi192 bash[24608]: cluster 2024-09-13T22:41:35.694057+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:41:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:36 smithi192 bash[24608]: cluster 2024-09-13T22:41:35.872840+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.c scrub starts 2024-09-13T22:41:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:36 smithi192 bash[24608]: cluster 2024-09-13T22:41:35.878340+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.c scrub ok 2024-09-13T22:41:37.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:36 smithi192 bash[24608]: cluster 2024-09-13T22:41:36.492604+0000 mgr.y (mgr.14613) 149 : cluster [DBG] pgmap v124: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:41:37.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:37.200 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 30 15 49774 49759 103.644 68.8125 0.129149 0.00961319 2024-09-13T22:41:37.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:37.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:37.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:37.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:37.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:37.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:37.623 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:37.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:37.753 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:37.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:37.883 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:37.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:37.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:38.011 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:38.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:38.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[29889]: cluster 2024-09-13T22:41:36.593460+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:41:38.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[29889]: cluster 2024-09-13T22:41:36.600149+0000 osd.4 (osd.4) 78 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:41:38.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[29889]: cluster 2024-09-13T22:41:36.918066+0000 osd.3 (osd.3) 97 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:41:38.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[29889]: cluster 2024-09-13T22:41:36.926230+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:41:38.125 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[22840]: cluster 2024-09-13T22:41:36.593460+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:41:38.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[22840]: cluster 2024-09-13T22:41:36.600149+0000 osd.4 (osd.4) 78 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:41:38.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[22840]: cluster 2024-09-13T22:41:36.918066+0000 osd.3 (osd.3) 97 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:41:38.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:37 smithi178 bash[22840]: cluster 2024-09-13T22:41:36.926230+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:41:38.148 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:38.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:37 smithi192 bash[24608]: cluster 2024-09-13T22:41:36.593460+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:41:38.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:37 smithi192 bash[24608]: cluster 2024-09-13T22:41:36.600149+0000 osd.4 (osd.4) 78 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:41:38.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:37 smithi192 bash[24608]: cluster 2024-09-13T22:41:36.918066+0000 osd.3 (osd.3) 97 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:41:38.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:37 smithi192 bash[24608]: cluster 2024-09-13T22:41:36.926230+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:41:38.199 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 31 16 51016 51000 102.803 77.5625 0.00285412 0.00971943 2024-09-13T22:41:38.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:38.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:38.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:38.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:38.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:38.544 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:38.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:38.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:38.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:38.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:38.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:38.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:38.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:38.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:39.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:38 smithi178 bash[22840]: cluster 2024-09-13T22:41:37.544286+0000 osd.0 (osd.0) 91 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:41:39.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:38 smithi178 bash[22840]: cluster 2024-09-13T22:41:37.549988+0000 osd.0 (osd.0) 92 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:41:39.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:38 smithi178 bash[22840]: cluster 2024-09-13T22:41:37.934693+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:41:39.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:38 smithi178 bash[22840]: cluster 2024-09-13T22:41:37.950029+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:41:39.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:38 smithi178 bash[22840]: cluster 2024-09-13T22:41:38.493353+0000 mgr.y (mgr.14613) 150 : cluster [DBG] pgmap v125: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:41:39.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:39.151 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: cluster 2024-09-13T22:41:37.544286+0000 osd.0 (osd.0) 91 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:41:39.151 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: cluster 2024-09-13T22:41:37.549988+0000 osd.0 (osd.0) 92 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:41:39.151 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: cluster 2024-09-13T22:41:37.934693+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:41:39.151 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: cluster 2024-09-13T22:41:37.950029+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:41:39.151 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: cluster 2024-09-13T22:41:38.493353+0000 mgr.y (mgr.14613) 150 : cluster [DBG] pgmap v125: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:41:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:38 smithi192 bash[24608]: cluster 2024-09-13T22:41:37.544286+0000 osd.0 (osd.0) 91 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:41:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:38 smithi192 bash[24608]: cluster 2024-09-13T22:41:37.549988+0000 osd.0 (osd.0) 92 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:41:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:38 smithi192 bash[24608]: cluster 2024-09-13T22:41:37.934693+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:41:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:38 smithi192 bash[24608]: cluster 2024-09-13T22:41:37.950029+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:41:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:38 smithi192 bash[24608]: cluster 2024-09-13T22:41:38.493353+0000 mgr.y (mgr.14613) 150 : cluster [DBG] pgmap v125: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:41:39.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:39.187 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:39.199 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 32 15 52072 52057 101.655 66.0625 0.0164385 0.00983164 2024-09-13T22:41:39.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:39.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:39.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:39.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:39.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:39.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:39.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:39.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:39.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:39.839 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:39.878 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T22:41:39.943 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:39.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:39.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:40.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:40.092 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[22840]: audit 2024-09-13T22:41:39.652688+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.178:0/1632461042' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:41:40.092 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[22840]: audit 2024-09-13T22:41:39.653490+0000 mon.a (mon.0) 946 : audit [INF] from='client.24784 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:41:40.092 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:41:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:41:40.093 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: audit 2024-09-13T22:41:39.652688+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.178:0/1632461042' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:41:40.093 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:39 smithi178 bash[29889]: audit 2024-09-13T22:41:39.653490+0000 mon.a (mon.0) 946 : audit [INF] from='client.24784 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:41:40.105 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:40.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:39 smithi192 bash[24608]: audit 2024-09-13T22:41:39.652688+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.178:0/1632461042' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:41:40.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:39 smithi192 bash[24608]: audit 2024-09-13T22:41:39.653490+0000 mon.a (mon.0) 946 : audit [INF] from='client.24784 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:41:40.201 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 33 15 53235 53220 100.772 72.6875 0.00482967 0.00991796 2024-09-13T22:41:40.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:40.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:40.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:40.382 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:40.390 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T22:41:40.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:40.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:40.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:40.647 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:40.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:40.775 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:40.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:40.902 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:40.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:41.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:41.024 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:41.130 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:41.144 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.571082+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:41:41.144 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.580375+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:41:41.144 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.610861+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.615589+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.745107+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.746643+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: audit 2024-09-13T22:41:39.878946+0000 mon.a (mon.0) 947 : audit [INF] from='client.24784 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:39.886686+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[29889]: cluster 2024-09-13T22:41:40.493776+0000 mgr.y (mgr.14613) 151 : cluster [DBG] pgmap v127: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.571082+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.580375+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.610861+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.615589+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:41:41.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.745107+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:41:41.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.746643+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:41:41.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: audit 2024-09-13T22:41:39.878946+0000 mon.a (mon.0) 947 : audit [INF] from='client.24784 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:41:41.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:39.886686+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-13T22:41:41.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:40 smithi178 bash[22840]: cluster 2024-09-13T22:41:40.493776+0000 mgr.y (mgr.14613) 151 : cluster [DBG] pgmap v127: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:41:41.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.571082+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:41:41.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.580375+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:41:41.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.610861+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:41:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.615589+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:41:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.745107+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:41:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.746643+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:41:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: audit 2024-09-13T22:41:39.878946+0000 mon.a (mon.0) 947 : audit [INF] from='client.24784 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:41:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:39.886686+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-13T22:41:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:40 smithi192 bash[24608]: cluster 2024-09-13T22:41:40.493776+0000 mgr.y (mgr.14613) 151 : cluster [DBG] pgmap v127: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:41:41.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:41.201 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 34 16 54239 54223 99.6523 62.6875 0.00854875 0.0100296 2024-09-13T22:41:41.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:41.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:41.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:41.418 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:41.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:41.546 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:41.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:41.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:41.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:41.809 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:41.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:40.677054+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:40.678629+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:40.740273+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:40.741734+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.028067+0000 osd.5 (osd.5) 93 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.032812+0000 osd.5 (osd.5) 94 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.526706+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:41:41.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:41 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.531830+0000 osd.1 (osd.1) 94 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:41:41.936 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:41.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:41.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T22:41:42.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:40.677054+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:40.678629+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:40.740273+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:40.741734+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.028067+0000 osd.5 (osd.5) 93 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.032812+0000 osd.5 (osd.5) 94 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.526706+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:41:42.056 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.531830+0000 osd.1 (osd.1) 94 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:40.677054+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:40.678629+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:40.740273+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:40.741734+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.028067+0000 osd.5 (osd.5) 93 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.032812+0000 osd.5 (osd.5) 94 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.526706+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:41:42.057 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:41 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.531830+0000 osd.1 (osd.1) 94 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:41:42.064 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:42.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:42.189 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:42.201 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 35 15 55376 55361 98.8372 71.125 0.0263844 0.0101116 2024-09-13T22:41:42.292 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:41:42.292 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:41:42.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:42.326 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:42.336 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:41:42.336 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:41:42.387 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:41:42.388 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:41:42.429 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:41:42.430 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:41:42.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:42.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:42.463 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:41:42.464 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:41:42.502 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:41:42.503 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:41:42.538 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:41:42.539 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:41:42.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:42.569 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:41:42.570 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:41:42.589 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:42.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:42.721 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:42.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:42.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:42.899 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:41:42.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:42.984 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:42.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:43.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:43.114 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.521920+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.525560+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.758988+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: cluster 2024-09-13T22:41:41.762934+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: audit 2024-09-13T22:41:42.227502+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.178:0/1157987564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: cluster 2024-09-13T22:41:42.494189+0000 mgr.y (mgr.14613) 152 : cluster [DBG] pgmap v128: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T22:41:43.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: audit 2024-09-13T22:41:42.643101+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.178:0/2342122248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[29889]: audit 2024-09-13T22:41:42.644427+0000 mon.a (mon.0) 949 : audit [INF] from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.521920+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.525560+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.758988+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: cluster 2024-09-13T22:41:41.762934+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: audit 2024-09-13T22:41:42.227502+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.178:0/1157987564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: cluster 2024-09-13T22:41:42.494189+0000 mgr.y (mgr.14613) 152 : cluster [DBG] pgmap v128: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: audit 2024-09-13T22:41:42.643101+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.178:0/2342122248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:42 smithi178 bash[22840]: audit 2024-09-13T22:41:42.644427+0000 mon.a (mon.0) 949 : audit [INF] from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:43.201 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 36 16 56496 56480 98.0343 69.9375 0.011666 0.010195 2024-09-13T22:41:43.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.521920+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.525560+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.758988+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.a scrub starts 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: cluster 2024-09-13T22:41:41.762934+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.a scrub ok 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: audit 2024-09-13T22:41:42.227502+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.178:0/1157987564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: cluster 2024-09-13T22:41:42.494189+0000 mgr.y (mgr.14613) 152 : cluster [DBG] pgmap v128: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: audit 2024-09-13T22:41:42.643101+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.178:0/2342122248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:43.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:42 smithi192 bash[24608]: audit 2024-09-13T22:41:42.644427+0000 mon.a (mon.0) 949 : audit [INF] from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:41:43.241 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:43.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:43.370 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:43.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:43.509 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:43.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:43.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:43.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:43.761 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:43.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:43.889 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:43.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:43.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:44.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:44.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:44.127 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:43 smithi192 bash[24608]: cluster 2024-09-13T22:41:42.896185+0000 mon.a (mon.0) 950 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:41:44.127 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:43 smithi192 bash[24608]: cluster 2024-09-13T22:41:42.896207+0000 mon.a (mon.0) 951 : cluster [INF] Cluster is now healthy 2024-09-13T22:41:44.127 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:43 smithi192 bash[24608]: audit 2024-09-13T22:41:42.897781+0000 mon.a (mon.0) 952 : audit [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:41:44.127 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:43 smithi192 bash[24608]: cluster 2024-09-13T22:41:42.906149+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-13T22:41:44.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[29889]: cluster 2024-09-13T22:41:42.896185+0000 mon.a (mon.0) 950 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[29889]: cluster 2024-09-13T22:41:42.896207+0000 mon.a (mon.0) 951 : cluster [INF] Cluster is now healthy 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[29889]: audit 2024-09-13T22:41:42.897781+0000 mon.a (mon.0) 952 : audit [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[29889]: cluster 2024-09-13T22:41:42.906149+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[22840]: cluster 2024-09-13T22:41:42.896185+0000 mon.a (mon.0) 950 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[22840]: cluster 2024-09-13T22:41:42.896207+0000 mon.a (mon.0) 951 : cluster [INF] Cluster is now healthy 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[22840]: audit 2024-09-13T22:41:42.897781+0000 mon.a (mon.0) 952 : audit [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:41:44.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:43 smithi178 bash[22840]: cluster 2024-09-13T22:41:42.906149+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-13T22:41:44.201 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 37 15 57603 57588 97.2562 69.25 0.00663087 0.0102764 2024-09-13T22:41:44.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:44.278 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:44.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:44.413 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:44.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:44.543 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:44.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:44.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:44.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:44.808 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:44.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:44.943 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:44.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:45.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:45.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:41:44 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:41:45.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:44 smithi192 bash[24608]: cluster 2024-09-13T22:41:43.723247+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T22:41:45.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:44 smithi192 bash[24608]: cluster 2024-09-13T22:41:43.728101+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T22:41:45.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:44 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.488452+0000 osd.1 (osd.1) 95 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:41:45.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:44 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.490920+0000 osd.1 (osd.1) 96 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:41:45.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:44 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.495087+0000 mgr.y (mgr.14613) 153 : cluster [DBG] pgmap v130: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-13T22:41:45.072 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:45.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:45.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[29889]: cluster 2024-09-13T22:41:43.723247+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T22:41:45.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[29889]: cluster 2024-09-13T22:41:43.728101+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T22:41:45.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.488452+0000 osd.1 (osd.1) 95 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:41:45.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.490920+0000 osd.1 (osd.1) 96 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:41:45.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.495087+0000 mgr.y (mgr.14613) 153 : cluster [DBG] pgmap v130: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-13T22:41:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[22840]: cluster 2024-09-13T22:41:43.723247+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T22:41:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[22840]: cluster 2024-09-13T22:41:43.728101+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T22:41:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.488452+0000 osd.1 (osd.1) 95 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:41:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.490920+0000 osd.1 (osd.1) 96 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:41:45.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:44 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.495087+0000 mgr.y (mgr.14613) 153 : cluster [DBG] pgmap v130: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-13T22:41:45.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 38 16 58815 58799 96.6839 75.6875 0.00483991 0.0103353 2024-09-13T22:41:45.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:45.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:45.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:45.439 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:45.465 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:45.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:45.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:45.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:45.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:45.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:45.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:45.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:45.988 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:45.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:46.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:46.117 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.556993+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.560365+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.681963+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.682856+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.886191+0000 osd.3 (osd.3) 99 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.888893+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:41:46.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.956913+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:44.960379+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: audit 2024-09-13T22:41:44.970484+0000 mgr.y (mgr.14613) 154 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[29889]: cluster 2024-09-13T22:41:45.896718+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.556993+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.560365+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.681963+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.682856+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.886191+0000 osd.3 (osd.3) 99 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.888893+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.956913+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:41:46.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:44.960379+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:41:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: audit 2024-09-13T22:41:44.970484+0000 mgr.y (mgr.14613) 154 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:45 smithi178 bash[22840]: cluster 2024-09-13T22:41:45.896718+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-09-13T22:41:46.203 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 39 16 59925 59909 95.9836 69.375 0.0160157 0.010413 2024-09-13T22:41:46.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:46.250 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:46.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.556993+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.560365+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.681963+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.682856+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.886191+0000 osd.3 (osd.3) 99 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.888893+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.956913+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:44.960379+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: audit 2024-09-13T22:41:44.970484+0000 mgr.y (mgr.14613) 154 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:46.365 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:45 smithi192 bash[24608]: cluster 2024-09-13T22:41:45.896718+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-09-13T22:41:46.376 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:46.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:46.511 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:46.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:46.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:46.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:46.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:46.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:46.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:46.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:47.005 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:47.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[29889]: cluster 2024-09-13T22:41:45.618214+0000 osd.7 (osd.7) 93 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:41:47.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[29889]: cluster 2024-09-13T22:41:45.619730+0000 osd.7 (osd.7) 94 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:41:47.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[29889]: cluster 2024-09-13T22:41:45.905007+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-09-13T22:41:47.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[29889]: cluster 2024-09-13T22:41:46.495452+0000 mgr.y (mgr.14613) 155 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T22:41:47.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[22840]: cluster 2024-09-13T22:41:45.618214+0000 osd.7 (osd.7) 93 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:41:47.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[22840]: cluster 2024-09-13T22:41:45.619730+0000 osd.7 (osd.7) 94 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:41:47.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[22840]: cluster 2024-09-13T22:41:45.905007+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-09-13T22:41:47.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:46 smithi178 bash[22840]: cluster 2024-09-13T22:41:46.495452+0000 mgr.y (mgr.14613) 155 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T22:41:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:47.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:47.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:47.163 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:47.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:41:47.203 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:41:47.208525+0000 min lat: 0.00140062 max lat: 0.407633 avg lat: 0.0104617 2024-09-13T22:41:47.203 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:41:47.203 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 40 16 61177 61161 95.5402 78.25 0.0117397 0.0104617 2024-09-13T22:41:47.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:41:47.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:47.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:47.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:46 smithi192 bash[24608]: cluster 2024-09-13T22:41:45.618214+0000 osd.7 (osd.7) 93 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:41:47.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:46 smithi192 bash[24608]: cluster 2024-09-13T22:41:45.619730+0000 osd.7 (osd.7) 94 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:41:47.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:46 smithi192 bash[24608]: cluster 2024-09-13T22:41:45.905007+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-09-13T22:41:47.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:46 smithi192 bash[24608]: cluster 2024-09-13T22:41:46.495452+0000 mgr.y (mgr.14613) 155 : cluster [DBG] pgmap v131: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T22:41:47.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:47.454 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:47.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:47.583 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:47.672 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:47.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:47.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:47.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:47.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:47.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:47.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:47.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:48.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:48.098 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:48.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:48.203 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 41 16 61971 61955 94.4203 49.625 0.306328 0.010518 2024-09-13T22:41:48.233 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:48.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:48.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:48.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:48.502 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:48.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:48.695 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:48.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:48.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:48.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:48.870 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:48.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:48.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.004 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:49.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:46.906883+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:46.916508+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.489514+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 deep-scrub starts 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.494384+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 deep-scrub ok 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.556607+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.556678+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.667629+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.671510+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:41:49.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.762134+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.13 deep-scrub starts 2024-09-13T22:41:49.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.765131+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.13 deep-scrub ok 2024-09-13T22:41:49.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: audit 2024-09-13T22:41:47.849022+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:49.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: audit 2024-09-13T22:41:47.850071+0000 mon.b (mon.1) 101 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:49.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:47.943582+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-13T22:41:49.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:48 smithi192 bash[24608]: cluster 2024-09-13T22:41:48.496504+0000 mgr.y (mgr.14613) 156 : cluster [DBG] pgmap v133: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:41:49.134 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:46.906883+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:46.916508+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.489514+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 deep-scrub starts 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.494384+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 deep-scrub ok 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.556607+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.556678+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.667629+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.671510+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.762134+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.13 deep-scrub starts 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.765131+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.13 deep-scrub ok 2024-09-13T22:41:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: audit 2024-09-13T22:41:47.849022+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: audit 2024-09-13T22:41:47.850071+0000 mon.b (mon.1) 101 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:47.943582+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[22840]: cluster 2024-09-13T22:41:48.496504+0000 mgr.y (mgr.14613) 156 : cluster [DBG] pgmap v133: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:46.906883+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:46.916508+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.489514+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 deep-scrub starts 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.494384+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 deep-scrub ok 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.556607+0000 osd.2 (osd.2) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.556678+0000 osd.2 (osd.2) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.667629+0000 osd.7 (osd.7) 95 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:41:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.671510+0000 osd.7 (osd.7) 96 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:41:49.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.762134+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.13 deep-scrub starts 2024-09-13T22:41:49.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.765131+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.13 deep-scrub ok 2024-09-13T22:41:49.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: audit 2024-09-13T22:41:47.849022+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:41:49.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: audit 2024-09-13T22:41:47.850071+0000 mon.b (mon.1) 101 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:41:49.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:47.943582+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-13T22:41:49.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:48 smithi178 bash[29889]: cluster 2024-09-13T22:41:48.496504+0000 mgr.y (mgr.14613) 156 : cluster [DBG] pgmap v133: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:41:49.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 42 16 62617 62601 93.1336 40.375 0.0126934 0.0107229 2024-09-13T22:41:49.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:49.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:49.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.409 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:49.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:49.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:49.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:49.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:49.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:49.895 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.908 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:41:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:41:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:41:49.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:49.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:50.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:50.047 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:50 smithi192 bash[24608]: cluster 2024-09-13T22:41:48.475702+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:41:50.047 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:50 smithi192 bash[24608]: cluster 2024-09-13T22:41:48.479125+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:41:50.064 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:50.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:50.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:50 smithi178 bash[29889]: cluster 2024-09-13T22:41:48.475702+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:41:50.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:50 smithi178 bash[29889]: cluster 2024-09-13T22:41:48.479125+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:41:50.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:50 smithi178 bash[22840]: cluster 2024-09-13T22:41:48.475702+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:41:50.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:50 smithi178 bash[22840]: cluster 2024-09-13T22:41:48.479125+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:41:50.193 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:50.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 43 16 63662 63646 92.4865 65.3125 0.0221188 0.0108079 2024-09-13T22:41:50.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:50.324 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:50.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:50.452 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:50.556 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:50.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:50.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:50.717 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:50.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:50.843 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:50.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:50.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:50.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:51.079 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:51.109 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:51.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:49.714581+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:41:51.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:49.715963+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:41:51.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:49.762485+0000 osd.3 (osd.3) 103 : cluster [DBG] 3.1e deep-scrub starts 2024-09-13T22:41:51.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:49.765985+0000 osd.3 (osd.3) 104 : cluster [DBG] 3.1e deep-scrub ok 2024-09-13T22:41:51.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.497076+0000 mgr.y (mgr.14613) 157 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.520393+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.524092+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:49.714581+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:49.715963+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:49.762485+0000 osd.3 (osd.3) 103 : cluster [DBG] 3.1e deep-scrub starts 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:49.765985+0000 osd.3 (osd.3) 104 : cluster [DBG] 3.1e deep-scrub ok 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.497076+0000 mgr.y (mgr.14613) 157 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.520393+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:41:51.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:51 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.524092+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:41:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:51.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 44 16 64808 64792 92.0122 71.625 0.00935302 0.0108499 2024-09-13T22:41:51.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:51.235 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:51.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:51.373 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:49.714581+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:49.715963+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:49.762485+0000 osd.3 (osd.3) 103 : cluster [DBG] 3.1e deep-scrub starts 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:49.765985+0000 osd.3 (osd.3) 104 : cluster [DBG] 3.1e deep-scrub ok 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.497076+0000 mgr.y (mgr.14613) 157 : cluster [DBG] pgmap v134: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.520393+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:41:51.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:51 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.524092+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:41:51.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:51.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:51.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:51.651 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:51.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:51.768 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:51.770 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:41:51.786 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:51.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:51.920 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:51.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:52.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:52.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.506657+0000 osd.0 (osd.0) 103 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.548922+0000 osd.0 (osd.0) 104 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.675175+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[29889]: cluster 2024-09-13T22:41:50.679295+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.506657+0000 osd.0 (osd.0) 103 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.548922+0000 osd.0 (osd.0) 104 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.675175+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:41:52.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:52 smithi178 bash[22840]: cluster 2024-09-13T22:41:50.679295+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:41:52.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:52.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:52.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 45 16 65669 65653 91.1633 53.8125 0.00846646 0.0109533 2024-09-13T22:41:52.212 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:41:52.212 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:41:52.212 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:41:52.212 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:41:52.212 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:41:52.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:52.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:52 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.506657+0000 osd.0 (osd.0) 103 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:41:52.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:52 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.548922+0000 osd.0 (osd.0) 104 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:41:52.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:52 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.675175+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:41:52.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:52 smithi192 bash[24608]: cluster 2024-09-13T22:41:50.679295+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:41:52.302 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:52.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:52.441 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:52.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:52.577 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:52.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:52.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:52.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:52.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:52.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:52.963 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:52.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:53.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:53.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:53 smithi192 bash[24608]: cluster 2024-09-13T22:41:51.720899+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:41:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:53 smithi192 bash[24608]: cluster 2024-09-13T22:41:51.724725+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:41:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:53 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.497524+0000 mgr.y (mgr.14613) 158 : cluster [DBG] pgmap v135: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T22:41:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:53 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.789229+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-13T22:41:53.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:53.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 46 16 66659 66643 90.5266 61.875 0.00563474 0.0110395 2024-09-13T22:41:53.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:53.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:53.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[29889]: cluster 2024-09-13T22:41:51.720899+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:41:53.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[29889]: cluster 2024-09-13T22:41:51.724725+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:41:53.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.497524+0000 mgr.y (mgr.14613) 158 : cluster [DBG] pgmap v135: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T22:41:53.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.789229+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-13T22:41:53.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[22840]: cluster 2024-09-13T22:41:51.720899+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:41:53.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[22840]: cluster 2024-09-13T22:41:51.724725+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:41:53.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.497524+0000 mgr.y (mgr.14613) 158 : cluster [DBG] pgmap v135: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T22:41:53.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:53 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.789229+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-13T22:41:53.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:53.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:53.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:53.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:53.613 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:53.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:53.752 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:53.854 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:53.878 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:53.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:53.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:54.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:54.147 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.635444+0000 osd.1 (osd.1) 99 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.638216+0000 osd.1 (osd.1) 100 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.796710+0000 osd.4 (osd.4) 83 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.800260+0000 osd.4 (osd.4) 84 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.919086+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[29889]: cluster 2024-09-13T22:41:52.950360+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.635444+0000 osd.1 (osd.1) 99 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:41:54.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.638216+0000 osd.1 (osd.1) 100 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:41:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.796710+0000 osd.4 (osd.4) 83 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:41:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.800260+0000 osd.4 (osd.4) 84 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:41:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.919086+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T22:41:54.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:54 smithi178 bash[22840]: cluster 2024-09-13T22:41:52.950360+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T22:41:54.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 47 16 67299 67283 89.4516 40 0.00440881 0.0111746 2024-09-13T22:41:54.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:54.278 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:54.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.635444+0000 osd.1 (osd.1) 99 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:41:54.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.638216+0000 osd.1 (osd.1) 100 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:41:54.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.796710+0000 osd.4 (osd.4) 83 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:41:54.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.800260+0000 osd.4 (osd.4) 84 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:41:54.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.919086+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T22:41:54.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[24608]: cluster 2024-09-13T22:41:52.950360+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T22:41:54.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:54.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:54.537 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:54.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:54.667 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:54.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:54.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.921 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:54.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:55.023 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:55.045 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T22:41:55.055 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:55.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:41:54 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:41:55.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:53.364470+0000 osd.0 (osd.0) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:55.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:53.364534+0000 osd.0 (osd.0) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:55.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:53.748043+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.e scrub starts 2024-09-13T22:41:55.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:53.752594+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.e scrub ok 2024-09-13T22:41:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:53.891331+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T22:41:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:53.894004+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T22:41:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.029779+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-13T22:41:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.497991+0000 mgr.y (mgr.14613) 159 : cluster [DBG] pgmap v138: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-13T22:41:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: audit 2024-09-13T22:41:55.013778+0000 mon.c (mon.2) 25 : audit [INF] from='client.? v1:172.21.15.178:0/3416134114' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:41:55.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:55 smithi192 bash[24608]: audit 2024-09-13T22:41:55.014262+0000 mon.a (mon.0) 958 : audit [INF] from='client.24800 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:41:55.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:53.364470+0000 osd.0 (osd.0) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:53.364534+0000 osd.0 (osd.0) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:53.748043+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.e scrub starts 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:53.752594+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.e scrub ok 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:53.891331+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:53.894004+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.029779+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.497991+0000 mgr.y (mgr.14613) 159 : cluster [DBG] pgmap v138: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: audit 2024-09-13T22:41:55.013778+0000 mon.c (mon.2) 25 : audit [INF] from='client.? v1:172.21.15.178:0/3416134114' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:41:55.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[29889]: audit 2024-09-13T22:41:55.014262+0000 mon.a (mon.0) 958 : audit [INF] from='client.24800 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:53.364470+0000 osd.0 (osd.0) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:53.364534+0000 osd.0 (osd.0) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:53.748043+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.e scrub starts 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:53.752594+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.e scrub ok 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:53.891331+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:53.894004+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.029779+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-13T22:41:55.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.497991+0000 mgr.y (mgr.14613) 159 : cluster [DBG] pgmap v138: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-13T22:41:55.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: audit 2024-09-13T22:41:55.013778+0000 mon.c (mon.2) 25 : audit [INF] from='client.? v1:172.21.15.178:0/3416134114' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:41:55.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:55 smithi178 bash[22840]: audit 2024-09-13T22:41:55.014262+0000 mon.a (mon.0) 958 : audit [INF] from='client.24800 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:41:55.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 48 16 68229 68213 88.799 58.125 0.0413554 0.011251 2024-09-13T22:41:55.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:55.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:55.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:55.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:55.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:55.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:55.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:55.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:55.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:55.817 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:55.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:55.940 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:55.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:56.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:56.065 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.666971+0000 osd.2 (osd.2) 73 : cluster [DBG] 6.f scrub starts 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.669990+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.880930+0000 osd.4 (osd.4) 87 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.887955+0000 osd.4 (osd.4) 88 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.891933+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:54.906869+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:41:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: audit 2024-09-13T22:41:54.978404+0000 mgr.y (mgr.14613) 160 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:56.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: audit 2024-09-13T22:41:55.036894+0000 mon.a (mon.0) 959 : audit [INF] from='client.24800 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:41:56.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:56 smithi192 bash[24608]: cluster 2024-09-13T22:41:55.043331+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-13T22:41:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:56.168 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:56.196 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:56.205 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 49 16 69129 69113 88.1347 56.25 0.00336524 0.0113363 2024-09-13T22:41:56.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.666971+0000 osd.2 (osd.2) 73 : cluster [DBG] 6.f scrub starts 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.669990+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.880930+0000 osd.4 (osd.4) 87 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.887955+0000 osd.4 (osd.4) 88 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.891933+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:54.906869+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: audit 2024-09-13T22:41:54.978404+0000 mgr.y (mgr.14613) 160 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:56.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: audit 2024-09-13T22:41:55.036894+0000 mon.a (mon.0) 959 : audit [INF] from='client.24800 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[29889]: cluster 2024-09-13T22:41:55.043331+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.666971+0000 osd.2 (osd.2) 73 : cluster [DBG] 6.f scrub starts 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.669990+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.880930+0000 osd.4 (osd.4) 87 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.887955+0000 osd.4 (osd.4) 88 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.891933+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:54.906869+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: audit 2024-09-13T22:41:54.978404+0000 mgr.y (mgr.14613) 160 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: audit 2024-09-13T22:41:55.036894+0000 mon.a (mon.0) 959 : audit [INF] from='client.24800 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:41:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:56 smithi178 bash[22840]: cluster 2024-09-13T22:41:55.043331+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-13T22:41:56.325 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:56.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:56.470 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:56.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:56.597 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:56.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:56.733 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:56.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:56.861 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:56.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:56.991 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:56.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:57.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:57.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[29889]: cluster 2024-09-13T22:41:55.746492+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:41:57.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[29889]: cluster 2024-09-13T22:41:55.748621+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:41:57.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[29889]: cluster 2024-09-13T22:41:55.778969+0000 osd.4 (osd.4) 89 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T22:41:57.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[29889]: cluster 2024-09-13T22:41:55.779971+0000 osd.4 (osd.4) 90 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T22:41:57.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[29889]: cluster 2024-09-13T22:41:56.498923+0000 mgr.y (mgr.14613) 161 : cluster [DBG] pgmap v140: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 757 op/s 2024-09-13T22:41:57.107 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[22840]: cluster 2024-09-13T22:41:55.746492+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:41:57.107 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[22840]: cluster 2024-09-13T22:41:55.748621+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:41:57.107 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[22840]: cluster 2024-09-13T22:41:55.778969+0000 osd.4 (osd.4) 89 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T22:41:57.107 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[22840]: cluster 2024-09-13T22:41:55.779971+0000 osd.4 (osd.4) 90 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T22:41:57.107 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:57 smithi178 bash[22840]: cluster 2024-09-13T22:41:56.498923+0000 mgr.y (mgr.14613) 161 : cluster [DBG] pgmap v140: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 757 op/s 2024-09-13T22:41:57.123 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:57.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T22:41:57.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 50 16 70044 70028 87.5157 57.1875 0.0181311 0.0114125 2024-09-13T22:41:57.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:57.249 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:57.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:57.380 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:41:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:41:57.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:57 smithi192 bash[24608]: cluster 2024-09-13T22:41:55.746492+0000 osd.7 (osd.7) 105 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:41:57.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:57 smithi192 bash[24608]: cluster 2024-09-13T22:41:55.748621+0000 osd.7 (osd.7) 106 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:41:57.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:57 smithi192 bash[24608]: cluster 2024-09-13T22:41:55.778969+0000 osd.4 (osd.4) 89 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T22:41:57.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:57 smithi192 bash[24608]: cluster 2024-09-13T22:41:55.779971+0000 osd.4 (osd.4) 90 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T22:41:57.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:57 smithi192 bash[24608]: cluster 2024-09-13T22:41:56.498923+0000 mgr.y (mgr.14613) 161 : cluster [DBG] pgmap v140: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 757 op/s 2024-09-13T22:41:57.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:57.509 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:57.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:57.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:57.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:57.786 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:57.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:57.920 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:57.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:57.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T22:41:58.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:58.069 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:56.716472+0000 osd.1 (osd.1) 101 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:56.720486+0000 osd.1 (osd.1) 102 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:56.793548+0000 osd.3 (osd.3) 107 : cluster [DBG] 3.b scrub starts 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:56.802205+0000 osd.3 (osd.3) 108 : cluster [DBG] 3.b scrub ok 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:57.758514+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:57.761318+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:41:58.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:58 smithi192 bash[24608]: cluster 2024-09-13T22:41:57.791255+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-13T22:41:58.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:58.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 51 15 70977 70962 86.9443 58.375 0.0152524 0.0114958 2024-09-13T22:41:58.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:56.716472+0000 osd.1 (osd.1) 101 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:56.720486+0000 osd.1 (osd.1) 102 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:56.793548+0000 osd.3 (osd.3) 107 : cluster [DBG] 3.b scrub starts 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:56.802205+0000 osd.3 (osd.3) 108 : cluster [DBG] 3.b scrub ok 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:57.758514+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:57.761318+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[22840]: cluster 2024-09-13T22:41:57.791255+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:56.716472+0000 osd.1 (osd.1) 101 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:41:58.307 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:56.720486+0000 osd.1 (osd.1) 102 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:41:58.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:56.793548+0000 osd.3 (osd.3) 107 : cluster [DBG] 3.b scrub starts 2024-09-13T22:41:58.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:56.802205+0000 osd.3 (osd.3) 108 : cluster [DBG] 3.b scrub ok 2024-09-13T22:41:58.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:57.758514+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:41:58.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:57.761318+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:41:58.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:58 smithi178 bash[29889]: cluster 2024-09-13T22:41:57.791255+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-13T22:41:58.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:58.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:58.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:58.483 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:58.545 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:41:58.545 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:41:58.585 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:41:58.586 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:41:58.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:58.608 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:58.622 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:41:58.623 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:41:58.667 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:41:58.667 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:41:58.707 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:41:58.708 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:41:58.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:58.742 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:58.750 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:41:58.750 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:41:58.790 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:41:58.791 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:41:58.824 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:41:58.825 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:41:58.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:58.881 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:41:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:58.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:41:58.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:59.006 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:41:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:41:59.077 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:41:59.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:57.520455+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:41:59.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:57.526499+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:41:59.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: audit 2024-09-13T22:41:58.485811+0000 mon.a (mon.0) 962 : audit [DBG] from='client.? v1:172.21.15.178:0/1346023394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.499717+0000 mgr.y (mgr.14613) 162 : cluster [DBG] pgmap v142: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 896 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.656372+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.672135+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.798373+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.799383+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[29889]: audit 2024-09-13T22:41:58.910245+0000 mon.a (mon.0) 963 : audit [INF] from='client.? v1:172.21.15.178:0/3885814045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:57.520455+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:57.526499+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: audit 2024-09-13T22:41:58.485811+0000 mon.a (mon.0) 962 : audit [DBG] from='client.? v1:172.21.15.178:0/1346023394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:59.102 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.499717+0000 mgr.y (mgr.14613) 162 : cluster [DBG] pgmap v142: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 896 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T22:41:59.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.656372+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:41:59.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.672135+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:41:59.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.798373+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:41:59.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.799383+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:41:59.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[22840]: audit 2024-09-13T22:41:58.910245+0000 mon.a (mon.0) 963 : audit [INF] from='client.? v1:172.21.15.178:0/3885814045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:41:59.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.151 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:59.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:41:59.205 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 52 16 71919 71903 86.4033 58.8125 0.0128865 0.0115678 2024-09-13T22:41:59.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:59.281 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:59.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:57.520455+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:57.526499+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: audit 2024-09-13T22:41:58.485811+0000 mon.a (mon.0) 962 : audit [DBG] from='client.? v1:172.21.15.178:0/1346023394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.499717+0000 mgr.y (mgr.14613) 162 : cluster [DBG] pgmap v142: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 896 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.656372+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.672135+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.798373+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.799383+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:41:59.395 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:41:59 smithi192 bash[24608]: audit 2024-09-13T22:41:58.910245+0000 mon.a (mon.0) 963 : audit [INF] from='client.? v1:172.21.15.178:0/3885814045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:41:59.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:41:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:41:59.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:59.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:41:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:41:59.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:59.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:41:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:41:59.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:59.794 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:41:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:41:59.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:59.923 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:41:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:41:59.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:00.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:00.061 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:00.061 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:41:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:41:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:42:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:00.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.749911+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[22840]: cluster 2024-09-13T22:41:58.751576+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[22840]: cluster 2024-09-13T22:41:59.068262+0000 mon.a (mon.0) 964 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[22840]: audit 2024-09-13T22:41:59.078072+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.178:0/3885814045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[22840]: cluster 2024-09-13T22:41:59.093218+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.749911+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[29889]: cluster 2024-09-13T22:41:58.751576+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:42:00.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[29889]: cluster 2024-09-13T22:41:59.068262+0000 mon.a (mon.0) 964 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[29889]: audit 2024-09-13T22:41:59.078072+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.178:0/3885814045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:42:00.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:00 smithi178 bash[29889]: cluster 2024-09-13T22:41:59.093218+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-13T22:42:00.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:00.204 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 53 16 73013 72997 86.0631 68.375 0.0279928 0.0116138 2024-09-13T22:42:00.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:00.336 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:00 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.749911+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:42:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:00 smithi192 bash[24608]: cluster 2024-09-13T22:41:58.751576+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:42:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:00 smithi192 bash[24608]: cluster 2024-09-13T22:41:59.068262+0000 mon.a (mon.0) 964 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:00 smithi192 bash[24608]: audit 2024-09-13T22:41:59.078072+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.178:0/3885814045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:42:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:00 smithi192 bash[24608]: cluster 2024-09-13T22:41:59.093218+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-13T22:42:00.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:00.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:00.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:00.594 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:00.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:00.721 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:00.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:00.849 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:00.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:00.986 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:00.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:01.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:01.113 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:01.205 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 54 16 74043 74027 85.6614 64.375 0.0103116 0.0116685 2024-09-13T22:42:01.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:01.241 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:01.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:01.352 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:01 smithi192 bash[24608]: cluster 2024-09-13T22:42:00.500141+0000 mgr.y (mgr.14613) 163 : cluster [DBG] pgmap v144: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 744 op/s 2024-09-13T22:42:01.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:01.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:01 smithi178 bash[29889]: cluster 2024-09-13T22:42:00.500141+0000 mgr.y (mgr.14613) 163 : cluster [DBG] pgmap v144: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 744 op/s 2024-09-13T22:42:01.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:01 smithi178 bash[22840]: cluster 2024-09-13T22:42:00.500141+0000 mgr.y (mgr.14613) 163 : cluster [DBG] pgmap v144: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 744 op/s 2024-09-13T22:42:01.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:01.509 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:01.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:01.637 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:01.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:01.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:01.893 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:01.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:01.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:02.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:02.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:02.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:02.205 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 55 16 75150 75134 85.3609 69.1875 0.0150384 0.0117101 2024-09-13T22:42:02.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:02.288 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:02.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:02.417 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:02.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:02.548 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:02.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:02.671 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:02.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:02.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:02.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:02.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:02.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:03.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:03.053 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:03.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:03.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:03.205 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 56 16 76276 76260 85.0933 70.375 0.00735849 0.0117471 2024-09-13T22:42:03.284 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:03.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:03.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:03.439 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:03.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:03.571 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:03.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:42:03.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:42:03.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[22840]: cluster 2024-09-13T22:42:02.500546+0000 mgr.y (mgr.14613) 164 : cluster [DBG] pgmap v145: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 686 B/s rd, 37 MiB/s wr, 598 op/s 2024-09-13T22:42:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[22840]: cluster 2024-09-13T22:42:02.791838+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-13T22:42:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[22840]: audit 2024-09-13T22:42:02.872161+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:03.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[22840]: audit 2024-09-13T22:42:02.873565+0000 mon.b (mon.1) 102 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[29889]: cluster 2024-09-13T22:42:02.500546+0000 mgr.y (mgr.14613) 164 : cluster [DBG] pgmap v145: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 686 B/s rd, 37 MiB/s wr, 598 op/s 2024-09-13T22:42:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[29889]: cluster 2024-09-13T22:42:02.791838+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-13T22:42:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[29889]: audit 2024-09-13T22:42:02.872161+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:03 smithi178 bash[29889]: audit 2024-09-13T22:42:02.873565+0000 mon.b (mon.1) 102 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:03.737 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:03.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:03.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:03 smithi192 bash[24608]: cluster 2024-09-13T22:42:02.500546+0000 mgr.y (mgr.14613) 164 : cluster [DBG] pgmap v145: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 686 B/s rd, 37 MiB/s wr, 598 op/s 2024-09-13T22:42:03.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:03 smithi192 bash[24608]: cluster 2024-09-13T22:42:02.791838+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-13T22:42:03.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:03 smithi192 bash[24608]: audit 2024-09-13T22:42:02.872161+0000 mon.a (mon.0) 968 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:03.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:03 smithi192 bash[24608]: audit 2024-09-13T22:42:02.873565+0000 mon.b (mon.1) 102 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:03.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:03.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:03.990 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:03.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:04.095 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:04.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:04.136 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:04.136 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:04.205 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 57 15 77261 77246 84.6815 61.625 0.011747 0.0118033 2024-09-13T22:42:04.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:04.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:04.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:04.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:04.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:04.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:04.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:04.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:04.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:04.779 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:04.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:04.900 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:04.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:05.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:05.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:05.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:05.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:05.206 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 58 16 78204 78188 84.2365 58.875 0.0136726 0.011866 2024-09-13T22:42:05.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:05.306 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:05.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:42:04 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:42:05.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:05.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:05.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:05.564 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:05.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:05.709 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:05.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:05.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:05 smithi192 bash[24608]: cluster 2024-09-13T22:42:04.501494+0000 mgr.y (mgr.14613) 165 : cluster [DBG] pgmap v147: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-13T22:42:05.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:05.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:05 smithi178 bash[29889]: cluster 2024-09-13T22:42:04.501494+0000 mgr.y (mgr.14613) 165 : cluster [DBG] pgmap v147: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-13T22:42:05.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:05 smithi178 bash[22840]: cluster 2024-09-13T22:42:04.501494+0000 mgr.y (mgr.14613) 165 : cluster [DBG] pgmap v147: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-13T22:42:05.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:05.955 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:05.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:06.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:06.079 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:06.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:06.209 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 59 16 79139 79123 83.7945 58.4375 0.0143061 0.0119282 2024-09-13T22:42:06.228 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:06.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:06.379 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:06.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:06.507 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:06.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:06.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:06.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:06.772 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:06.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:06.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:06 smithi192 bash[24608]: audit 2024-09-13T22:42:04.987098+0000 mgr.y (mgr.14613) 166 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:06.903 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:06.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:06.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:06 smithi178 bash[29889]: audit 2024-09-13T22:42:04.987098+0000 mgr.y (mgr.14613) 166 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:06.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:06 smithi178 bash[22840]: audit 2024-09-13T22:42:04.987098+0000 mgr.y (mgr.14613) 166 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:07.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.032 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:07.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.175 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:07.209 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:42:07.214332+0000 min lat: 0.00140062 max lat: 0.414894 avg lat: 0.0119792 2024-09-13T22:42:07.209 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:42:07.209 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 60 16 80142 80126 83.4426 62.6875 0.0146741 0.0119792 2024-09-13T22:42:07.284 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:07.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:07.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:07.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:07.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:07.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:07.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:07 smithi192 bash[24608]: cluster 2024-09-13T22:42:06.501909+0000 mgr.y (mgr.14613) 167 : cluster [DBG] pgmap v148: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 984 op/s 2024-09-13T22:42:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:07 smithi192 bash[24608]: cluster 2024-09-13T22:42:06.613618+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:07 smithi192 bash[24608]: cluster 2024-09-13T22:42:06.613680+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:07.909 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:07 smithi178 bash[22840]: cluster 2024-09-13T22:42:06.501909+0000 mgr.y (mgr.14613) 167 : cluster [DBG] pgmap v148: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 984 op/s 2024-09-13T22:42:07.909 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:07 smithi178 bash[22840]: cluster 2024-09-13T22:42:06.613618+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:07.909 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:07 smithi178 bash[22840]: cluster 2024-09-13T22:42:06.613680+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:07.909 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:07 smithi178 bash[29889]: cluster 2024-09-13T22:42:06.501909+0000 mgr.y (mgr.14613) 167 : cluster [DBG] pgmap v148: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 984 op/s 2024-09-13T22:42:07.909 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:07 smithi178 bash[29889]: cluster 2024-09-13T22:42:06.613618+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:07.909 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:07 smithi178 bash[29889]: cluster 2024-09-13T22:42:06.613680+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:07.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:07.989 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:07.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:08.093 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:08.129 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:08.162 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:42:08.163 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:42:08.209 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 61 16 81078 81062 83.0337 58.5 0.0138962 0.0120359 2024-09-13T22:42:08.213 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:08.213 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:08.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:08.268 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:08.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:08.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:08.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:08.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:08.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:08.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:08 smithi192 bash[24608]: cluster 2024-09-13T22:42:07.576330+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-13T22:42:08.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:08 smithi192 bash[24608]: cluster 2024-09-13T22:42:07.794239+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-13T22:42:08.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:08 smithi192 bash[24608]: cluster 2024-09-13T22:42:08.502921+0000 mgr.y (mgr.14613) 168 : cluster [DBG] pgmap v151: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:42:08.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:08 smithi192 bash[24608]: audit 2024-09-13T22:42:08.544605+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.178:0/3674618720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T22:42:08.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:08 smithi192 bash[24608]: audit 2024-09-13T22:42:08.545582+0000 mon.a (mon.0) 971 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T22:42:08.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:08.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:08.770 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[29889]: cluster 2024-09-13T22:42:07.576330+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[29889]: cluster 2024-09-13T22:42:07.794239+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[29889]: cluster 2024-09-13T22:42:08.502921+0000 mgr.y (mgr.14613) 168 : cluster [DBG] pgmap v151: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[29889]: audit 2024-09-13T22:42:08.544605+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.178:0/3674618720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[29889]: audit 2024-09-13T22:42:08.545582+0000 mon.a (mon.0) 971 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[22840]: cluster 2024-09-13T22:42:07.576330+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[22840]: cluster 2024-09-13T22:42:07.794239+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-13T22:42:08.771 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[22840]: cluster 2024-09-13T22:42:08.502921+0000 mgr.y (mgr.14613) 168 : cluster [DBG] pgmap v151: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:42:08.772 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[22840]: audit 2024-09-13T22:42:08.544605+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.178:0/3674618720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T22:42:08.772 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:08 smithi178 bash[22840]: audit 2024-09-13T22:42:08.545582+0000 mon.a (mon.0) 971 : audit [INF] from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T22:42:08.789 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:08.804 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T22:42:08.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:08.917 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:08.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:09.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:09.044 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:09.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:09.177 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:09.209 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 62 16 81927 81911 82.5503 53.0625 0.0083897 0.0121093 2024-09-13T22:42:09.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:09.305 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:09.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:09.429 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:09.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:09.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:09.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:09.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:09.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:09.824 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:09.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:09.938 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:09 smithi192 bash[24608]: audit 2024-09-13T22:42:08.805485+0000 mon.a (mon.0) 972 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T22:42:09.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:09 smithi192 bash[24608]: cluster 2024-09-13T22:42:08.810612+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-13T22:42:09.952 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:09.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:10.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:10.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:10.177 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:42:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:42:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:42:10.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:09 smithi178 bash[29889]: audit 2024-09-13T22:42:08.805485+0000 mon.a (mon.0) 972 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T22:42:10.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:09 smithi178 bash[29889]: cluster 2024-09-13T22:42:08.810612+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-13T22:42:10.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:09 smithi178 bash[22840]: audit 2024-09-13T22:42:08.805485+0000 mon.a (mon.0) 972 : audit [INF] from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T22:42:10.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:09 smithi178 bash[22840]: cluster 2024-09-13T22:42:08.810612+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-13T22:42:10.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:10.209 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:10.210 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 63 16 82835 82819 82.1403 56.75 0.00570161 0.0121691 2024-09-13T22:42:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:10.313 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:10.354 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:10.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:10.496 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:10.600 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:10.617 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:10.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:10.751 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:10.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:10.886 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:10.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:10.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:11.014 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:11.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:11.127 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:10 smithi192 bash[24608]: cluster 2024-09-13T22:42:10.503486+0000 mgr.y (mgr.14613) 169 : cluster [DBG] pgmap v153: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-09-13T22:42:11.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:11.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:10 smithi178 bash[29889]: cluster 2024-09-13T22:42:10.503486+0000 mgr.y (mgr.14613) 169 : cluster [DBG] pgmap v153: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-09-13T22:42:11.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:10 smithi178 bash[22840]: cluster 2024-09-13T22:42:10.503486+0000 mgr.y (mgr.14613) 169 : cluster [DBG] pgmap v153: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 794 op/s 2024-09-13T22:42:11.210 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 64 16 83737 83721 81.7377 56.375 0.0263961 0.012229 2024-09-13T22:42:11.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:11.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:11.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:11.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:11.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:11.625 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:11.658 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:11.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:11.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:11.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T22:42:11.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:11.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:11.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:11.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T22:42:12.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:12.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:12.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:12.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:12.210 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 65 16 84676 84660 81.3827 58.6875 0.0147297 0.0122834 2024-09-13T22:42:12.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.301 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:12.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:12.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:12.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:12.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.705 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:12.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:12.831 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:12.866 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:42:12.867 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:42:12.905 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:42:12.905 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:42:12.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.960 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:42:12.960 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:42:12.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:12.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:13.002 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:42:13.003 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:42:13.035 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:42:13.035 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:42:13.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:13.073 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:42:13.073 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:42:13.092 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:13.104 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:42:13.104 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:42:13.149 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:42:13.150 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:42:13.194 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:13.210 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 66 16 85528 85512 80.9565 53.25 0.0109725 0.0123461 2024-09-13T22:42:13.223 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:13.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:13.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:13.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:13.494 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:13.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:13.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:13.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:13.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[22840]: cluster 2024-09-13T22:42:12.503849+0000 mgr.y (mgr.14613) 170 : cluster [DBG] pgmap v154: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 794 op/s 2024-09-13T22:42:13.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[22840]: cluster 2024-09-13T22:42:12.796838+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-13T22:42:13.742 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[22840]: audit 2024-09-13T22:42:12.810313+0000 mon.b (mon.1) 104 : audit [DBG] from='client.? v1:172.21.15.178:0/2787403247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:13.742 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[22840]: audit 2024-09-13T22:42:13.164922+0000 mon.a (mon.0) 975 : audit [INF] from='client.? v1:172.21.15.178:0/1493446413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:13.742 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[29889]: cluster 2024-09-13T22:42:12.503849+0000 mgr.y (mgr.14613) 170 : cluster [DBG] pgmap v154: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 794 op/s 2024-09-13T22:42:13.743 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[29889]: cluster 2024-09-13T22:42:12.796838+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-13T22:42:13.743 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[29889]: audit 2024-09-13T22:42:12.810313+0000 mon.b (mon.1) 104 : audit [DBG] from='client.? v1:172.21.15.178:0/2787403247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:13.743 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:13 smithi178 bash[29889]: audit 2024-09-13T22:42:13.164922+0000 mon.a (mon.0) 975 : audit [INF] from='client.? v1:172.21.15.178:0/1493446413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:13.760 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:13.801 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:42:13.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:13.874 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:13 smithi192 bash[24608]: cluster 2024-09-13T22:42:12.503849+0000 mgr.y (mgr.14613) 170 : cluster [DBG] pgmap v154: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 794 op/s 2024-09-13T22:42:13.874 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:13 smithi192 bash[24608]: cluster 2024-09-13T22:42:12.796838+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-13T22:42:13.874 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:13 smithi192 bash[24608]: audit 2024-09-13T22:42:12.810313+0000 mon.b (mon.1) 104 : audit [DBG] from='client.? v1:172.21.15.178:0/2787403247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:13.874 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:13 smithi192 bash[24608]: audit 2024-09-13T22:42:13.164922+0000 mon.a (mon.0) 975 : audit [INF] from='client.? v1:172.21.15.178:0/1493446413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:13.889 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:13.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:13.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:14.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:14.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:14.144 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:14.210 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 67 16 86509 86493 80.6632 61.3125 0.0142571 0.0123921 2024-09-13T22:42:14.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:14.267 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:14.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:14.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:14.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:14.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:14.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:14.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:14.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:14.814 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:14.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:14 smithi192 bash[24608]: cluster 2024-09-13T22:42:13.800951+0000 mon.a (mon.0) 976 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:14.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:14 smithi192 bash[24608]: audit 2024-09-13T22:42:13.802586+0000 mon.a (mon.0) 977 : audit [INF] from='client.? v1:172.21.15.178:0/1493446413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:42:14.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:14 smithi192 bash[24608]: cluster 2024-09-13T22:42:13.806485+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-13T22:42:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:14.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:14.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:14 smithi178 bash[29889]: cluster 2024-09-13T22:42:13.800951+0000 mon.a (mon.0) 976 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:14.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:14 smithi178 bash[29889]: audit 2024-09-13T22:42:13.802586+0000 mon.a (mon.0) 977 : audit [INF] from='client.? v1:172.21.15.178:0/1493446413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:42:14.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:14 smithi178 bash[29889]: cluster 2024-09-13T22:42:13.806485+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-13T22:42:14.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:14 smithi178 bash[22840]: cluster 2024-09-13T22:42:13.800951+0000 mon.a (mon.0) 976 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:14.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:14 smithi178 bash[22840]: audit 2024-09-13T22:42:13.802586+0000 mon.a (mon.0) 977 : audit [INF] from='client.? v1:172.21.15.178:0/1493446413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:42:14.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:14 smithi178 bash[22840]: cluster 2024-09-13T22:42:13.806485+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-13T22:42:14.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:14.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:15.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:15.059 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:42:14 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:42:15.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:15.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:15.206 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:15.211 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 68 16 87231 87215 80.1406 45.125 0.0033469 0.0124499 2024-09-13T22:42:15.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:15.341 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:15.444 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:15.483 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:15.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:15.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:15.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:15 smithi192 bash[24608]: cluster 2024-09-13T22:42:14.504402+0000 mgr.y (mgr.14613) 171 : cluster [DBG] pgmap v157: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-13T22:42:15.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:15.717 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:15 smithi178 bash[29889]: cluster 2024-09-13T22:42:14.504402+0000 mgr.y (mgr.14613) 171 : cluster [DBG] pgmap v157: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-13T22:42:15.717 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:15 smithi178 bash[22840]: cluster 2024-09-13T22:42:14.504402+0000 mgr.y (mgr.14613) 171 : cluster [DBG] pgmap v157: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-13T22:42:15.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:15.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:15.865 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:15.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:15.972 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:16.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:16.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:16.161 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:16.212 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 69 16 87991 87975 79.6658 47.5 0.0343942 0.0125479 2024-09-13T22:42:16.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:16.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:16.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:16.441 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:16.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:16.579 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:16.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:16.692 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:16 smithi192 bash[24608]: audit 2024-09-13T22:42:14.997550+0000 mgr.y (mgr.14613) 172 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:16.692 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:16 smithi192 bash[24608]: audit 2024-09-13T22:42:15.877466+0000 mon.b (mon.1) 105 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:42:16.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:16.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:16.821 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:16 smithi178 bash[29889]: audit 2024-09-13T22:42:14.997550+0000 mgr.y (mgr.14613) 172 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:16.821 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:16 smithi178 bash[29889]: audit 2024-09-13T22:42:15.877466+0000 mon.b (mon.1) 105 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:42:16.821 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:16 smithi178 bash[22840]: audit 2024-09-13T22:42:14.997550+0000 mgr.y (mgr.14613) 172 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:16.821 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:16 smithi178 bash[22840]: audit 2024-09-13T22:42:15.877466+0000 mon.b (mon.1) 105 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:42:16.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:16.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:16.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:16.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:17.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:42:17.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:17.142 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:17.212 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 70 16 88869 88853 79.3117 54.875 0.00465582 0.012603 2024-09-13T22:42:17.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:17.275 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:17.380 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:17.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:17.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:17.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:17.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:42:17.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:17.658 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[22840]: cluster 2024-09-13T22:42:16.504880+0000 mgr.y (mgr.14613) 173 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 698 op/s 2024-09-13T22:42:17.658 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[22840]: audit 2024-09-13T22:42:16.818978+0000 mon.b (mon.1) 106 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:42:17.658 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[22840]: audit 2024-09-13T22:42:16.820063+0000 mon.b (mon.1) 107 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:42:17.658 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[22840]: audit 2024-09-13T22:42:16.829476+0000 mon.a (mon.0) 979 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:17.658 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[29889]: cluster 2024-09-13T22:42:16.504880+0000 mgr.y (mgr.14613) 173 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 698 op/s 2024-09-13T22:42:17.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[29889]: audit 2024-09-13T22:42:16.818978+0000 mon.b (mon.1) 106 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:42:17.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[29889]: audit 2024-09-13T22:42:16.820063+0000 mon.b (mon.1) 107 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:42:17.659 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:17 smithi178 bash[29889]: audit 2024-09-13T22:42:16.829476+0000 mon.a (mon.0) 979 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:17.669 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:17.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:17.809 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:17 smithi192 bash[24608]: cluster 2024-09-13T22:42:16.504880+0000 mgr.y (mgr.14613) 173 : cluster [DBG] pgmap v158: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 698 op/s 2024-09-13T22:42:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:17 smithi192 bash[24608]: audit 2024-09-13T22:42:16.818978+0000 mon.b (mon.1) 106 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:42:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:17 smithi192 bash[24608]: audit 2024-09-13T22:42:16.820063+0000 mon.b (mon.1) 107 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:42:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:17 smithi192 bash[24608]: audit 2024-09-13T22:42:16.829476+0000 mon.a (mon.0) 979 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:17.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:17.935 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:17.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:18.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:18.061 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:18.159 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:18.159 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:18.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:18.188 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:18.212 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 71 16 89784 89768 79 57.1875 0.0189374 0.0126527 2024-09-13T22:42:18.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:18.316 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:18.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:18.448 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:18.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:18.576 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:18.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:18.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:18.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:18.832 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:18.936 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:18.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:18.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:19.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:18 smithi178 bash[22840]: cluster 2024-09-13T22:42:17.807020+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-13T22:42:19.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:18 smithi178 bash[22840]: audit 2024-09-13T22:42:17.889151+0000 mon.b (mon.1) 108 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:19.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:19.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:18 smithi192 bash[24608]: cluster 2024-09-13T22:42:17.807020+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-13T22:42:19.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:18 smithi192 bash[24608]: audit 2024-09-13T22:42:17.889151+0000 mon.b (mon.1) 108 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:19.080 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:18 smithi192 bash[24608]: cluster 2024-09-13T22:42:18.505585+0000 mgr.y (mgr.14613) 174 : cluster [DBG] pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-13T22:42:19.095 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:19.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:19 smithi178 bash[22840]: cluster 2024-09-13T22:42:18.505585+0000 mgr.y (mgr.14613) 174 : cluster [DBG] pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-13T22:42:19.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:19 smithi178 bash[29889]: cluster 2024-09-13T22:42:17.807020+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-13T22:42:19.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:19 smithi178 bash[29889]: audit 2024-09-13T22:42:17.889151+0000 mon.b (mon.1) 108 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:19.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:19 smithi178 bash[29889]: cluster 2024-09-13T22:42:18.505585+0000 mgr.y (mgr.14613) 174 : cluster [DBG] pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-13T22:42:19.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:19.214 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 72 16 90739 90723 78.7303 59.6875 0.0257824 0.0126957 2024-09-13T22:42:19.241 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:19.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:19.371 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:19.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:19.504 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:19.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:19.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:19.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:19.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:19.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:19.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:19.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:20.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:20.036 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:20.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:20.150 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:42:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:42:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:42:20.170 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:20.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:20.215 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 73 16 91695 91679 78.4688 59.75 0.0091666 0.0127379 2024-09-13T22:42:20.277 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:20.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:20.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:20.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:20.548 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:20.589 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:20.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:20.718 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:20.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:20.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:20.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:20.969 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:20.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:21.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:21.096 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:21.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.215 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 74 15 92645 92630 78.2116 59.4375 0.013188 0.0127811 2024-09-13T22:42:21.230 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.234 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:21.336 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.364 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:21.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:21.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:21.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.623 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:21.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.742 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:21 smithi178 bash[29889]: cluster 2024-09-13T22:42:20.506212+0000 mgr.y (mgr.14613) 175 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T22:42:21.742 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:21 smithi178 bash[22840]: cluster 2024-09-13T22:42:20.506212+0000 mgr.y (mgr.14613) 175 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T22:42:21.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:21.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:21.901 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:21.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:21.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:21 smithi192 bash[24608]: cluster 2024-09-13T22:42:20.506212+0000 mgr.y (mgr.14613) 175 : cluster [DBG] pgmap v161: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T22:42:22.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:22.042 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:22.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:22.176 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:22.215 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 75 16 93612 93596 77.9737 60.375 0.0101693 0.01282 2024-09-13T22:42:22.254 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:22.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:22.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:22.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:22.441 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:22.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:22.570 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:22.663 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:42:22.663 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:42:22.663 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.24767.0:93974 7.2 e139/116 7:46bdacf5:::benchmark_data_smithi178_64772_object93973:head v 139'5773, mlcod=139'5770)", 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:42:22.633672+0000", 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.011206422000000001, 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011242650999999999, 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.633672+0000", 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.664 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.633672+0000", 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.633673+0000", 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3300000000000001e-07 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:42:22.665 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.633699+0000", 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6129999999999999e-05 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.633707+0000", 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7500000000000003e-06 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.633712+0000", 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.677e-06 2024-09-13T22:42:22.666 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.640310+0000", 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0065973769999999998 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.640327+0000", 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7779999999999999e-05 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.667 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.24767.0:93983 7.3 e139/116 7:c6df4e83:::benchmark_data_smithi178_64772_object93982:head v 139'5872, mlcod=139'5870)", 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:42:22.642714+0000", 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0021648050000000001, 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0022658349999999999, 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642714+0000", 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.668 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642714+0000", 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642715+0000", 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.916e-06 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.669 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642743+0000", 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.7619000000000001e-05 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642755+0000", 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1491000000000001e-05 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642762+0000", 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2989999999999999e-06 2024-09-13T22:42:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643482+0000", 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00072003799999999999 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643497+0000", 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5503e-05 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.24767.0:93982 7.2 e139/116 7:490defb7:::benchmark_data_smithi178_64772_object93981:head v 139'5774, mlcod=139'5770)", 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:42:22.642786+0000", 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0020922610000000002, 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0022466349999999999, 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:42:22.672 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642786+0000", 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642786+0000", 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642787+0000", 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1999999999999999e-07 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.673 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642808+0000", 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1776999999999998e-05 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642815+0000", 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7000000000000002e-06 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.642820+0000", 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.2700000000000004e-06 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.674 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643218+0000", 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00039784500000000002 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643357+0000", 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000138555 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.24767.0:93991 7.8 7:19ee9840:::benchmark_data_smithi178_64772_object93990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T22:42:22.675 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:42:22.643754+0000", 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.001124372, 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001336737, 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.24767", 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 93991 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643754+0000", 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643754+0000", 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643756+0000", 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.978e-06 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.677 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643777+0000", 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1098e-05 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643787+0000", 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0383e-05 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643795+0000", 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3000000000000004e-06 2024-09-13T22:42:22.678 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.643838+0000", 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.3077000000000001e-05 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.644052+0000", 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00021444899999999999 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 4,7", 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:42:22.644447+0000", 2024-09-13T22:42:22.679 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00039482100000000001 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout: "num_ops": 4 2024-09-13T22:42:22.680 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:42:22.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:22.699 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:22.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:22.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:22.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:22.956 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:22.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:23.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:23.087 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:23.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:23.216 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 76 16 94344 94328 77.5497 45.75 0.0202505 0.012889 2024-09-13T22:42:23.219 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:23.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:23.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:23.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:23.487 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:23.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:23.620 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:23.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:23.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:23 smithi192 bash[24608]: cluster 2024-09-13T22:42:22.506664+0000 mgr.y (mgr.14613) 176 : cluster [DBG] pgmap v162: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 586 op/s 2024-09-13T22:42:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:23 smithi192 bash[24608]: audit 2024-09-13T22:42:23.112177+0000 mon.c (mon.2) 26 : audit [INF] from='client.? v1:172.21.15.178:0/2022208428' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:42:23.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:23 smithi192 bash[24608]: audit 2024-09-13T22:42:23.112468+0000 mon.a (mon.0) 981 : audit [INF] from='client.24827 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:42:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:23.796 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T22:42:23.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:23 smithi178 bash[22840]: cluster 2024-09-13T22:42:22.506664+0000 mgr.y (mgr.14613) 176 : cluster [DBG] pgmap v162: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 586 op/s 2024-09-13T22:42:23.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:23 smithi178 bash[22840]: audit 2024-09-13T22:42:23.112177+0000 mon.c (mon.2) 26 : audit [INF] from='client.? v1:172.21.15.178:0/2022208428' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:42:23.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:23 smithi178 bash[22840]: audit 2024-09-13T22:42:23.112468+0000 mon.a (mon.0) 981 : audit [INF] from='client.24827 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:42:23.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:23 smithi178 bash[29889]: cluster 2024-09-13T22:42:22.506664+0000 mgr.y (mgr.14613) 176 : cluster [DBG] pgmap v162: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 586 op/s 2024-09-13T22:42:23.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:23 smithi178 bash[29889]: audit 2024-09-13T22:42:23.112177+0000 mon.c (mon.2) 26 : audit [INF] from='client.? v1:172.21.15.178:0/2022208428' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:42:23.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:23 smithi178 bash[29889]: audit 2024-09-13T22:42:23.112468+0000 mon.a (mon.0) 981 : audit [INF] from='client.24827 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:42:23.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:23.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:23.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:23.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:24.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:24.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:24.216 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 77 16 94975 94959 77.0548 39.4375 0.0058134 0.0129587 2024-09-13T22:42:24.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:24.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:24.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:24.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:24.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:24.524 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:24.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:24.654 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:24.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:24.784 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:24.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:24.922 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:24.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:25.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:25.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:24 smithi178 bash[22840]: audit 2024-09-13T22:42:23.796455+0000 mon.a (mon.0) 982 : audit [INF] from='client.24827 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:42:25.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:24 smithi178 bash[22840]: cluster 2024-09-13T22:42:23.801710+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-13T22:42:25.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:24 smithi178 bash[22840]: cluster 2024-09-13T22:42:24.507241+0000 mgr.y (mgr.14613) 177 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-13T22:42:25.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:24 smithi178 bash[29889]: audit 2024-09-13T22:42:23.796455+0000 mon.a (mon.0) 982 : audit [INF] from='client.24827 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:42:25.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:24 smithi178 bash[29889]: cluster 2024-09-13T22:42:23.801710+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-13T22:42:25.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:24 smithi178 bash[29889]: cluster 2024-09-13T22:42:24.507241+0000 mgr.y (mgr.14613) 177 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-13T22:42:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:25.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:25.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:42:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:42:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:24 smithi192 bash[24608]: audit 2024-09-13T22:42:23.796455+0000 mon.a (mon.0) 982 : audit [INF] from='client.24827 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:42:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:24 smithi192 bash[24608]: cluster 2024-09-13T22:42:23.801710+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-13T22:42:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:24 smithi192 bash[24608]: cluster 2024-09-13T22:42:24.507241+0000 mgr.y (mgr.14613) 177 : cluster [DBG] pgmap v164: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-13T22:42:25.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:25.184 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:25.216 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 78 16 95729 95713 76.671 47.125 0.00936538 0.0130244 2024-09-13T22:42:25.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:25.314 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:25.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:25.443 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:25.506 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T22:42:25.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:25.582 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:25.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:25.709 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:25.803 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:25 smithi178 bash[22840]: audit 2024-09-13T22:42:25.006440+0000 mgr.y (mgr.14613) 178 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:25.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:25.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:25 smithi192 bash[24608]: audit 2024-09-13T22:42:25.006440+0000 mgr.y (mgr.14613) 178 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:25.841 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:25.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:25.967 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:25.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:26.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:26.095 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:26.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:25 smithi178 bash[29889]: audit 2024-09-13T22:42:25.006440+0000 mgr.y (mgr.14613) 178 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:26.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:26.220 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 79 16 96448 96432 76.2654 44.9375 0.0246205 0.0131061 2024-09-13T22:42:26.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:26.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:26.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:26.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:26.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:26.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:26.619 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:26.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T22:42:26.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:26.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:26.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:26.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:26.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:26.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:26 smithi178 bash[29889]: cluster 2024-09-13T22:42:26.507626+0000 mgr.y (mgr.14613) 179 : cluster [DBG] pgmap v165: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-13T22:42:26.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:26 smithi178 bash[22840]: cluster 2024-09-13T22:42:26.507626+0000 mgr.y (mgr.14613) 179 : cluster [DBG] pgmap v165: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-13T22:42:26.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:26.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:26 smithi192 bash[24608]: cluster 2024-09-13T22:42:26.507626+0000 mgr.y (mgr.14613) 179 : cluster [DBG] pgmap v165: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-13T22:42:26.998 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:42:26.998 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:42:27.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:27.033 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:42:27.033 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:42:27.080 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:42:27.080 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:42:27.104 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:27.110 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:42:27.110 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:42:27.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:27.143 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:42:27.144 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:42:27.179 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:42:27.180 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:42:27.211 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:42:27.211 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:42:27.220 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:42:27.225315+0000 min lat: 0.00140062 max lat: 0.414894 avg lat: 0.0131608 2024-09-13T22:42:27.220 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:42:27.220 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 80 16 97269 97253 75.9535 51.3125 0.0190956 0.0131608 2024-09-13T22:42:27.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:27.256 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:42:27.261 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:42:27.267 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:27.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:27.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:27.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:27.528 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:27.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:27.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:27.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:27.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:27.841 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:42:27.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:27.901 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:27 smithi192 bash[24608]: audit 2024-09-13T22:42:26.937675+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.178:0/3228961123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:27.901 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:27 smithi192 bash[24608]: audit 2024-09-13T22:42:27.474848+0000 mon.c (mon.2) 28 : audit [INF] from='client.? v1:172.21.15.178:0/2565510149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:42:27.901 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:27 smithi192 bash[24608]: audit 2024-09-13T22:42:27.475308+0000 mon.a (mon.0) 984 : audit [INF] from='client.24839 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:42:27.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:27.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:28.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:28.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:28.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:28.152 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:27 smithi178 bash[29889]: audit 2024-09-13T22:42:26.937675+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.178:0/3228961123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:28.152 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:27 smithi178 bash[29889]: audit 2024-09-13T22:42:27.474848+0000 mon.c (mon.2) 28 : audit [INF] from='client.? v1:172.21.15.178:0/2565510149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:42:28.152 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:27 smithi178 bash[29889]: audit 2024-09-13T22:42:27.475308+0000 mon.a (mon.0) 984 : audit [INF] from='client.24839 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:42:28.152 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:27 smithi178 bash[22840]: audit 2024-09-13T22:42:26.937675+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? v1:172.21.15.178:0/3228961123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:28.152 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:27 smithi178 bash[22840]: audit 2024-09-13T22:42:27.474848+0000 mon.c (mon.2) 28 : audit [INF] from='client.? v1:172.21.15.178:0/2565510149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:42:28.152 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:27 smithi178 bash[22840]: audit 2024-09-13T22:42:27.475308+0000 mon.a (mon.0) 984 : audit [INF] from='client.24839 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:42:28.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:28.220 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 81 16 97929 97913 75.5251 41.25 0.0111005 0.013236 2024-09-13T22:42:28.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:28.307 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:28.412 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:28.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:28.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:28.564 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:28.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:28.804 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:28.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:28.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:28.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:28.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:29.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:29.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:28 smithi192 bash[24608]: cluster 2024-09-13T22:42:27.817556+0000 mon.a (mon.0) 985 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:29.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:28 smithi192 bash[24608]: audit 2024-09-13T22:42:27.822094+0000 mon.a (mon.0) 986 : audit [INF] from='client.24839 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:42:29.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:28 smithi192 bash[24608]: cluster 2024-09-13T22:42:27.827504+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-13T22:42:29.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:28 smithi192 bash[24608]: cluster 2024-09-13T22:42:28.335199+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:42:29.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:28 smithi192 bash[24608]: cluster 2024-09-13T22:42:28.345145+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:42:29.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:28 smithi192 bash[24608]: cluster 2024-09-13T22:42:28.508296+0000 mgr.y (mgr.14613) 180 : cluster [DBG] pgmap v167: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-13T22:42:29.093 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:29.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[29889]: cluster 2024-09-13T22:42:27.817556+0000 mon.a (mon.0) 985 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:29.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[29889]: audit 2024-09-13T22:42:27.822094+0000 mon.a (mon.0) 986 : audit [INF] from='client.24839 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:42:29.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[29889]: cluster 2024-09-13T22:42:27.827504+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-13T22:42:29.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[29889]: cluster 2024-09-13T22:42:28.335199+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:42:29.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[29889]: cluster 2024-09-13T22:42:28.345145+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[29889]: cluster 2024-09-13T22:42:28.508296+0000 mgr.y (mgr.14613) 180 : cluster [DBG] pgmap v167: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[22840]: cluster 2024-09-13T22:42:27.817556+0000 mon.a (mon.0) 985 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[22840]: audit 2024-09-13T22:42:27.822094+0000 mon.a (mon.0) 986 : audit [INF] from='client.24839 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[22840]: cluster 2024-09-13T22:42:27.827504+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[22840]: cluster 2024-09-13T22:42:28.335199+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[22840]: cluster 2024-09-13T22:42:28.345145+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:42:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:28 smithi178 bash[22840]: cluster 2024-09-13T22:42:28.508296+0000 mgr.y (mgr.14613) 180 : cluster [DBG] pgmap v167: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-13T22:42:29.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:29.220 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 82 16 98904 98888 75.3472 60.9375 0.0186041 0.0132673 2024-09-13T22:42:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:29.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:29.346 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:29.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:29.471 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:29.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:29.602 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:29.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:29.750 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:29.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:29.859 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:42:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:42:29.878 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:29.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:29.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:30.004 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:30.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:28.924176+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.c scrub starts 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:28.929611+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.c scrub ok 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.187152+0000 osd.5 (osd.5) 99 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.188832+0000 osd.5 (osd.5) 100 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.315887+0000 osd.0 (osd.0) 109 : cluster [DBG] 7.9 scrub starts 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.328314+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:42:30.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:29 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.335014+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:42:30.133 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:30.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:28.924176+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.c scrub starts 2024-09-13T22:42:30.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:28.929611+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.c scrub ok 2024-09-13T22:42:30.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.187152+0000 osd.5 (osd.5) 99 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:42:30.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.188832+0000 osd.5 (osd.5) 100 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.315887+0000 osd.0 (osd.0) 109 : cluster [DBG] 7.9 scrub starts 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.328314+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.335014+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:28.924176+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.c scrub starts 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:28.929611+0000 osd.1 (osd.1) 106 : cluster [DBG] 5.c scrub ok 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.187152+0000 osd.5 (osd.5) 99 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.188832+0000 osd.5 (osd.5) 100 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.315887+0000 osd.0 (osd.0) 109 : cluster [DBG] 7.9 scrub starts 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.328314+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:42:30.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:29 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.335014+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:42:30.221 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 83 16 99837 99821 75.1419 58.3125 0.0159409 0.0133031 2024-09-13T22:42:30.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:30.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:30.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:30.396 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:30.504 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:30.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:30.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:42:30.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:30.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:30.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:30.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:30.880 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.568252+0000 osd.4 (osd.4) 93 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:30.880 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.568316+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:30.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[22840]: cluster 2024-09-13T22:42:29.877172+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-13T22:42:30.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[22840]: cluster 2024-09-13T22:42:30.508720+0000 mgr.y (mgr.14613) 181 : cluster [DBG] pgmap v169: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 42 MiB/s wr, 667 op/s 2024-09-13T22:42:30.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:30 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.568252+0000 osd.4 (osd.4) 93 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:30 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.568316+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:30 smithi192 bash[24608]: cluster 2024-09-13T22:42:29.877172+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-13T22:42:30.899 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:30 smithi192 bash[24608]: cluster 2024-09-13T22:42:30.508720+0000 mgr.y (mgr.14613) 181 : cluster [DBG] pgmap v169: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 42 MiB/s wr, 667 op/s 2024-09-13T22:42:30.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:30.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:31.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:31.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:31.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:31.153 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.568252+0000 osd.4 (osd.4) 93 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:31.153 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.568316+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:31.153 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[29889]: cluster 2024-09-13T22:42:29.877172+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-13T22:42:31.153 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:30 smithi178 bash[29889]: cluster 2024-09-13T22:42:30.508720+0000 mgr.y (mgr.14613) 181 : cluster [DBG] pgmap v169: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 42 MiB/s wr, 667 op/s 2024-09-13T22:42:31.177 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:31.221 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 84 16 100710 100694 74.8969 54.5625 0.0119837 0.0133454 2024-09-13T22:42:31.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:42:31.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:31.329 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:31.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:31.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:31.601 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:31.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:31.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:31.741 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:31.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:31.880 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:31.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:31.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:32.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:32.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:32.135 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:31 smithi192 bash[24608]: cluster 2024-09-13T22:42:30.762936+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:31 smithi192 bash[24608]: cluster 2024-09-13T22:42:30.787261+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:31 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.097733+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:31 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.100084+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:31 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.271976+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.b scrub starts 2024-09-13T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:31 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.280427+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.b scrub ok 2024-09-13T22:42:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:32.221 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 85 16 101564 101548 74.6437 53.375 0.00692727 0.0133927 2024-09-13T22:42:32.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:32.265 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:32.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:32.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:32.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:32.526 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:32.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:32.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:32.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:30.762936+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:42:32.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:30.787261+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:42:32.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.097733+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:42:32.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.100084+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:42:32.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.271976+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.b scrub starts 2024-09-13T22:42:32.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.280427+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.b scrub ok 2024-09-13T22:42:32.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:30.762936+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:42:32.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:30.787261+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:42:32.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.097733+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:42:32.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.100084+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:42:32.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.271976+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.b scrub starts 2024-09-13T22:42:32.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.280427+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.b scrub ok 2024-09-13T22:42:32.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:32.778 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:32.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:32.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:32.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:32.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:33.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:33.023 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.664034+0000 osd.0 (osd.0) 110 : cluster [DBG] 7.9 scrub ok 2024-09-13T22:42:33.023 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.772516+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e scrub starts 2024-09-13T22:42:33.024 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:31.787437+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e scrub ok 2024-09-13T22:42:33.024 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[22840]: cluster 2024-09-13T22:42:32.509170+0000 mgr.y (mgr.14613) 182 : cluster [DBG] pgmap v170: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 29 MiB/s wr, 457 op/s 2024-09-13T22:42:33.024 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.664034+0000 osd.0 (osd.0) 110 : cluster [DBG] 7.9 scrub ok 2024-09-13T22:42:33.024 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.772516+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e scrub starts 2024-09-13T22:42:33.024 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:31.787437+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e scrub ok 2024-09-13T22:42:33.024 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:32 smithi178 bash[29889]: cluster 2024-09-13T22:42:32.509170+0000 mgr.y (mgr.14613) 182 : cluster [DBG] pgmap v170: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 29 MiB/s wr, 457 op/s 2024-09-13T22:42:33.041 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:33.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:33.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:32 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.664034+0000 osd.0 (osd.0) 110 : cluster [DBG] 7.9 scrub ok 2024-09-13T22:42:33.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:32 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.772516+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e scrub starts 2024-09-13T22:42:33.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:32 smithi192 bash[24608]: cluster 2024-09-13T22:42:31.787437+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e scrub ok 2024-09-13T22:42:33.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:32 smithi192 bash[24608]: cluster 2024-09-13T22:42:32.509170+0000 mgr.y (mgr.14613) 182 : cluster [DBG] pgmap v170: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 29 MiB/s wr, 457 op/s 2024-09-13T22:42:33.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:33.221 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 86 16 102447 102431 74.4175 55.1875 0.00601418 0.0134326 2024-09-13T22:42:33.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:33.284 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:33.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:33.420 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:33.526 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:33.555 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:33.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:33.684 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:33.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:33.809 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:33.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:33.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:33 smithi178 bash[29889]: audit 2024-09-13T22:42:32.904014+0000 mon.b (mon.1) 109 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:33.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:33 smithi178 bash[29889]: cluster 2024-09-13T22:42:32.969304+0000 osd.1 (osd.1) 107 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:42:33.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:33 smithi178 bash[29889]: cluster 2024-09-13T22:42:32.975422+0000 osd.1 (osd.1) 108 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:42:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:33 smithi178 bash[22840]: audit 2024-09-13T22:42:32.904014+0000 mon.b (mon.1) 109 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:33 smithi178 bash[22840]: cluster 2024-09-13T22:42:32.969304+0000 osd.1 (osd.1) 107 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:42:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:33 smithi178 bash[22840]: cluster 2024-09-13T22:42:32.975422+0000 osd.1 (osd.1) 108 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:42:33.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:33.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:34.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:34.074 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:33 smithi192 bash[24608]: audit 2024-09-13T22:42:32.904014+0000 mon.b (mon.1) 109 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:33 smithi192 bash[24608]: cluster 2024-09-13T22:42:32.969304+0000 osd.1 (osd.1) 107 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:42:34.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:33 smithi192 bash[24608]: cluster 2024-09-13T22:42:32.975422+0000 osd.1 (osd.1) 108 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:42:34.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:34.203 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:34.221 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 87 16 103256 103240 74.1433 50.5625 0.0187923 0.0134824 2024-09-13T22:42:34.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:34.335 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:34.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:34.461 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:34.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:34.586 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:34.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:34.717 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:34.814 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:34.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:34.849 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:34.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:34.979 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:34.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:35.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:35.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:42:35.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:33.825083+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:42:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:33.826322+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:42:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.296968+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:42:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.299246+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:42:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.352983+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.353186+0000 osd.3 (osd.3) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:35.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:34 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.510092+0000 mgr.y (mgr.14613) 183 : cluster [DBG] pgmap v171: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-09-13T22:42:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:33.825083+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:42:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:33.826322+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:42:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.296968+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:42:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.299246+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:42:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.352983+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:35.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.353186+0000 osd.3 (osd.3) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.510092+0000 mgr.y (mgr.14613) 183 : cluster [DBG] pgmap v171: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:33.825083+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:33.826322+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.296968+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.299246+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.352983+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.353186+0000 osd.3 (osd.3) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:35.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:34 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.510092+0000 mgr.y (mgr.14613) 183 : cluster [DBG] pgmap v171: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-09-13T22:42:35.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:35.221 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 88 16 104207 104191 73.9762 59.4375 0.00731508 0.0135138 2024-09-13T22:42:35.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:35.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:35.366 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:35.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:35.498 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:35.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:35.631 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:35.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:35.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:35.826 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:35.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:35.893 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:35.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:35.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:36.018 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:36.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:36.147 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:36.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.853272+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.859642+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.927898+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.986998+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:34.994409+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: audit 2024-09-13T22:42:35.014436+0000 mgr.y (mgr.14613) 184 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:35.197425+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:35.206287+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:42:36.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:35 smithi192 bash[24608]: cluster 2024-09-13T22:42:35.828356+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-13T22:42:36.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.853272+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:42:36.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.859642+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:42:36.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.927898+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-13T22:42:36.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.986998+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:42:36.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:34.994409+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:42:36.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: audit 2024-09-13T22:42:35.014436+0000 mgr.y (mgr.14613) 184 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:35.197425+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:35.206287+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[29889]: cluster 2024-09-13T22:42:35.828356+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.853272+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.859642+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.927898+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.986998+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:34.994409+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: audit 2024-09-13T22:42:35.014436+0000 mgr.y (mgr.14613) 184 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:35.197425+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:35.206287+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:42:36.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:35 smithi178 bash[22840]: cluster 2024-09-13T22:42:35.828356+0000 osd.6 (osd.6) 71 : cluster [DBG] 7.c scrub starts 2024-09-13T22:42:36.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 89 16 105008 104992 73.707 50.0625 0.0131329 0.0135631 2024-09-13T22:42:36.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:36.280 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:36.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:36.416 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:36.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:36.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:36.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:36.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:36.767 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:36.804 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:36.909 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:36.931 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:36.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:37.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:37.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:37.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[22840]: cluster 2024-09-13T22:42:35.834538+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:42:37.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[22840]: cluster 2024-09-13T22:42:35.836930+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:42:37.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.030869+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:42:37.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.032349+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:42:37.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.510516+0000 mgr.y (mgr.14613) 185 : cluster [DBG] pgmap v173: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 844 op/s 2024-09-13T22:42:37.128 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[29889]: cluster 2024-09-13T22:42:35.834538+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:42:37.128 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[29889]: cluster 2024-09-13T22:42:35.836930+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:42:37.128 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.030869+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:42:37.128 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.032349+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:42:37.128 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:36 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.510516+0000 mgr.y (mgr.14613) 185 : cluster [DBG] pgmap v173: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 844 op/s 2024-09-13T22:42:37.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:37.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total time run: 90.0119 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total writes made: 105750 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Write size: 65536 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object size: 65536 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Bandwidth (MB/sec): 73.4278 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Bandwidth: 28.86 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max bandwidth (MB/sec): 175.75 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min bandwidth (MB/sec): 39.4375 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average IOPS: 1174 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev IOPS: 461.76 2024-09-13T22:42:37.222 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max IOPS: 2812 2024-09-13T22:42:37.223 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min IOPS: 631 2024-09-13T22:42:37.223 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average Latency(s): 0.0136147 2024-09-13T22:42:37.223 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Latency(s): 0.0181665 2024-09-13T22:42:37.223 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max latency(s): 0.414894 2024-09-13T22:42:37.223 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min latency(s): 0.00140062 2024-09-13T22:42:37.228 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T22:42:37.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:37.329 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:36 smithi192 bash[24608]: cluster 2024-09-13T22:42:35.834538+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:42:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:36 smithi192 bash[24608]: cluster 2024-09-13T22:42:35.836930+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:42:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:36 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.030869+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:42:37.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:36 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.032349+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:42:37.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:36 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.510516+0000 mgr.y (mgr.14613) 185 : cluster [DBG] pgmap v173: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 844 op/s 2024-09-13T22:42:37.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:37.460 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:37.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:37.587 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:37.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:37.726 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:37.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:37.862 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:37.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:37.984 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:37.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:38.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:38.109 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.813059+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.814603+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.949507+0000 osd.6 (osd.6) 72 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:36.949629+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:37.023148+0000 osd.5 (osd.5) 105 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:37.031322+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:37.458136+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:37.463882+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:42:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:37 smithi192 bash[24608]: cluster 2024-09-13T22:42:37.860290+0000 osd.6 (osd.6) 74 : cluster [DBG] 7.c scrub ok 2024-09-13T22:42:38.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:38.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:38.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:38.375 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:38.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.813059+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:42:38.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.814603+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.949507+0000 osd.6 (osd.6) 72 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:36.949629+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:37.023148+0000 osd.5 (osd.5) 105 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:37.031322+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:37.458136+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:37.463882+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[22840]: cluster 2024-09-13T22:42:37.860290+0000 osd.6 (osd.6) 74 : cluster [DBG] 7.c scrub ok 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.813059+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.814603+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.949507+0000 osd.6 (osd.6) 72 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:36.949629+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:38.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:37.023148+0000 osd.5 (osd.5) 105 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:42:38.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:37.031322+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:42:38.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:37.458136+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:42:38.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:37.463882+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:42:38.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:37 smithi178 bash[29889]: cluster 2024-09-13T22:42:37.860290+0000 osd.6 (osd.6) 74 : cluster [DBG] 7.c scrub ok 2024-09-13T22:42:38.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:38.501 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:38.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.645 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:38.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:38.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:38.875 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:38.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:38.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:39.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:39.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:39.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:39.160 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:39.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:37.937750+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.039437+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.c scrub starts 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.042785+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.c scrub ok 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.222932+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.222979+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.261761+0000 osd.0 (osd.0) 113 : cluster [DBG] 7.4 scrub starts 2024-09-13T22:42:39.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:38 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.511564+0000 mgr.y (mgr.14613) 186 : cluster [DBG] pgmap v175: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 767 B/s rd, 44 MiB/s wr, 751 op/s 2024-09-13T22:42:39.295 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:39.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:39.426 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:39.487 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:37.937750+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-13T22:42:39.488 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.039437+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.c scrub starts 2024-09-13T22:42:39.488 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.042785+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.c scrub ok 2024-09-13T22:42:39.488 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.222932+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:39.488 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.222979+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:39.488 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.261761+0000 osd.0 (osd.0) 113 : cluster [DBG] 7.4 scrub starts 2024-09-13T22:42:39.488 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.511564+0000 mgr.y (mgr.14613) 186 : cluster [DBG] pgmap v175: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 767 B/s rd, 44 MiB/s wr, 751 op/s 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:37.937750+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.039437+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.c scrub starts 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.042785+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.c scrub ok 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.222932+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.222979+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.261761+0000 osd.0 (osd.0) 113 : cluster [DBG] 7.4 scrub starts 2024-09-13T22:42:39.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.511564+0000 mgr.y (mgr.14613) 186 : cluster [DBG] pgmap v175: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 767 B/s rd, 44 MiB/s wr, 751 op/s 2024-09-13T22:42:39.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:39.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:39.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:39.686 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:39.727 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:42:39.727 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:42:39.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:39.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:39.927 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:42:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:42:39.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:39.965 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:39.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:40.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:40.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.888387+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:42:40.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.889858+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:42:40.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:38.968481+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-13T22:42:40.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.150777+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.153978+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.273826+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.282131+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.500962+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.2 scrub starts 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.854627+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[29889]: cluster 2024-09-13T22:42:39.863151+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.888387+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.889858+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:38.968481+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.150777+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.153978+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:42:40.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.273826+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:42:40.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.282131+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:42:40.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.500962+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.2 scrub starts 2024-09-13T22:42:40.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.854627+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:42:40.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:39 smithi178 bash[22840]: cluster 2024-09-13T22:42:39.863151+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:42:40.112 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:40.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:40.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:40.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.888387+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.889858+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:38.968481+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.150777+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.153978+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.273826+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.282131+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.500962+0000 osd.3 (osd.3) 121 : cluster [DBG] 7.2 scrub starts 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.854627+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:42:40.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:39 smithi192 bash[24608]: cluster 2024-09-13T22:42:39.863151+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:42:40.242 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:40.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:40.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:40.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:40.473 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:42:40.473 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:42:40.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:40.533 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:42:40.534 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:42:40.561 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:42:40.561 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:42:40.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:40.596 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:42:40.597 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:42:40.629 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:40.633 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:42:40.633 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:42:40.671 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:42:40.671 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:42:40.705 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:42:40.705 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:42:40.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:40.745 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:42:40.745 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:42:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.761 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:40.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:40.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:40.899 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:40.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:40.966 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T22:42:41.004 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.024950+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:42:41.004 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.026157+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:42:41.004 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.305382+0000 osd.0 (osd.0) 116 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.311803+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: audit 2024-09-13T22:42:40.318963+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.178:0/2428137047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: audit 2024-09-13T22:42:40.319734+0000 mon.a (mon.0) 992 : audit [INF] from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: audit 2024-09-13T22:42:40.382606+0000 mon.a (mon.0) 993 : audit [DBG] from='client.? v1:172.21.15.178:0/2922092635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.512080+0000 mgr.y (mgr.14613) 187 : cluster [DBG] pgmap v177: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 662 op/s 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.024950+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.026157+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.305382+0000 osd.0 (osd.0) 116 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.311803+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: audit 2024-09-13T22:42:40.318963+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.178:0/2428137047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:42:41.005 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: audit 2024-09-13T22:42:40.319734+0000 mon.a (mon.0) 992 : audit [INF] from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:42:41.006 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: audit 2024-09-13T22:42:40.382606+0000 mon.a (mon.0) 993 : audit [DBG] from='client.? v1:172.21.15.178:0/2922092635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:41.006 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:40 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.512080+0000 mgr.y (mgr.14613) 187 : cluster [DBG] pgmap v177: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 662 op/s 2024-09-13T22:42:41.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:41.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:41.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:41.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.024950+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.026157+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.305382+0000 osd.0 (osd.0) 116 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.311803+0000 osd.0 (osd.0) 117 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: audit 2024-09-13T22:42:40.318963+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.178:0/2428137047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: audit 2024-09-13T22:42:40.319734+0000 mon.a (mon.0) 992 : audit [INF] from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: audit 2024-09-13T22:42:40.382606+0000 mon.a (mon.0) 993 : audit [DBG] from='client.? v1:172.21.15.178:0/2922092635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:41.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:40 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.512080+0000 mgr.y (mgr.14613) 187 : cluster [DBG] pgmap v177: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 662 op/s 2024-09-13T22:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:41.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:41.288 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:41.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:41.423 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:41.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:41.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:41.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:41.682 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:41.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:41.821 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:41.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:41.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:42.170 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:42.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.742532+0000 osd.0 (osd.0) 118 : cluster [DBG] 7.4 scrub ok 2024-09-13T22:42:42.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[22840]: audit 2024-09-13T22:42:40.964657+0000 mon.a (mon.0) 994 : audit [INF] from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T22:42:42.177 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[22840]: cluster 2024-09-13T22:42:40.970804+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[22840]: cluster 2024-09-13T22:42:41.072287+0000 osd.5 (osd.5) 113 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[22840]: cluster 2024-09-13T22:42:41.074410+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[22840]: cluster 2024-09-13T22:42:41.294071+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.6 scrub starts 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.742532+0000 osd.0 (osd.0) 118 : cluster [DBG] 7.4 scrub ok 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[29889]: audit 2024-09-13T22:42:40.964657+0000 mon.a (mon.0) 994 : audit [INF] from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[29889]: cluster 2024-09-13T22:42:40.970804+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[29889]: cluster 2024-09-13T22:42:41.072287+0000 osd.5 (osd.5) 113 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[29889]: cluster 2024-09-13T22:42:41.074410+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:42:42.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:41 smithi178 bash[29889]: cluster 2024-09-13T22:42:41.294071+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.6 scrub starts 2024-09-13T22:42:42.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.305 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:42.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:41 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.742532+0000 osd.0 (osd.0) 118 : cluster [DBG] 7.4 scrub ok 2024-09-13T22:42:42.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:41 smithi192 bash[24608]: audit 2024-09-13T22:42:40.964657+0000 mon.a (mon.0) 994 : audit [INF] from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T22:42:42.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:41 smithi192 bash[24608]: cluster 2024-09-13T22:42:40.970804+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-13T22:42:42.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:41 smithi192 bash[24608]: cluster 2024-09-13T22:42:41.072287+0000 osd.5 (osd.5) 113 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:42:42.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:41 smithi192 bash[24608]: cluster 2024-09-13T22:42:41.074410+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:42:42.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:41 smithi192 bash[24608]: cluster 2024-09-13T22:42:41.294071+0000 osd.0 (osd.0) 119 : cluster [DBG] 7.6 scrub starts 2024-09-13T22:42:42.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.437 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:42.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.573 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:42.674 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:42.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:42.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:42.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:42.967 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:42.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:43.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:43.081 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:42 smithi192 bash[24608]: cluster 2024-09-13T22:42:42.512510+0000 mgr.y (mgr.14613) 188 : cluster [DBG] pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 662 op/s 2024-09-13T22:42:43.081 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:42 smithi192 bash[24608]: cluster 2024-09-13T22:42:42.638848+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:43.081 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:42 smithi192 bash[24608]: cluster 2024-09-13T22:42:42.638922+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:43.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:43.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:43.236 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:43.340 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:43.364 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:43.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:43.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:43.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:43 smithi178 bash[29889]: cluster 2024-09-13T22:42:42.512510+0000 mgr.y (mgr.14613) 188 : cluster [DBG] pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 662 op/s 2024-09-13T22:42:43.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:43 smithi178 bash[29889]: cluster 2024-09-13T22:42:42.638848+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:43.541 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:43 smithi178 bash[29889]: cluster 2024-09-13T22:42:42.638922+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:43.541 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:43 smithi178 bash[22840]: cluster 2024-09-13T22:42:42.512510+0000 mgr.y (mgr.14613) 188 : cluster [DBG] pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 31 MiB/s wr, 662 op/s 2024-09-13T22:42:43.542 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:43 smithi178 bash[22840]: cluster 2024-09-13T22:42:42.638848+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:43.542 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:43 smithi178 bash[22840]: cluster 2024-09-13T22:42:42.638922+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:43.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:43.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:43.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T22:42:43.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:43.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:43.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:43.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:43.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:43.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:43.980 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:42:44.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:44.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:44.137 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:44.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: cluster 2024-09-13T22:42:42.864460+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.2 scrub ok 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.175230+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: audit 2024-09-13T22:42:43.175314+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.178:0/2106478105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: audit 2024-09-13T22:42:43.176870+0000 mon.a (mon.0) 997 : audit [INF] from='client.24854 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.547694+0000 osd.3 (osd.3) 123 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.551982+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:42:44.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:43 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.846857+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-13T22:42:44.274 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:44.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:44.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: cluster 2024-09-13T22:42:42.864460+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.2 scrub ok 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.175230+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: audit 2024-09-13T22:42:43.175314+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.178:0/2106478105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: audit 2024-09-13T22:42:43.176870+0000 mon.a (mon.0) 997 : audit [INF] from='client.24854 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.547694+0000 osd.3 (osd.3) 123 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.551982+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.846857+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: cluster 2024-09-13T22:42:42.864460+0000 osd.3 (osd.3) 122 : cluster [DBG] 7.2 scrub ok 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.175230+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-13T22:42:44.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: audit 2024-09-13T22:42:43.175314+0000 mon.c (mon.2) 29 : audit [INF] from='client.? v1:172.21.15.178:0/2106478105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:44.379 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: audit 2024-09-13T22:42:43.176870+0000 mon.a (mon.0) 997 : audit [INF] from='client.24854 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:42:44.379 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.547694+0000 osd.3 (osd.3) 123 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:42:44.379 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.551982+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:42:44.379 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:44 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.846857+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.7 scrub starts 2024-09-13T22:42:44.409 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:44.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:44.548 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:44.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:44.678 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:44.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:44.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:44.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:44.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:45.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:45.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:45.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:45.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:42:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.976870+0000 mon.a (mon.0) 998 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.976900+0000 mon.a (mon.0) 999 : cluster [INF] Cluster is now healthy 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: audit 2024-09-13T22:42:43.978325+0000 mon.a (mon.0) 1000 : audit [INF] from='client.24854 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:43.981472+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.078546+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.6 scrub ok 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.124623+0000 osd.5 (osd.5) 117 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.126114+0000 osd.5 (osd.5) 118 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.243486+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.246160+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:42:45.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.339456+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.f scrub starts 2024-09-13T22:42:45.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:44 smithi192 bash[24608]: cluster 2024-09-13T22:42:44.513036+0000 mgr.y (mgr.14613) 189 : cluster [DBG] pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 5.3 MiB/s wr, 750 op/s 2024-09-13T22:42:45.179 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:45.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:45.316 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:45.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:45.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:45.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:45.562 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.976870+0000 mon.a (mon.0) 998 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:42:45.562 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.976900+0000 mon.a (mon.0) 999 : cluster [INF] Cluster is now healthy 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: audit 2024-09-13T22:42:43.978325+0000 mon.a (mon.0) 1000 : audit [INF] from='client.24854 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:43.981472+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.078546+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.6 scrub ok 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.124623+0000 osd.5 (osd.5) 117 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.126114+0000 osd.5 (osd.5) 118 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.243486+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.246160+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.339456+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.f scrub starts 2024-09-13T22:42:45.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:44.513036+0000 mgr.y (mgr.14613) 189 : cluster [DBG] pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 5.3 MiB/s wr, 750 op/s 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.976870+0000 mon.a (mon.0) 998 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.976900+0000 mon.a (mon.0) 999 : cluster [INF] Cluster is now healthy 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: audit 2024-09-13T22:42:43.978325+0000 mon.a (mon.0) 1000 : audit [INF] from='client.24854 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:43.981472+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.078546+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.6 scrub ok 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.124623+0000 osd.5 (osd.5) 117 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.126114+0000 osd.5 (osd.5) 118 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.243486+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.246160+0000 osd.1 (osd.1) 114 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:42:45.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.339456+0000 osd.0 (osd.0) 121 : cluster [DBG] 7.f scrub starts 2024-09-13T22:42:45.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:44.513036+0000 mgr.y (mgr.14613) 189 : cluster [DBG] pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 5.3 MiB/s wr, 750 op/s 2024-09-13T22:42:45.579 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:45.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:45.721 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:45.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:45.845 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:45.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:45.979 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:45.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:46.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:46.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:46.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: audit 2024-09-13T22:42:45.022678+0000 mgr.y (mgr.14613) 190 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:46.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.548516+0000 osd.3 (osd.3) 125 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:42:46.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.551186+0000 osd.3 (osd.3) 126 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:42:46.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.855900+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.d scrub starts 2024-09-13T22:42:46.144 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: audit 2024-09-13T22:42:45.022678+0000 mgr.y (mgr.14613) 190 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:46.144 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.548516+0000 osd.3 (osd.3) 125 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:42:46.144 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.551186+0000 osd.3 (osd.3) 126 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:42:46.144 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:45 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.855900+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.d scrub starts 2024-09-13T22:42:46.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:45 smithi192 bash[24608]: audit 2024-09-13T22:42:45.022678+0000 mgr.y (mgr.14613) 190 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:46.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:45 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.548516+0000 osd.3 (osd.3) 125 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:42:46.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:45 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.551186+0000 osd.3 (osd.3) 126 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:42:46.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:45 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.855900+0000 osd.6 (osd.6) 78 : cluster [DBG] 7.d scrub starts 2024-09-13T22:42:46.225 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:46.243 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:46.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:46.369 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:46.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:46.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:46.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:46.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:46.643 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:46.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T22:42:46.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:46.772 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:46.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:46.921 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:46.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:47.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.681709+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:42:47.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.683890+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:42:47.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.742238+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:42:47.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.743261+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.934511+0000 osd.4 (osd.4) 97 : cluster [DBG] 2.f scrub starts 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:45.982960+0000 osd.4 (osd.4) 98 : cluster [DBG] 2.f scrub ok 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.199074+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.200049+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.264155+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.a scrub starts 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.267339+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.a scrub ok 2024-09-13T22:42:47.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.513475+0000 mgr.y (mgr.14613) 191 : cluster [DBG] pgmap v183: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 575 op/s 2024-09-13T22:42:47.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:47.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:47.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:47.181 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:47.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:47.293 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.681709+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:42:47.294 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.683890+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:42:47.294 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.742238+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:42:47.294 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.743261+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:42:47.294 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.934511+0000 osd.4 (osd.4) 97 : cluster [DBG] 2.f scrub starts 2024-09-13T22:42:47.294 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:45.982960+0000 osd.4 (osd.4) 98 : cluster [DBG] 2.f scrub ok 2024-09-13T22:42:47.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.199074+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:42:47.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.200049+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:42:47.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.264155+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.a scrub starts 2024-09-13T22:42:47.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.267339+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.a scrub ok 2024-09-13T22:42:47.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:47 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.513475+0000 mgr.y (mgr.14613) 191 : cluster [DBG] pgmap v183: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 575 op/s 2024-09-13T22:42:47.307 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:47.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:47.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.681709+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:42:47.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.683890+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:42:47.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.742238+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:42:47.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.743261+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.934511+0000 osd.4 (osd.4) 97 : cluster [DBG] 2.f scrub starts 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:45.982960+0000 osd.4 (osd.4) 98 : cluster [DBG] 2.f scrub ok 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.199074+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.200049+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.264155+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.a scrub starts 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.267339+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.a scrub ok 2024-09-13T22:42:47.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:47 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.513475+0000 mgr.y (mgr.14613) 191 : cluster [DBG] pgmap v183: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 575 op/s 2024-09-13T22:42:47.438 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:47.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:47.571 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:47.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:47.696 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:47.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:47.824 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:47.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:47.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:47.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:48.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.710278+0000 osd.2 (osd.2) 81 : cluster [DBG] 5.b scrub starts 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: cluster 2024-09-13T22:42:46.715130+0000 osd.2 (osd.2) 82 : cluster [DBG] 5.b scrub ok 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: cluster 2024-09-13T22:42:47.043529+0000 osd.6 (osd.6) 79 : cluster [DBG] 7.7 scrub ok 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: cluster 2024-09-13T22:42:47.749223+0000 osd.6 (osd.6) 80 : cluster [DBG] 7.d scrub ok 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: cluster 2024-09-13T22:42:47.803484+0000 osd.6 (osd.6) 81 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: cluster 2024-09-13T22:42:47.806481+0000 osd.6 (osd.6) 82 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: audit 2024-09-13T22:42:47.940111+0000 mon.a (mon.0) 1002 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:48.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:48 smithi192 bash[24608]: audit 2024-09-13T22:42:47.941934+0000 mon.b (mon.1) 111 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:48.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:48.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:48.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:48.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:48.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:48.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.710278+0000 osd.2 (osd.2) 81 : cluster [DBG] 5.b scrub starts 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: cluster 2024-09-13T22:42:46.715130+0000 osd.2 (osd.2) 82 : cluster [DBG] 5.b scrub ok 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: cluster 2024-09-13T22:42:47.043529+0000 osd.6 (osd.6) 79 : cluster [DBG] 7.7 scrub ok 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: cluster 2024-09-13T22:42:47.749223+0000 osd.6 (osd.6) 80 : cluster [DBG] 7.d scrub ok 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: cluster 2024-09-13T22:42:47.803484+0000 osd.6 (osd.6) 81 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: cluster 2024-09-13T22:42:47.806481+0000 osd.6 (osd.6) 82 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: audit 2024-09-13T22:42:47.940111+0000 mon.a (mon.0) 1002 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[29889]: audit 2024-09-13T22:42:47.941934+0000 mon.b (mon.1) 111 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.710278+0000 osd.2 (osd.2) 81 : cluster [DBG] 5.b scrub starts 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: cluster 2024-09-13T22:42:46.715130+0000 osd.2 (osd.2) 82 : cluster [DBG] 5.b scrub ok 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: cluster 2024-09-13T22:42:47.043529+0000 osd.6 (osd.6) 79 : cluster [DBG] 7.7 scrub ok 2024-09-13T22:42:48.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: cluster 2024-09-13T22:42:47.749223+0000 osd.6 (osd.6) 80 : cluster [DBG] 7.d scrub ok 2024-09-13T22:42:48.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: cluster 2024-09-13T22:42:47.803484+0000 osd.6 (osd.6) 81 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:42:48.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: cluster 2024-09-13T22:42:47.806481+0000 osd.6 (osd.6) 82 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:42:48.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: audit 2024-09-13T22:42:47.940111+0000 mon.a (mon.0) 1002 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:42:48.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:48 smithi178 bash[22840]: audit 2024-09-13T22:42:47.941934+0000 mon.b (mon.1) 111 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:42:48.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:48.468 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:48.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:48.597 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:48.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:42:48.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:48.730 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:48.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:48.869 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:48.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:48.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:42:48.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:49.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:49.113 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:47.688622+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.d scrub starts 2024-09-13T22:42:49.113 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:47.690100+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.d scrub ok 2024-09-13T22:42:49.113 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:48.017311+0000 osd.4 (osd.4) 99 : cluster [DBG] 7.1 scrub starts 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:48.154023+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.f scrub ok 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:48.514294+0000 mgr.y (mgr.14613) 192 : cluster [DBG] pgmap v184: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 660 op/s 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:48.549833+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[29889]: cluster 2024-09-13T22:42:48.550841+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:47.688622+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.d scrub starts 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:47.690100+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.d scrub ok 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:48.017311+0000 osd.4 (osd.4) 99 : cluster [DBG] 7.1 scrub starts 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:48.154023+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.f scrub ok 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:48.514294+0000 mgr.y (mgr.14613) 192 : cluster [DBG] pgmap v184: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 660 op/s 2024-09-13T22:42:49.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:48.549833+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:42:49.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[22840]: cluster 2024-09-13T22:42:48.550841+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:42:49.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:49.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:47.688622+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.d scrub starts 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:47.690100+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.d scrub ok 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:48.017311+0000 osd.4 (osd.4) 99 : cluster [DBG] 7.1 scrub starts 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:48.154023+0000 osd.0 (osd.0) 122 : cluster [DBG] 7.f scrub ok 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:48.514294+0000 mgr.y (mgr.14613) 192 : cluster [DBG] pgmap v184: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 660 op/s 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:48.549833+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:42:49.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:49 smithi192 bash[24608]: cluster 2024-09-13T22:42:48.550841+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:42:49.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:49.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:49.406 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:49.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:49.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:49.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.659 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:49.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.783 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:49.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.915 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:49.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:50.019 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:50.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:50.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:50.177 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:42:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:42:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:42:50.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[29889]: cluster 2024-09-13T22:42:49.259075+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.7 deep-scrub starts 2024-09-13T22:42:50.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[29889]: cluster 2024-09-13T22:42:49.271172+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.7 deep-scrub ok 2024-09-13T22:42:50.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[29889]: cluster 2024-09-13T22:42:49.865971+0000 osd.6 (osd.6) 83 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:42:50.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[29889]: cluster 2024-09-13T22:42:49.867075+0000 osd.6 (osd.6) 84 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:42:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[22840]: cluster 2024-09-13T22:42:49.259075+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.7 deep-scrub starts 2024-09-13T22:42:50.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[22840]: cluster 2024-09-13T22:42:49.271172+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.7 deep-scrub ok 2024-09-13T22:42:50.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[22840]: cluster 2024-09-13T22:42:49.865971+0000 osd.6 (osd.6) 83 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:42:50.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:50 smithi178 bash[22840]: cluster 2024-09-13T22:42:49.867075+0000 osd.6 (osd.6) 84 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:42:50.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:50.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:50.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:50 smithi192 bash[24608]: cluster 2024-09-13T22:42:49.259075+0000 osd.5 (osd.5) 121 : cluster [DBG] 5.7 deep-scrub starts 2024-09-13T22:42:50.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:50 smithi192 bash[24608]: cluster 2024-09-13T22:42:49.271172+0000 osd.5 (osd.5) 122 : cluster [DBG] 5.7 deep-scrub ok 2024-09-13T22:42:50.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:50 smithi192 bash[24608]: cluster 2024-09-13T22:42:49.865971+0000 osd.6 (osd.6) 83 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:42:50.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:50 smithi192 bash[24608]: cluster 2024-09-13T22:42:49.867075+0000 osd.6 (osd.6) 84 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:42:50.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:50.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:50.445 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:50.460 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:50.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:50.577 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:50.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:50.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:50.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:50.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:50.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:50.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:50.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:51.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:51.087 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:51.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:51.214 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:51.317 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:51.326 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[22840]: cluster 2024-09-13T22:42:49.489056+0000 osd.4 (osd.4) 100 : cluster [DBG] 7.1 scrub ok 2024-09-13T22:42:51.326 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[22840]: cluster 2024-09-13T22:42:50.036577+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.e scrub starts 2024-09-13T22:42:51.326 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[22840]: cluster 2024-09-13T22:42:50.045092+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.e scrub ok 2024-09-13T22:42:51.327 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[22840]: cluster 2024-09-13T22:42:50.515029+0000 mgr.y (mgr.14613) 193 : cluster [DBG] pgmap v185: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-09-13T22:42:51.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[29889]: cluster 2024-09-13T22:42:49.489056+0000 osd.4 (osd.4) 100 : cluster [DBG] 7.1 scrub ok 2024-09-13T22:42:51.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[29889]: cluster 2024-09-13T22:42:50.036577+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.e scrub starts 2024-09-13T22:42:51.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[29889]: cluster 2024-09-13T22:42:50.045092+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.e scrub ok 2024-09-13T22:42:51.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:51 smithi178 bash[29889]: cluster 2024-09-13T22:42:50.515029+0000 mgr.y (mgr.14613) 193 : cluster [DBG] pgmap v185: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-09-13T22:42:51.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:51.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:51 smithi192 bash[24608]: cluster 2024-09-13T22:42:49.489056+0000 osd.4 (osd.4) 100 : cluster [DBG] 7.1 scrub ok 2024-09-13T22:42:51.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:51 smithi192 bash[24608]: cluster 2024-09-13T22:42:50.036577+0000 osd.4 (osd.4) 101 : cluster [DBG] 5.e scrub starts 2024-09-13T22:42:51.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:51 smithi192 bash[24608]: cluster 2024-09-13T22:42:50.045092+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.e scrub ok 2024-09-13T22:42:51.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:51 smithi192 bash[24608]: cluster 2024-09-13T22:42:50.515029+0000 mgr.y (mgr.14613) 193 : cluster [DBG] pgmap v185: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-09-13T22:42:51.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:51.465 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:51.479 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:51.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:51.593 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:51.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:51.721 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:51.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:51.847 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:51.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:51.972 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:51.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:51.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:52.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:52.100 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:52.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:52.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:52 smithi192 bash[24608]: cluster 2024-09-13T22:42:50.634019+0000 osd.3 (osd.3) 129 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:42:52.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:52 smithi192 bash[24608]: cluster 2024-09-13T22:42:50.643275+0000 osd.3 (osd.3) 130 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:42:52.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:52 smithi192 bash[24608]: cluster 2024-09-13T22:42:51.187634+0000 osd.1 (osd.1) 117 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:42:52.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:52 smithi192 bash[24608]: cluster 2024-09-13T22:42:51.196410+0000 osd.1 (osd.1) 118 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:42:52.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:52 smithi192 bash[24608]: cluster 2024-09-13T22:42:51.842976+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:42:52.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:52 smithi192 bash[24608]: cluster 2024-09-13T22:42:51.845701+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:42:52.230 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:52.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:52.356 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:52.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[29889]: cluster 2024-09-13T22:42:50.634019+0000 osd.3 (osd.3) 129 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:42:52.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[29889]: cluster 2024-09-13T22:42:50.643275+0000 osd.3 (osd.3) 130 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:42:52.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[29889]: cluster 2024-09-13T22:42:51.187634+0000 osd.1 (osd.1) 117 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:42:52.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[29889]: cluster 2024-09-13T22:42:51.196410+0000 osd.1 (osd.1) 118 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:42:52.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[29889]: cluster 2024-09-13T22:42:51.842976+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:42:52.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[29889]: cluster 2024-09-13T22:42:51.845701+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:42:52.425 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[22840]: cluster 2024-09-13T22:42:50.634019+0000 osd.3 (osd.3) 129 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:42:52.425 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[22840]: cluster 2024-09-13T22:42:50.643275+0000 osd.3 (osd.3) 130 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:42:52.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[22840]: cluster 2024-09-13T22:42:51.187634+0000 osd.1 (osd.1) 117 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:42:52.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[22840]: cluster 2024-09-13T22:42:51.196410+0000 osd.1 (osd.1) 118 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:42:52.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[22840]: cluster 2024-09-13T22:42:51.842976+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:42:52.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:52 smithi178 bash[22840]: cluster 2024-09-13T22:42:51.845701+0000 osd.6 (osd.6) 86 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:42:52.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:52.497 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:52.499 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:52.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:52.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:52.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:52.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:52.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:52.878 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:52.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:52.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:53.025 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:53.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:53.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:53.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.053282+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.059583+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.150544+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.a scrub starts 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.153180+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.a scrub ok 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.333808+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.339022+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:42:53.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.515457+0000 mgr.y (mgr.14613) 194 : cluster [DBG] pgmap v186: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.053282+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.059583+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.150544+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.a scrub starts 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.153180+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.a scrub ok 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.333808+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.339022+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:42:53.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:53 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.515457+0000 mgr.y (mgr.14613) 194 : cluster [DBG] pgmap v186: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s 2024-09-13T22:42:53.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:53.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:53.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.053282+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:42:53.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.059583+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:42:53.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.150544+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.a scrub starts 2024-09-13T22:42:53.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.153180+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.a scrub ok 2024-09-13T22:42:53.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.333808+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:42:53.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.339022+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:42:53.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:53 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.515457+0000 mgr.y (mgr.14613) 194 : cluster [DBG] pgmap v186: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s 2024-09-13T22:42:53.422 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:53.521 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:42:53.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:53.552 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:53.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:53.676 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:53.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:53.810 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:53.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:53.935 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:53.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:54.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:54.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:54.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:54.217 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:54.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:54.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.777333+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:52.779351+0000 osd.7 (osd.7) 120 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.180571+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.183127+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.284791+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.286641+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.856049+0000 osd.6 (osd.6) 87 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.857430+0000 osd.6 (osd.6) 88 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.777333+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:52.779351+0000 osd.7 (osd.7) 120 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:42:54.337 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.180571+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:42:54.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.183127+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:42:54.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.284791+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:42:54.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.286641+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:42:54.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.856049+0000 osd.6 (osd.6) 87 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:42:54.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:54 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.857430+0000 osd.6 (osd.6) 88 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:42:54.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:54.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:54.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.777333+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:42:54.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:52.779351+0000 osd.7 (osd.7) 120 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:42:54.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.180571+0000 osd.1 (osd.1) 121 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:42:54.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.183127+0000 osd.1 (osd.1) 122 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:42:54.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.284791+0000 osd.0 (osd.0) 125 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:42:54.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.286641+0000 osd.0 (osd.0) 126 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:42:54.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.856049+0000 osd.6 (osd.6) 87 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:42:54.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:54 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.857430+0000 osd.6 (osd.6) 88 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:42:54.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:54.474 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:54.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:54.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:54.608 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:54.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:54.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:54.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:54.864 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:54.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:54.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:54.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:55.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:55.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:55.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.773269+0000 osd.3 (osd.3) 131 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.788714+0000 osd.3 (osd.3) 132 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.803679+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:53.803748+0000 osd.2 (osd.2) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.067641+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.073416+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.080525+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.148354+0000 osd.1 (osd.1) 123 : cluster [DBG] 7.b deep-scrub starts 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.273671+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 deep-scrub starts 2024-09-13T22:42:55.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.275355+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 deep-scrub ok 2024-09-13T22:42:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.515875+0000 mgr.y (mgr.14613) 195 : cluster [DBG] pgmap v188: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 841 op/s 2024-09-13T22:42:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.748113+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:42:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.751213+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:42:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.836282+0000 osd.6 (osd.6) 89 : cluster [DBG] 3.a scrub starts 2024-09-13T22:42:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:55 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.838186+0000 osd.6 (osd.6) 90 : cluster [DBG] 3.a scrub ok 2024-09-13T22:42:55.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.773269+0000 osd.3 (osd.3) 131 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.788714+0000 osd.3 (osd.3) 132 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.803679+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:53.803748+0000 osd.2 (osd.2) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.067641+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.073416+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-13T22:42:55.229 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.080525+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.148354+0000 osd.1 (osd.1) 123 : cluster [DBG] 7.b deep-scrub starts 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.273671+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 deep-scrub starts 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.275355+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 deep-scrub ok 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.515875+0000 mgr.y (mgr.14613) 195 : cluster [DBG] pgmap v188: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 841 op/s 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.748113+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.751213+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.836282+0000 osd.6 (osd.6) 89 : cluster [DBG] 3.a scrub starts 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.838186+0000 osd.6 (osd.6) 90 : cluster [DBG] 3.a scrub ok 2024-09-13T22:42:55.230 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.773269+0000 osd.3 (osd.3) 131 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.788714+0000 osd.3 (osd.3) 132 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.803679+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:53.803748+0000 osd.2 (osd.2) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.067641+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.073416+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.080525+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.148354+0000 osd.1 (osd.1) 123 : cluster [DBG] 7.b deep-scrub starts 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.273671+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 deep-scrub starts 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.275355+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 deep-scrub ok 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.515875+0000 mgr.y (mgr.14613) 195 : cluster [DBG] pgmap v188: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 841 op/s 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.748113+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.751213+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:42:55.231 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.836282+0000 osd.6 (osd.6) 89 : cluster [DBG] 3.a scrub starts 2024-09-13T22:42:55.232 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:55 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.838186+0000 osd.6 (osd.6) 90 : cluster [DBG] 3.a scrub ok 2024-09-13T22:42:55.250 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:55.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:55.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:55.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:55.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:55.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:55.636 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:55.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:55.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:55.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:55.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:55.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:56.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:56.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:56.144 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:56 smithi192 bash[24608]: cluster 2024-09-13T22:42:54.765101+0000 osd.2 (osd.2) 87 : cluster [DBG] 7.e scrub starts 2024-09-13T22:42:56.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:56 smithi192 bash[24608]: audit 2024-09-13T22:42:55.030404+0000 mgr.y (mgr.14613) 196 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:56.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:56 smithi192 bash[24608]: cluster 2024-09-13T22:42:55.791355+0000 osd.6 (osd.6) 91 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:42:56.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:56 smithi192 bash[24608]: cluster 2024-09-13T22:42:55.792352+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:42:56.183 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:56.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:56.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[22840]: cluster 2024-09-13T22:42:54.765101+0000 osd.2 (osd.2) 87 : cluster [DBG] 7.e scrub starts 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[22840]: audit 2024-09-13T22:42:55.030404+0000 mgr.y (mgr.14613) 196 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[22840]: cluster 2024-09-13T22:42:55.791355+0000 osd.6 (osd.6) 91 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[22840]: cluster 2024-09-13T22:42:55.792352+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[29889]: cluster 2024-09-13T22:42:54.765101+0000 osd.2 (osd.2) 87 : cluster [DBG] 7.e scrub starts 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[29889]: audit 2024-09-13T22:42:55.030404+0000 mgr.y (mgr.14613) 196 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:42:56.458 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[29889]: cluster 2024-09-13T22:42:55.791355+0000 osd.6 (osd.6) 91 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:42:56.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:56 smithi178 bash[29889]: cluster 2024-09-13T22:42:55.792352+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:42:56.459 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:56.565 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:56.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.730 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:56.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:56.861 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:56.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:56.989 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:56.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:57.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:57.114 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:57.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:57.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:57.252 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:57 smithi192 bash[24608]: cluster 2024-09-13T22:42:55.830606+0000 osd.2 (osd.2) 88 : cluster [DBG] 7.8 scrub starts 2024-09-13T22:42:57.252 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:57 smithi192 bash[24608]: cluster 2024-09-13T22:42:56.293031+0000 osd.0 (osd.0) 129 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:42:57.252 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:57 smithi192 bash[24608]: cluster 2024-09-13T22:42:56.297983+0000 osd.0 (osd.0) 130 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:42:57.252 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:57 smithi192 bash[24608]: cluster 2024-09-13T22:42:56.516279+0000 mgr.y (mgr.14613) 197 : cluster [DBG] pgmap v189: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 841 op/s 2024-09-13T22:42:57.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:57.374 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:57.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[29889]: cluster 2024-09-13T22:42:55.830606+0000 osd.2 (osd.2) 88 : cluster [DBG] 7.8 scrub starts 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[29889]: cluster 2024-09-13T22:42:56.293031+0000 osd.0 (osd.0) 129 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[29889]: cluster 2024-09-13T22:42:56.297983+0000 osd.0 (osd.0) 130 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[29889]: cluster 2024-09-13T22:42:56.516279+0000 mgr.y (mgr.14613) 197 : cluster [DBG] pgmap v189: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 841 op/s 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[22840]: cluster 2024-09-13T22:42:55.830606+0000 osd.2 (osd.2) 88 : cluster [DBG] 7.8 scrub starts 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[22840]: cluster 2024-09-13T22:42:56.293031+0000 osd.0 (osd.0) 129 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[22840]: cluster 2024-09-13T22:42:56.297983+0000 osd.0 (osd.0) 130 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:42:57.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:57 smithi178 bash[22840]: cluster 2024-09-13T22:42:56.516279+0000 mgr.y (mgr.14613) 197 : cluster [DBG] pgmap v189: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 841 op/s 2024-09-13T22:42:57.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:57.512 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:57.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:57.655 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:57.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:42:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:42:57.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:57.906 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:57.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:58.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:58.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:42:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:42:58.150 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:58.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:58.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:58.298 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[22840]: cluster 2024-09-13T22:42:56.610958+0000 osd.3 (osd.3) 133 : cluster [DBG] 7.a scrub starts 2024-09-13T22:42:58.298 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[22840]: cluster 2024-09-13T22:42:57.287886+0000 osd.2 (osd.2) 89 : cluster [DBG] 7.e scrub ok 2024-09-13T22:42:58.298 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[22840]: cluster 2024-09-13T22:42:57.300312+0000 osd.5 (osd.5) 123 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:42:58.298 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[22840]: cluster 2024-09-13T22:42:57.304407+0000 osd.5 (osd.5) 124 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:42:58.298 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[29889]: cluster 2024-09-13T22:42:56.610958+0000 osd.3 (osd.3) 133 : cluster [DBG] 7.a scrub starts 2024-09-13T22:42:58.298 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[29889]: cluster 2024-09-13T22:42:57.287886+0000 osd.2 (osd.2) 89 : cluster [DBG] 7.e scrub ok 2024-09-13T22:42:58.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[29889]: cluster 2024-09-13T22:42:57.300312+0000 osd.5 (osd.5) 123 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:42:58.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:58 smithi178 bash[29889]: cluster 2024-09-13T22:42:57.304407+0000 osd.5 (osd.5) 124 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:42:58.319 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:58.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:58 smithi192 bash[24608]: cluster 2024-09-13T22:42:56.610958+0000 osd.3 (osd.3) 133 : cluster [DBG] 7.a scrub starts 2024-09-13T22:42:58.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:58 smithi192 bash[24608]: cluster 2024-09-13T22:42:57.287886+0000 osd.2 (osd.2) 89 : cluster [DBG] 7.e scrub ok 2024-09-13T22:42:58.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:58 smithi192 bash[24608]: cluster 2024-09-13T22:42:57.300312+0000 osd.5 (osd.5) 123 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:42:58.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:58 smithi192 bash[24608]: cluster 2024-09-13T22:42:57.304407+0000 osd.5 (osd.5) 124 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:42:58.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:58.443 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:58.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:58.576 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:58.629 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:42:58.629 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:42:58.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:58.703 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:58.729 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:42:58.729 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:42:58.777 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:42:58.777 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:42:58.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:58.827 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:42:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:42:58.830 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:42:58.830 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:42:58.854 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:42:58.855 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:42:58.870 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:42:58.871 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:42:58.903 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:42:58.903 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:42:58.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:58.940 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:42:58.940 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:42:58.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:58.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:42:58.985 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:42:58.985 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:42:59.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:59.122 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:42:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:42:59.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:57.541319+0000 osd.2 (osd.2) 90 : cluster [DBG] 7.8 scrub ok 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:57.854130+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:57.858133+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.236271+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.239963+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.277472+0000 osd.5 (osd.5) 125 : cluster [DBG] 7.0 deep-scrub starts 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.516884+0000 mgr.y (mgr.14613) 198 : cluster [DBG] pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 827 op/s 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.574872+0000 osd.3 (osd.3) 134 : cluster [DBG] 7.3 scrub starts 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[29889]: audit 2024-09-13T22:42:58.589068+0000 mon.b (mon.1) 112 : audit [DBG] from='client.? v1:172.21.15.178:0/149868145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:57.541319+0000 osd.2 (osd.2) 90 : cluster [DBG] 7.8 scrub ok 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:57.854130+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:57.858133+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:42:59.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.236271+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:42:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.239963+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:42:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.277472+0000 osd.5 (osd.5) 125 : cluster [DBG] 7.0 deep-scrub starts 2024-09-13T22:42:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.516884+0000 mgr.y (mgr.14613) 198 : cluster [DBG] pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 827 op/s 2024-09-13T22:42:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.574872+0000 osd.3 (osd.3) 134 : cluster [DBG] 7.3 scrub starts 2024-09-13T22:42:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[22840]: audit 2024-09-13T22:42:58.589068+0000 mon.b (mon.1) 112 : audit [DBG] from='client.? v1:172.21.15.178:0/149868145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:59.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:59.264 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:42:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:42:59.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:59.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:57.541319+0000 osd.2 (osd.2) 90 : cluster [DBG] 7.8 scrub ok 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:57.854130+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:57.858133+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.236271+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.239963+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.277472+0000 osd.5 (osd.5) 125 : cluster [DBG] 7.0 deep-scrub starts 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.516884+0000 mgr.y (mgr.14613) 198 : cluster [DBG] pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 827 op/s 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.574872+0000 osd.3 (osd.3) 134 : cluster [DBG] 7.3 scrub starts 2024-09-13T22:42:59.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:42:59 smithi192 bash[24608]: audit 2024-09-13T22:42:58.589068+0000 mon.b (mon.1) 112 : audit [DBG] from='client.? v1:172.21.15.178:0/149868145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:42:59.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:59.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:59.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:59.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:59.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:42:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:42:59.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:59.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:42:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:42:59.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:59.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:42:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:42:59.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:00.023 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.053 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:00.108 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:42:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:42:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:43:00.120 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T22:43:00.125 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.825047+0000 osd.2 (osd.2) 93 : cluster [DBG] 2.c scrub starts 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.827864+0000 osd.2 (osd.2) 94 : cluster [DBG] 2.c scrub ok 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:58.834947+0000 osd.3 (osd.3) 135 : cluster [DBG] 7.a scrub ok 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:59.274181+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:59.279499+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:59.305366+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.19 deep-scrub starts 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: cluster 2024-09-13T22:42:59.307058+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.19 deep-scrub ok 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: audit 2024-09-13T22:42:59.321963+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? v1:172.21.15.178:0/583456206' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:43:00.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: audit 2024-09-13T22:42:59.339216+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.178:0/2636204231' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[29889]: audit 2024-09-13T22:42:59.340731+0000 mon.a (mon.0) 1005 : audit [INF] from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.825047+0000 osd.2 (osd.2) 93 : cluster [DBG] 2.c scrub starts 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.827864+0000 osd.2 (osd.2) 94 : cluster [DBG] 2.c scrub ok 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:58.834947+0000 osd.3 (osd.3) 135 : cluster [DBG] 7.a scrub ok 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:59.274181+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:59.279499+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:59.305366+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.19 deep-scrub starts 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: cluster 2024-09-13T22:42:59.307058+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.19 deep-scrub ok 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: audit 2024-09-13T22:42:59.321963+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? v1:172.21.15.178:0/583456206' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: audit 2024-09-13T22:42:59.339216+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.178:0/2636204231' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:43:00.158 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:00 smithi178 bash[22840]: audit 2024-09-13T22:42:59.340731+0000 mon.a (mon.0) 1005 : audit [INF] from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:43:00.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:00.191 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:00.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.321 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.825047+0000 osd.2 (osd.2) 93 : cluster [DBG] 2.c scrub starts 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.827864+0000 osd.2 (osd.2) 94 : cluster [DBG] 2.c scrub ok 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:58.834947+0000 osd.3 (osd.3) 135 : cluster [DBG] 7.a scrub ok 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:59.274181+0000 osd.0 (osd.0) 133 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:59.279499+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:59.305366+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.19 deep-scrub starts 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: cluster 2024-09-13T22:42:59.307058+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.19 deep-scrub ok 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: audit 2024-09-13T22:42:59.321963+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? v1:172.21.15.178:0/583456206' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: audit 2024-09-13T22:42:59.339216+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.178:0/2636204231' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:43:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:00 smithi192 bash[24608]: audit 2024-09-13T22:42:59.340731+0000 mon.a (mon.0) 1005 : audit [INF] from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:43:00.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:00.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:00.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:00.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.709 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:00.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:00.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:00.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:00.977 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:00.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:01.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:01.110 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:01.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:01.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:01.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:01.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: cluster 2024-09-13T22:42:59.872005+0000 osd.2 (osd.2) 95 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: cluster 2024-09-13T22:42:59.874436+0000 osd.2 (osd.2) 96 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: cluster 2024-09-13T22:43:00.108740+0000 mon.a (mon.0) 1006 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: audit 2024-09-13T22:43:00.123977+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? v1:172.21.15.178:0/583456206' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: audit 2024-09-13T22:43:00.124153+0000 mon.a (mon.0) 1008 : audit [INF] from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: cluster 2024-09-13T22:43:00.129097+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[29889]: cluster 2024-09-13T22:43:00.517324+0000 mgr.y (mgr.14613) 199 : cluster [DBG] pgmap v192: 148 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 145 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: cluster 2024-09-13T22:42:59.872005+0000 osd.2 (osd.2) 95 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:43:01.359 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: cluster 2024-09-13T22:42:59.874436+0000 osd.2 (osd.2) 96 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:43:01.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: cluster 2024-09-13T22:43:00.108740+0000 mon.a (mon.0) 1006 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:01.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: audit 2024-09-13T22:43:00.123977+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? v1:172.21.15.178:0/583456206' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:43:01.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: audit 2024-09-13T22:43:00.124153+0000 mon.a (mon.0) 1008 : audit [INF] from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:43:01.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: cluster 2024-09-13T22:43:00.129097+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-13T22:43:01.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:01 smithi178 bash[22840]: cluster 2024-09-13T22:43:00.517324+0000 mgr.y (mgr.14613) 199 : cluster [DBG] pgmap v192: 148 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 145 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T22:43:01.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:01.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: cluster 2024-09-13T22:42:59.872005+0000 osd.2 (osd.2) 95 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:43:01.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: cluster 2024-09-13T22:42:59.874436+0000 osd.2 (osd.2) 96 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:43:01.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: cluster 2024-09-13T22:43:00.108740+0000 mon.a (mon.0) 1006 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:01.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: audit 2024-09-13T22:43:00.123977+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? v1:172.21.15.178:0/583456206' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:43:01.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: audit 2024-09-13T22:43:00.124153+0000 mon.a (mon.0) 1008 : audit [INF] from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:43:01.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: cluster 2024-09-13T22:43:00.129097+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-13T22:43:01.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:01 smithi192 bash[24608]: cluster 2024-09-13T22:43:00.517324+0000 mgr.y (mgr.14613) 199 : cluster [DBG] pgmap v192: 148 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 145 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T22:43:01.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.518 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:01.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:01.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:01.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:01.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.789 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:01.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:01.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:01.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:01.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:02.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:02.030 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:02.133 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:02.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:02.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:02.284 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:02.387 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:02.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:02.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:02.540 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:02.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:02.673 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:02.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:02.809 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:02.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:02.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:02.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:03.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:03.066 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:03.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:03.191 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:03.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:03.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:03.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:03.444 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:03.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:03.577 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:03.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:03.720 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:03.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:03.834 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:03 smithi178 bash[29889]: cluster 2024-09-13T22:43:02.517726+0000 mgr.y (mgr.14613) 200 : cluster [DBG] pgmap v193: 148 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 145 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s 2024-09-13T22:43:03.834 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:03 smithi178 bash[29889]: audit 2024-09-13T22:43:02.967456+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:43:03.834 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:03 smithi178 bash[29889]: audit 2024-09-13T22:43:02.968777+0000 mon.b (mon.1) 114 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:03.834 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:03 smithi178 bash[22840]: cluster 2024-09-13T22:43:02.517726+0000 mgr.y (mgr.14613) 200 : cluster [DBG] pgmap v193: 148 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 145 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s 2024-09-13T22:43:03.835 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:03 smithi178 bash[22840]: audit 2024-09-13T22:43:02.967456+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:43:03.835 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:03 smithi178 bash[22840]: audit 2024-09-13T22:43:02.968777+0000 mon.b (mon.1) 114 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:03.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:03.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:03 smithi192 bash[24608]: cluster 2024-09-13T22:43:02.517726+0000 mgr.y (mgr.14613) 200 : cluster [DBG] pgmap v193: 148 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 145 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s 2024-09-13T22:43:03.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:03 smithi192 bash[24608]: audit 2024-09-13T22:43:02.967456+0000 mon.a (mon.0) 1010 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:43:03.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:03 smithi192 bash[24608]: audit 2024-09-13T22:43:02.968777+0000 mon.b (mon.1) 114 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:03.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:03.988 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:03.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:04.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T22:43:04.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:04.112 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:04.133 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T22:43:04.133 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:43:04.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:04.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:04.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:04.407 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:04.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:04.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:04.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:04.664 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:04.766 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:04.804 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:04.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:04.942 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:04.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:05.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:05.078 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:05.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:05.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:43:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:43:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.212 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:05.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:05.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:05.335 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:05.439 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:05.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:05.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:05.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:05.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:05.729 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:05.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:05.842 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:05 smithi192 bash[24608]: cluster 2024-09-13T22:43:04.518625+0000 mgr.y (mgr.14613) 201 : cluster [DBG] pgmap v194: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 698 op/s 2024-09-13T22:43:05.858 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:05.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:05 smithi178 bash[29889]: cluster 2024-09-13T22:43:04.518625+0000 mgr.y (mgr.14613) 201 : cluster [DBG] pgmap v194: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 698 op/s 2024-09-13T22:43:05.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:05 smithi178 bash[22840]: cluster 2024-09-13T22:43:04.518625+0000 mgr.y (mgr.14613) 201 : cluster [DBG] pgmap v194: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 698 op/s 2024-09-13T22:43:05.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:05.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:05.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:06.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:06.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:43:06.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:06.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:43:06.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:06.260 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:06.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:06.386 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:06.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:06.509 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:06.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:06.648 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:06.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:06.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:06 smithi192 bash[24608]: audit 2024-09-13T22:43:05.038433+0000 mgr.y (mgr.14613) 202 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:06.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:06.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:06.916 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:06.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:06.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:06 smithi178 bash[29889]: audit 2024-09-13T22:43:05.038433+0000 mgr.y (mgr.14613) 202 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:06.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:06 smithi178 bash[22840]: audit 2024-09-13T22:43:05.038433+0000 mgr.y (mgr.14613) 202 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:07.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:07.042 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:07.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:07.164 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:07.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:07.295 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:07.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:07.420 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:07.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:07.549 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:07.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:07.677 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:07.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:07.802 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:07.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:07 smithi192 bash[24608]: cluster 2024-09-13T22:43:06.519078+0000 mgr.y (mgr.14613) 203 : cluster [DBG] pgmap v195: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 698 op/s 2024-09-13T22:43:07.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:07.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:07 smithi178 bash[29889]: cluster 2024-09-13T22:43:06.519078+0000 mgr.y (mgr.14613) 203 : cluster [DBG] pgmap v195: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 698 op/s 2024-09-13T22:43:07.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:07 smithi178 bash[22840]: cluster 2024-09-13T22:43:06.519078+0000 mgr.y (mgr.14613) 203 : cluster [DBG] pgmap v195: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 698 op/s 2024-09-13T22:43:07.930 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:07.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:08.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:08.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:08.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:08.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:08.195 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:08.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.333 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:08.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:08.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:08.463 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:08.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:08.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:08.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.739 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:08.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:08.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:08.879 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:08.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:08.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:08 smithi192 bash[24608]: cluster 2024-09-13T22:43:08.519719+0000 mgr.y (mgr.14613) 204 : cluster [DBG] pgmap v196: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 699 op/s 2024-09-13T22:43:08.961 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:08.963 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:08.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:08.996 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:08 smithi178 bash[22840]: cluster 2024-09-13T22:43:08.519719+0000 mgr.y (mgr.14613) 204 : cluster [DBG] pgmap v196: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 699 op/s 2024-09-13T22:43:08.998 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:08 smithi178 bash[29889]: cluster 2024-09-13T22:43:08.519719+0000 mgr.y (mgr.14613) 204 : cluster [DBG] pgmap v196: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 699 op/s 2024-09-13T22:43:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.027 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:09.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:09.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:09.156 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:09.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:09.287 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:09.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:09.423 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:09.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:09.559 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:09.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:09.689 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:09.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:09.817 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:09.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:09.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:09.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:10.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:10.067 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:43:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:43:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:43:10.087 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:10.190 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:10.213 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:10.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:10.339 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:10.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:10.470 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:10.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:10.612 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:10.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:10.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:10.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:10.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:10.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:10.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:10.999 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:11.002 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:11.002 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:11.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:11.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:11.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:11.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:11.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:11.362 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:11.386 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:11.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:11.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:11.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:11.666 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:11.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:11.784 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:11 smithi192 bash[24608]: cluster 2024-09-13T22:43:10.520381+0000 mgr.y (mgr.14613) 205 : cluster [DBG] pgmap v197: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 674 op/s 2024-09-13T22:43:11.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:11.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:11.928 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:11.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:11.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:11 smithi178 bash[29889]: cluster 2024-09-13T22:43:10.520381+0000 mgr.y (mgr.14613) 205 : cluster [DBG] pgmap v197: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 674 op/s 2024-09-13T22:43:11.942 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:11 smithi178 bash[22840]: cluster 2024-09-13T22:43:10.520381+0000 mgr.y (mgr.14613) 205 : cluster [DBG] pgmap v197: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 674 op/s 2024-09-13T22:43:12.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:12.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:12.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:12.199 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:12.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:12.324 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:12.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:12.453 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:12.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:12.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:12.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:12.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:12.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:12.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.846 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:12.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:12.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:12.973 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:12.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:13.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:13.099 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:13.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:13.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:13.373 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:13.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:13.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:13.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:13.618 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:13 smithi192 bash[24608]: cluster 2024-09-13T22:43:12.520813+0000 mgr.y (mgr.14613) 206 : cluster [DBG] pgmap v198: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 584 op/s 2024-09-13T22:43:13.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:13.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.769 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:13.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:13.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:13.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:13 smithi178 bash[29889]: cluster 2024-09-13T22:43:12.520813+0000 mgr.y (mgr.14613) 206 : cluster [DBG] pgmap v198: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 584 op/s 2024-09-13T22:43:13.880 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:13 smithi178 bash[22840]: cluster 2024-09-13T22:43:12.520813+0000 mgr.y (mgr.14613) 206 : cluster [DBG] pgmap v198: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 584 op/s 2024-09-13T22:43:13.894 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:13.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:13.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:14.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:14.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.147 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:14.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:14.281 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:14.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.422 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:14.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:14.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.685 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:14.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:14.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:14.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:14.959 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:14.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:15.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:15.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:43:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:43:15.092 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:15.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:15.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:15.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:15.347 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:15.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:15.487 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:15.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:15.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:15.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:15 smithi192 bash[24608]: cluster 2024-09-13T22:43:14.521754+0000 mgr.y (mgr.14613) 207 : cluster [DBG] pgmap v199: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 878 op/s 2024-09-13T22:43:15.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:15.749 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:15.750 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:15 smithi178 bash[29889]: cluster 2024-09-13T22:43:14.521754+0000 mgr.y (mgr.14613) 207 : cluster [DBG] pgmap v199: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 878 op/s 2024-09-13T22:43:15.751 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:15 smithi178 bash[22840]: cluster 2024-09-13T22:43:14.521754+0000 mgr.y (mgr.14613) 207 : cluster [DBG] pgmap v199: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 878 op/s 2024-09-13T22:43:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:15.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:15.885 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:15.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:15.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.018 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:16.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:16.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:16.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.292 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:16.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:16.423 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:16.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.552 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:16.655 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:16.686 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:16.687 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:16 smithi178 bash[29889]: audit 2024-09-13T22:43:15.046441+0000 mgr.y (mgr.14613) 208 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:16.688 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:16 smithi178 bash[22840]: audit 2024-09-13T22:43:15.046441+0000 mgr.y (mgr.14613) 208 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:16.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.798 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:16 smithi192 bash[24608]: audit 2024-09-13T22:43:15.046441+0000 mgr.y (mgr.14613) 208 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:16.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:16.867 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:43:16.867 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:43:16.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.939 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:16.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:17.014 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:43:17.014 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:43:17.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:17.098 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:43:17.098 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:43:17.107 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:17.131 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:43:17.131 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:43:17.186 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:43:17.186 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:43:17.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:43:17.197 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:43:17.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:17.223 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:43:17.223 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:43:17.262 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:43:17.262 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:43:17.271 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:17.301 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:43:17.301 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:43:17.330 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:43:17.331 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:43:17.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.413 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:17.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:17.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:17.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:17.618 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:43:17.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:17.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:17 smithi192 bash[24608]: cluster 2024-09-13T22:43:16.522183+0000 mgr.y (mgr.14613) 209 : cluster [DBG] pgmap v200: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 606 op/s 2024-09-13T22:43:17.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:17 smithi192 bash[24608]: audit 2024-09-13T22:43:16.956178+0000 mon.b (mon.1) 115 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:43:17.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:17 smithi192 bash[24608]: audit 2024-09-13T22:43:16.968012+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? v1:172.21.15.178:0/3736225430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:17.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:17 smithi192 bash[24608]: audit 2024-09-13T22:43:17.170207+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.178:0/3489484222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:17.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:17 smithi192 bash[24608]: audit 2024-09-13T22:43:17.170621+0000 mgr.y (mgr.14613) 210 : audit [DBG] from='client.14979 v1:172.21.15.178:0/3002301284' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:43:17.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:17 smithi192 bash[24608]: audit 2024-09-13T22:43:17.171045+0000 mon.a (mon.0) 1011 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:17.682 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:17.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:17.811 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:17.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:17.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[29889]: cluster 2024-09-13T22:43:16.522183+0000 mgr.y (mgr.14613) 209 : cluster [DBG] pgmap v200: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 606 op/s 2024-09-13T22:43:17.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[29889]: audit 2024-09-13T22:43:16.956178+0000 mon.b (mon.1) 115 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:43:17.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[29889]: audit 2024-09-13T22:43:16.968012+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? v1:172.21.15.178:0/3736225430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:17.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[29889]: audit 2024-09-13T22:43:17.170207+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.178:0/3489484222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:17.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[29889]: audit 2024-09-13T22:43:17.170621+0000 mgr.y (mgr.14613) 210 : audit [DBG] from='client.14979 v1:172.21.15.178:0/3002301284' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[29889]: audit 2024-09-13T22:43:17.171045+0000 mon.a (mon.0) 1011 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[22840]: cluster 2024-09-13T22:43:16.522183+0000 mgr.y (mgr.14613) 209 : cluster [DBG] pgmap v200: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 606 op/s 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[22840]: audit 2024-09-13T22:43:16.956178+0000 mon.b (mon.1) 115 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[22840]: audit 2024-09-13T22:43:16.968012+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? v1:172.21.15.178:0/3736225430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[22840]: audit 2024-09-13T22:43:17.170207+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.178:0/3489484222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[22840]: audit 2024-09-13T22:43:17.170621+0000 mgr.y (mgr.14613) 210 : audit [DBG] from='client.14979 v1:172.21.15.178:0/3002301284' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:43:17.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:17 smithi178 bash[22840]: audit 2024-09-13T22:43:17.171045+0000 mon.a (mon.0) 1011 : audit [INF] from='client.24874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:17.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:17.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:18.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:18.066 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:18.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:18.193 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:18.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:18.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:18.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:18.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:18.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:18.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:18.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:18.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:18.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:18.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: cluster 2024-09-13T22:43:17.610121+0000 mon.a (mon.0) 1012 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:18.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: audit 2024-09-13T22:43:17.614156+0000 mon.a (mon.0) 1013 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:43:18.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: cluster 2024-09-13T22:43:17.618196+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-13T22:43:18.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: audit 2024-09-13T22:43:17.899049+0000 mon.b (mon.1) 117 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:43:18.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: audit 2024-09-13T22:43:17.900198+0000 mon.b (mon.1) 118 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:43:18.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: audit 2024-09-13T22:43:17.908140+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:43:18.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: audit 2024-09-13T22:43:17.986003+0000 mon.b (mon.1) 119 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:18.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:18 smithi192 bash[24608]: cluster 2024-09-13T22:43:18.522835+0000 mgr.y (mgr.14613) 211 : cluster [DBG] pgmap v202: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 706 op/s 2024-09-13T22:43:18.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:18.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:18.964 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:18.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:19.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:19.096 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:19.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:19.217 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:19.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:19.223 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: cluster 2024-09-13T22:43:17.610121+0000 mon.a (mon.0) 1012 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:19.223 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: audit 2024-09-13T22:43:17.614156+0000 mon.a (mon.0) 1013 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:43:19.223 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: cluster 2024-09-13T22:43:17.618196+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-13T22:43:19.223 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: audit 2024-09-13T22:43:17.899049+0000 mon.b (mon.1) 117 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:43:19.223 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: audit 2024-09-13T22:43:17.900198+0000 mon.b (mon.1) 118 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:43:19.223 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: audit 2024-09-13T22:43:17.908140+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: audit 2024-09-13T22:43:17.986003+0000 mon.b (mon.1) 119 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[22840]: cluster 2024-09-13T22:43:18.522835+0000 mgr.y (mgr.14613) 211 : cluster [DBG] pgmap v202: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 706 op/s 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: cluster 2024-09-13T22:43:17.610121+0000 mon.a (mon.0) 1012 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: audit 2024-09-13T22:43:17.614156+0000 mon.a (mon.0) 1013 : audit [INF] from='client.24874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: cluster 2024-09-13T22:43:17.618196+0000 mon.a (mon.0) 1014 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: audit 2024-09-13T22:43:17.899049+0000 mon.b (mon.1) 117 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: audit 2024-09-13T22:43:17.900198+0000 mon.b (mon.1) 118 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: audit 2024-09-13T22:43:17.908140+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: audit 2024-09-13T22:43:17.986003+0000 mon.b (mon.1) 119 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:19.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:18 smithi178 bash[29889]: cluster 2024-09-13T22:43:18.522835+0000 mgr.y (mgr.14613) 211 : cluster [DBG] pgmap v202: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 706 op/s 2024-09-13T22:43:19.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:19.344 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:19.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:19.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:19.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:19.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:19.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T22:43:19.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:19.731 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:19.834 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:19.864 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:19.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:19.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:19.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:20.024 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-13T22:43:16.521932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5154413107,"num_objects":78841,"num_object_clones":0,"num_object_copies":236523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78841,"num_whiteouts":0,"num_read":1264,"num_read_kb":1110,"num_write":133475,"num_write_kb":6770462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90761,"ondisk_log_size":90761,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15862504,"kb_used_data":15204648,"kb_used_omap":12,"kb_used_meta":657459,"kb_avail":734098712,"statfs":{"total":767960285184,"available":751717081088,"internally_reserved":0,"allocated":15569559552,"data_stored":15566627177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":673238576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-476184576,"num_objects":-7266,"num_object_clones":0,"num_object_copies":-21798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7266,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":7266,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004021"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":166,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146222+0000","last_change":"2024-09-13T22:41:44.889037+0000","last_active":"2024-09-13T22:43:00.146222+0000","last_peered":"2024-09-13T22:43:00.146222+0000","last_clean":"2024-09-13T22:43:00.146222+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:43:00.146222+0000","last_undegraded":"2024-09-13T22:43:00.146222+0000","last_fullsized":"2024-09-13T22:43:00.146222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.888900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.888900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.153720+0000","last_change":"2024-09-13T22:41:33.637917+0000","last_active":"2024-09-13T22:43:00.153720+0000","last_peered":"2024-09-13T22:43:00.153720+0000","last_clean":"2024-09-13T22:43:00.153720+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:43:00.153720+0000","last_undegraded":"2024-09-13T22:43:00.153720+0000","last_fullsized":"2024-09-13T22:43:00.153720+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:33.637519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.637519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":202,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143453+0000","last_change":"2024-09-13T22:41:49.766349+0000","last_active":"2024-09-13T22:43:00.143453+0000","last_peered":"2024-09-13T22:43:00.143453+0000","last_clean":"2024-09-13T22:43:00.143453+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:43:00.143453+0000","last_undegraded":"2024-09-13T22:43:00.143453+0000","last_fullsized":"2024-09-13T22:43:00.143453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:41:49.765993+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:02.705641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129491+0000","last_change":"2024-09-13T22:41:55.780046+0000","last_active":"2024-09-13T22:43:00.129491+0000","last_peered":"2024-09-13T22:43:00.129491+0000","last_clean":"2024-09-13T22:43:00.129491+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:43:00.129491+0000","last_undegraded":"2024-09-13T22:43:00.129491+0000","last_fullsized":"2024-09-13T22:43:00.129491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:41:55.779975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:07.536927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143545+0000","last_change":"2024-09-13T22:41:26.073240+0000","last_active":"2024-09-13T22:43:00.143545+0000","last_peered":"2024-09-13T22:43:00.143545+0000","last_clean":"2024-09-13T22:43:00.143545+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:43:00.143545+0000","last_undegraded":"2024-09-13T22:43:00.143545+0000","last_fullsized":"2024-09-13T22:43:00.143545+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:26.073058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:26.073058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":206,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.153463+0000","last_change":"2024-09-13T22:41:57.526625+0000","last_active":"2024-09-13T22:43:00.153463+0000","last_peered":"2024-09-13T22:43:00.153463+0000","last_clean":"2024-09-13T22:43:00.153463+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:43:00.153463+0000","last_undegraded":"2024-09-13T22:43:00.153463+0000","last_fullsized":"2024-09-13T22:43:00.153463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:41:57.526506+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:41:57.526506+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.257711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151274+0000","last_change":"2024-09-13T22:42:59.307163+0000","last_active":"2024-09-13T22:43:00.151274+0000","last_peered":"2024-09-13T22:43:00.151274+0000","last_clean":"2024-09-13T22:43:00.151274+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:43:00.151274+0000","last_undegraded":"2024-09-13T22:43:00.151274+0000","last_fullsized":"2024-09-13T22:43:00.151274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.307067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:18.937510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129648+0000","last_change":"2024-09-13T22:41:53.897309+0000","last_active":"2024-09-13T22:43:00.129648+0000","last_peered":"2024-09-13T22:43:00.129648+0000","last_clean":"2024-09-13T22:43:00.129648+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:43:00.129648+0000","last_undegraded":"2024-09-13T22:43:00.129648+0000","last_fullsized":"2024-09-13T22:43:00.129648+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_clean_scrub_stamp":"2024-09-13T22:41:53.894021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:07.236970+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151979+0000","last_change":"2024-09-13T22:42:35.837037+0000","last_active":"2024-09-13T22:43:00.151979+0000","last_peered":"2024-09-13T22:43:00.151979+0000","last_clean":"2024-09-13T22:43:00.151979+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:43:00.151979+0000","last_undegraded":"2024-09-13T22:43:00.151979+0000","last_fullsized":"2024-09-13T22:43:00.151979+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:35.836936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:35.836936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:49.472291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":213,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146247+0000","last_change":"2024-09-13T22:42:39.154087+0000","last_active":"2024-09-13T22:43:00.146247+0000","last_peered":"2024-09-13T22:43:00.146247+0000","last_clean":"2024-09-13T22:43:00.146247+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:43:00.146247+0000","last_undegraded":"2024-09-13T22:43:00.146247+0000","last_fullsized":"2024-09-13T22:43:00.146247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:42:39.153983+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.153983+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:43.318456+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151305+0000","last_change":"2024-09-13T22:41:49.716084+0000","last_active":"2024-09-13T22:43:00.151305+0000","last_peered":"2024-09-13T22:43:00.151305+0000","last_clean":"2024-09-13T22:43:00.151305+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:43:00.151305+0000","last_undegraded":"2024-09-13T22:43:00.151305+0000","last_fullsized":"2024-09-13T22:43:00.151305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:49.715971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:41:49.715971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:19.642530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145879+0000","last_change":"2024-09-13T22:41:41.033132+0000","last_active":"2024-09-13T22:43:00.145879+0000","last_peered":"2024-09-13T22:43:00.145879+0000","last_clean":"2024-09-13T22:43:00.145879+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:43:00.145879+0000","last_undegraded":"2024-09-13T22:43:00.145879+0000","last_fullsized":"2024-09-13T22:43:00.145879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.032827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.032827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151787+0000","last_change":"2024-09-13T22:41:57.761398+0000","last_active":"2024-09-13T22:43:00.151787+0000","last_peered":"2024-09-13T22:43:00.151787+0000","last_clean":"2024-09-13T22:43:00.151787+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:43:00.151787+0000","last_undegraded":"2024-09-13T22:43:00.151787+0000","last_fullsized":"2024-09-13T22:43:00.151787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:57.761322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:57.761322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:00.168271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146728+0000","last_change":"2024-09-13T22:42:29.188915+0000","last_active":"2024-09-13T22:43:00.146728+0000","last_peered":"2024-09-13T22:43:00.146728+0000","last_clean":"2024-09-13T22:43:00.146728+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:43:00.146728+0000","last_undegraded":"2024-09-13T22:43:00.146728+0000","last_fullsized":"2024-09-13T22:43:00.146728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:42:29.188836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:29.188836+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:39.007169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146057+0000","last_change":"2024-09-13T22:42:37.031507+0000","last_active":"2024-09-13T22:43:00.146057+0000","last_peered":"2024-09-13T22:43:00.146057+0000","last_clean":"2024-09-13T22:43:00.146057+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:43:00.146057+0000","last_undegraded":"2024-09-13T22:43:00.146057+0000","last_fullsized":"2024-09-13T22:43:00.146057+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:37.031329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.031329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.995792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.148788+0000","last_change":"2024-09-13T22:41:48.479208+0000","last_active":"2024-09-13T22:43:00.148788+0000","last_peered":"2024-09-13T22:43:00.148788+0000","last_clean":"2024-09-13T22:43:00.148788+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:43:00.148788+0000","last_undegraded":"2024-09-13T22:43:00.148788+0000","last_fullsized":"2024-09-13T22:43:00.148788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:48.479129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:48.479129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:04.485410+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"150'8233","reported_seq":8377,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.577242+0000","last_change":"2024-09-13T22:42:57.288156+0000","last_active":"2024-09-13T22:43:13.577242+0000","last_peered":"2024-09-13T22:43:13.577242+0000","last_clean":"2024-09-13T22:43:13.577242+0000","last_became_active":"2024-09-13T22:40:54.965149+0000","last_became_peered":"2024-09-13T22:40:54.965149+0000","last_unstale":"2024-09-13T22:43:13.577242+0000","last_undegraded":"2024-09-13T22:43:13.577242+0000","last_fullsized":"2024-09-13T22:43:13.577242+0000","mapping_epoch":116,"log_start":"121'1500","ondisk_log_start":"121'1500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7538","last_scrub_stamp":"2024-09-13T22:42:57.287895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.287895+0000","objects_scrubbed":5635,"log_size":6733,"log_dups_size":0,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.157500+0000","scrub_duration":2527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320929792,"num_objects":4897,"num_object_clones":0,"num_object_copies":14691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8233,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"114'28","reported_seq":230,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144428+0000","last_change":"2024-09-13T22:42:54.838293+0000","last_active":"2024-09-13T22:43:00.144428+0000","last_peered":"2024-09-13T22:43:00.144428+0000","last_clean":"2024-09-13T22:43:00.144428+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:43:00.144428+0000","last_undegraded":"2024-09-13T22:43:00.144428+0000","last_fullsized":"2024-09-13T22:43:00.144428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'28","last_scrub_stamp":"2024-09-13T22:42:54.838192+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.838192+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:00.553515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.152053+0000","last_change":"2024-09-13T22:40:48.576923+0000","last_active":"2024-09-13T22:43:00.152053+0000","last_peered":"2024-09-13T22:43:00.152053+0000","last_clean":"2024-09-13T22:43:00.152053+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:43:00.152053+0000","last_undegraded":"2024-09-13T22:43:00.152053+0000","last_fullsized":"2024-09-13T22:43:00.152053+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:48.576808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:48.576808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.142437+0000","last_change":"2024-09-13T22:42:28.929922+0000","last_active":"2024-09-13T22:43:00.142437+0000","last_peered":"2024-09-13T22:43:00.142437+0000","last_clean":"2024-09-13T22:43:00.142437+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:43:00.142437+0000","last_undegraded":"2024-09-13T22:43:00.142437+0000","last_fullsized":"2024-09-13T22:43:00.142437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:28.929617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:28.929617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:43.287158+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145223+0000","last_change":"2024-09-13T22:41:54.670111+0000","last_active":"2024-09-13T22:43:00.145223+0000","last_peered":"2024-09-13T22:43:00.145223+0000","last_clean":"2024-09-13T22:43:00.145223+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:43:00.145223+0000","last_undegraded":"2024-09-13T22:43:00.145223+0000","last_fullsized":"2024-09-13T22:43:00.145223+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:54.669996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.669996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:24.550540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"150'8271","reported_seq":8420,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.825717+0000","last_change":"2024-09-13T22:42:48.154144+0000","last_active":"2024-09-13T22:43:13.825717+0000","last_peered":"2024-09-13T22:43:13.825717+0000","last_clean":"2024-09-13T22:43:13.825717+0000","last_became_active":"2024-09-13T22:40:54.957954+0000","last_became_peered":"2024-09-13T22:40:54.957954+0000","last_unstale":"2024-09-13T22:43:13.825717+0000","last_undegraded":"2024-09-13T22:43:13.825717+0000","last_fullsized":"2024-09-13T22:43:13.825717+0000","mapping_epoch":116,"log_start":"124'3000","ondisk_log_start":"124'3000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7054","last_scrub_stamp":"2024-09-13T22:42:48.154030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:48.154030+0000","objects_scrubbed":6079,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:00.734631+0000","scrub_duration":3823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312672256,"num_objects":4771,"num_object_clones":0,"num_object_copies":14313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":417344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"110'13","reported_seq":202,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146728+0000","last_change":"2024-09-13T22:41:56.802293+0000","last_active":"2024-09-13T22:43:00.146728+0000","last_peered":"2024-09-13T22:43:00.146728+0000","last_clean":"2024-09-13T22:43:00.146728+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:43:00.146728+0000","last_undegraded":"2024-09-13T22:43:00.146728+0000","last_fullsized":"2024-09-13T22:43:00.146728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-09-13T22:41:56.802209+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:41:56.802209+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:18.425832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144049+0000","last_change":"2024-09-13T22:42:52.153298+0000","last_active":"2024-09-13T22:43:00.144049+0000","last_peered":"2024-09-13T22:43:00.144049+0000","last_clean":"2024-09-13T22:43:00.144049+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:43:00.144049+0000","last_undegraded":"2024-09-13T22:43:00.144049+0000","last_fullsized":"2024-09-13T22:43:00.144049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:52.153187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.153187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.767977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":255,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.056321+0000","last_change":"2024-09-13T22:42:47.690211+0000","last_active":"2024-09-13T22:43:06.056321+0000","last_peered":"2024-09-13T22:43:06.056321+0000","last_clean":"2024-09-13T22:43:06.056321+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:43:06.056321+0000","last_undegraded":"2024-09-13T22:43:06.056321+0000","last_fullsized":"2024-09-13T22:43:06.056321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:47.690105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.690105+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.251912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129906+0000","last_change":"2024-09-13T22:41:29.671454+0000","last_active":"2024-09-13T22:43:00.129906+0000","last_peered":"2024-09-13T22:43:00.129906+0000","last_clean":"2024-09-13T22:43:00.129906+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:43:00.129906+0000","last_undegraded":"2024-09-13T22:43:00.129906+0000","last_fullsized":"2024-09-13T22:43:00.129906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:29.671369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:29.671369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"150'8380","reported_seq":8532,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.482942+0000","last_change":"2024-09-13T22:42:37.860665+0000","last_active":"2024-09-13T22:43:13.482942+0000","last_peered":"2024-09-13T22:43:13.482942+0000","last_clean":"2024-09-13T22:43:13.482942+0000","last_became_active":"2024-09-13T22:40:54.952746+0000","last_became_peered":"2024-09-13T22:40:54.952746+0000","last_unstale":"2024-09-13T22:43:13.482942+0000","last_undegraded":"2024-09-13T22:43:13.482942+0000","last_fullsized":"2024-09-13T22:43:13.482942+0000","mapping_epoch":116,"log_start":"122'1900","ondisk_log_start":"122'1900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'6674","last_scrub_stamp":"2024-09-13T22:42:37.860298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.860298+0000","objects_scrubbed":6611,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:07.575624+0000","scrub_duration":2031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321388544,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8380,"num_write_kb":425088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":207,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.172095+0000","last_change":"2024-09-13T22:42:28.345281+0000","last_active":"2024-09-13T22:43:00.172095+0000","last_peered":"2024-09-13T22:43:00.172095+0000","last_clean":"2024-09-13T22:43:00.172095+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:43:00.172095+0000","last_undegraded":"2024-09-13T22:43:00.172095+0000","last_fullsized":"2024-09-13T22:43:00.172095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:42:28.345153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:28.345153+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:41.482075+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151499+0000","last_change":"2024-09-13T22:41:52.638569+0000","last_active":"2024-09-13T22:43:00.151499+0000","last_peered":"2024-09-13T22:43:00.151499+0000","last_clean":"2024-09-13T22:43:00.151499+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:43:00.151499+0000","last_undegraded":"2024-09-13T22:43:00.151499+0000","last_fullsized":"2024-09-13T22:43:00.151499+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:52.638224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.638224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":256,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.055199+0000","last_change":"2024-09-13T22:42:50.045223+0000","last_active":"2024-09-13T22:43:06.055199+0000","last_peered":"2024-09-13T22:43:06.055199+0000","last_clean":"2024-09-13T22:43:06.055199+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:43:06.055199+0000","last_undegraded":"2024-09-13T22:43:06.055199+0000","last_fullsized":"2024-09-13T22:43:06.055199+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:42:50.045100+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.045100+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.086555+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146144+0000","last_change":"2024-09-13T22:41:32.995433+0000","last_active":"2024-09-13T22:43:00.146144+0000","last_peered":"2024-09-13T22:43:00.146144+0000","last_clean":"2024-09-13T22:43:00.146144+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:43:00.146144+0000","last_undegraded":"2024-09-13T22:43:00.146144+0000","last_fullsized":"2024-09-13T22:43:00.146144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:32.995357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_clean_scrub_stamp":"2024-09-13T22:41:32.995357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:00.834000+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"150'8388","reported_seq":8540,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.508505+0000","last_change":"2024-09-13T22:42:47.749351+0000","last_active":"2024-09-13T22:43:13.508505+0000","last_peered":"2024-09-13T22:43:13.508505+0000","last_clean":"2024-09-13T22:43:13.508505+0000","last_became_active":"2024-09-13T22:40:54.959205+0000","last_became_peered":"2024-09-13T22:40:54.959205+0000","last_unstale":"2024-09-13T22:43:13.508505+0000","last_undegraded":"2024-09-13T22:43:13.508505+0000","last_fullsized":"2024-09-13T22:43:13.508505+0000","mapping_epoch":116,"log_start":"123'2000","ondisk_log_start":"123'2000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7167","last_scrub_stamp":"2024-09-13T22:42:47.749227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.749227+0000","objects_scrubbed":6263,"log_size":6388,"log_dups_size":0,"ondisk_log_size":6388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:52.740875+0000","scrub_duration":1899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327876632,"num_objects":5004,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8388,"num_write_kb":428481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":205,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129747+0000","last_change":"2024-09-13T22:41:54.888290+0000","last_active":"2024-09-13T22:43:00.129747+0000","last_peered":"2024-09-13T22:43:00.129747+0000","last_clean":"2024-09-13T22:43:00.129747+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:43:00.129747+0000","last_undegraded":"2024-09-13T22:43:00.129747+0000","last_fullsized":"2024-09-13T22:43:00.129747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:41:54.887968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.887968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:16.966191+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151185+0000","last_change":"2024-09-13T22:41:47.671619+0000","last_active":"2024-09-13T22:43:00.151185+0000","last_peered":"2024-09-13T22:43:00.151185+0000","last_clean":"2024-09-13T22:43:00.151185+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:43:00.151185+0000","last_undegraded":"2024-09-13T22:43:00.151185+0000","last_fullsized":"2024-09-13T22:43:00.151185+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.671515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.671515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:05.830624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146543+0000","last_change":"2024-09-13T22:41:22.943141+0000","last_active":"2024-09-13T22:43:00.146543+0000","last_peered":"2024-09-13T22:43:00.146543+0000","last_clean":"2024-09-13T22:43:00.146543+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:43:00.146543+0000","last_undegraded":"2024-09-13T22:43:00.146543+0000","last_fullsized":"2024-09-13T22:43:00.146543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:22.943025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:22.943025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":166,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.135887+0000","last_change":"2024-09-13T22:41:35.880618+0000","last_active":"2024-09-13T22:43:00.135887+0000","last_peered":"2024-09-13T22:43:00.135887+0000","last_clean":"2024-09-13T22:43:00.135887+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:43:00.135887+0000","last_undegraded":"2024-09-13T22:43:00.135887+0000","last_fullsized":"2024-09-13T22:43:00.135887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:35.878354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:35.878354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:02.930623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"150'8245","reported_seq":8396,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:14.315304+0000","last_change":"2024-09-13T22:42:42.864731+0000","last_active":"2024-09-13T22:43:14.315304+0000","last_peered":"2024-09-13T22:43:14.315304+0000","last_clean":"2024-09-13T22:43:14.315304+0000","last_became_active":"2024-09-13T22:40:54.961774+0000","last_became_peered":"2024-09-13T22:40:54.961774+0000","last_unstale":"2024-09-13T22:43:14.315304+0000","last_undegraded":"2024-09-13T22:43:14.315304+0000","last_fullsized":"2024-09-13T22:43:14.315304+0000","mapping_epoch":116,"log_start":"124'3100","ondisk_log_start":"124'3100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6785","last_scrub_stamp":"2024-09-13T22:42:42.864469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:42.864469+0000","objects_scrubbed":6365,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.071905+0000","scrub_duration":3367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316735488,"num_objects":4833,"num_object_clones":0,"num_object_copies":14499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":200,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.153021+0000","last_change":"2024-09-13T22:42:40.311926+0000","last_active":"2024-09-13T22:43:00.153021+0000","last_peered":"2024-09-13T22:43:00.153021+0000","last_clean":"2024-09-13T22:43:00.153021+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:43:00.153021+0000","last_undegraded":"2024-09-13T22:43:00.153021+0000","last_fullsized":"2024-09-13T22:43:00.153021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:42:40.311810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.311810+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:48.816888+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146708+0000","last_change":"2024-09-13T22:42:49.867175+0000","last_active":"2024-09-13T22:43:00.146708+0000","last_peered":"2024-09-13T22:43:00.146708+0000","last_clean":"2024-09-13T22:43:00.146708+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:43:00.146708+0000","last_undegraded":"2024-09-13T22:43:00.146708+0000","last_fullsized":"2024-09-13T22:43:00.146708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:49.867081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.867081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.457345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":190,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129739+0000","last_change":"2024-09-13T22:42:38.889988+0000","last_active":"2024-09-13T22:43:00.129739+0000","last_peered":"2024-09-13T22:43:00.129739+0000","last_clean":"2024-09-13T22:43:00.129739+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:43:00.129739+0000","last_undegraded":"2024-09-13T22:43:00.129739+0000","last_fullsized":"2024-09-13T22:43:00.129739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:42:38.889867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:38.889867+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.145041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":247,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.058537+0000","last_change":"2024-09-13T22:41:33.857435+0000","last_active":"2024-09-13T22:43:06.058537+0000","last_peered":"2024-09-13T22:43:06.058537+0000","last_clean":"2024-09-13T22:43:06.058537+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:43:06.058537+0000","last_undegraded":"2024-09-13T22:43:06.058537+0000","last_fullsized":"2024-09-13T22:43:06.058537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:33.857359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.857359+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151462+0000","last_change":"2024-09-13T22:41:35.694198+0000","last_active":"2024-09-13T22:43:00.151462+0000","last_peered":"2024-09-13T22:43:00.151462+0000","last_clean":"2024-09-13T22:43:00.151462+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:43:00.151462+0000","last_undegraded":"2024-09-13T22:43:00.151462+0000","last_fullsized":"2024-09-13T22:43:00.151462+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:35.694064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:35.694064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"150'8284","reported_seq":8408,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:14.277194+0000","last_change":"2024-09-13T22:43:00.227649+0000","last_active":"2024-09-13T22:43:14.277194+0000","last_peered":"2024-09-13T22:43:14.277194+0000","last_clean":"2024-09-13T22:43:14.277194+0000","last_became_active":"2024-09-13T22:40:54.961695+0000","last_became_peered":"2024-09-13T22:40:54.961695+0000","last_unstale":"2024-09-13T22:43:14.277194+0000","last_undegraded":"2024-09-13T22:43:14.277194+0000","last_fullsized":"2024-09-13T22:43:14.277194+0000","mapping_epoch":116,"log_start":"124'3100","ondisk_log_start":"124'3100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":4074,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:17.428014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320733184,"num_objects":4894,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8284,"num_write_kb":421696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":214,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.136745+0000","last_change":"2024-09-13T22:42:53.788869+0000","last_active":"2024-09-13T22:43:00.136745+0000","last_peered":"2024-09-13T22:43:00.136745+0000","last_clean":"2024-09-13T22:43:00.136745+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:43:00.136745+0000","last_undegraded":"2024-09-13T22:43:00.136745+0000","last_fullsized":"2024-09-13T22:43:00.136745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:42:53.788721+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.788721+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.693042+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":187,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.141185+0000","last_change":"2024-09-13T22:42:46.200128+0000","last_active":"2024-09-13T22:43:00.141185+0000","last_peered":"2024-09-13T22:43:00.141185+0000","last_clean":"2024-09-13T22:43:00.141185+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:43:00.141185+0000","last_undegraded":"2024-09-13T22:43:00.141185+0000","last_fullsized":"2024-09-13T22:43:00.141185+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:46.200054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.200054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:46.600075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":497,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:12.765113+0000","last_change":"2024-09-13T22:41:31.810930+0000","last_active":"2024-09-13T22:43:12.765113+0000","last_peered":"2024-09-13T22:43:12.765113+0000","last_clean":"2024-09-13T22:43:12.765113+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:43:12.765113+0000","last_undegraded":"2024-09-13T22:43:12.765113+0000","last_fullsized":"2024-09-13T22:43:12.765113+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:41:31.810814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:31.810814+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:17.428099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129534+0000","last_change":"2024-09-13T22:41:36.600260+0000","last_active":"2024-09-13T22:43:00.129534+0000","last_peered":"2024-09-13T22:43:00.129534+0000","last_clean":"2024-09-13T22:43:00.129534+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:43:00.129534+0000","last_undegraded":"2024-09-13T22:43:00.129534+0000","last_fullsized":"2024-09-13T22:43:00.129534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:36.600156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:36.600156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:02.155689+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129537+0000","last_change":"2024-09-13T22:42:54.080648+0000","last_active":"2024-09-13T22:43:00.129537+0000","last_peered":"2024-09-13T22:43:00.129537+0000","last_clean":"2024-09-13T22:43:00.129537+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:43:00.129537+0000","last_undegraded":"2024-09-13T22:43:00.129537+0000","last_fullsized":"2024-09-13T22:43:00.129537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.080532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.080532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:06.210400+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"150'11063","reported_seq":11243,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-13T22:43:13.511541+0000","last_change":"2024-09-13T22:42:58.252972+0000","last_active":"2024-09-13T22:43:13.511541+0000","last_peered":"2024-09-13T22:43:13.511541+0000","last_clean":"2024-09-13T22:43:13.511541+0000","last_became_active":"2024-09-13T22:40:54.952389+0000","last_became_peered":"2024-09-13T22:40:54.952389+0000","last_unstale":"2024-09-13T22:43:13.511541+0000","last_undegraded":"2024-09-13T22:43:13.511541+0000","last_fullsized":"2024-09-13T22:43:13.511541+0000","mapping_epoch":116,"log_start":"139'5702","ondisk_log_start":"139'5702","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":2849,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325910528,"num_objects":4973,"num_object_clones":0,"num_object_copies":14919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":423744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":299,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.061732+0000","last_change":"2024-09-13T22:42:51.196536+0000","last_active":"2024-09-13T22:43:06.061732+0000","last_peered":"2024-09-13T22:43:06.061732+0000","last_clean":"2024-09-13T22:43:06.061732+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:43:06.061732+0000","last_undegraded":"2024-09-13T22:43:06.061732+0000","last_fullsized":"2024-09-13T22:43:06.061732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:42:51.196417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:51.196417+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:10.030113+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.152014+0000","last_change":"2024-09-13T22:40:46.579205+0000","last_active":"2024-09-13T22:43:00.152014+0000","last_peered":"2024-09-13T22:43:00.152014+0000","last_clean":"2024-09-13T22:43:00.152014+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:43:00.152014+0000","last_undegraded":"2024-09-13T22:43:00.152014+0000","last_fullsized":"2024-09-13T22:43:00.152014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:46.579126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:46.579126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144350+0000","last_change":"2024-09-13T22:42:51.845782+0000","last_active":"2024-09-13T22:43:00.144350+0000","last_peered":"2024-09-13T22:43:00.144350+0000","last_clean":"2024-09-13T22:43:00.144350+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:43:00.144350+0000","last_undegraded":"2024-09-13T22:43:00.144350+0000","last_fullsized":"2024-09-13T22:43:00.144350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:51.845704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:51.845704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:09.906670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151184+0000","last_change":"2024-09-13T22:42:44.246269+0000","last_active":"2024-09-13T22:43:00.151184+0000","last_peered":"2024-09-13T22:43:00.151184+0000","last_clean":"2024-09-13T22:43:00.151184+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:43:00.151184+0000","last_undegraded":"2024-09-13T22:43:00.151184+0000","last_fullsized":"2024-09-13T22:43:00.151184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:44.246165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.246165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:58.274315+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"150'8316","reported_seq":8465,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.505974+0000","last_change":"2024-09-13T22:42:49.489354+0000","last_active":"2024-09-13T22:43:13.505974+0000","last_peered":"2024-09-13T22:43:13.505974+0000","last_clean":"2024-09-13T22:43:13.505974+0000","last_became_active":"2024-09-13T22:40:54.952306+0000","last_became_peered":"2024-09-13T22:40:54.952306+0000","last_unstale":"2024-09-13T22:43:13.505974+0000","last_undegraded":"2024-09-13T22:43:13.505974+0000","last_fullsized":"2024-09-13T22:43:13.505974+0000","mapping_epoch":116,"log_start":"124'3300","ondisk_log_start":"124'3300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7216","last_scrub_stamp":"2024-09-13T22:42:49.489064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.489064+0000","objects_scrubbed":6088,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.508108+0000","scrub_duration":1475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324009984,"num_objects":4944,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8316,"num_write_kb":424320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":270,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.055654+0000","last_change":"2024-09-13T22:41:24.936384+0000","last_active":"2024-09-13T22:43:06.055654+0000","last_peered":"2024-09-13T22:43:06.055654+0000","last_clean":"2024-09-13T22:43:06.055654+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:43:06.055654+0000","last_undegraded":"2024-09-13T22:43:06.055654+0000","last_fullsized":"2024-09-13T22:43:06.055654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:41:24.936274+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:41:24.936274+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.142527+0000","last_change":"2024-09-13T22:41:58.672475+0000","last_active":"2024-09-13T22:43:00.142527+0000","last_peered":"2024-09-13T22:43:00.142527+0000","last_clean":"2024-09-13T22:43:00.142527+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:43:00.142527+0000","last_undegraded":"2024-09-13T22:43:00.142527+0000","last_fullsized":"2024-09-13T22:43:00.142527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:41:58.672143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.672143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:14.462146+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":191,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143921+0000","last_change":"2024-09-13T22:42:34.994631+0000","last_active":"2024-09-13T22:43:00.143921+0000","last_peered":"2024-09-13T22:43:00.143921+0000","last_clean":"2024-09-13T22:43:00.143921+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:43:00.143921+0000","last_undegraded":"2024-09-13T22:43:00.143921+0000","last_fullsized":"2024-09-13T22:43:00.143921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:42:34.994416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.994416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:45.727568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.055360+0000","last_change":"2024-09-13T22:42:53.286723+0000","last_active":"2024-09-13T22:43:06.055360+0000","last_peered":"2024-09-13T22:43:06.055360+0000","last_clean":"2024-09-13T22:43:06.055360+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:43:06.055360+0000","last_undegraded":"2024-09-13T22:43:06.055360+0000","last_fullsized":"2024-09-13T22:43:06.055360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:42:53.286646+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.286646+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:16.002562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.149034+0000","last_change":"2024-09-13T22:42:39.282264+0000","last_active":"2024-09-13T22:43:00.149034+0000","last_peered":"2024-09-13T22:43:00.149034+0000","last_clean":"2024-09-13T22:43:00.149034+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:43:00.149034+0000","last_undegraded":"2024-09-13T22:43:00.149034+0000","last_fullsized":"2024-09-13T22:43:00.149034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:39.282139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.282139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.104358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"150'8552","reported_seq":8704,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.505279+0000","last_change":"2024-09-13T22:42:47.043643+0000","last_active":"2024-09-13T22:43:13.505279+0000","last_peered":"2024-09-13T22:43:13.505279+0000","last_clean":"2024-09-13T22:43:13.505279+0000","last_became_active":"2024-09-13T22:40:54.952085+0000","last_became_peered":"2024-09-13T22:40:54.952085+0000","last_unstale":"2024-09-13T22:43:13.505279+0000","last_undegraded":"2024-09-13T22:43:13.505279+0000","last_fullsized":"2024-09-13T22:43:13.505279+0000","mapping_epoch":116,"log_start":"123'2100","ondisk_log_start":"123'2100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7211","last_scrub_stamp":"2024-09-13T22:42:47.043534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.043534+0000","objects_scrubbed":6429,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:50.140235+0000","scrub_duration":3215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328990720,"num_objects":5020,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8552,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":218,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129761+0000","last_change":"2024-09-13T22:41:52.800424+0000","last_active":"2024-09-13T22:43:00.129761+0000","last_peered":"2024-09-13T22:43:00.129761+0000","last_clean":"2024-09-13T22:43:00.129761+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:43:00.129761+0000","last_undegraded":"2024-09-13T22:43:00.129761+0000","last_fullsized":"2024-09-13T22:43:00.129761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:41:52.800270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.800270+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146278+0000","last_change":"2024-09-13T22:42:31.100216+0000","last_active":"2024-09-13T22:43:00.146278+0000","last_peered":"2024-09-13T22:43:00.146278+0000","last_clean":"2024-09-13T22:43:00.146278+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:43:00.146278+0000","last_undegraded":"2024-09-13T22:43:00.146278+0000","last_fullsized":"2024-09-13T22:43:00.146278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.100088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.100088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:35.433339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.149193+0000","last_change":"2024-09-13T22:41:47.494639+0000","last_active":"2024-09-13T22:43:00.149193+0000","last_peered":"2024-09-13T22:43:00.149193+0000","last_clean":"2024-09-13T22:43:00.149193+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:43:00.149193+0000","last_undegraded":"2024-09-13T22:43:00.149193+0000","last_fullsized":"2024-09-13T22:43:00.149193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.494388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:01.309544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":166,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143250+0000","last_change":"2024-09-13T22:41:27.793557+0000","last_active":"2024-09-13T22:43:00.143250+0000","last_peered":"2024-09-13T22:43:00.143250+0000","last_clean":"2024-09-13T22:43:00.143250+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:43:00.143250+0000","last_undegraded":"2024-09-13T22:43:00.143250+0000","last_fullsized":"2024-09-13T22:43:00.143250+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:27.793491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:27.793491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"150'8110","reported_seq":8261,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.822193+0000","last_change":"2024-09-13T22:42:40.742692+0000","last_active":"2024-09-13T22:43:13.822193+0000","last_peered":"2024-09-13T22:43:13.822193+0000","last_clean":"2024-09-13T22:43:13.822193+0000","last_became_active":"2024-09-13T22:40:54.959564+0000","last_became_peered":"2024-09-13T22:40:54.959564+0000","last_unstale":"2024-09-13T22:43:13.822193+0000","last_undegraded":"2024-09-13T22:43:13.822193+0000","last_fullsized":"2024-09-13T22:43:13.822193+0000","mapping_epoch":116,"log_start":"124'2900","ondisk_log_start":"124'2900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6596","last_scrub_stamp":"2024-09-13T22:42:40.742539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.742539+0000","objects_scrubbed":6364,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:47.311474+0000","scrub_duration":2487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314703872,"num_objects":4802,"num_object_clones":0,"num_object_copies":14406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8110,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":222,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.149834+0000","last_change":"2024-09-13T22:42:32.975550+0000","last_active":"2024-09-13T22:43:00.149834+0000","last_peered":"2024-09-13T22:43:00.149834+0000","last_clean":"2024-09-13T22:43:00.149834+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:43:00.149834+0000","last_undegraded":"2024-09-13T22:43:00.149834+0000","last_fullsized":"2024-09-13T22:43:00.149834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-09-13T22:42:32.975430+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:42:32.975430+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.970807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143534+0000","last_change":"2024-09-13T22:42:30.787370+0000","last_active":"2024-09-13T22:43:00.143534+0000","last_peered":"2024-09-13T22:43:00.143534+0000","last_clean":"2024-09-13T22:43:00.143534+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:43:00.143534+0000","last_undegraded":"2024-09-13T22:43:00.143534+0000","last_fullsized":"2024-09-13T22:43:00.143534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:30.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:30.787265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:55.200968+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145356+0000","last_change":"2024-09-13T22:41:39.746764+0000","last_active":"2024-09-13T22:43:00.145356+0000","last_peered":"2024-09-13T22:43:00.145356+0000","last_clean":"2024-09-13T22:43:00.145356+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:43:00.145356+0000","last_undegraded":"2024-09-13T22:43:00.145356+0000","last_fullsized":"2024-09-13T22:43:00.145356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:39.746649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:41:39.746649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151772+0000","last_change":"2024-09-13T22:42:45.743372+0000","last_active":"2024-09-13T22:43:00.151772+0000","last_peered":"2024-09-13T22:43:00.151772+0000","last_clean":"2024-09-13T22:43:00.151772+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:43:00.151772+0000","last_undegraded":"2024-09-13T22:43:00.151772+0000","last_fullsized":"2024-09-13T22:43:00.151772+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.743269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.743269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.530343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"150'8328","reported_seq":8406,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.472419+0000","last_change":"2024-09-13T22:40:54.961075+0000","last_active":"2024-09-13T22:43:13.472419+0000","last_peered":"2024-09-13T22:43:13.472419+0000","last_clean":"2024-09-13T22:43:13.472419+0000","last_became_active":"2024-09-13T22:40:54.960739+0000","last_became_peered":"2024-09-13T22:40:54.960739+0000","last_unstale":"2024-09-13T22:43:13.472419+0000","last_undegraded":"2024-09-13T22:43:13.472419+0000","last_fullsized":"2024-09-13T22:43:13.472419+0000","mapping_epoch":116,"log_start":"126'3600","ondisk_log_start":"126'3600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:17.934304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323747840,"num_objects":4940,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":424576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":203,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.154719+0000","last_change":"2024-09-13T22:41:41.525997+0000","last_active":"2024-09-13T22:43:00.154719+0000","last_peered":"2024-09-13T22:43:00.154719+0000","last_clean":"2024-09-13T22:43:00.154719+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:43:00.154719+0000","last_undegraded":"2024-09-13T22:43:00.154719+0000","last_fullsized":"2024-09-13T22:43:00.154719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:41:41.525568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.525568+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:01.568124+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143352+0000","last_change":"2024-09-13T22:42:45.684106+0000","last_active":"2024-09-13T22:43:00.143352+0000","last_peered":"2024-09-13T22:43:00.143352+0000","last_clean":"2024-09-13T22:43:00.143352+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:43:00.143352+0000","last_undegraded":"2024-09-13T22:43:00.143352+0000","last_fullsized":"2024-09-13T22:43:00.143352+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.683896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.683896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:57.709636+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146661+0000","last_change":"2024-09-13T22:42:49.271307+0000","last_active":"2024-09-13T22:43:00.146661+0000","last_peered":"2024-09-13T22:43:00.146661+0000","last_clean":"2024-09-13T22:43:00.146661+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:43:00.146661+0000","last_undegraded":"2024-09-13T22:43:00.146661+0000","last_fullsized":"2024-09-13T22:43:00.146661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.271179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:05.043890+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.142370+0000","last_change":"2024-09-13T22:41:50.524433+0000","last_active":"2024-09-13T22:43:00.142370+0000","last_peered":"2024-09-13T22:43:00.142370+0000","last_clean":"2024-09-13T22:43:00.142370+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:43:00.142370+0000","last_undegraded":"2024-09-13T22:43:00.142370+0000","last_fullsized":"2024-09-13T22:43:00.142370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:50.524098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.524098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:09.106937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"150'8161","reported_seq":8310,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.788581+0000","last_change":"2024-09-13T22:42:44.078796+0000","last_active":"2024-09-13T22:43:13.788581+0000","last_peered":"2024-09-13T22:43:13.788581+0000","last_clean":"2024-09-13T22:43:13.788581+0000","last_became_active":"2024-09-13T22:40:54.955979+0000","last_became_peered":"2024-09-13T22:40:54.955979+0000","last_unstale":"2024-09-13T22:43:13.788581+0000","last_undegraded":"2024-09-13T22:43:13.788581+0000","last_fullsized":"2024-09-13T22:43:13.788581+0000","mapping_epoch":116,"log_start":"124'2900","ondisk_log_start":"124'2900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'6814","last_scrub_stamp":"2024-09-13T22:42:44.078550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.078550+0000","objects_scrubbed":6262,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:10.325784+0000","scrub_duration":2783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317652992,"num_objects":4847,"num_object_clones":0,"num_object_copies":14541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8161,"num_write_kb":416256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.157315+0000","last_change":"2024-09-13T22:42:43.552263+0000","last_active":"2024-09-13T22:43:00.157315+0000","last_peered":"2024-09-13T22:43:00.157315+0000","last_clean":"2024-09-13T22:43:00.157315+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:43:00.157315+0000","last_undegraded":"2024-09-13T22:43:00.157315+0000","last_fullsized":"2024-09-13T22:43:00.157315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:42:43.551997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:43.551997+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.760390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146215+0000","last_change":"2024-09-13T22:42:36.032459+0000","last_active":"2024-09-13T22:43:00.146215+0000","last_peered":"2024-09-13T22:43:00.146215+0000","last_clean":"2024-09-13T22:43:00.146215+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:43:00.146215+0000","last_undegraded":"2024-09-13T22:43:00.146215+0000","last_fullsized":"2024-09-13T22:43:00.146215+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:36.032356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.032356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.001671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":439,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151347+0000","last_change":"2024-09-13T22:41:45.619812+0000","last_active":"2024-09-13T22:43:00.151347+0000","last_peered":"2024-09-13T22:43:00.151347+0000","last_clean":"2024-09-13T22:43:00.151347+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:43:00.151347+0000","last_undegraded":"2024-09-13T22:43:00.151347+0000","last_fullsized":"2024-09-13T22:43:00.151347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:41:45.619735+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:41:45.619735+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:17.934339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151600+0000","last_change":"2024-09-13T22:41:55.748722+0000","last_active":"2024-09-13T22:43:00.151600+0000","last_peered":"2024-09-13T22:43:00.151600+0000","last_clean":"2024-09-13T22:43:00.151600+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:43:00.151600+0000","last_undegraded":"2024-09-13T22:43:00.151600+0000","last_fullsized":"2024-09-13T22:43:00.151600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:55.748627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:41:55.748627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:25.670554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146535+0000","last_change":"2024-09-13T22:41:44.960471+0000","last_active":"2024-09-13T22:43:00.146535+0000","last_peered":"2024-09-13T22:43:00.146535+0000","last_clean":"2024-09-13T22:43:00.146535+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:43:00.146535+0000","last_undegraded":"2024-09-13T22:43:00.146535+0000","last_fullsized":"2024-09-13T22:43:00.146535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.960382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.960382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:12.124586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"150'8340","reported_seq":8487,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.804949+0000","last_change":"2024-09-13T22:42:31.664279+0000","last_active":"2024-09-13T22:43:13.804949+0000","last_peered":"2024-09-13T22:43:13.804949+0000","last_clean":"2024-09-13T22:43:13.804949+0000","last_became_active":"2024-09-13T22:40:54.959549+0000","last_became_peered":"2024-09-13T22:40:54.959549+0000","last_unstale":"2024-09-13T22:43:13.804949+0000","last_undegraded":"2024-09-13T22:43:13.804949+0000","last_fullsized":"2024-09-13T22:43:13.804949+0000","mapping_epoch":116,"log_start":"124'3100","ondisk_log_start":"124'3100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'6299","last_scrub_stamp":"2024-09-13T22:42:31.664041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.664041+0000","objects_scrubbed":6215,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:45.679181+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8340,"num_write_kb":423232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":201,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151907+0000","last_change":"2024-09-13T22:41:33.652663+0000","last_active":"2024-09-13T22:43:00.151907+0000","last_peered":"2024-09-13T22:43:00.151907+0000","last_clean":"2024-09-13T22:43:00.151907+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:43:00.151907+0000","last_undegraded":"2024-09-13T22:43:00.151907+0000","last_fullsized":"2024-09-13T22:43:00.151907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:41:33.652584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.652584+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:03.179618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144971+0000","last_change":"2024-09-13T22:42:58.827975+0000","last_active":"2024-09-13T22:43:00.144971+0000","last_peered":"2024-09-13T22:43:00.144971+0000","last_clean":"2024-09-13T22:43:00.144971+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:43:00.144971+0000","last_undegraded":"2024-09-13T22:43:00.144971+0000","last_fullsized":"2024-09-13T22:43:00.144971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.827870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.827870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.240808+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146365+0000","last_change":"2024-09-13T22:42:46.715235+0000","last_active":"2024-09-13T22:43:00.146365+0000","last_peered":"2024-09-13T22:43:00.146365+0000","last_clean":"2024-09-13T22:43:00.146365+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:43:00.146365+0000","last_undegraded":"2024-09-13T22:43:00.146365+0000","last_fullsized":"2024-09-13T22:43:00.146365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:46.715135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.715135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.458531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151190+0000","last_change":"2024-09-13T22:41:40.678879+0000","last_active":"2024-09-13T22:43:00.151190+0000","last_peered":"2024-09-13T22:43:00.151190+0000","last_clean":"2024-09-13T22:43:00.151190+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:43:00.151190+0000","last_undegraded":"2024-09-13T22:43:00.151190+0000","last_fullsized":"2024-09-13T22:43:00.151190+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:40.678635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:40.678635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"150'8193","reported_seq":8336,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:13.552137+0000","last_change":"2024-09-13T22:42:57.541592+0000","last_active":"2024-09-13T22:43:13.552137+0000","last_peered":"2024-09-13T22:43:13.552137+0000","last_clean":"2024-09-13T22:43:13.552137+0000","last_became_active":"2024-09-13T22:40:54.960207+0000","last_became_peered":"2024-09-13T22:40:54.960207+0000","last_unstale":"2024-09-13T22:43:13.552137+0000","last_undegraded":"2024-09-13T22:43:13.552137+0000","last_fullsized":"2024-09-13T22:43:13.552137+0000","mapping_epoch":116,"log_start":"121'1500","ondisk_log_start":"121'1500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7482","last_scrub_stamp":"2024-09-13T22:42:57.541325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.541325+0000","objects_scrubbed":5583,"log_size":6693,"log_dups_size":0,"ondisk_log_size":6693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.613669+0000","scrub_duration":1715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316211200,"num_objects":4825,"num_object_clones":0,"num_object_copies":14475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146386+0000","last_change":"2024-09-13T22:42:38.043048+0000","last_active":"2024-09-13T22:43:00.146386+0000","last_peered":"2024-09-13T22:43:00.146386+0000","last_clean":"2024-09-13T22:43:00.146386+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:43:00.146386+0000","last_undegraded":"2024-09-13T22:43:00.146386+0000","last_fullsized":"2024-09-13T22:43:00.146386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:42:38.042796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:38.042796+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:53.974317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.141651+0000","last_change":"2024-09-13T22:41:31.544822+0000","last_active":"2024-09-13T22:43:00.141651+0000","last_peered":"2024-09-13T22:43:00.141651+0000","last_clean":"2024-09-13T22:43:00.141651+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:43:00.141651+0000","last_undegraded":"2024-09-13T22:43:00.141651+0000","last_fullsized":"2024-09-13T22:43:00.141651+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:31.544520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:31.544520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145986+0000","last_change":"2024-09-13T22:41:41.763050+0000","last_active":"2024-09-13T22:43:00.145986+0000","last_peered":"2024-09-13T22:43:00.145986+0000","last_clean":"2024-09-13T22:43:00.145986+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:43:00.145986+0000","last_undegraded":"2024-09-13T22:43:00.145986+0000","last_fullsized":"2024-09-13T22:43:00.145986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.762942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.762942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.154553+0000","last_change":"2024-09-13T22:42:54.275527+0000","last_active":"2024-09-13T22:43:00.154553+0000","last_peered":"2024-09-13T22:43:00.154553+0000","last_clean":"2024-09-13T22:43:00.154553+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:43:00.154553+0000","last_undegraded":"2024-09-13T22:43:00.154553+0000","last_fullsized":"2024-09-13T22:43:00.154553+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.275361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.824268+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"150'11417","reported_seq":11629,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-13T22:43:13.755610+0000","last_change":"2024-09-13T22:42:54.147403+0000","last_active":"2024-09-13T22:43:13.755610+0000","last_peered":"2024-09-13T22:43:13.755610+0000","last_clean":"2024-09-13T22:43:13.755610+0000","last_became_active":"2024-09-13T22:40:54.949821+0000","last_became_peered":"2024-09-13T22:40:54.949821+0000","last_unstale":"2024-09-13T22:43:13.755610+0000","last_undegraded":"2024-09-13T22:43:13.755610+0000","last_fullsized":"2024-09-13T22:43:13.755610+0000","mapping_epoch":116,"log_start":"139'5701","ondisk_log_start":"139'5701","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":3249,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 19s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332136448,"num_objects":5068,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":428160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":205,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151681+0000","last_change":"2024-09-13T22:41:32.631943+0000","last_active":"2024-09-13T22:43:00.151681+0000","last_peered":"2024-09-13T22:43:00.151681+0000","last_clean":"2024-09-13T22:43:00.151681+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:43:00.151681+0000","last_undegraded":"2024-09-13T22:43:00.151681+0000","last_fullsized":"2024-09-13T22:43:00.151681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:41:32.631815+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:41:32.631815+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:01.739812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145521+0000","last_change":"2024-09-13T22:42:31.787949+0000","last_active":"2024-09-13T22:43:00.145521+0000","last_peered":"2024-09-13T22:43:00.145521+0000","last_clean":"2024-09-13T22:43:00.145521+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:43:00.145521+0000","last_undegraded":"2024-09-13T22:43:00.145521+0000","last_fullsized":"2024-09-13T22:43:00.145521+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.787457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.787457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:51.479082+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":254,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.057730+0000","last_change":"2024-09-13T22:41:50.679387+0000","last_active":"2024-09-13T22:43:06.057730+0000","last_peered":"2024-09-13T22:43:06.057730+0000","last_clean":"2024-09-13T22:43:06.057730+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:43:06.057730+0000","last_undegraded":"2024-09-13T22:43:06.057730+0000","last_fullsized":"2024-09-13T22:43:06.057730+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:50.679300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.679300+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:12.847984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145940+0000","last_change":"2024-09-13T22:42:46.267477+0000","last_active":"2024-09-13T22:43:00.145940+0000","last_peered":"2024-09-13T22:43:00.145940+0000","last_clean":"2024-09-13T22:43:00.145940+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:43:00.145940+0000","last_undegraded":"2024-09-13T22:43:00.145940+0000","last_fullsized":"2024-09-13T22:43:00.145940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:46.267344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.267344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.234794+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"150'8478","reported_seq":8622,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:14.333220+0000","last_change":"2024-09-13T22:42:58.835575+0000","last_active":"2024-09-13T22:43:14.333220+0000","last_peered":"2024-09-13T22:43:14.333220+0000","last_clean":"2024-09-13T22:43:14.333220+0000","last_became_active":"2024-09-13T22:40:54.964195+0000","last_became_peered":"2024-09-13T22:40:54.964195+0000","last_unstale":"2024-09-13T22:43:14.333220+0000","last_undegraded":"2024-09-13T22:43:14.333220+0000","last_fullsized":"2024-09-13T22:43:14.333220+0000","mapping_epoch":116,"log_start":"124'3300","ondisk_log_start":"124'3300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7811","last_scrub_stamp":"2024-09-13T22:42:58.835009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.835009+0000","objects_scrubbed":5759,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:22.402928+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329908224,"num_objects":5034,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":432384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":200,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151789+0000","last_change":"2024-09-13T22:41:53.752721+0000","last_active":"2024-09-13T22:43:00.151789+0000","last_peered":"2024-09-13T22:43:00.151789+0000","last_clean":"2024-09-13T22:43:00.151789+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:43:00.151789+0000","last_undegraded":"2024-09-13T22:43:00.151789+0000","last_fullsized":"2024-09-13T22:43:00.151789+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:53.752600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:53.752600+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:15.389215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.130185+0000","last_change":"2024-09-13T22:42:45.983095+0000","last_active":"2024-09-13T22:43:00.130185+0000","last_peered":"2024-09-13T22:43:00.130185+0000","last_clean":"2024-09-13T22:43:00.130185+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:43:00.130185+0000","last_undegraded":"2024-09-13T22:43:00.130185+0000","last_fullsized":"2024-09-13T22:43:00.130185+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:42:45.982969+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.982969+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:51.531936+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143472+0000","last_change":"2024-09-13T22:42:57.858562+0000","last_active":"2024-09-13T22:43:00.143472+0000","last_peered":"2024-09-13T22:43:00.143472+0000","last_clean":"2024-09-13T22:43:00.143472+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:43:00.143472+0000","last_undegraded":"2024-09-13T22:43:00.143472+0000","last_fullsized":"2024-09-13T22:43:00.143472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:57.858139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.858139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:27.371302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146477+0000","last_change":"2024-09-13T22:42:31.280678+0000","last_active":"2024-09-13T22:43:00.146477+0000","last_peered":"2024-09-13T22:43:00.146477+0000","last_clean":"2024-09-13T22:43:00.146477+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:43:00.146477+0000","last_undegraded":"2024-09-13T22:43:00.146477+0000","last_fullsized":"2024-09-13T22:43:00.146477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.280433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.280433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:41.469625+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"112'16","reported_seq":205,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151345+0000","last_change":"2024-09-13T22:42:36.814689+0000","last_active":"2024-09-13T22:43:00.151345+0000","last_peered":"2024-09-13T22:43:00.151345+0000","last_clean":"2024-09-13T22:43:00.151345+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:43:00.151345+0000","last_undegraded":"2024-09-13T22:43:00.151345+0000","last_fullsized":"2024-09-13T22:43:00.151345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'16","last_scrub_stamp":"2024-09-13T22:42:36.814607+0000","last_deep_scrub":"112'16","last_deep_scrub_stamp":"2024-09-13T22:40:26.503359+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.814607+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.577055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.143548+0000","last_change":"2024-09-13T22:41:43.728379+0000","last_active":"2024-09-13T22:43:00.143548+0000","last_peered":"2024-09-13T22:43:00.143548+0000","last_clean":"2024-09-13T22:43:00.143548+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:43:00.143548+0000","last_undegraded":"2024-09-13T22:43:00.143548+0000","last_fullsized":"2024-09-13T22:43:00.143548+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:41:43.728107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.136581+0000","last_change":"2024-09-13T22:42:29.335168+0000","last_active":"2024-09-13T22:43:00.136581+0000","last_peered":"2024-09-13T22:43:00.136581+0000","last_clean":"2024-09-13T22:43:00.136581+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:43:00.136581+0000","last_undegraded":"2024-09-13T22:43:00.136581+0000","last_fullsized":"2024-09-13T22:43:00.136581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:29.335020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:29.335020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:39.684520+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144616+0000","last_change":"2024-09-13T22:42:59.874545+0000","last_active":"2024-09-13T22:43:00.144616+0000","last_peered":"2024-09-13T22:43:00.144616+0000","last_clean":"2024-09-13T22:43:00.144616+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:43:00.144616+0000","last_undegraded":"2024-09-13T22:43:00.144616+0000","last_fullsized":"2024-09-13T22:43:00.144616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.874442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.874442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:20.548793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144234+0000","last_change":"2024-09-13T22:42:53.857519+0000","last_active":"2024-09-13T22:43:00.144234+0000","last_peered":"2024-09-13T22:43:00.144234+0000","last_clean":"2024-09-13T22:43:00.144234+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:43:00.144234+0000","last_undegraded":"2024-09-13T22:43:00.144234+0000","last_fullsized":"2024-09-13T22:43:00.144234+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:42:53.857434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.857434+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.762599+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144480+0000","last_change":"2024-09-13T22:42:39.863272+0000","last_active":"2024-09-13T22:43:00.144480+0000","last_peered":"2024-09-13T22:43:00.144480+0000","last_clean":"2024-09-13T22:43:00.144480+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:43:00.144480+0000","last_undegraded":"2024-09-13T22:43:00.144480+0000","last_fullsized":"2024-09-13T22:43:00.144480+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:39.863156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.863156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.360048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":177,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146504+0000","last_change":"2024-09-13T22:42:41.074516+0000","last_active":"2024-09-13T22:43:00.146504+0000","last_peered":"2024-09-13T22:43:00.146504+0000","last_clean":"2024-09-13T22:43:00.146504+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:43:00.146504+0000","last_undegraded":"2024-09-13T22:43:00.146504+0000","last_fullsized":"2024-09-13T22:43:00.146504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:41.074413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:41.074413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:06.392680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151661+0000","last_change":"2024-09-13T22:40:38.582019+0000","last_active":"2024-09-13T22:43:00.151661+0000","last_peered":"2024-09-13T22:43:00.151661+0000","last_clean":"2024-09-13T22:43:00.151661+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:43:00.151661+0000","last_undegraded":"2024-09-13T22:43:00.151661+0000","last_fullsized":"2024-09-13T22:43:00.151661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:38.581799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:38.581799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":200,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151849+0000","last_change":"2024-09-13T22:42:34.859780+0000","last_active":"2024-09-13T22:43:00.151849+0000","last_peered":"2024-09-13T22:43:00.151849+0000","last_clean":"2024-09-13T22:43:00.151849+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:43:00.151849+0000","last_undegraded":"2024-09-13T22:43:00.151849+0000","last_fullsized":"2024-09-13T22:43:00.151849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:34.859649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.859649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.462988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146190+0000","last_change":"2024-09-13T22:41:52.950538+0000","last_active":"2024-09-13T22:43:00.146190+0000","last_peered":"2024-09-13T22:43:00.146190+0000","last_clean":"2024-09-13T22:43:00.146190+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:43:00.146190+0000","last_undegraded":"2024-09-13T22:43:00.146190+0000","last_fullsized":"2024-09-13T22:43:00.146190+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.950367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:06.744404+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":257,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.055461+0000","last_change":"2024-09-13T22:41:37.950160+0000","last_active":"2024-09-13T22:43:06.055461+0000","last_peered":"2024-09-13T22:43:06.055461+0000","last_clean":"2024-09-13T22:43:06.055461+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:43:06.055461+0000","last_undegraded":"2024-09-13T22:43:06.055461+0000","last_fullsized":"2024-09-13T22:43:06.055461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:37.950036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:37.950036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.152729+0000","last_change":"2024-09-13T22:42:35.206528+0000","last_active":"2024-09-13T22:43:00.152729+0000","last_peered":"2024-09-13T22:43:00.152729+0000","last_clean":"2024-09-13T22:43:00.152729+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:43:00.152729+0000","last_undegraded":"2024-09-13T22:43:00.152729+0000","last_fullsized":"2024-09-13T22:43:00.152729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:35.206294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:35.206294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:51.189905+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":198,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.154835+0000","last_change":"2024-09-13T22:41:50.549051+0000","last_active":"2024-09-13T22:43:00.154835+0000","last_peered":"2024-09-13T22:43:00.154835+0000","last_clean":"2024-09-13T22:43:00.154835+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:43:00.154835+0000","last_undegraded":"2024-09-13T22:43:00.154835+0000","last_fullsized":"2024-09-13T22:43:00.154835+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:41:50.548929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.548929+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:09.565840+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.152347+0000","last_change":"2024-09-13T22:42:56.298096+0000","last_active":"2024-09-13T22:43:00.152347+0000","last_peered":"2024-09-13T22:43:00.152347+0000","last_clean":"2024-09-13T22:43:00.152347+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:43:00.152347+0000","last_undegraded":"2024-09-13T22:43:00.152347+0000","last_fullsized":"2024-09-13T22:43:00.152347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:56.297990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:56.297990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.122881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":255,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.058128+0000","last_change":"2024-09-13T22:41:36.926417+0000","last_active":"2024-09-13T22:43:06.058128+0000","last_peered":"2024-09-13T22:43:06.058128+0000","last_clean":"2024-09-13T22:43:06.058128+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:43:06.058128+0000","last_undegraded":"2024-09-13T22:43:06.058128+0000","last_fullsized":"2024-09-13T22:43:06.058128+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:36.926235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:36.926235+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129979+0000","last_change":"2024-09-13T22:41:58.751646+0000","last_active":"2024-09-13T22:43:00.129979+0000","last_peered":"2024-09-13T22:43:00.129979+0000","last_clean":"2024-09-13T22:43:00.129979+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:43:00.129979+0000","last_undegraded":"2024-09-13T22:43:00.129979+0000","last_fullsized":"2024-09-13T22:43:00.129979+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:58.751579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.751579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:22.483186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151909+0000","last_change":"2024-09-13T22:42:52.779439+0000","last_active":"2024-09-13T22:43:00.151909+0000","last_peered":"2024-09-13T22:43:00.151909+0000","last_clean":"2024-09-13T22:43:00.151909+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:43:00.151909+0000","last_undegraded":"2024-09-13T22:43:00.151909+0000","last_fullsized":"2024-09-13T22:43:00.151909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.779356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.779356+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.029073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144858+0000","last_change":"2024-09-13T22:41:46.916586+0000","last_active":"2024-09-13T22:43:00.144858+0000","last_peered":"2024-09-13T22:43:00.144858+0000","last_clean":"2024-09-13T22:43:00.144858+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:43:00.144858+0000","last_undegraded":"2024-09-13T22:43:00.144858+0000","last_fullsized":"2024-09-13T22:43:00.144858+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:46.916513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:46.916513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:06.534270+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146571+0000","last_change":"2024-09-13T22:42:50.643442+0000","last_active":"2024-09-13T22:43:00.146571+0000","last_peered":"2024-09-13T22:43:00.146571+0000","last_clean":"2024-09-13T22:43:00.146571+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:43:00.146571+0000","last_undegraded":"2024-09-13T22:43:00.146571+0000","last_fullsized":"2024-09-13T22:43:00.146571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:50.643281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.643281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.030417+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.148573+0000","last_change":"2024-09-13T22:42:58.240077+0000","last_active":"2024-09-13T22:43:00.148573+0000","last_peered":"2024-09-13T22:43:00.148573+0000","last_clean":"2024-09-13T22:43:00.148573+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:43:00.148573+0000","last_undegraded":"2024-09-13T22:43:00.148573+0000","last_fullsized":"2024-09-13T22:43:00.148573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.239969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.239969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:18.532690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.130061+0000","last_change":"2024-09-13T22:41:26.621774+0000","last_active":"2024-09-13T22:43:00.130061+0000","last_peered":"2024-09-13T22:43:00.130061+0000","last_clean":"2024-09-13T22:43:00.130061+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:43:00.130061+0000","last_undegraded":"2024-09-13T22:43:00.130061+0000","last_fullsized":"2024-09-13T22:43:00.130061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:41:26.621656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:26.621656+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.141564+0000","last_change":"2024-09-13T22:41:44.491145+0000","last_active":"2024-09-13T22:43:00.141564+0000","last_peered":"2024-09-13T22:43:00.141564+0000","last_clean":"2024-09-13T22:43:00.141564+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:43:00.141564+0000","last_undegraded":"2024-09-13T22:43:00.141564+0000","last_fullsized":"2024-09-13T22:43:00.141564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.490926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.490926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:07.753604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144113+0000","last_change":"2024-09-13T22:41:56.720968+0000","last_active":"2024-09-13T22:43:00.144113+0000","last_peered":"2024-09-13T22:43:00.144113+0000","last_clean":"2024-09-13T22:43:00.144113+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:43:00.144113+0000","last_undegraded":"2024-09-13T22:43:00.144113+0000","last_fullsized":"2024-09-13T22:43:00.144113+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:56.720495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:56.720495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:07.193014+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.136621+0000","last_change":"2024-09-13T22:42:34.299357+0000","last_active":"2024-09-13T22:43:00.136621+0000","last_peered":"2024-09-13T22:43:00.136621+0000","last_clean":"2024-09-13T22:43:00.136621+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:43:00.136621+0000","last_undegraded":"2024-09-13T22:43:00.136621+0000","last_fullsized":"2024-09-13T22:43:00.136621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:34.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:47.934174+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":191,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.153544+0000","last_change":"2024-09-13T22:41:44.560484+0000","last_active":"2024-09-13T22:43:00.153544+0000","last_peered":"2024-09-13T22:43:00.153544+0000","last_clean":"2024-09-13T22:43:00.153544+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:43:00.153544+0000","last_undegraded":"2024-09-13T22:43:00.153544+0000","last_fullsized":"2024-09-13T22:43:00.153544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:41:44.560372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.560372+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146629+0000","last_change":"2024-09-13T22:42:40.026232+0000","last_active":"2024-09-13T22:43:00.146629+0000","last_peered":"2024-09-13T22:43:00.146629+0000","last_clean":"2024-09-13T22:43:00.146629+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:43:00.146629+0000","last_undegraded":"2024-09-13T22:43:00.146629+0000","last_fullsized":"2024-09-13T22:43:00.146629+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:40.026160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.026160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:03.829266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144276+0000","last_change":"2024-09-13T22:41:58.799450+0000","last_active":"2024-09-13T22:43:00.144276+0000","last_peered":"2024-09-13T22:43:00.144276+0000","last_clean":"2024-09-13T22:43:00.144276+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:43:00.144276+0000","last_undegraded":"2024-09-13T22:43:00.144276+0000","last_fullsized":"2024-09-13T22:43:00.144276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:58.799387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.799387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:21.551080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151190+0000","last_change":"2024-09-13T22:41:51.724840+0000","last_active":"2024-09-13T22:43:00.151190+0000","last_peered":"2024-09-13T22:43:00.151190+0000","last_clean":"2024-09-13T22:43:00.151190+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:43:00.151190+0000","last_undegraded":"2024-09-13T22:43:00.151190+0000","last_fullsized":"2024-09-13T22:43:00.151190+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:51.724731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:51.724731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:12.219336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146514+0000","last_change":"2024-09-13T22:42:44.126239+0000","last_active":"2024-09-13T22:43:00.146514+0000","last_peered":"2024-09-13T22:43:00.146514+0000","last_clean":"2024-09-13T22:43:00.146514+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:43:00.146514+0000","last_undegraded":"2024-09-13T22:43:00.146514+0000","last_fullsized":"2024-09-13T22:43:00.146514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:44.126118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.126118+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.929366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146697+0000","last_change":"2024-09-13T22:42:55.792471+0000","last_active":"2024-09-13T22:43:00.146697+0000","last_peered":"2024-09-13T22:43:00.146697+0000","last_clean":"2024-09-13T22:43:00.146697+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:43:00.146697+0000","last_undegraded":"2024-09-13T22:43:00.146697+0000","last_fullsized":"2024-09-13T22:43:00.146697+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:55.792364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:42:55.792364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.743688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151006+0000","last_change":"2024-09-13T22:40:45.622305+0000","last_active":"2024-09-13T22:43:00.151006+0000","last_peered":"2024-09-13T22:43:00.151006+0000","last_clean":"2024-09-13T22:43:00.151006+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:43:00.151006+0000","last_undegraded":"2024-09-13T22:43:00.151006+0000","last_fullsized":"2024-09-13T22:43:00.151006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:45.622167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:45.622167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146644+0000","last_change":"2024-09-13T22:41:47.765249+0000","last_active":"2024-09-13T22:43:00.146644+0000","last_peered":"2024-09-13T22:43:00.146644+0000","last_clean":"2024-09-13T22:43:00.146644+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:43:00.146644+0000","last_undegraded":"2024-09-13T22:43:00.146644+0000","last_fullsized":"2024-09-13T22:43:00.146644+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.765137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":175,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151483+0000","last_change":"2024-09-13T22:42:54.751350+0000","last_active":"2024-09-13T22:43:00.151483+0000","last_peered":"2024-09-13T22:43:00.151483+0000","last_clean":"2024-09-13T22:43:00.151483+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:43:00.151483+0000","last_undegraded":"2024-09-13T22:43:00.151483+0000","last_fullsized":"2024-09-13T22:43:00.151483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:42:54.751219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.751219+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.922863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"111'22","reported_seq":214,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.151360+0000","last_change":"2024-09-13T22:41:24.620491+0000","last_active":"2024-09-13T22:43:00.151360+0000","last_peered":"2024-09-13T22:43:00.151360+0000","last_clean":"2024-09-13T22:43:00.151360+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:43:00.151360+0000","last_undegraded":"2024-09-13T22:43:00.151360+0000","last_fullsized":"2024-09-13T22:43:00.151360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-13T22:41:24.620413+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:41:24.620413+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.145864+0000","last_change":"2024-09-13T22:42:57.304510+0000","last_active":"2024-09-13T22:43:00.145864+0000","last_peered":"2024-09-13T22:43:00.145864+0000","last_clean":"2024-09-13T22:43:00.145864+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:43:00.145864+0000","last_undegraded":"2024-09-13T22:43:00.145864+0000","last_fullsized":"2024-09-13T22:43:00.145864+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:57.304411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.304411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:15.973967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:06.055204+0000","last_change":"2024-09-13T22:42:33.826398+0000","last_active":"2024-09-13T22:43:06.055204+0000","last_peered":"2024-09-13T22:43:06.055204+0000","last_clean":"2024-09-13T22:43:06.055204+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:43:06.055204+0000","last_undegraded":"2024-09-13T22:43:06.055204+0000","last_fullsized":"2024-09-13T22:43:06.055204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:33.826326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:33.826326+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:46.172148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.140610+0000","last_change":"2024-09-13T22:41:41.532153+0000","last_active":"2024-09-13T22:43:00.140610+0000","last_peered":"2024-09-13T22:43:00.140610+0000","last_clean":"2024-09-13T22:43:00.140610+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:43:00.140610+0000","last_undegraded":"2024-09-13T22:43:00.140610+0000","last_fullsized":"2024-09-13T22:43:00.140610+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.531837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.531837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":198,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.172016+0000","last_change":"2024-09-13T22:42:37.463998+0000","last_active":"2024-09-13T22:43:00.172016+0000","last_peered":"2024-09-13T22:43:00.172016+0000","last_clean":"2024-09-13T22:43:00.172016+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:43:00.172016+0000","last_undegraded":"2024-09-13T22:43:00.172016+0000","last_fullsized":"2024-09-13T22:43:00.172016+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:37.463889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.463889+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:46.425283+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":187,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.136900+0000","last_change":"2024-09-13T22:42:45.551596+0000","last_active":"2024-09-13T22:43:00.136900+0000","last_peered":"2024-09-13T22:43:00.136900+0000","last_clean":"2024-09-13T22:43:00.136900+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:43:00.136900+0000","last_undegraded":"2024-09-13T22:43:00.136900+0000","last_fullsized":"2024-09-13T22:43:00.136900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:45.551192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.551192+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:07.564548+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.152901+0000","last_change":"2024-09-13T22:42:59.279764+0000","last_active":"2024-09-13T22:43:00.152901+0000","last_peered":"2024-09-13T22:43:00.152901+0000","last_clean":"2024-09-13T22:43:00.152901+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:43:00.152901+0000","last_undegraded":"2024-09-13T22:43:00.152901+0000","last_fullsized":"2024-09-13T22:43:00.152901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.279506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.279506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.337637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129970+0000","last_change":"2024-09-13T22:41:44.682932+0000","last_active":"2024-09-13T22:43:00.129970+0000","last_peered":"2024-09-13T22:43:00.129970+0000","last_clean":"2024-09-13T22:43:00.129970+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:43:00.129970+0000","last_undegraded":"2024-09-13T22:43:00.129970+0000","last_fullsized":"2024-09-13T22:43:00.129970+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.682861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.682861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:11.993566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.144499+0000","last_change":"2024-09-13T22:42:47.806567+0000","last_active":"2024-09-13T22:43:00.144499+0000","last_peered":"2024-09-13T22:43:00.144499+0000","last_clean":"2024-09-13T22:43:00.144499+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:43:00.144499+0000","last_undegraded":"2024-09-13T22:43:00.144499+0000","last_fullsized":"2024-09-13T22:43:00.144499+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:47.806485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.806485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.641266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":202,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.149091+0000","last_change":"2024-09-13T22:42:52.339140+0000","last_active":"2024-09-13T22:43:00.149091+0000","last_peered":"2024-09-13T22:43:00.149091+0000","last_clean":"2024-09-13T22:43:00.149091+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:43:00.149091+0000","last_undegraded":"2024-09-13T22:43:00.149091+0000","last_fullsized":"2024-09-13T22:43:00.149091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:42:52.339029+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.339029+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:57.801246+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.141415+0000","last_change":"2024-09-13T22:42:53.183255+0000","last_active":"2024-09-13T22:43:00.141415+0000","last_peered":"2024-09-13T22:43:00.141415+0000","last_clean":"2024-09-13T22:43:00.141415+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:43:00.141415+0000","last_undegraded":"2024-09-13T22:43:00.141415+0000","last_fullsized":"2024-09-13T22:43:00.141415+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:53.183134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.183134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.729375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":175,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.136433+0000","last_change":"2024-09-13T22:41:54.907202+0000","last_active":"2024-09-13T22:43:00.136433+0000","last_peered":"2024-09-13T22:43:00.136433+0000","last_clean":"2024-09-13T22:43:00.136433+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:43:00.136433+0000","last_undegraded":"2024-09-13T22:43:00.136433+0000","last_fullsized":"2024-09-13T22:43:00.136433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:41:54.906878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.906878+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:00.212785+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.146973+0000","last_change":"2024-09-13T22:42:48.550930+0000","last_active":"2024-09-13T22:43:00.146973+0000","last_peered":"2024-09-13T22:43:00.146973+0000","last_clean":"2024-09-13T22:43:00.146973+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:43:00.146973+0000","last_undegraded":"2024-09-13T22:43:00.146973+0000","last_fullsized":"2024-09-13T22:43:00.146973+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:48.550845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:48.550845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.389475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":198,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.130276+0000","last_change":"2024-09-13T22:42:52.059935+0000","last_active":"2024-09-13T22:43:00.130276+0000","last_peered":"2024-09-13T22:43:00.130276+0000","last_clean":"2024-09-13T22:43:00.130276+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:43:00.130276+0000","last_undegraded":"2024-09-13T22:43:00.130276+0000","last_fullsized":"2024-09-13T22:43:00.130276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.059600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.059600+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:01.524878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":178,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T22:43:00.129795+0000","last_change":"2024-09-13T22:41:39.615720+0000","last_active":"2024-09-13T22:43:00.129795+0000","last_peered":"2024-09-13T22:43:00.129795+0000","last_clean":"2024-09-13T22:43:00.129795+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:43:00.129795+0000","last_undegraded":"2024-09-13T22:43:00.129795+0000","last_fullsized":"2024-09-13T22:43:00.129795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:39.615598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:41:39.615598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5153816600,"num_objects":78642,"num_object_clones":0,"num_object_copies":235926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132860,"num_write_kb":6768001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15463886848,"data_stored":15463874632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90056,"ondisk_log_size":90056,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808907,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629020,"kb_used_data":1565900,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92116132,"statfs":{"total":95995035648,"available":94326919168,"internally_reserved":0,"allocated":1603481600,"data_stored":1603101249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135958,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1687748,"kb_used_data":1607988,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":92057404,"statfs":{"total":95995035648,"available":94266781696,"internally_reserved":0,"allocated":1646579712,"data_stored":1646221639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528417,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1357952,"kb_used_data":1286512,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92387200,"statfs":{"total":95995035648,"available":94604492800,"internally_reserved":0,"allocated":1317388288,"data_stored":1317034145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888171,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636512,"kb_used_data":1573392,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92108640,"statfs":{"total":95995035648,"available":94319247360,"internally_reserved":0,"allocated":1611153408,"data_stored":1610767027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084405,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2946816,"kb_used_data":2843248,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90798336,"statfs":{"total":95995035648,"available":92977496064,"internally_reserved":0,"allocated":2911485952,"data_stored":2911132230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378753,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1983744,"kb_used_data":1908144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91761408,"statfs":{"total":95995035648,"available":93963681792,"internally_reserved":0,"allocated":1953939456,"data_stored":1953585827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1997520,"kb_used_data":1903808,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91747632,"statfs":{"total":95995035648,"available":93949575168,"internally_reserved":0,"allocated":1949499392,"data_stored":1949129798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738518,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2623192,"kb_used_data":2515656,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91121960,"statfs":{"total":95995035648,"available":93308887040,"internally_reserved":0,"allocated":2576031744,"data_stored":2575655262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2563510272,"data_stored":2563506200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1938690048,"data_stored":1938685976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1935147008,"data_stored":1935147008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2885156864,"data_stored":2885156864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1600847872,"data_stored":1600847872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1311899648,"data_stored":1311899648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1634930688,"data_stored":1634926616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1593704448,"data_stored":1593704448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:43:20.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:43:20.039 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:43:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:43:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:43:20.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:20.116 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:20.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:20.244 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:20.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:20.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:20.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:20.500 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:20.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:20.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:20.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:20.771 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:20.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:20.899 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:20.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:21.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:21.029 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:21.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:21.162 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:21.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:21.292 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:21.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:21.425 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:21.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:21.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:21.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:21.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:21 smithi192 bash[24608]: cluster 2024-09-13T22:43:20.526021+0000 mgr.y (mgr.14613) 212 : cluster [DBG] pgmap v203: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 670 op/s 2024-09-13T22:43:21.695 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:21.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:21.812 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:21 smithi178 bash[29889]: cluster 2024-09-13T22:43:20.526021+0000 mgr.y (mgr.14613) 212 : cluster [DBG] pgmap v203: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 670 op/s 2024-09-13T22:43:21.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:21 smithi178 bash[22840]: cluster 2024-09-13T22:43:20.526021+0000 mgr.y (mgr.14613) 212 : cluster [DBG] pgmap v203: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 670 op/s 2024-09-13T22:43:21.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:21.932 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:21.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:21.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:22.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:22.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:22.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:22.217 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:22.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:22.352 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:22.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:22.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:22.608 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:22.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:22.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:22.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:43:22.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:43:22.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:22.903 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:22.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:23.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:23.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:23.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:23.178 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:23.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:23.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:23.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:23.441 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:23.546 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:23.581 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:23.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:23.708 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:23.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:23.819 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:23 smithi192 bash[24608]: cluster 2024-09-13T22:43:21.712869+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:23.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:23 smithi192 bash[24608]: cluster 2024-09-13T22:43:21.712930+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:23.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:23 smithi192 bash[24608]: cluster 2024-09-13T22:43:22.527702+0000 mgr.y (mgr.14613) 213 : cluster [DBG] pgmap v204: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 670 op/s 2024-09-13T22:43:23.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:23 smithi192 bash[24608]: cluster 2024-09-13T22:43:22.590098+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-13T22:43:23.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:23.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[29889]: cluster 2024-09-13T22:43:21.712869+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:23.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[29889]: cluster 2024-09-13T22:43:21.712930+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:23.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[29889]: cluster 2024-09-13T22:43:22.527702+0000 mgr.y (mgr.14613) 213 : cluster [DBG] pgmap v204: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 670 op/s 2024-09-13T22:43:23.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[29889]: cluster 2024-09-13T22:43:22.590098+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-13T22:43:23.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[22840]: cluster 2024-09-13T22:43:21.712869+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:23.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[22840]: cluster 2024-09-13T22:43:21.712930+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:23.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[22840]: cluster 2024-09-13T22:43:22.527702+0000 mgr.y (mgr.14613) 213 : cluster [DBG] pgmap v204: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 670 op/s 2024-09-13T22:43:23.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:23 smithi178 bash[22840]: cluster 2024-09-13T22:43:22.590098+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-13T22:43:23.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:23.966 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:23.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:24.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:24.092 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:24.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:24.225 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:24.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:24.352 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:24.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:24.477 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:24.552 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:24.553 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:24.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:24.608 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:24.710 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:24.739 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:24.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:24.876 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:24.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:24.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:24.999 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:25.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:25.124 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:25.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:25.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:25.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:43:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:43:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:25.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:25.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:25.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:25.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:25.618 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:25.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:25.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:25.781 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:25.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:25.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:25 smithi192 bash[24608]: cluster 2024-09-13T22:43:24.528691+0000 mgr.y (mgr.14613) 214 : cluster [DBG] pgmap v206: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-09-13T22:43:25.906 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:25.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:25.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:25 smithi178 bash[22840]: cluster 2024-09-13T22:43:24.528691+0000 mgr.y (mgr.14613) 214 : cluster [DBG] pgmap v206: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-09-13T22:43:25.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:25 smithi178 bash[29889]: cluster 2024-09-13T22:43:24.528691+0000 mgr.y (mgr.14613) 214 : cluster [DBG] pgmap v206: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-09-13T22:43:26.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:26.036 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:26.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:26.168 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:26.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:26.310 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:26.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:26.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:26.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:26.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:26.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:26.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:26 smithi192 bash[24608]: audit 2024-09-13T22:43:25.057050+0000 mgr.y (mgr.14613) 215 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:26.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:26 smithi192 bash[24608]: cluster 2024-09-13T22:43:25.862866+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:26.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:26 smithi192 bash[24608]: cluster 2024-09-13T22:43:25.862931+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:26.689 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:26.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:26.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:26 smithi178 bash[29889]: audit 2024-09-13T22:43:25.057050+0000 mgr.y (mgr.14613) 215 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:26.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:26 smithi178 bash[29889]: cluster 2024-09-13T22:43:25.862866+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:26.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:26 smithi178 bash[29889]: cluster 2024-09-13T22:43:25.862931+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:26.805 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:26 smithi178 bash[22840]: audit 2024-09-13T22:43:25.057050+0000 mgr.y (mgr.14613) 215 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:26.805 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:26 smithi178 bash[22840]: cluster 2024-09-13T22:43:25.862866+0000 osd.7 (osd.7) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:26.805 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:26 smithi178 bash[22840]: cluster 2024-09-13T22:43:25.862931+0000 osd.7 (osd.7) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:26.822 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:26.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:26.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:26.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:27.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:27.076 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:27.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:27.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:27.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:27.329 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:27.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:27.458 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:27.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:27.584 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:27.610 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:27.610 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:27.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:27 smithi178 bash[29889]: cluster 2024-09-13T22:43:26.529079+0000 mgr.y (mgr.14613) 216 : cluster [DBG] pgmap v207: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 870 op/s 2024-09-13T22:43:27.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:27 smithi178 bash[29889]: cluster 2024-09-13T22:43:26.608217+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-13T22:43:27.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:27 smithi178 bash[22840]: cluster 2024-09-13T22:43:26.529079+0000 mgr.y (mgr.14613) 216 : cluster [DBG] pgmap v207: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 870 op/s 2024-09-13T22:43:27.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:27 smithi178 bash[22840]: cluster 2024-09-13T22:43:26.608217+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-13T22:43:27.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:27.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:27.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:27.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:27 smithi192 bash[24608]: cluster 2024-09-13T22:43:26.529079+0000 mgr.y (mgr.14613) 216 : cluster [DBG] pgmap v207: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 870 op/s 2024-09-13T22:43:27.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:27 smithi192 bash[24608]: cluster 2024-09-13T22:43:26.608217+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-13T22:43:27.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:27.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:27.980 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:27.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:28.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:28.104 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:28.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:28.240 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:28.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:28.370 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:28.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:28.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:28.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:28.633 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:28.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:28.763 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:28.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:28.886 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:28.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:28.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:28 smithi192 bash[24608]: cluster 2024-09-13T22:43:28.529720+0000 mgr.y (mgr.14613) 217 : cluster [DBG] pgmap v209: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 795 op/s 2024-09-13T22:43:28.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:29.084 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:28 smithi178 bash[22840]: cluster 2024-09-13T22:43:28.529720+0000 mgr.y (mgr.14613) 217 : cluster [DBG] pgmap v209: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 795 op/s 2024-09-13T22:43:29.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:28 smithi178 bash[29889]: cluster 2024-09-13T22:43:28.529720+0000 mgr.y (mgr.14613) 217 : cluster [DBG] pgmap v209: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 795 op/s 2024-09-13T22:43:29.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:29.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:29.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:29.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.427 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:29.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:29.657 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:29.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:29.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.927 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:43:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:43:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:43:29.951 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:29.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:30.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:30.087 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:30.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:30.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:30.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:30.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:30.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:30.489 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:30.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:30.614 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:30.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:30.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:30.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:30.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:30.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:30.991 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:30.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:31.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:31.119 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:31.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:31.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:31.369 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:31.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:31.506 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:31.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:31.631 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:31.677 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:31 smithi178 bash[29889]: cluster 2024-09-13T22:43:30.530397+0000 mgr.y (mgr.14613) 218 : cluster [DBG] pgmap v210: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T22:43:31.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:31 smithi178 bash[22840]: cluster 2024-09-13T22:43:30.530397+0000 mgr.y (mgr.14613) 218 : cluster [DBG] pgmap v210: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T22:43:31.735 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:31.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:31.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:31.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:31 smithi192 bash[24608]: cluster 2024-09-13T22:43:30.530397+0000 mgr.y (mgr.14613) 218 : cluster [DBG] pgmap v210: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T22:43:31.894 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:31.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:31.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:32.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:32.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:32.155 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:32.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:32.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:32.314 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:43:32.314 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:43:32.314 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:43:32.314 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:43:32.314 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:43:32.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:32.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:32.461 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:32.554 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:43:32.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:32.620 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:43:32.682 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:32.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:32.808 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:32.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:32.943 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:32.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:33.049 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:33.084 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:33.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:33.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:33.263 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:43:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:43:33.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:33.317 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:43:33.317 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:43:33.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:33.361 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:43:33.361 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:43:33.397 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:43:33.398 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:43:33.437 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:43:33.437 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:43:33.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:33.471 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:33.475 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:43:33.475 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:43:33.510 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:43:33.511 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:43:33.541 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:43:33.541 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:43:33.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:33.598 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:33.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:33.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:33 smithi192 bash[24608]: cluster 2024-09-13T22:43:32.530840+0000 mgr.y (mgr.14613) 219 : cluster [DBG] pgmap v211: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 935 op/s 2024-09-13T22:43:33.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:33 smithi192 bash[24608]: audit 2024-09-13T22:43:32.538483+0000 mgr.y (mgr.14613) 220 : audit [DBG] from='client.14997 v1:172.21.15.178:0/617802620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:43:33.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:33 smithi192 bash[24608]: audit 2024-09-13T22:43:32.961575+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? v1:172.21.15.178:0/3152465484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:43:33.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:33 smithi192 bash[24608]: audit 2024-09-13T22:43:33.005686+0000 mon.b (mon.1) 120 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:33.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:33 smithi192 bash[24608]: audit 2024-09-13T22:43:33.227131+0000 mon.a (mon.0) 1019 : audit [DBG] from='client.? v1:172.21.15.178:0/167758260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:33.725 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:33.809 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:43:33.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:33.853 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:33.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[29889]: cluster 2024-09-13T22:43:32.530840+0000 mgr.y (mgr.14613) 219 : cluster [DBG] pgmap v211: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 935 op/s 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[29889]: audit 2024-09-13T22:43:32.538483+0000 mgr.y (mgr.14613) 220 : audit [DBG] from='client.14997 v1:172.21.15.178:0/617802620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[29889]: audit 2024-09-13T22:43:32.961575+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? v1:172.21.15.178:0/3152465484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[29889]: audit 2024-09-13T22:43:33.005686+0000 mon.b (mon.1) 120 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[29889]: audit 2024-09-13T22:43:33.227131+0000 mon.a (mon.0) 1019 : audit [DBG] from='client.? v1:172.21.15.178:0/167758260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[22840]: cluster 2024-09-13T22:43:32.530840+0000 mgr.y (mgr.14613) 219 : cluster [DBG] pgmap v211: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 935 op/s 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[22840]: audit 2024-09-13T22:43:32.538483+0000 mgr.y (mgr.14613) 220 : audit [DBG] from='client.14997 v1:172.21.15.178:0/617802620' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[22840]: audit 2024-09-13T22:43:32.961575+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? v1:172.21.15.178:0/3152465484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[22840]: audit 2024-09-13T22:43:33.005686+0000 mon.b (mon.1) 120 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:33 smithi178 bash[22840]: audit 2024-09-13T22:43:33.227131+0000 mon.a (mon.0) 1019 : audit [DBG] from='client.? v1:172.21.15.178:0/167758260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:33.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:33.983 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:33.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:34.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:34.114 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:34.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:34.251 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:34.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:34.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:34.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:34.510 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:34.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:34.645 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:34.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:34.759 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:34 smithi192 bash[24608]: cluster 2024-09-13T22:43:33.811055+0000 mon.a (mon.0) 1020 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:34.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:34 smithi192 bash[24608]: audit 2024-09-13T22:43:33.813262+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? v1:172.21.15.178:0/3152465484' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:43:34.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:34 smithi192 bash[24608]: cluster 2024-09-13T22:43:33.819077+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-13T22:43:34.773 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:34.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:34.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:34.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:34.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:34 smithi178 bash[29889]: cluster 2024-09-13T22:43:33.811055+0000 mon.a (mon.0) 1020 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:34.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:34 smithi178 bash[29889]: audit 2024-09-13T22:43:33.813262+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? v1:172.21.15.178:0/3152465484' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:43:34.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:34 smithi178 bash[29889]: cluster 2024-09-13T22:43:33.819077+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-13T22:43:34.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:34 smithi178 bash[22840]: cluster 2024-09-13T22:43:33.811055+0000 mon.a (mon.0) 1020 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:43:34.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:34 smithi178 bash[22840]: audit 2024-09-13T22:43:33.813262+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? v1:172.21.15.178:0/3152465484' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:43:34.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:34 smithi178 bash[22840]: cluster 2024-09-13T22:43:33.819077+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-13T22:43:35.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:35.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:35.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:35.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:35.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:35.281 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:35.282 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:43:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:43:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:35.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:35.407 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:35.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:35.534 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:35.606 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T22:43:35.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:35.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:35.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:35.805 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:35.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:35 smithi192 bash[24608]: cluster 2024-09-13T22:43:34.383371+0000 osd.3 (osd.3) 136 : cluster [DBG] 7.3 scrub starts 2024-09-13T22:43:35.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:35 smithi192 bash[24608]: cluster 2024-09-13T22:43:34.531515+0000 mgr.y (mgr.14613) 221 : cluster [DBG] pgmap v213: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-13T22:43:35.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:35 smithi192 bash[24608]: cluster 2024-09-13T22:43:34.539243+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:43:35.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:35 smithi192 bash[24608]: cluster 2024-09-13T22:43:34.541076+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:43:35.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:35.942 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:35.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:35.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:36.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:36.067 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:36.068 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[22840]: cluster 2024-09-13T22:43:34.383371+0000 osd.3 (osd.3) 136 : cluster [DBG] 7.3 scrub starts 2024-09-13T22:43:36.068 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[22840]: cluster 2024-09-13T22:43:34.531515+0000 mgr.y (mgr.14613) 221 : cluster [DBG] pgmap v213: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-13T22:43:36.068 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[22840]: cluster 2024-09-13T22:43:34.539243+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:43:36.068 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[22840]: cluster 2024-09-13T22:43:34.541076+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:43:36.069 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[29889]: cluster 2024-09-13T22:43:34.383371+0000 osd.3 (osd.3) 136 : cluster [DBG] 7.3 scrub starts 2024-09-13T22:43:36.069 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[29889]: cluster 2024-09-13T22:43:34.531515+0000 mgr.y (mgr.14613) 221 : cluster [DBG] pgmap v213: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-13T22:43:36.069 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[29889]: cluster 2024-09-13T22:43:34.539243+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:43:36.069 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:35 smithi178 bash[29889]: cluster 2024-09-13T22:43:34.541076+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:43:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:36.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:36.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:36.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:36.316 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:36.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:36.441 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:36.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:36.573 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:36.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:36.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:36 smithi192 bash[24608]: audit 2024-09-13T22:43:35.066699+0000 mgr.y (mgr.14613) 222 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:36.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:36 smithi192 bash[24608]: cluster 2024-09-13T22:43:35.741137+0000 osd.3 (osd.3) 137 : cluster [DBG] 7.3 scrub ok 2024-09-13T22:43:36.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:36 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.036790+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:43:36.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:36 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.040684+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:43:36.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:36.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:36.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[29889]: audit 2024-09-13T22:43:35.066699+0000 mgr.y (mgr.14613) 222 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:36.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[29889]: cluster 2024-09-13T22:43:35.741137+0000 osd.3 (osd.3) 137 : cluster [DBG] 7.3 scrub ok 2024-09-13T22:43:36.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.036790+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:43:36.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.040684+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:43:36.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[22840]: audit 2024-09-13T22:43:35.066699+0000 mgr.y (mgr.14613) 222 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:36.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[22840]: cluster 2024-09-13T22:43:35.741137+0000 osd.3 (osd.3) 137 : cluster [DBG] 7.3 scrub ok 2024-09-13T22:43:36.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.036790+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:43:36.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:36 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.040684+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:43:36.826 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-13T22:43:32.530579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4337572403,"num_objects":66377,"num_object_clones":0,"num_object_copies":199131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66377,"num_whiteouts":0,"num_read":1279,"num_read_kb":1125,"num_write":145939,"num_write_kb":6770462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90325,"ondisk_log_size":90325,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13492952,"kb_used_data":12801672,"kb_used_omap":12,"kb_used_meta":690803,"kb_avail":736468264,"statfs":{"total":767960285184,"available":754143502336,"internally_reserved":0,"allocated":13108912128,"data_stored":13105848025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":707382832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,7,11,7,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-608894976,"num_objects":-9291,"num_object_clones":0,"num_object_copies":-27873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9291,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.941071"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":172,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.608173+0000","last_change":"2024-09-13T22:41:44.889037+0000","last_active":"2024-09-13T22:43:26.608173+0000","last_peered":"2024-09-13T22:43:26.608173+0000","last_clean":"2024-09-13T22:43:26.608173+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:43:26.608173+0000","last_undegraded":"2024-09-13T22:43:26.608173+0000","last_fullsized":"2024-09-13T22:43:26.608173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.888900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.888900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.619235+0000","last_change":"2024-09-13T22:41:33.637917+0000","last_active":"2024-09-13T22:43:26.619235+0000","last_peered":"2024-09-13T22:43:26.619235+0000","last_clean":"2024-09-13T22:43:26.619235+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:43:26.619235+0000","last_undegraded":"2024-09-13T22:43:26.619235+0000","last_fullsized":"2024-09-13T22:43:26.619235+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:33.637519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.637519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.521822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.607003+0000","last_change":"2024-09-13T22:41:49.766349+0000","last_active":"2024-09-13T22:43:26.607003+0000","last_peered":"2024-09-13T22:43:26.607003+0000","last_clean":"2024-09-13T22:43:26.607003+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:43:26.607003+0000","last_undegraded":"2024-09-13T22:43:26.607003+0000","last_fullsized":"2024-09-13T22:43:26.607003+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:41:49.765993+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614025+0000","last_change":"2024-09-13T22:41:55.780046+0000","last_active":"2024-09-13T22:43:26.614025+0000","last_peered":"2024-09-13T22:43:26.614025+0000","last_clean":"2024-09-13T22:43:26.614025+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:43:26.614025+0000","last_undegraded":"2024-09-13T22:43:26.614025+0000","last_fullsized":"2024-09-13T22:43:26.614025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:41:55.779975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.607039+0000","last_change":"2024-09-13T22:41:26.073240+0000","last_active":"2024-09-13T22:43:26.607039+0000","last_peered":"2024-09-13T22:43:26.607039+0000","last_clean":"2024-09-13T22:43:26.607039+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:43:26.607039+0000","last_undegraded":"2024-09-13T22:43:26.607039+0000","last_fullsized":"2024-09-13T22:43:26.607039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:26.073058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:26.073058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":212,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618962+0000","last_change":"2024-09-13T22:41:57.526625+0000","last_active":"2024-09-13T22:43:26.618962+0000","last_peered":"2024-09-13T22:43:26.618962+0000","last_clean":"2024-09-13T22:43:26.618962+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:43:26.618962+0000","last_undegraded":"2024-09-13T22:43:26.618962+0000","last_fullsized":"2024-09-13T22:43:26.618962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:41:57.526506+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:41:57.526506+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614352+0000","last_change":"2024-09-13T22:42:59.307163+0000","last_active":"2024-09-13T22:43:26.614352+0000","last_peered":"2024-09-13T22:43:26.614352+0000","last_clean":"2024-09-13T22:43:26.614352+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:43:26.614352+0000","last_undegraded":"2024-09-13T22:43:26.614352+0000","last_fullsized":"2024-09-13T22:43:26.614352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.307067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:18.937510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614334+0000","last_change":"2024-09-13T22:41:53.897309+0000","last_active":"2024-09-13T22:43:26.614334+0000","last_peered":"2024-09-13T22:43:26.614334+0000","last_clean":"2024-09-13T22:43:26.614334+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:43:26.614334+0000","last_undegraded":"2024-09-13T22:43:26.614334+0000","last_fullsized":"2024-09-13T22:43:26.614334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_clean_scrub_stamp":"2024-09-13T22:41:53.894021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027306+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605595+0000","last_change":"2024-09-13T22:42:35.837037+0000","last_active":"2024-09-13T22:43:26.605595+0000","last_peered":"2024-09-13T22:43:26.605595+0000","last_clean":"2024-09-13T22:43:26.605595+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:43:26.605595+0000","last_undegraded":"2024-09-13T22:43:26.605595+0000","last_fullsized":"2024-09-13T22:43:26.605595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:35.836936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:35.836936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:49.472291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":219,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616037+0000","last_change":"2024-09-13T22:42:39.154087+0000","last_active":"2024-09-13T22:43:26.616037+0000","last_peered":"2024-09-13T22:43:26.616037+0000","last_clean":"2024-09-13T22:43:26.616037+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:43:26.616037+0000","last_undegraded":"2024-09-13T22:43:26.616037+0000","last_fullsized":"2024-09-13T22:43:26.616037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:42:39.153983+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.153983+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:43.318456+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":175,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604632+0000","last_change":"2024-09-13T22:41:49.716084+0000","last_active":"2024-09-13T22:43:26.604632+0000","last_peered":"2024-09-13T22:43:26.604632+0000","last_clean":"2024-09-13T22:43:26.604632+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:43:26.604632+0000","last_undegraded":"2024-09-13T22:43:26.604632+0000","last_fullsized":"2024-09-13T22:43:26.604632+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:49.715971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:41:49.715971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615784+0000","last_change":"2024-09-13T22:41:41.033132+0000","last_active":"2024-09-13T22:43:26.615784+0000","last_peered":"2024-09-13T22:43:26.615784+0000","last_clean":"2024-09-13T22:43:26.615784+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:43:26.615784+0000","last_undegraded":"2024-09-13T22:43:26.615784+0000","last_fullsized":"2024-09-13T22:43:26.615784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.032827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.032827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077210+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605173+0000","last_change":"2024-09-13T22:41:57.761398+0000","last_active":"2024-09-13T22:43:26.605173+0000","last_peered":"2024-09-13T22:43:26.605173+0000","last_clean":"2024-09-13T22:43:26.605173+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:43:26.605173+0000","last_undegraded":"2024-09-13T22:43:26.605173+0000","last_fullsized":"2024-09-13T22:43:26.605173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:57.761322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:57.761322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616337+0000","last_change":"2024-09-13T22:42:29.188915+0000","last_active":"2024-09-13T22:43:26.616337+0000","last_peered":"2024-09-13T22:43:26.616337+0000","last_clean":"2024-09-13T22:43:26.616337+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:43:26.616337+0000","last_undegraded":"2024-09-13T22:43:26.616337+0000","last_fullsized":"2024-09-13T22:43:26.616337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:42:29.188836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:29.188836+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:39.007169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615941+0000","last_change":"2024-09-13T22:42:37.031507+0000","last_active":"2024-09-13T22:43:26.615941+0000","last_peered":"2024-09-13T22:43:26.615941+0000","last_clean":"2024-09-13T22:43:26.615941+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:43:26.615941+0000","last_undegraded":"2024-09-13T22:43:26.615941+0000","last_fullsized":"2024-09-13T22:43:26.615941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:37.031329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.031329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.995792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614269+0000","last_change":"2024-09-13T22:41:48.479208+0000","last_active":"2024-09-13T22:43:26.614269+0000","last_peered":"2024-09-13T22:43:26.614269+0000","last_clean":"2024-09-13T22:43:26.614269+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:43:26.614269+0000","last_undegraded":"2024-09-13T22:43:26.614269+0000","last_fullsized":"2024-09-13T22:43:26.614269+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:48.479129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:48.479129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.521931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"153'8973","reported_seq":9122,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.706937+0000","last_change":"2024-09-13T22:42:57.288156+0000","last_active":"2024-09-13T22:43:28.706937+0000","last_peered":"2024-09-13T22:43:28.706937+0000","last_clean":"2024-09-13T22:43:28.706937+0000","last_became_active":"2024-09-13T22:40:54.965149+0000","last_became_peered":"2024-09-13T22:40:54.965149+0000","last_unstale":"2024-09-13T22:43:28.706937+0000","last_undegraded":"2024-09-13T22:43:28.706937+0000","last_fullsized":"2024-09-13T22:43:28.706937+0000","mapping_epoch":116,"log_start":"123'2300","ondisk_log_start":"123'2300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7538","last_scrub_stamp":"2024-09-13T22:42:57.287895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.287895+0000","objects_scrubbed":5635,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.157500+0000","scrub_duration":2527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"114'28","reported_seq":236,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610501+0000","last_change":"2024-09-13T22:42:54.838293+0000","last_active":"2024-09-13T22:43:26.610501+0000","last_peered":"2024-09-13T22:43:26.610501+0000","last_clean":"2024-09-13T22:43:26.610501+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:43:26.610501+0000","last_undegraded":"2024-09-13T22:43:26.610501+0000","last_fullsized":"2024-09-13T22:43:26.610501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'28","last_scrub_stamp":"2024-09-13T22:42:54.838192+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.838192+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:00.553515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605214+0000","last_change":"2024-09-13T22:40:48.576923+0000","last_active":"2024-09-13T22:43:26.605214+0000","last_peered":"2024-09-13T22:43:26.605214+0000","last_clean":"2024-09-13T22:43:26.605214+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:43:26.605214+0000","last_undegraded":"2024-09-13T22:43:26.605214+0000","last_fullsized":"2024-09-13T22:43:26.605214+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:48.576808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:48.576808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.934474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618138+0000","last_change":"2024-09-13T22:42:28.929922+0000","last_active":"2024-09-13T22:43:26.618138+0000","last_peered":"2024-09-13T22:43:26.618138+0000","last_clean":"2024-09-13T22:43:26.618138+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:43:26.618138+0000","last_undegraded":"2024-09-13T22:43:26.618138+0000","last_fullsized":"2024-09-13T22:43:26.618138+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:28.929617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:28.929617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:43.287158+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.617918+0000","last_change":"2024-09-13T22:41:54.670111+0000","last_active":"2024-09-13T22:43:26.617918+0000","last_peered":"2024-09-13T22:43:26.617918+0000","last_clean":"2024-09-13T22:43:26.617918+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:43:26.617918+0000","last_undegraded":"2024-09-13T22:43:26.617918+0000","last_fullsized":"2024-09-13T22:43:26.617918+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:54.669996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.669996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"153'9014","reported_seq":9170,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.817046+0000","last_change":"2024-09-13T22:42:48.154144+0000","last_active":"2024-09-13T22:43:28.817046+0000","last_peered":"2024-09-13T22:43:28.817046+0000","last_clean":"2024-09-13T22:43:28.817046+0000","last_became_active":"2024-09-13T22:40:54.957954+0000","last_became_peered":"2024-09-13T22:40:54.957954+0000","last_unstale":"2024-09-13T22:43:28.817046+0000","last_undegraded":"2024-09-13T22:43:28.817046+0000","last_fullsized":"2024-09-13T22:43:28.817046+0000","mapping_epoch":116,"log_start":"126'3800","ondisk_log_start":"126'3800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7054","last_scrub_stamp":"2024-09-13T22:42:48.154030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:48.154030+0000","objects_scrubbed":6079,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:00.734631+0000","scrub_duration":3823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263979008,"num_objects":4028,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9014,"num_write_kb":417344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"110'13","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.608463+0000","last_change":"2024-09-13T22:41:56.802293+0000","last_active":"2024-09-13T22:43:26.608463+0000","last_peered":"2024-09-13T22:43:26.608463+0000","last_clean":"2024-09-13T22:43:26.608463+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:43:26.608463+0000","last_undegraded":"2024-09-13T22:43:26.608463+0000","last_fullsized":"2024-09-13T22:43:26.608463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-09-13T22:41:56.802209+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:41:56.802209+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.617310+0000","last_change":"2024-09-13T22:42:52.153298+0000","last_active":"2024-09-13T22:43:26.617310+0000","last_peered":"2024-09-13T22:43:26.617310+0000","last_clean":"2024-09-13T22:43:26.617310+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:43:26.617310+0000","last_undegraded":"2024-09-13T22:43:26.617310+0000","last_fullsized":"2024-09-13T22:43:26.617310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:52.153187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.153187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.767977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":264,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615041+0000","last_change":"2024-09-13T22:42:47.690211+0000","last_active":"2024-09-13T22:43:26.615041+0000","last_peered":"2024-09-13T22:43:26.615041+0000","last_clean":"2024-09-13T22:43:26.615041+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:43:26.615041+0000","last_undegraded":"2024-09-13T22:43:26.615041+0000","last_fullsized":"2024-09-13T22:43:26.615041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:47.690105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.690105+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.251912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614591+0000","last_change":"2024-09-13T22:41:29.671454+0000","last_active":"2024-09-13T22:43:26.614591+0000","last_peered":"2024-09-13T22:43:26.614591+0000","last_clean":"2024-09-13T22:43:26.614591+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:43:26.614591+0000","last_undegraded":"2024-09-13T22:43:26.614591+0000","last_fullsized":"2024-09-13T22:43:26.614591+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:29.671369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:29.671369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027262+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"153'9115","reported_seq":9274,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.437320+0000","last_change":"2024-09-13T22:42:37.860665+0000","last_active":"2024-09-13T22:43:28.437320+0000","last_peered":"2024-09-13T22:43:28.437320+0000","last_clean":"2024-09-13T22:43:28.437320+0000","last_became_active":"2024-09-13T22:40:54.952746+0000","last_became_peered":"2024-09-13T22:40:54.952746+0000","last_unstale":"2024-09-13T22:43:28.437320+0000","last_undegraded":"2024-09-13T22:43:28.437320+0000","last_fullsized":"2024-09-13T22:43:28.437320+0000","mapping_epoch":116,"log_start":"123'2700","ondisk_log_start":"123'2700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'6674","last_scrub_stamp":"2024-09-13T22:42:37.860298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.860298+0000","objects_scrubbed":6611,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:07.575624+0000","scrub_duration":2031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273219584,"num_objects":4169,"num_object_clones":0,"num_object_copies":12507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9115,"num_write_kb":425088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":213,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.606208+0000","last_change":"2024-09-13T22:42:28.345281+0000","last_active":"2024-09-13T22:43:26.606208+0000","last_peered":"2024-09-13T22:43:26.606208+0000","last_clean":"2024-09-13T22:43:26.606208+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:43:26.606208+0000","last_undegraded":"2024-09-13T22:43:26.606208+0000","last_fullsized":"2024-09-13T22:43:26.606208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:42:28.345153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:28.345153+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:41.482075+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614558+0000","last_change":"2024-09-13T22:41:52.638569+0000","last_active":"2024-09-13T22:43:26.614558+0000","last_peered":"2024-09-13T22:43:26.614558+0000","last_clean":"2024-09-13T22:43:26.614558+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:43:26.614558+0000","last_undegraded":"2024-09-13T22:43:26.614558+0000","last_fullsized":"2024-09-13T22:43:26.614558+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:52.638224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.638224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":265,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614750+0000","last_change":"2024-09-13T22:42:50.045223+0000","last_active":"2024-09-13T22:43:26.614750+0000","last_peered":"2024-09-13T22:43:26.614750+0000","last_clean":"2024-09-13T22:43:26.614750+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:43:26.614750+0000","last_undegraded":"2024-09-13T22:43:26.614750+0000","last_fullsized":"2024-09-13T22:43:26.614750+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:42:50.045100+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.045100+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.086555+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616111+0000","last_change":"2024-09-13T22:41:32.995433+0000","last_active":"2024-09-13T22:43:26.616111+0000","last_peered":"2024-09-13T22:43:26.616111+0000","last_clean":"2024-09-13T22:43:26.616111+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:43:26.616111+0000","last_undegraded":"2024-09-13T22:43:26.616111+0000","last_fullsized":"2024-09-13T22:43:26.616111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:32.995357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:33.392318+0000","last_clean_scrub_stamp":"2024-09-13T22:41:32.995357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077262+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"153'9187","reported_seq":9345,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.492609+0000","last_change":"2024-09-13T22:42:47.749351+0000","last_active":"2024-09-13T22:43:28.492609+0000","last_peered":"2024-09-13T22:43:28.492609+0000","last_clean":"2024-09-13T22:43:28.492609+0000","last_became_active":"2024-09-13T22:40:54.959205+0000","last_became_peered":"2024-09-13T22:40:54.959205+0000","last_unstale":"2024-09-13T22:43:28.492609+0000","last_undegraded":"2024-09-13T22:43:28.492609+0000","last_fullsized":"2024-09-13T22:43:28.492609+0000","mapping_epoch":116,"log_start":"124'2800","ondisk_log_start":"124'2800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7167","last_scrub_stamp":"2024-09-13T22:42:47.749227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.749227+0000","objects_scrubbed":6263,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:52.740875+0000","scrub_duration":1899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275513368,"num_objects":4205,"num_object_clones":0,"num_object_copies":12615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9187,"num_write_kb":428481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614290+0000","last_change":"2024-09-13T22:41:54.888290+0000","last_active":"2024-09-13T22:43:26.614290+0000","last_peered":"2024-09-13T22:43:26.614290+0000","last_clean":"2024-09-13T22:43:26.614290+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:43:26.614290+0000","last_undegraded":"2024-09-13T22:43:26.614290+0000","last_fullsized":"2024-09-13T22:43:26.614290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:41:54.887968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.887968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027351+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604854+0000","last_change":"2024-09-13T22:41:47.671619+0000","last_active":"2024-09-13T22:43:26.604854+0000","last_peered":"2024-09-13T22:43:26.604854+0000","last_clean":"2024-09-13T22:43:26.604854+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:43:26.604854+0000","last_undegraded":"2024-09-13T22:43:26.604854+0000","last_fullsized":"2024-09-13T22:43:26.604854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.671515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.671515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936196+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616460+0000","last_change":"2024-09-13T22:41:22.943141+0000","last_active":"2024-09-13T22:43:26.616460+0000","last_peered":"2024-09-13T22:43:26.616460+0000","last_clean":"2024-09-13T22:43:26.616460+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:43:26.616460+0000","last_undegraded":"2024-09-13T22:43:26.616460+0000","last_fullsized":"2024-09-13T22:43:26.616460+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:22.943025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:22.943025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":172,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.609186+0000","last_change":"2024-09-13T22:41:35.880618+0000","last_active":"2024-09-13T22:43:26.609186+0000","last_peered":"2024-09-13T22:43:26.609186+0000","last_clean":"2024-09-13T22:43:26.609186+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:43:26.609186+0000","last_undegraded":"2024-09-13T22:43:26.609186+0000","last_fullsized":"2024-09-13T22:43:26.609186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:35.878354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:35.878354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"153'9011","reported_seq":9168,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:29.352108+0000","last_change":"2024-09-13T22:42:42.864731+0000","last_active":"2024-09-13T22:43:29.352108+0000","last_peered":"2024-09-13T22:43:29.352108+0000","last_clean":"2024-09-13T22:43:29.352108+0000","last_became_active":"2024-09-13T22:40:54.961774+0000","last_became_peered":"2024-09-13T22:40:54.961774+0000","last_unstale":"2024-09-13T22:43:29.352108+0000","last_undegraded":"2024-09-13T22:43:29.352108+0000","last_fullsized":"2024-09-13T22:43:29.352108+0000","mapping_epoch":116,"log_start":"127'3900","ondisk_log_start":"127'3900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6785","last_scrub_stamp":"2024-09-13T22:42:42.864469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:42.864469+0000","objects_scrubbed":6365,"log_size":5111,"log_dups_size":0,"ondisk_log_size":5111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.071905+0000","scrub_duration":3367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266534912,"num_objects":4067,"num_object_clones":0,"num_object_copies":12201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9011,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":206,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.624172+0000","last_change":"2024-09-13T22:42:40.311926+0000","last_active":"2024-09-13T22:43:26.624172+0000","last_peered":"2024-09-13T22:43:26.624172+0000","last_clean":"2024-09-13T22:43:26.624172+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:43:26.624172+0000","last_undegraded":"2024-09-13T22:43:26.624172+0000","last_fullsized":"2024-09-13T22:43:26.624172+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:42:40.311810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.311810+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:48.816888+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610977+0000","last_change":"2024-09-13T22:42:49.867175+0000","last_active":"2024-09-13T22:43:26.610977+0000","last_peered":"2024-09-13T22:43:26.610977+0000","last_clean":"2024-09-13T22:43:26.610977+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:43:26.610977+0000","last_undegraded":"2024-09-13T22:43:26.610977+0000","last_fullsized":"2024-09-13T22:43:26.610977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:49.867081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.867081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.457345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614237+0000","last_change":"2024-09-13T22:42:38.889988+0000","last_active":"2024-09-13T22:43:26.614237+0000","last_peered":"2024-09-13T22:43:26.614237+0000","last_clean":"2024-09-13T22:43:26.614237+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:43:26.614237+0000","last_undegraded":"2024-09-13T22:43:26.614237+0000","last_fullsized":"2024-09-13T22:43:26.614237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:42:38.889867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:38.889867+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.145041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":256,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605882+0000","last_change":"2024-09-13T22:41:33.857435+0000","last_active":"2024-09-13T22:43:26.605882+0000","last_peered":"2024-09-13T22:43:26.605882+0000","last_clean":"2024-09-13T22:43:26.605882+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:43:26.605882+0000","last_undegraded":"2024-09-13T22:43:26.605882+0000","last_fullsized":"2024-09-13T22:43:26.605882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:33.857359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.857359+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604903+0000","last_change":"2024-09-13T22:41:35.694198+0000","last_active":"2024-09-13T22:43:26.604903+0000","last_peered":"2024-09-13T22:43:26.604903+0000","last_clean":"2024-09-13T22:43:26.604903+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:43:26.604903+0000","last_undegraded":"2024-09-13T22:43:26.604903+0000","last_fullsized":"2024-09-13T22:43:26.604903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:35.694064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:35.694064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"153'9078","reported_seq":9208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:29.351959+0000","last_change":"2024-09-13T22:43:00.227649+0000","last_active":"2024-09-13T22:43:29.351959+0000","last_peered":"2024-09-13T22:43:29.351959+0000","last_clean":"2024-09-13T22:43:29.351959+0000","last_became_active":"2024-09-13T22:40:54.961695+0000","last_became_peered":"2024-09-13T22:40:54.961695+0000","last_unstale":"2024-09-13T22:43:29.351959+0000","last_undegraded":"2024-09-13T22:43:29.351959+0000","last_fullsized":"2024-09-13T22:43:29.351959+0000","mapping_epoch":116,"log_start":"127'3900","ondisk_log_start":"127'3900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":4074,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:32.378335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697600,"num_objects":4100,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":421696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":220,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.611695+0000","last_change":"2024-09-13T22:42:53.788869+0000","last_active":"2024-09-13T22:43:26.611695+0000","last_peered":"2024-09-13T22:43:26.611695+0000","last_clean":"2024-09-13T22:43:26.611695+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:43:26.611695+0000","last_undegraded":"2024-09-13T22:43:26.611695+0000","last_fullsized":"2024-09-13T22:43:26.611695+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:42:53.788721+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.788721+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.693042+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614696+0000","last_change":"2024-09-13T22:42:46.200128+0000","last_active":"2024-09-13T22:43:26.614696+0000","last_peered":"2024-09-13T22:43:26.614696+0000","last_clean":"2024-09-13T22:43:26.614696+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:43:26.614696+0000","last_undegraded":"2024-09-13T22:43:26.614696+0000","last_fullsized":"2024-09-13T22:43:26.614696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:46.200054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.200054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:46.600075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":518,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:27.952041+0000","last_change":"2024-09-13T22:41:31.810930+0000","last_active":"2024-09-13T22:43:27.952041+0000","last_peered":"2024-09-13T22:43:27.952041+0000","last_clean":"2024-09-13T22:43:27.952041+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:43:27.952041+0000","last_undegraded":"2024-09-13T22:43:27.952041+0000","last_fullsized":"2024-09-13T22:43:27.952041+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:41:31.810814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:31.810814+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:32.378414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":328,"num_read_kb":323,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614082+0000","last_change":"2024-09-13T22:41:36.600260+0000","last_active":"2024-09-13T22:43:26.614082+0000","last_peered":"2024-09-13T22:43:26.614082+0000","last_clean":"2024-09-13T22:43:26.614082+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:43:26.614082+0000","last_undegraded":"2024-09-13T22:43:26.614082+0000","last_fullsized":"2024-09-13T22:43:26.614082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:36.600156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:36.600156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027292+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614113+0000","last_change":"2024-09-13T22:42:54.080648+0000","last_active":"2024-09-13T22:43:26.614113+0000","last_peered":"2024-09-13T22:43:26.614113+0000","last_clean":"2024-09-13T22:43:26.614113+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:43:26.614113+0000","last_undegraded":"2024-09-13T22:43:26.614113+0000","last_fullsized":"2024-09-13T22:43:26.614113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.080532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.080532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:06.210400+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"153'12472","reported_seq":12700,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.487859+0000","last_change":"2024-09-13T22:43:17.736224+0000","last_active":"2024-09-13T22:43:28.487859+0000","last_peered":"2024-09-13T22:43:28.487859+0000","last_clean":"2024-09-13T22:43:28.487859+0000","last_became_active":"2024-09-13T22:40:54.952389+0000","last_became_peered":"2024-09-13T22:40:54.952389+0000","last_unstale":"2024-09-13T22:43:28.487859+0000","last_undegraded":"2024-09-13T22:43:28.487859+0000","last_fullsized":"2024-09-13T22:43:28.487859+0000","mapping_epoch":116,"log_start":"148'7202","ondisk_log_start":"148'7202","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":3499,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276168704,"num_objects":4214,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9028,"num_write_kb":423744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":308,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618279+0000","last_change":"2024-09-13T22:42:51.196536+0000","last_active":"2024-09-13T22:43:26.618279+0000","last_peered":"2024-09-13T22:43:26.618279+0000","last_clean":"2024-09-13T22:43:26.618279+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:43:26.618279+0000","last_undegraded":"2024-09-13T22:43:26.618279+0000","last_fullsized":"2024-09-13T22:43:26.618279+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:42:51.196417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:51.196417+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:10.030113+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605641+0000","last_change":"2024-09-13T22:40:46.579205+0000","last_active":"2024-09-13T22:43:26.605641+0000","last_peered":"2024-09-13T22:43:26.605641+0000","last_clean":"2024-09-13T22:43:26.605641+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:43:26.605641+0000","last_undegraded":"2024-09-13T22:43:26.605641+0000","last_fullsized":"2024-09-13T22:43:26.605641+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:46.579126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:40:46.579126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.934404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610338+0000","last_change":"2024-09-13T22:42:51.845782+0000","last_active":"2024-09-13T22:43:26.610338+0000","last_peered":"2024-09-13T22:43:26.610338+0000","last_clean":"2024-09-13T22:43:26.610338+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:43:26.610338+0000","last_undegraded":"2024-09-13T22:43:26.610338+0000","last_fullsized":"2024-09-13T22:43:26.610338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:51.845704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:51.845704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:09.906670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614267+0000","last_change":"2024-09-13T22:42:44.246269+0000","last_active":"2024-09-13T22:43:26.614267+0000","last_peered":"2024-09-13T22:43:26.614267+0000","last_clean":"2024-09-13T22:43:26.614267+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:43:26.614267+0000","last_undegraded":"2024-09-13T22:43:26.614267+0000","last_fullsized":"2024-09-13T22:43:26.614267+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:44.246165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.246165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:58.274315+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"153'9108","reported_seq":9263,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.497320+0000","last_change":"2024-09-13T22:42:49.489354+0000","last_active":"2024-09-13T22:43:28.497320+0000","last_peered":"2024-09-13T22:43:28.497320+0000","last_clean":"2024-09-13T22:43:28.497320+0000","last_became_active":"2024-09-13T22:40:54.952306+0000","last_became_peered":"2024-09-13T22:40:54.952306+0000","last_unstale":"2024-09-13T22:43:28.497320+0000","last_undegraded":"2024-09-13T22:43:28.497320+0000","last_fullsized":"2024-09-13T22:43:28.497320+0000","mapping_epoch":116,"log_start":"127'4100","ondisk_log_start":"127'4100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7216","last_scrub_stamp":"2024-09-13T22:42:49.489064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.489064+0000","objects_scrubbed":6088,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.508108+0000","scrub_duration":1475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272105472,"num_objects":4152,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":424320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":279,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616345+0000","last_change":"2024-09-13T22:41:24.936384+0000","last_active":"2024-09-13T22:43:26.616345+0000","last_peered":"2024-09-13T22:43:26.616345+0000","last_clean":"2024-09-13T22:43:26.616345+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:43:26.616345+0000","last_undegraded":"2024-09-13T22:43:26.616345+0000","last_fullsized":"2024-09-13T22:43:26.616345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:41:24.936274+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:41:24.936274+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618370+0000","last_change":"2024-09-13T22:41:58.672475+0000","last_active":"2024-09-13T22:43:26.618370+0000","last_peered":"2024-09-13T22:43:26.618370+0000","last_clean":"2024-09-13T22:43:26.618370+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:43:26.618370+0000","last_undegraded":"2024-09-13T22:43:26.618370+0000","last_fullsized":"2024-09-13T22:43:26.618370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:41:58.672143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.672143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.617188+0000","last_change":"2024-09-13T22:42:34.994631+0000","last_active":"2024-09-13T22:43:26.617188+0000","last_peered":"2024-09-13T22:43:26.617188+0000","last_clean":"2024-09-13T22:43:26.617188+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:43:26.617188+0000","last_undegraded":"2024-09-13T22:43:26.617188+0000","last_fullsized":"2024-09-13T22:43:26.617188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:42:34.994416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.994416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:45.727568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":273,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618172+0000","last_change":"2024-09-13T22:42:53.286723+0000","last_active":"2024-09-13T22:43:26.618172+0000","last_peered":"2024-09-13T22:43:26.618172+0000","last_clean":"2024-09-13T22:43:26.618172+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:43:26.618172+0000","last_undegraded":"2024-09-13T22:43:26.618172+0000","last_fullsized":"2024-09-13T22:43:26.618172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:42:53.286646+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.286646+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:16.002562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614409+0000","last_change":"2024-09-13T22:42:39.282264+0000","last_active":"2024-09-13T22:43:26.614409+0000","last_peered":"2024-09-13T22:43:26.614409+0000","last_clean":"2024-09-13T22:43:26.614409+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:43:26.614409+0000","last_undegraded":"2024-09-13T22:43:26.614409+0000","last_fullsized":"2024-09-13T22:43:26.614409+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:39.282139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.282139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.104358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"153'9349","reported_seq":9507,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.420766+0000","last_change":"2024-09-13T22:42:47.043643+0000","last_active":"2024-09-13T22:43:28.420766+0000","last_peered":"2024-09-13T22:43:28.420766+0000","last_clean":"2024-09-13T22:43:28.420766+0000","last_became_active":"2024-09-13T22:40:54.952085+0000","last_became_peered":"2024-09-13T22:40:54.952085+0000","last_unstale":"2024-09-13T22:43:28.420766+0000","last_undegraded":"2024-09-13T22:43:28.420766+0000","last_fullsized":"2024-09-13T22:43:28.420766+0000","mapping_epoch":116,"log_start":"124'2900","ondisk_log_start":"124'2900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7211","last_scrub_stamp":"2024-09-13T22:42:47.043534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.043534+0000","objects_scrubbed":6429,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:50.140235+0000","scrub_duration":3215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276758528,"num_objects":4223,"num_object_clones":0,"num_object_copies":12669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":224,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614556+0000","last_change":"2024-09-13T22:41:52.800424+0000","last_active":"2024-09-13T22:43:26.614556+0000","last_peered":"2024-09-13T22:43:26.614556+0000","last_clean":"2024-09-13T22:43:26.614556+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:43:26.614556+0000","last_undegraded":"2024-09-13T22:43:26.614556+0000","last_fullsized":"2024-09-13T22:43:26.614556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:41:52.800270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.800270+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027278+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616537+0000","last_change":"2024-09-13T22:42:31.100216+0000","last_active":"2024-09-13T22:43:26.616537+0000","last_peered":"2024-09-13T22:43:26.616537+0000","last_clean":"2024-09-13T22:43:26.616537+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:43:26.616537+0000","last_undegraded":"2024-09-13T22:43:26.616537+0000","last_fullsized":"2024-09-13T22:43:26.616537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.100088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.100088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:35.433339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618307+0000","last_change":"2024-09-13T22:41:47.494639+0000","last_active":"2024-09-13T22:43:26.618307+0000","last_peered":"2024-09-13T22:43:26.618307+0000","last_clean":"2024-09-13T22:43:26.618307+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:43:26.618307+0000","last_undegraded":"2024-09-13T22:43:26.618307+0000","last_fullsized":"2024-09-13T22:43:26.618307+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.494388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.521879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":172,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.606847+0000","last_change":"2024-09-13T22:41:27.793557+0000","last_active":"2024-09-13T22:43:26.606847+0000","last_peered":"2024-09-13T22:43:26.606847+0000","last_clean":"2024-09-13T22:43:26.606847+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:43:26.606847+0000","last_undegraded":"2024-09-13T22:43:26.606847+0000","last_fullsized":"2024-09-13T22:43:26.606847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:27.793491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:27.793491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"153'8886","reported_seq":9043,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.933509+0000","last_change":"2024-09-13T22:42:40.742692+0000","last_active":"2024-09-13T22:43:28.933509+0000","last_peered":"2024-09-13T22:43:28.933509+0000","last_clean":"2024-09-13T22:43:28.933509+0000","last_became_active":"2024-09-13T22:40:54.959564+0000","last_became_peered":"2024-09-13T22:40:54.959564+0000","last_unstale":"2024-09-13T22:43:28.933509+0000","last_undegraded":"2024-09-13T22:43:28.933509+0000","last_fullsized":"2024-09-13T22:43:28.933509+0000","mapping_epoch":116,"log_start":"126'3700","ondisk_log_start":"126'3700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6596","last_scrub_stamp":"2024-09-13T22:42:40.742539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.742539+0000","objects_scrubbed":6364,"log_size":5186,"log_dups_size":0,"ondisk_log_size":5186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:47.311474+0000","scrub_duration":2487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8886,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":228,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616527+0000","last_change":"2024-09-13T22:42:32.975550+0000","last_active":"2024-09-13T22:43:26.616527+0000","last_peered":"2024-09-13T22:43:26.616527+0000","last_clean":"2024-09-13T22:43:26.616527+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:43:26.616527+0000","last_undegraded":"2024-09-13T22:43:26.616527+0000","last_fullsized":"2024-09-13T22:43:26.616527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-09-13T22:42:32.975430+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:42:32.975430+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.970807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614436+0000","last_change":"2024-09-13T22:42:30.787370+0000","last_active":"2024-09-13T22:43:26.614436+0000","last_peered":"2024-09-13T22:43:26.614436+0000","last_clean":"2024-09-13T22:43:26.614436+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:43:26.614436+0000","last_undegraded":"2024-09-13T22:43:26.614436+0000","last_fullsized":"2024-09-13T22:43:26.614436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:30.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:30.787265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:55.200968+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614816+0000","last_change":"2024-09-13T22:41:39.746764+0000","last_active":"2024-09-13T22:43:26.614816+0000","last_peered":"2024-09-13T22:43:26.614816+0000","last_clean":"2024-09-13T22:43:26.614816+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:43:26.614816+0000","last_undegraded":"2024-09-13T22:43:26.614816+0000","last_fullsized":"2024-09-13T22:43:26.614816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:39.746649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:41:39.746649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":179,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605295+0000","last_change":"2024-09-13T22:42:45.743372+0000","last_active":"2024-09-13T22:43:26.605295+0000","last_peered":"2024-09-13T22:43:26.605295+0000","last_clean":"2024-09-13T22:43:26.605295+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:43:26.605295+0000","last_undegraded":"2024-09-13T22:43:26.605295+0000","last_fullsized":"2024-09-13T22:43:26.605295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.743269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.743269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.530343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"153'9090","reported_seq":9174,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.469028+0000","last_change":"2024-09-13T22:40:54.961075+0000","last_active":"2024-09-13T22:43:28.469028+0000","last_peered":"2024-09-13T22:43:28.469028+0000","last_clean":"2024-09-13T22:43:28.469028+0000","last_became_active":"2024-09-13T22:40:54.960739+0000","last_became_peered":"2024-09-13T22:40:54.960739+0000","last_unstale":"2024-09-13T22:43:28.469028+0000","last_undegraded":"2024-09-13T22:43:28.469028+0000","last_fullsized":"2024-09-13T22:43:28.469028+0000","mapping_epoch":116,"log_start":"131'4400","ondisk_log_start":"131'4400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273809408,"num_objects":4178,"num_object_clones":0,"num_object_copies":12534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":424576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":209,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.617838+0000","last_change":"2024-09-13T22:41:41.525997+0000","last_active":"2024-09-13T22:43:26.617838+0000","last_peered":"2024-09-13T22:43:26.617838+0000","last_clean":"2024-09-13T22:43:26.617838+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:43:26.617838+0000","last_undegraded":"2024-09-13T22:43:26.617838+0000","last_fullsized":"2024-09-13T22:43:26.617838+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:41:41.525568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.525568+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.521906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":122,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.625473+0000","last_change":"2024-09-13T22:42:45.684106+0000","last_active":"2024-09-13T22:43:26.625473+0000","last_peered":"2024-09-13T22:43:26.625473+0000","last_clean":"2024-09-13T22:43:26.625473+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:43:26.625473+0000","last_undegraded":"2024-09-13T22:43:26.625473+0000","last_fullsized":"2024-09-13T22:43:26.625473+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.683896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.683896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:57.709636+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616578+0000","last_change":"2024-09-13T22:42:49.271307+0000","last_active":"2024-09-13T22:43:26.616578+0000","last_peered":"2024-09-13T22:43:26.616578+0000","last_clean":"2024-09-13T22:43:26.616578+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:43:26.616578+0000","last_undegraded":"2024-09-13T22:43:26.616578+0000","last_fullsized":"2024-09-13T22:43:26.616578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.271179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:05.043890+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618045+0000","last_change":"2024-09-13T22:41:50.524433+0000","last_active":"2024-09-13T22:43:26.618045+0000","last_peered":"2024-09-13T22:43:26.618045+0000","last_clean":"2024-09-13T22:43:26.618045+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:43:26.618045+0000","last_undegraded":"2024-09-13T22:43:26.618045+0000","last_fullsized":"2024-09-13T22:43:26.618045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:50.524098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.524098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"153'8960","reported_seq":9115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.819059+0000","last_change":"2024-09-13T22:42:44.078796+0000","last_active":"2024-09-13T22:43:28.819059+0000","last_peered":"2024-09-13T22:43:28.819059+0000","last_clean":"2024-09-13T22:43:28.819059+0000","last_became_active":"2024-09-13T22:40:54.955979+0000","last_became_peered":"2024-09-13T22:40:54.955979+0000","last_unstale":"2024-09-13T22:43:28.819059+0000","last_undegraded":"2024-09-13T22:43:28.819059+0000","last_fullsized":"2024-09-13T22:43:28.819059+0000","mapping_epoch":116,"log_start":"126'3700","ondisk_log_start":"126'3700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'6814","last_scrub_stamp":"2024-09-13T22:42:44.078550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.078550+0000","objects_scrubbed":6262,"log_size":5260,"log_dups_size":0,"ondisk_log_size":5260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:10.325784+0000","scrub_duration":2783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265289728,"num_objects":4048,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":416256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.613243+0000","last_change":"2024-09-13T22:42:43.552263+0000","last_active":"2024-09-13T22:43:26.613243+0000","last_peered":"2024-09-13T22:43:26.613243+0000","last_clean":"2024-09-13T22:43:26.613243+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:43:26.613243+0000","last_undegraded":"2024-09-13T22:43:26.613243+0000","last_fullsized":"2024-09-13T22:43:26.613243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:42:43.551997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:43.551997+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.760390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616043+0000","last_change":"2024-09-13T22:42:36.032459+0000","last_active":"2024-09-13T22:43:26.616043+0000","last_peered":"2024-09-13T22:43:26.616043+0000","last_clean":"2024-09-13T22:43:26.616043+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:43:26.616043+0000","last_undegraded":"2024-09-13T22:43:26.616043+0000","last_fullsized":"2024-09-13T22:43:26.616043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:36.032356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.032356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.001671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":445,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605094+0000","last_change":"2024-09-13T22:41:45.619812+0000","last_active":"2024-09-13T22:43:26.605094+0000","last_peered":"2024-09-13T22:43:26.605094+0000","last_clean":"2024-09-13T22:43:26.605094+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:43:26.605094+0000","last_undegraded":"2024-09-13T22:43:26.605094+0000","last_fullsized":"2024-09-13T22:43:26.605094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:41:45.619735+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:41:45.619735+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936046+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604919+0000","last_change":"2024-09-13T22:41:55.748722+0000","last_active":"2024-09-13T22:43:26.604919+0000","last_peered":"2024-09-13T22:43:26.604919+0000","last_clean":"2024-09-13T22:43:26.604919+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:43:26.604919+0000","last_undegraded":"2024-09-13T22:43:26.604919+0000","last_fullsized":"2024-09-13T22:43:26.604919+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:55.748627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:41:55.748627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":179,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616364+0000","last_change":"2024-09-13T22:41:44.960471+0000","last_active":"2024-09-13T22:43:26.616364+0000","last_peered":"2024-09-13T22:43:26.616364+0000","last_clean":"2024-09-13T22:43:26.616364+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:43:26.616364+0000","last_undegraded":"2024-09-13T22:43:26.616364+0000","last_fullsized":"2024-09-13T22:43:26.616364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.960382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.960382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"153'9107","reported_seq":9262,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.784900+0000","last_change":"2024-09-13T22:42:31.664279+0000","last_active":"2024-09-13T22:43:28.784900+0000","last_peered":"2024-09-13T22:43:28.784900+0000","last_clean":"2024-09-13T22:43:28.784900+0000","last_became_active":"2024-09-13T22:40:54.959549+0000","last_became_peered":"2024-09-13T22:40:54.959549+0000","last_unstale":"2024-09-13T22:43:28.784900+0000","last_undegraded":"2024-09-13T22:43:28.784900+0000","last_fullsized":"2024-09-13T22:43:28.784900+0000","mapping_epoch":116,"log_start":"127'3900","ondisk_log_start":"127'3900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'6299","last_scrub_stamp":"2024-09-13T22:42:31.664041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.664041+0000","objects_scrubbed":6215,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:45.679181+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269942784,"num_objects":4119,"num_object_clones":0,"num_object_copies":12357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":423232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":207,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605530+0000","last_change":"2024-09-13T22:41:33.652663+0000","last_active":"2024-09-13T22:43:26.605530+0000","last_peered":"2024-09-13T22:43:26.605530+0000","last_clean":"2024-09-13T22:43:26.605530+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:43:26.605530+0000","last_undegraded":"2024-09-13T22:43:26.605530+0000","last_fullsized":"2024-09-13T22:43:26.605530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:41:33.652584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.652584+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615039+0000","last_change":"2024-09-13T22:42:58.827975+0000","last_active":"2024-09-13T22:43:26.615039+0000","last_peered":"2024-09-13T22:43:26.615039+0000","last_clean":"2024-09-13T22:43:26.615039+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:43:26.615039+0000","last_undegraded":"2024-09-13T22:43:26.615039+0000","last_fullsized":"2024-09-13T22:43:26.615039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.827870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.827870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.240808+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.622124+0000","last_change":"2024-09-13T22:42:46.715235+0000","last_active":"2024-09-13T22:43:26.622124+0000","last_peered":"2024-09-13T22:43:26.622124+0000","last_clean":"2024-09-13T22:43:26.622124+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:43:26.622124+0000","last_undegraded":"2024-09-13T22:43:26.622124+0000","last_fullsized":"2024-09-13T22:43:26.622124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:46.715135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.715135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.458531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604936+0000","last_change":"2024-09-13T22:41:40.678879+0000","last_active":"2024-09-13T22:43:26.604936+0000","last_peered":"2024-09-13T22:43:26.604936+0000","last_clean":"2024-09-13T22:43:26.604936+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:43:26.604936+0000","last_undegraded":"2024-09-13T22:43:26.604936+0000","last_fullsized":"2024-09-13T22:43:26.604936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:40.678635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:40.678635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936018+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"153'8995","reported_seq":9143,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.704948+0000","last_change":"2024-09-13T22:42:57.541592+0000","last_active":"2024-09-13T22:43:28.704948+0000","last_peered":"2024-09-13T22:43:28.704948+0000","last_clean":"2024-09-13T22:43:28.704948+0000","last_became_active":"2024-09-13T22:40:54.960207+0000","last_became_peered":"2024-09-13T22:40:54.960207+0000","last_unstale":"2024-09-13T22:43:28.704948+0000","last_undegraded":"2024-09-13T22:43:28.704948+0000","last_fullsized":"2024-09-13T22:43:28.704948+0000","mapping_epoch":116,"log_start":"123'2300","ondisk_log_start":"123'2300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7482","last_scrub_stamp":"2024-09-13T22:42:57.541325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.541325+0000","objects_scrubbed":5583,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.613669+0000","scrub_duration":1715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263651328,"num_objects":4023,"num_object_clones":0,"num_object_copies":12069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8995,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":202,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616253+0000","last_change":"2024-09-13T22:42:38.043048+0000","last_active":"2024-09-13T22:43:26.616253+0000","last_peered":"2024-09-13T22:43:26.616253+0000","last_clean":"2024-09-13T22:43:26.616253+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:43:26.616253+0000","last_undegraded":"2024-09-13T22:43:26.616253+0000","last_fullsized":"2024-09-13T22:43:26.616253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:42:38.042796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:38.042796+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:53.974317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.619468+0000","last_change":"2024-09-13T22:41:31.544822+0000","last_active":"2024-09-13T22:43:26.619468+0000","last_peered":"2024-09-13T22:43:26.619468+0000","last_clean":"2024-09-13T22:43:26.619468+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:43:26.619468+0000","last_undegraded":"2024-09-13T22:43:26.619468+0000","last_fullsized":"2024-09-13T22:43:26.619468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:31.544520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:31.544520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615335+0000","last_change":"2024-09-13T22:41:41.763050+0000","last_active":"2024-09-13T22:43:26.615335+0000","last_peered":"2024-09-13T22:43:26.615335+0000","last_clean":"2024-09-13T22:43:26.615335+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:43:26.615335+0000","last_undegraded":"2024-09-13T22:43:26.615335+0000","last_fullsized":"2024-09-13T22:43:26.615335+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.762942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.762942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.617739+0000","last_change":"2024-09-13T22:42:54.275527+0000","last_active":"2024-09-13T22:43:26.617739+0000","last_peered":"2024-09-13T22:43:26.617739+0000","last_clean":"2024-09-13T22:43:26.617739+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:43:26.617739+0000","last_undegraded":"2024-09-13T22:43:26.617739+0000","last_fullsized":"2024-09-13T22:43:26.617739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.275361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.824268+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"153'12857","reported_seq":13101,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:28.755158+0000","last_change":"2024-09-13T22:43:17.787104+0000","last_active":"2024-09-13T22:43:28.755158+0000","last_peered":"2024-09-13T22:43:28.755158+0000","last_clean":"2024-09-13T22:43:28.755158+0000","last_became_active":"2024-09-13T22:40:54.949821+0000","last_became_peered":"2024-09-13T22:40:54.949821+0000","last_unstale":"2024-09-13T22:43:28.755158+0000","last_undegraded":"2024-09-13T22:43:28.755158+0000","last_fullsized":"2024-09-13T22:43:28.755158+0000","mapping_epoch":116,"log_start":"148'7101","ondisk_log_start":"148'7101","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:40:53.917678+0000","objects_scrubbed":3899,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.186984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280363008,"num_objects":4278,"num_object_clones":0,"num_object_copies":12834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":428160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605446+0000","last_change":"2024-09-13T22:41:32.631943+0000","last_active":"2024-09-13T22:43:26.605446+0000","last_peered":"2024-09-13T22:43:26.605446+0000","last_clean":"2024-09-13T22:43:26.605446+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:43:26.605446+0000","last_undegraded":"2024-09-13T22:43:26.605446+0000","last_fullsized":"2024-09-13T22:43:26.605446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:41:32.631815+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:41:32.631815+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614952+0000","last_change":"2024-09-13T22:42:31.787949+0000","last_active":"2024-09-13T22:43:26.614952+0000","last_peered":"2024-09-13T22:43:26.614952+0000","last_clean":"2024-09-13T22:43:26.614952+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:43:26.614952+0000","last_undegraded":"2024-09-13T22:43:26.614952+0000","last_fullsized":"2024-09-13T22:43:26.614952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.787457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.787457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:51.479082+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":263,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604708+0000","last_change":"2024-09-13T22:41:50.679387+0000","last_active":"2024-09-13T22:43:26.604708+0000","last_peered":"2024-09-13T22:43:26.604708+0000","last_clean":"2024-09-13T22:43:26.604708+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:43:26.604708+0000","last_undegraded":"2024-09-13T22:43:26.604708+0000","last_fullsized":"2024-09-13T22:43:26.604708+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:50.679300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.679300+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936073+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":179,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615849+0000","last_change":"2024-09-13T22:42:46.267477+0000","last_active":"2024-09-13T22:43:26.615849+0000","last_peered":"2024-09-13T22:43:26.615849+0000","last_clean":"2024-09-13T22:43:26.615849+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:43:26.615849+0000","last_undegraded":"2024-09-13T22:43:26.615849+0000","last_fullsized":"2024-09-13T22:43:26.615849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:46.267344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.267344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.234794+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"153'9321","reported_seq":9470,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:29.388229+0000","last_change":"2024-09-13T22:42:58.835575+0000","last_active":"2024-09-13T22:43:29.388229+0000","last_peered":"2024-09-13T22:43:29.388229+0000","last_clean":"2024-09-13T22:43:29.388229+0000","last_became_active":"2024-09-13T22:40:54.964195+0000","last_became_peered":"2024-09-13T22:40:54.964195+0000","last_unstale":"2024-09-13T22:43:29.388229+0000","last_undegraded":"2024-09-13T22:43:29.388229+0000","last_fullsized":"2024-09-13T22:43:29.388229+0000","mapping_epoch":116,"log_start":"127'4200","ondisk_log_start":"127'4200","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7811","last_scrub_stamp":"2024-09-13T22:42:58.835009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.835009+0000","objects_scrubbed":5759,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:22.402928+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274661376,"num_objects":4191,"num_object_clones":0,"num_object_copies":12573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9321,"num_write_kb":432384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":206,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605441+0000","last_change":"2024-09-13T22:41:53.752721+0000","last_active":"2024-09-13T22:43:26.605441+0000","last_peered":"2024-09-13T22:43:26.605441+0000","last_clean":"2024-09-13T22:43:26.605441+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:43:26.605441+0000","last_undegraded":"2024-09-13T22:43:26.605441+0000","last_fullsized":"2024-09-13T22:43:26.605441+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:53.752600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:53.752600+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614979+0000","last_change":"2024-09-13T22:42:45.983095+0000","last_active":"2024-09-13T22:43:26.614979+0000","last_peered":"2024-09-13T22:43:26.614979+0000","last_clean":"2024-09-13T22:43:26.614979+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:43:26.614979+0000","last_undegraded":"2024-09-13T22:43:26.614979+0000","last_fullsized":"2024-09-13T22:43:26.614979+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:42:45.982969+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.982969+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:51.531936+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.625946+0000","last_change":"2024-09-13T22:42:57.858562+0000","last_active":"2024-09-13T22:43:26.625946+0000","last_peered":"2024-09-13T22:43:26.625946+0000","last_clean":"2024-09-13T22:43:26.625946+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:43:26.625946+0000","last_undegraded":"2024-09-13T22:43:26.625946+0000","last_fullsized":"2024-09-13T22:43:26.625946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:57.858139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.858139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:27.371302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.608252+0000","last_change":"2024-09-13T22:42:31.280678+0000","last_active":"2024-09-13T22:43:26.608252+0000","last_peered":"2024-09-13T22:43:26.608252+0000","last_clean":"2024-09-13T22:43:26.608252+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:43:26.608252+0000","last_undegraded":"2024-09-13T22:43:26.608252+0000","last_fullsized":"2024-09-13T22:43:26.608252+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.280433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.280433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:41.469625+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"112'16","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604851+0000","last_change":"2024-09-13T22:42:36.814689+0000","last_active":"2024-09-13T22:43:26.604851+0000","last_peered":"2024-09-13T22:43:26.604851+0000","last_clean":"2024-09-13T22:43:26.604851+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:43:26.604851+0000","last_undegraded":"2024-09-13T22:43:26.604851+0000","last_fullsized":"2024-09-13T22:43:26.604851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'16","last_scrub_stamp":"2024-09-13T22:42:36.814607+0000","last_deep_scrub":"112'16","last_deep_scrub_stamp":"2024-09-13T22:40:26.503359+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.814607+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.577055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.626096+0000","last_change":"2024-09-13T22:41:43.728379+0000","last_active":"2024-09-13T22:43:26.626096+0000","last_peered":"2024-09-13T22:43:26.626096+0000","last_clean":"2024-09-13T22:43:26.626096+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:43:26.626096+0000","last_undegraded":"2024-09-13T22:43:26.626096+0000","last_fullsized":"2024-09-13T22:43:26.626096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:41:43.728107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.611525+0000","last_change":"2024-09-13T22:42:29.335168+0000","last_active":"2024-09-13T22:43:26.611525+0000","last_peered":"2024-09-13T22:43:26.611525+0000","last_clean":"2024-09-13T22:43:26.611525+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:43:26.611525+0000","last_undegraded":"2024-09-13T22:43:26.611525+0000","last_fullsized":"2024-09-13T22:43:26.611525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:29.335020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:29.335020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:39.684520+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614627+0000","last_change":"2024-09-13T22:42:59.874545+0000","last_active":"2024-09-13T22:43:26.614627+0000","last_peered":"2024-09-13T22:43:26.614627+0000","last_clean":"2024-09-13T22:43:26.614627+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:43:26.614627+0000","last_undegraded":"2024-09-13T22:43:26.614627+0000","last_fullsized":"2024-09-13T22:43:26.614627+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.874442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.874442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:20.548793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":202,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610254+0000","last_change":"2024-09-13T22:42:53.857519+0000","last_active":"2024-09-13T22:43:26.610254+0000","last_peered":"2024-09-13T22:43:26.610254+0000","last_clean":"2024-09-13T22:43:26.610254+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:43:26.610254+0000","last_undegraded":"2024-09-13T22:43:26.610254+0000","last_fullsized":"2024-09-13T22:43:26.610254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:42:53.857434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.857434+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.762599+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610691+0000","last_change":"2024-09-13T22:42:39.863272+0000","last_active":"2024-09-13T22:43:26.610691+0000","last_peered":"2024-09-13T22:43:26.610691+0000","last_clean":"2024-09-13T22:43:26.610691+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:43:26.610691+0000","last_undegraded":"2024-09-13T22:43:26.610691+0000","last_fullsized":"2024-09-13T22:43:26.610691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:39.863156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.863156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.360048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":183,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616328+0000","last_change":"2024-09-13T22:42:41.074516+0000","last_active":"2024-09-13T22:43:26.616328+0000","last_peered":"2024-09-13T22:43:26.616328+0000","last_clean":"2024-09-13T22:43:26.616328+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:43:26.616328+0000","last_undegraded":"2024-09-13T22:43:26.616328+0000","last_fullsized":"2024-09-13T22:43:26.616328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:41.074413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:41.074413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:06.392680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605183+0000","last_change":"2024-09-13T22:40:38.582019+0000","last_active":"2024-09-13T22:43:26.605183+0000","last_peered":"2024-09-13T22:43:26.605183+0000","last_clean":"2024-09-13T22:43:26.605183+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:43:26.605183+0000","last_undegraded":"2024-09-13T22:43:26.605183+0000","last_fullsized":"2024-09-13T22:43:26.605183+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:38.581799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:40:38.581799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.934436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":206,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605056+0000","last_change":"2024-09-13T22:42:34.859780+0000","last_active":"2024-09-13T22:43:26.605056+0000","last_peered":"2024-09-13T22:43:26.605056+0000","last_clean":"2024-09-13T22:43:26.605056+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:43:26.605056+0000","last_undegraded":"2024-09-13T22:43:26.605056+0000","last_fullsized":"2024-09-13T22:43:26.605056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:34.859649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.859649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.462988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616114+0000","last_change":"2024-09-13T22:41:52.950538+0000","last_active":"2024-09-13T22:43:26.616114+0000","last_peered":"2024-09-13T22:43:26.616114+0000","last_clean":"2024-09-13T22:43:26.616114+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:43:26.616114+0000","last_undegraded":"2024-09-13T22:43:26.616114+0000","last_fullsized":"2024-09-13T22:43:26.616114+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.950367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077287+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":266,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616237+0000","last_change":"2024-09-13T22:41:37.950160+0000","last_active":"2024-09-13T22:43:26.616237+0000","last_peered":"2024-09-13T22:43:26.616237+0000","last_clean":"2024-09-13T22:43:26.616237+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:43:26.616237+0000","last_undegraded":"2024-09-13T22:43:26.616237+0000","last_fullsized":"2024-09-13T22:43:26.616237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:37.950036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:37.950036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.077238+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.623829+0000","last_change":"2024-09-13T22:42:35.206528+0000","last_active":"2024-09-13T22:43:26.623829+0000","last_peered":"2024-09-13T22:43:26.623829+0000","last_clean":"2024-09-13T22:43:26.623829+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:43:26.623829+0000","last_undegraded":"2024-09-13T22:43:26.623829+0000","last_fullsized":"2024-09-13T22:43:26.623829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:35.206294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:35.206294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:51.189905+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.617963+0000","last_change":"2024-09-13T22:41:50.549051+0000","last_active":"2024-09-13T22:43:26.617963+0000","last_peered":"2024-09-13T22:43:26.617963+0000","last_clean":"2024-09-13T22:43:26.617963+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:43:26.617963+0000","last_undegraded":"2024-09-13T22:43:26.617963+0000","last_fullsized":"2024-09-13T22:43:26.617963+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:41:50.548929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.548929+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.521958+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618513+0000","last_change":"2024-09-13T22:42:56.298096+0000","last_active":"2024-09-13T22:43:26.618513+0000","last_peered":"2024-09-13T22:43:26.618513+0000","last_clean":"2024-09-13T22:43:26.618513+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:43:26.618513+0000","last_undegraded":"2024-09-13T22:43:26.618513+0000","last_fullsized":"2024-09-13T22:43:26.618513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:56.297990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:56.297990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.122881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":264,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.609327+0000","last_change":"2024-09-13T22:41:36.926417+0000","last_active":"2024-09-13T22:43:26.609327+0000","last_peered":"2024-09-13T22:43:26.609327+0000","last_clean":"2024-09-13T22:43:26.609327+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:43:26.609327+0000","last_undegraded":"2024-09-13T22:43:26.609327+0000","last_fullsized":"2024-09-13T22:43:26.609327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:36.926235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:36.926235+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614796+0000","last_change":"2024-09-13T22:41:58.751646+0000","last_active":"2024-09-13T22:43:26.614796+0000","last_peered":"2024-09-13T22:43:26.614796+0000","last_clean":"2024-09-13T22:43:26.614796+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:43:26.614796+0000","last_undegraded":"2024-09-13T22:43:26.614796+0000","last_fullsized":"2024-09-13T22:43:26.614796+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:58.751579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.751579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.605440+0000","last_change":"2024-09-13T22:42:52.779439+0000","last_active":"2024-09-13T22:43:26.605440+0000","last_peered":"2024-09-13T22:43:26.605440+0000","last_clean":"2024-09-13T22:43:26.605440+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:43:26.605440+0000","last_undegraded":"2024-09-13T22:43:26.605440+0000","last_fullsized":"2024-09-13T22:43:26.605440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.779356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.779356+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.029073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610657+0000","last_change":"2024-09-13T22:41:46.916586+0000","last_active":"2024-09-13T22:43:26.610657+0000","last_peered":"2024-09-13T22:43:26.610657+0000","last_clean":"2024-09-13T22:43:26.610657+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:43:26.610657+0000","last_undegraded":"2024-09-13T22:43:26.610657+0000","last_fullsized":"2024-09-13T22:43:26.610657+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:46.916513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:46.916513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.803542+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.608322+0000","last_change":"2024-09-13T22:42:50.643442+0000","last_active":"2024-09-13T22:43:26.608322+0000","last_peered":"2024-09-13T22:43:26.608322+0000","last_clean":"2024-09-13T22:43:26.608322+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:43:26.608322+0000","last_undegraded":"2024-09-13T22:43:26.608322+0000","last_fullsized":"2024-09-13T22:43:26.608322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:50.643281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.643281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.030417+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614125+0000","last_change":"2024-09-13T22:42:58.240077+0000","last_active":"2024-09-13T22:43:26.614125+0000","last_peered":"2024-09-13T22:43:26.614125+0000","last_clean":"2024-09-13T22:43:26.614125+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:43:26.614125+0000","last_undegraded":"2024-09-13T22:43:26.614125+0000","last_fullsized":"2024-09-13T22:43:26.614125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.239969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.239969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:18.532690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614791+0000","last_change":"2024-09-13T22:41:26.621774+0000","last_active":"2024-09-13T22:43:26.614791+0000","last_peered":"2024-09-13T22:43:26.614791+0000","last_clean":"2024-09-13T22:43:26.614791+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:43:26.614791+0000","last_undegraded":"2024-09-13T22:43:26.614791+0000","last_fullsized":"2024-09-13T22:43:26.614791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:41:26.621656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:26.621656+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.619389+0000","last_change":"2024-09-13T22:41:44.491145+0000","last_active":"2024-09-13T22:43:26.619389+0000","last_peered":"2024-09-13T22:43:26.619389+0000","last_clean":"2024-09-13T22:43:26.619389+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:43:26.619389+0000","last_undegraded":"2024-09-13T22:43:26.619389+0000","last_fullsized":"2024-09-13T22:43:26.619389+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.490926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.490926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614844+0000","last_change":"2024-09-13T22:41:56.720968+0000","last_active":"2024-09-13T22:43:26.614844+0000","last_peered":"2024-09-13T22:43:26.614844+0000","last_clean":"2024-09-13T22:43:26.614844+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:43:26.614844+0000","last_undegraded":"2024-09-13T22:43:26.614844+0000","last_fullsized":"2024-09-13T22:43:26.614844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:56.720495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:56.720495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.606423+0000","last_change":"2024-09-13T22:42:34.299357+0000","last_active":"2024-09-13T22:43:26.606423+0000","last_peered":"2024-09-13T22:43:26.606423+0000","last_clean":"2024-09-13T22:43:26.606423+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:43:26.606423+0000","last_undegraded":"2024-09-13T22:43:26.606423+0000","last_fullsized":"2024-09-13T22:43:26.606423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:34.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:47.934174+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.619054+0000","last_change":"2024-09-13T22:41:44.560484+0000","last_active":"2024-09-13T22:43:26.619054+0000","last_peered":"2024-09-13T22:43:26.619054+0000","last_clean":"2024-09-13T22:43:26.619054+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:43:26.619054+0000","last_undegraded":"2024-09-13T22:43:26.619054+0000","last_fullsized":"2024-09-13T22:43:26.619054+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:41:44.560372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.560372+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.521853+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616496+0000","last_change":"2024-09-13T22:42:40.026232+0000","last_active":"2024-09-13T22:43:26.616496+0000","last_peered":"2024-09-13T22:43:26.616496+0000","last_clean":"2024-09-13T22:43:26.616496+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:43:26.616496+0000","last_undegraded":"2024-09-13T22:43:26.616496+0000","last_fullsized":"2024-09-13T22:43:26.616496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:40.026160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.026160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:03.829266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610470+0000","last_change":"2024-09-13T22:41:58.799450+0000","last_active":"2024-09-13T22:43:26.610470+0000","last_peered":"2024-09-13T22:43:26.610470+0000","last_clean":"2024-09-13T22:43:26.610470+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:43:26.610470+0000","last_undegraded":"2024-09-13T22:43:26.610470+0000","last_fullsized":"2024-09-13T22:43:26.610470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:58.799387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.799387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:28.780157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604533+0000","last_change":"2024-09-13T22:41:51.724840+0000","last_active":"2024-09-13T22:43:26.604533+0000","last_peered":"2024-09-13T22:43:26.604533+0000","last_clean":"2024-09-13T22:43:26.604533+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:43:26.604533+0000","last_undegraded":"2024-09-13T22:43:26.604533+0000","last_fullsized":"2024-09-13T22:43:26.604533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:51.724731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:51.724731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.936220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.616392+0000","last_change":"2024-09-13T22:42:44.126239+0000","last_active":"2024-09-13T22:43:26.616392+0000","last_peered":"2024-09-13T22:43:26.616392+0000","last_clean":"2024-09-13T22:43:26.616392+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:43:26.616392+0000","last_undegraded":"2024-09-13T22:43:26.616392+0000","last_fullsized":"2024-09-13T22:43:26.616392+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:44.126118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.126118+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.929366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610914+0000","last_change":"2024-09-13T22:42:55.792471+0000","last_active":"2024-09-13T22:43:26.610914+0000","last_peered":"2024-09-13T22:43:26.610914+0000","last_clean":"2024-09-13T22:43:26.610914+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:43:26.610914+0000","last_undegraded":"2024-09-13T22:43:26.610914+0000","last_fullsized":"2024-09-13T22:43:26.610914+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:55.792364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:42:55.792364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.743688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":175,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604403+0000","last_change":"2024-09-13T22:40:45.622305+0000","last_active":"2024-09-13T22:43:26.604403+0000","last_peered":"2024-09-13T22:43:26.604403+0000","last_clean":"2024-09-13T22:43:26.604403+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:43:26.604403+0000","last_undegraded":"2024-09-13T22:43:26.604403+0000","last_fullsized":"2024-09-13T22:43:26.604403+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:40:45.622167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:40:45.622167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:26.934491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.608391+0000","last_change":"2024-09-13T22:41:47.765249+0000","last_active":"2024-09-13T22:43:26.608391+0000","last_peered":"2024-09-13T22:43:26.608391+0000","last_clean":"2024-09-13T22:43:26.608391+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:43:26.608391+0000","last_undegraded":"2024-09-13T22:43:26.608391+0000","last_fullsized":"2024-09-13T22:43:26.608391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.765137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":181,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.604851+0000","last_change":"2024-09-13T22:42:54.751350+0000","last_active":"2024-09-13T22:43:26.604851+0000","last_peered":"2024-09-13T22:43:26.604851+0000","last_clean":"2024-09-13T22:43:26.604851+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:43:26.604851+0000","last_undegraded":"2024-09-13T22:43:26.604851+0000","last_fullsized":"2024-09-13T22:43:26.604851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:42:54.751219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.751219+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.922863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"111'22","reported_seq":220,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614425+0000","last_change":"2024-09-13T22:41:24.620491+0000","last_active":"2024-09-13T22:43:26.614425+0000","last_peered":"2024-09-13T22:43:26.614425+0000","last_clean":"2024-09-13T22:43:26.614425+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:43:26.614425+0000","last_undegraded":"2024-09-13T22:43:26.614425+0000","last_fullsized":"2024-09-13T22:43:26.614425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-13T22:41:24.620413+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:41:24.620413+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187015+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.615790+0000","last_change":"2024-09-13T22:42:57.304510+0000","last_active":"2024-09-13T22:43:26.615790+0000","last_peered":"2024-09-13T22:43:26.615790+0000","last_clean":"2024-09-13T22:43:26.615790+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:43:26.615790+0000","last_undegraded":"2024-09-13T22:43:26.615790+0000","last_fullsized":"2024-09-13T22:43:26.615790+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:57.304411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.304411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:15.973967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":267,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610893+0000","last_change":"2024-09-13T22:42:33.826398+0000","last_active":"2024-09-13T22:43:26.610893+0000","last_peered":"2024-09-13T22:43:26.610893+0000","last_clean":"2024-09-13T22:43:26.610893+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:43:26.610893+0000","last_undegraded":"2024-09-13T22:43:26.610893+0000","last_fullsized":"2024-09-13T22:43:26.610893+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:33.826326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:33.826326+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:46.172148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.619551+0000","last_change":"2024-09-13T22:41:41.532153+0000","last_active":"2024-09-13T22:43:26.619551+0000","last_peered":"2024-09-13T22:43:26.619551+0000","last_clean":"2024-09-13T22:43:26.619551+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:43:26.619551+0000","last_undegraded":"2024-09-13T22:43:26.619551+0000","last_fullsized":"2024-09-13T22:43:26.619551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.531837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.531837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:29.187064+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.606121+0000","last_change":"2024-09-13T22:42:37.463998+0000","last_active":"2024-09-13T22:43:26.606121+0000","last_peered":"2024-09-13T22:43:26.606121+0000","last_clean":"2024-09-13T22:43:26.606121+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:43:26.606121+0000","last_undegraded":"2024-09-13T22:43:26.606121+0000","last_fullsized":"2024-09-13T22:43:26.606121+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:37.463889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.463889+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:46.425283+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.606667+0000","last_change":"2024-09-13T22:42:45.551596+0000","last_active":"2024-09-13T22:43:26.606667+0000","last_peered":"2024-09-13T22:43:26.606667+0000","last_clean":"2024-09-13T22:43:26.606667+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:43:26.606667+0000","last_undegraded":"2024-09-13T22:43:26.606667+0000","last_fullsized":"2024-09-13T22:43:26.606667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:45.551192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.551192+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:07.564548+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.624023+0000","last_change":"2024-09-13T22:42:59.279764+0000","last_active":"2024-09-13T22:43:26.624023+0000","last_peered":"2024-09-13T22:43:26.624023+0000","last_clean":"2024-09-13T22:43:26.624023+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:43:26.624023+0000","last_undegraded":"2024-09-13T22:43:26.624023+0000","last_fullsized":"2024-09-13T22:43:26.624023+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.279506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.279506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.337637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614678+0000","last_change":"2024-09-13T22:41:44.682932+0000","last_active":"2024-09-13T22:43:26.614678+0000","last_peered":"2024-09-13T22:43:26.614678+0000","last_clean":"2024-09-13T22:43:26.614678+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:43:26.614678+0000","last_undegraded":"2024-09-13T22:43:26.614678+0000","last_fullsized":"2024-09-13T22:43:26.614678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.682861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.682861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.610617+0000","last_change":"2024-09-13T22:42:47.806567+0000","last_active":"2024-09-13T22:43:26.610617+0000","last_peered":"2024-09-13T22:43:26.610617+0000","last_clean":"2024-09-13T22:43:26.610617+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:43:26.610617+0000","last_undegraded":"2024-09-13T22:43:26.610617+0000","last_fullsized":"2024-09-13T22:43:26.610617+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:47.806485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.806485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.641266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.618494+0000","last_change":"2024-09-13T22:42:52.339140+0000","last_active":"2024-09-13T22:43:26.618494+0000","last_peered":"2024-09-13T22:43:26.618494+0000","last_clean":"2024-09-13T22:43:26.618494+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:43:26.618494+0000","last_undegraded":"2024-09-13T22:43:26.618494+0000","last_fullsized":"2024-09-13T22:43:26.618494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:42:52.339029+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.339029+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:57.801246+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.619243+0000","last_change":"2024-09-13T22:42:53.183255+0000","last_active":"2024-09-13T22:43:26.619243+0000","last_peered":"2024-09-13T22:43:26.619243+0000","last_clean":"2024-09-13T22:43:26.619243+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:43:26.619243+0000","last_undegraded":"2024-09-13T22:43:26.619243+0000","last_fullsized":"2024-09-13T22:43:26.619243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:53.183134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.183134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.729375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":181,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.611404+0000","last_change":"2024-09-13T22:41:54.907202+0000","last_active":"2024-09-13T22:43:26.611404+0000","last_peered":"2024-09-13T22:43:26.611404+0000","last_clean":"2024-09-13T22:43:26.611404+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:43:26.611404+0000","last_undegraded":"2024-09-13T22:43:26.611404+0000","last_fullsized":"2024-09-13T22:43:26.611404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:41:54.906878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.906878+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.608661+0000","last_change":"2024-09-13T22:42:48.550930+0000","last_active":"2024-09-13T22:43:26.608661+0000","last_peered":"2024-09-13T22:43:26.608661+0000","last_clean":"2024-09-13T22:43:26.608661+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:43:26.608661+0000","last_undegraded":"2024-09-13T22:43:26.608661+0000","last_fullsized":"2024-09-13T22:43:26.608661+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:48.550845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:48.550845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.389475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614493+0000","last_change":"2024-09-13T22:42:52.059935+0000","last_active":"2024-09-13T22:43:26.614493+0000","last_peered":"2024-09-13T22:43:26.614493+0000","last_clean":"2024-09-13T22:43:26.614493+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:43:26.614493+0000","last_undegraded":"2024-09-13T22:43:26.614493+0000","last_fullsized":"2024-09-13T22:43:26.614493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.059600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.059600+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:01.524878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T22:43:26.614617+0000","last_change":"2024-09-13T22:41:39.615720+0000","last_active":"2024-09-13T22:43:26.614617+0000","last_peered":"2024-09-13T22:43:26.614617+0000","last_clean":"2024-09-13T22:43:26.614617+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:43:26.614617+0000","last_undegraded":"2024-09-13T22:43:26.614617+0000","last_fullsized":"2024-09-13T22:43:26.614617+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:39.615598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:41:39.615598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:30.027247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4336975896,"num_objects":66178,"num_object_clones":0,"num_object_copies":198534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145324,"num_write_kb":6768001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13015658496,"data_stored":13015646280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89620,"ondisk_log_size":89620,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":328,"num_read_kb":323,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808910,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391044,"kb_used_data":1327856,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92354108,"statfs":{"total":95995035648,"available":94570606592,"internally_reserved":0,"allocated":1359724544,"data_stored":1359327727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135961,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1424616,"kb_used_data":1361368,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92320536,"statfs":{"total":95995035648,"available":94536228864,"internally_reserved":0,"allocated":1394040832,"data_stored":1393666293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528420,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1169576,"kb_used_data":1093972,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92575576,"statfs":{"total":95995035648,"available":94797389824,"internally_reserved":0,"allocated":1120227328,"data_stored":1119856719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888174,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1386248,"kb_used_data":1323124,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92358904,"statfs":{"total":95995035648,"available":94575517696,"internally_reserved":0,"allocated":1354878976,"data_stored":1354476129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084408,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2499108,"kb_used_data":2369364,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91246044,"statfs":{"total":95995035648,"available":93435949056,"internally_reserved":0,"allocated":2426228736,"data_stored":2425858548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132839879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378756,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1674788,"kb_used_data":1593620,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92070364,"statfs":{"total":95995035648,"available":94280052736,"internally_reserved":0,"allocated":1631866880,"data_stored":1631496785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1710260,"kb_used_data":1610980,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92034892,"statfs":{"total":95995035648,"available":94243729408,"internally_reserved":0,"allocated":1649643520,"data_stored":1649257460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2237312,"kb_used_data":2121388,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":91507840,"statfs":{"total":95995035648,"available":93704028160,"internally_reserved":0,"allocated":2172301312,"data_stored":2171908364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2157187072,"data_stored":2157183000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1632571392,"data_stored":1632567320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1625423872,"data_stored":1625423872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2422734848,"data_stored":2422734848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1345716224,"data_stored":1345716224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1109000192,"data_stored":1109000192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1379078144,"data_stored":1379074072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1343946752,"data_stored":1343946752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:43:36.832 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T22:43:36.832 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:43:36.832 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T22:43:36.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T22:43:36.833 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:36.936 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:36.992 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:36.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:37.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:37.121 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:37.224 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:37.258 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:37.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:37.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:37.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:37.527 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:37.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:37.653 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:37.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:37.765 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.144358+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:43:37.765 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.146960+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:43:37.765 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.509311+0000 osd.3 (osd.3) 138 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:43:37.765 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.511484+0000 osd.3 (osd.3) 139 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:43:37.766 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[22840]: cluster 2024-09-13T22:43:36.531995+0000 mgr.y (mgr.14613) 223 : cluster [DBG] pgmap v214: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s 2024-09-13T22:43:37.766 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.144358+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:43:37.766 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.146960+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:43:37.766 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.509311+0000 osd.3 (osd.3) 138 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:43:37.766 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.511484+0000 osd.3 (osd.3) 139 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:43:37.766 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:37 smithi178 bash[29889]: cluster 2024-09-13T22:43:36.531995+0000 mgr.y (mgr.14613) 223 : cluster [DBG] pgmap v214: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s 2024-09-13T22:43:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.781 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:37.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:37.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:37.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:37 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.144358+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:43:37.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:37 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.146960+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:43:37.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:37 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.509311+0000 osd.3 (osd.3) 138 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:43:37.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:37 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.511484+0000 osd.3 (osd.3) 139 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:43:37.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:37 smithi192 bash[24608]: cluster 2024-09-13T22:43:36.531995+0000 mgr.y (mgr.14613) 223 : cluster [DBG] pgmap v214: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s 2024-09-13T22:43:37.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:37.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:38.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:38.044 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:38.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:38.171 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:38.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:38.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:38.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:38.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:38.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:38.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:38.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:38.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:38.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:37.514987+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.a scrub starts 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:37.536967+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.a scrub ok 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:37.548807+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:37.549861+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:37.795156+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:37.803582+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.059014+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:43:38.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.061686+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:43:38.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.108794+0000 osd.4 (osd.4) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:38.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:38 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.108859+0000 osd.4 (osd.4) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:38.844 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:37.514987+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.a scrub starts 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:37.536967+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.a scrub ok 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:37.548807+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:37.549861+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:37.795156+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:37.803582+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.059014+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.061686+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:43:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.108794+0000 osd.4 (osd.4) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.108859+0000 osd.4 (osd.4) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:37.514987+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.a scrub starts 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:37.536967+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.a scrub ok 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:37.548807+0000 osd.3 (osd.3) 140 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:37.549861+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:37.795156+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:37.803582+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.059014+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.061686+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.108794+0000 osd.4 (osd.4) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:38 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.108859+0000 osd.4 (osd.4) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:38.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:38.972 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:38.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:39.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:39.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:39.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:39.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:39.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:39.370 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:39.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:39.502 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:39.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:39.637 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:39.651 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:43:39.721 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:43:39.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:39.753 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:39 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.532669+0000 mgr.y (mgr.14613) 224 : cluster [DBG] pgmap v215: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s 2024-09-13T22:43:39.753 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:39 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.552801+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:43:39.753 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:39 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.560247+0000 osd.3 (osd.3) 143 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:43:39.753 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:39 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.624236+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-13T22:43:39.753 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:39 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.821930+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:43:39.753 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:39 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.824325+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:43:39.765 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:39.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:39.880 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:43:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:43:39.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.532669+0000 mgr.y (mgr.14613) 224 : cluster [DBG] pgmap v215: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s 2024-09-13T22:43:39.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.552801+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.560247+0000 osd.3 (osd.3) 143 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.624236+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.821930+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.824325+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.532669+0000 mgr.y (mgr.14613) 224 : cluster [DBG] pgmap v215: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.552801+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.560247+0000 osd.3 (osd.3) 143 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.624236+0000 mon.a (mon.0) 1023 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.821930+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:43:39.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:39 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.824325+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:43:39.910 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:39.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:40.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:40.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:40.137 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:40.162 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:40.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:40.287 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:40.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:40.424 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:40.451 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:40.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:40.564 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:40.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:40.677 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:38.934170+0000 osd.5 (osd.5) 126 : cluster [DBG] 7.0 scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.273832+0000 osd.0 (osd.0) 137 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.277670+0000 osd.0 (osd.0) 138 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.855379+0000 osd.6 (osd.6) 95 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.856557+0000 osd.6 (osd.6) 96 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.079359+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.080315+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.e scrub ok 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.533347+0000 mgr.y (mgr.14613) 225 : cluster [DBG] pgmap v217: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:38.934170+0000 osd.5 (osd.5) 126 : cluster [DBG] 7.0 scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.273832+0000 osd.0 (osd.0) 137 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.277670+0000 osd.0 (osd.0) 138 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.855379+0000 osd.6 (osd.6) 95 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:43:40.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.856557+0000 osd.6 (osd.6) 96 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:43:40.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.079359+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub starts 2024-09-13T22:43:40.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.080315+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.e scrub ok 2024-09-13T22:43:40.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:40 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.533347+0000 mgr.y (mgr.14613) 225 : cluster [DBG] pgmap v217: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T22:43:40.700 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:40.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:40.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:40.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:38.934170+0000 osd.5 (osd.5) 126 : cluster [DBG] 7.0 scrub starts 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.273832+0000 osd.0 (osd.0) 137 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.277670+0000 osd.0 (osd.0) 138 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.855379+0000 osd.6 (osd.6) 95 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.856557+0000 osd.6 (osd.6) 96 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.079359+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub starts 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.080315+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.e scrub ok 2024-09-13T22:43:40.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:40 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.533347+0000 mgr.y (mgr.14613) 225 : cluster [DBG] pgmap v217: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T22:43:40.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:40.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:40.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:41.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:41.100 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:41.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:41.232 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:41.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:41.367 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:41.470 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:41.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:41.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:41.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:41.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:41.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:41.751 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:41.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.923988+0000 osd.5 (osd.5) 127 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.932319+0000 osd.5 (osd.5) 128 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:39.950380+0000 osd.5 (osd.5) 129 : cluster [DBG] 7.0 scrub ok 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.551655+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.553639+0000 osd.3 (osd.3) 145 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.562060+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.564532+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.669091+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:43:41.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.670203+0000 osd.7 (osd.7) 128 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:43:41.871 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.681051+0000 osd.6 (osd.6) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:41.871 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:41 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.681116+0000 osd.6 (osd.6) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:41.886 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:41.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:41.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.923988+0000 osd.5 (osd.5) 127 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:43:41.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.932319+0000 osd.5 (osd.5) 128 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:39.950380+0000 osd.5 (osd.5) 129 : cluster [DBG] 7.0 scrub ok 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.551655+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.553639+0000 osd.3 (osd.3) 145 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.562060+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.564532+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.669091+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.670203+0000 osd.7 (osd.7) 128 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.681051+0000 osd.6 (osd.6) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:41.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.681116+0000 osd.6 (osd.6) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.923988+0000 osd.5 (osd.5) 127 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.932319+0000 osd.5 (osd.5) 128 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:39.950380+0000 osd.5 (osd.5) 129 : cluster [DBG] 7.0 scrub ok 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.551655+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.553639+0000 osd.3 (osd.3) 145 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.562060+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.564532+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.669091+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.670203+0000 osd.7 (osd.7) 128 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.681051+0000 osd.6 (osd.6) 97 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:41.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:41 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.681116+0000 osd.6 (osd.6) 98 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:41.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:42.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:42.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:42.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:42.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:42.288 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:42.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:42.418 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:42.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:42.542 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:42.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:42.671 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:42.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:42.794 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:42.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:42.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:42 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.972374+0000 osd.5 (osd.5) 130 : cluster [DBG] 5.f scrub starts 2024-09-13T22:43:42.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:42 smithi192 bash[24608]: cluster 2024-09-13T22:43:40.973391+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.f scrub ok 2024-09-13T22:43:42.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:42 smithi192 bash[24608]: cluster 2024-09-13T22:43:41.637002+0000 mon.a (mon.0) 1024 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-13T22:43:42.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:42 smithi192 bash[24608]: cluster 2024-09-13T22:43:42.535268+0000 mgr.y (mgr.14613) 226 : cluster [DBG] pgmap v219: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 512 op/s 2024-09-13T22:43:42.906 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.972374+0000 osd.5 (osd.5) 130 : cluster [DBG] 5.f scrub starts 2024-09-13T22:43:42.906 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[22840]: cluster 2024-09-13T22:43:40.973391+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.f scrub ok 2024-09-13T22:43:42.907 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[22840]: cluster 2024-09-13T22:43:41.637002+0000 mon.a (mon.0) 1024 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-13T22:43:42.907 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[22840]: cluster 2024-09-13T22:43:42.535268+0000 mgr.y (mgr.14613) 226 : cluster [DBG] pgmap v219: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 512 op/s 2024-09-13T22:43:42.907 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.972374+0000 osd.5 (osd.5) 130 : cluster [DBG] 5.f scrub starts 2024-09-13T22:43:42.907 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[29889]: cluster 2024-09-13T22:43:40.973391+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.f scrub ok 2024-09-13T22:43:42.907 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[29889]: cluster 2024-09-13T22:43:41.637002+0000 mon.a (mon.0) 1024 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-13T22:43:42.907 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:42 smithi178 bash[29889]: cluster 2024-09-13T22:43:42.535268+0000 mgr.y (mgr.14613) 226 : cluster [DBG] pgmap v219: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 512 op/s 2024-09-13T22:43:42.929 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:42.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:43.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:43.070 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:43.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:43.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:43.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:43.326 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:43.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:43.452 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:43.555 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:43.584 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:43.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[22840]: cluster 2024-09-13T22:43:42.507653+0000 osd.3 (osd.3) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[22840]: cluster 2024-09-13T22:43:42.507718+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[22840]: cluster 2024-09-13T22:43:42.592506+0000 osd.3 (osd.3) 148 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[22840]: cluster 2024-09-13T22:43:42.603282+0000 osd.3 (osd.3) 149 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[22840]: cluster 2024-09-13T22:43:42.664594+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[29889]: cluster 2024-09-13T22:43:42.507653+0000 osd.3 (osd.3) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[29889]: cluster 2024-09-13T22:43:42.507718+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[29889]: cluster 2024-09-13T22:43:42.592506+0000 osd.3 (osd.3) 148 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[29889]: cluster 2024-09-13T22:43:42.603282+0000 osd.3 (osd.3) 149 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:43:43.696 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:43 smithi178 bash[29889]: cluster 2024-09-13T22:43:42.664594+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-13T22:43:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.727 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:43.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:43.828 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:43.863 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:43.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:43 smithi192 bash[24608]: cluster 2024-09-13T22:43:42.507653+0000 osd.3 (osd.3) 146 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:43 smithi192 bash[24608]: cluster 2024-09-13T22:43:42.507718+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:43 smithi192 bash[24608]: cluster 2024-09-13T22:43:42.592506+0000 osd.3 (osd.3) 148 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:43:43.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:43 smithi192 bash[24608]: cluster 2024-09-13T22:43:42.603282+0000 osd.3 (osd.3) 149 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:43:43.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:43 smithi192 bash[24608]: cluster 2024-09-13T22:43:42.664594+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-13T22:43:43.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:44.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:44.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:44.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:44.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:44.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:44.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:44.401 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:44.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:44.529 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:44.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:44.530 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:44.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:44.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:44.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:44.782 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:44.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:44.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:44 smithi192 bash[24608]: cluster 2024-09-13T22:43:43.549421+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:43:44.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:44 smithi192 bash[24608]: cluster 2024-09-13T22:43:43.560287+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:43:44.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:44 smithi192 bash[24608]: cluster 2024-09-13T22:43:44.536264+0000 mgr.y (mgr.14613) 227 : cluster [DBG] pgmap v221: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T22:43:44.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:44.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:44.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:44 smithi178 bash[22840]: cluster 2024-09-13T22:43:43.549421+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:43:44.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:44 smithi178 bash[22840]: cluster 2024-09-13T22:43:43.560287+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:43:44.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:44 smithi178 bash[22840]: cluster 2024-09-13T22:43:44.536264+0000 mgr.y (mgr.14613) 227 : cluster [DBG] pgmap v221: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T22:43:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:44 smithi178 bash[29889]: cluster 2024-09-13T22:43:43.549421+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:43:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:44 smithi178 bash[29889]: cluster 2024-09-13T22:43:43.560287+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:43:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:44 smithi178 bash[29889]: cluster 2024-09-13T22:43:44.536264+0000 mgr.y (mgr.14613) 227 : cluster [DBG] pgmap v221: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T22:43:45.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.038 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:45.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:45.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:43:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:43:45.167 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:45.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:45.293 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:45.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:45.419 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:45.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:45.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.675 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:45.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:45.811 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:45.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:45 smithi192 bash[24608]: cluster 2024-09-13T22:43:44.220951+0000 osd.1 (osd.1) 128 : cluster [DBG] 7.b scrub starts 2024-09-13T22:43:45.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:45 smithi192 bash[24608]: audit 2024-09-13T22:43:45.074410+0000 mgr.y (mgr.14613) 228 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:45.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:45 smithi178 bash[29889]: cluster 2024-09-13T22:43:44.220951+0000 osd.1 (osd.1) 128 : cluster [DBG] 7.b scrub starts 2024-09-13T22:43:45.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:45 smithi178 bash[29889]: audit 2024-09-13T22:43:45.074410+0000 mgr.y (mgr.14613) 228 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:45.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:45 smithi178 bash[22840]: cluster 2024-09-13T22:43:44.220951+0000 osd.1 (osd.1) 128 : cluster [DBG] 7.b scrub starts 2024-09-13T22:43:45.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:45 smithi178 bash[22840]: audit 2024-09-13T22:43:45.074410+0000 mgr.y (mgr.14613) 228 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:45.941 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:45.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:46.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:46.078 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:46.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.207 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:46.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:46.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:46.328 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:46.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:46.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:46.537 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:43:46.537 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:43:46.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:46.577 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:46.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:46.716 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:46.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:46.850 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:46.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.040818+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.042036+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.561485+0000 osd.1 (osd.1) 129 : cluster [DBG] 7.b scrub ok 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.601074+0000 osd.3 (osd.3) 152 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.602181+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.603348+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:43:46.957 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:45.608678+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-09-13T22:43:46.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[29889]: cluster 2024-09-13T22:43:46.536639+0000 mgr.y (mgr.14613) 229 : cluster [DBG] pgmap v222: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 915 op/s 2024-09-13T22:43:46.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.040818+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:43:46.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.042036+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:43:46.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.561485+0000 osd.1 (osd.1) 129 : cluster [DBG] 7.b scrub ok 2024-09-13T22:43:46.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.601074+0000 osd.3 (osd.3) 152 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:43:46.958 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.602181+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-09-13T22:43:46.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.603348+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:43:46.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:45.608678+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-09-13T22:43:46.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:46 smithi178 bash[22840]: cluster 2024-09-13T22:43:46.536639+0000 mgr.y (mgr.14613) 229 : cluster [DBG] pgmap v222: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 915 op/s 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.040818+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.042036+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.561485+0000 osd.1 (osd.1) 129 : cluster [DBG] 7.b scrub ok 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.601074+0000 osd.3 (osd.3) 152 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.602181+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.603348+0000 osd.3 (osd.3) 153 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:45.608678+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-09-13T22:43:46.963 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:46 smithi192 bash[24608]: cluster 2024-09-13T22:43:46.536639+0000 mgr.y (mgr.14613) 229 : cluster [DBG] pgmap v222: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 915 op/s 2024-09-13T22:43:46.980 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:46.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:47.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:47.113 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:47.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:47.241 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:47.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:47.378 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:47.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:47.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:47.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:47.638 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:47.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:47.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:47.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:47.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:47.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:47.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:47 smithi192 bash[24608]: cluster 2024-09-13T22:43:46.250760+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:43:47.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:47 smithi192 bash[24608]: cluster 2024-09-13T22:43:46.254535+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:43:47.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:47 smithi178 bash[29889]: cluster 2024-09-13T22:43:46.250760+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:43:47.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:47 smithi178 bash[29889]: cluster 2024-09-13T22:43:46.254535+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:43:47.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:47 smithi178 bash[22840]: cluster 2024-09-13T22:43:46.250760+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:43:47.976 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:47 smithi178 bash[22840]: cluster 2024-09-13T22:43:46.254535+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:43:48.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:48.022 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:48.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:48.150 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:48.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:48.280 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:48.358 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 157, 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "created": "2024-09-13T22:27:31.500176+0000", 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "modified": "2024-09-13T22:43:42.641081+0000", 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "last_up_change": "2024-09-13T22:37:07.372146+0000", 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "last_in_change": "2024-09-13T22:36:38.469358+0000", 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_num": 5804032, 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_set": [ 2024-09-13T22:43:48.359 INFO:teuthology.orchestra.run.smithi178.stdout: "nodeep-scrub", 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "pglog_hardlimit", 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "purged_snapdirs", 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_deletes", 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "sortbitwise" 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_version": 18, 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_snapshot": "", 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_max": 7, 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "max_osd": 8, 2024-09-13T22:43:48.360 INFO:teuthology.orchestra.run.smithi178.stdout: "require_min_compat_client": "luminous", 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "min_compat_client": "luminous", 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "require_osd_release": "squid", 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "allow_crimson": false, 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "pools": [ 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 1, 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".mgr", 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:32:40.780155+0000", 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.361 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 1, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 1, 2024-09-13T22:43:48.362 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 1, 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 1, 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 1, 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.363 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "22", 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.364 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.365 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.366 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_max": 32, 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_min": 1 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr": {} 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 7.8899998664855957, 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 7.8899998664855957, 2024-09-13T22:43:48.367 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 3, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 3, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 2, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".rgw.root", 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:39.227580+0000", 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:43:48.368 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:43:48.369 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "76", 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.371 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.372 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.373 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.374 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 3, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.log", 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.229251+0000", 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.376 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "78", 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.377 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.378 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.379 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:43:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 4, 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "datapool", 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.624329+0000", 2024-09-13T22:43:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 8193, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.382 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 3, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 3, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 3, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 3, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 3, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.383 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "85", 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.384 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 2, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 85, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.385 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.386 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.387 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd": {} 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.6500000953674316, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.6500000953674316, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T22:43:48.388 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 5, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.control", 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:43.199974+0000", 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.389 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.390 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "80", 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:43:48.391 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.392 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.393 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.25, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.25, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.25, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.25, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.394 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 6, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.meta", 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:45.423695+0000", 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.395 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:43:48.396 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "82", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.397 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.398 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.399 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_bias": 4 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.75, 2024-09-13T22:43:48.400 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.75, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.75, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.75, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 7, 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "unique_pool_0", 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:40:53.509919+0000", 2024-09-13T22:43:48.401 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:43:48.402 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 16, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 16, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 16, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 16, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 16, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:43:48.403 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "119", 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:43:48.404 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:43:48.405 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:43:48.406 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "rados": {} 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:43:48.407 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:43:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac", 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 8, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6801", 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6802", 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.410 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6804", 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.411 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6803", 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6801/1489719354", 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6802/1489719354", 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6804/1489719354", 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6803/1489719354", 2024-09-13T22:43:48.412 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa", 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.413 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 13, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.414 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.416 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.416 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6805", 2024-09-13T22:43:48.416 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6806", 2024-09-13T22:43:48.417 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6808", 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6807", 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6805/4187084287", 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6806/4187084287", 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6808/4187084287", 2024-09-13T22:43:48.419 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6807/4187084287", 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "40a79075-5102-4f6c-b41a-255102064624", 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.420 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 19, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6809", 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:43:48.421 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6810", 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.422 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6812", 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6811", 2024-09-13T22:43:48.423 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6809/2303802730", 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6810/2303802730", 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6812/2303802730", 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6811/2303802730", 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.424 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad", 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 28, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.425 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6813", 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.426 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6814", 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6816", 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.427 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6815", 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6813/320675660", 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6814/320675660", 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6816/320675660", 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6815/320675660", 2024-09-13T22:43:48.428 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b", 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.429 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 34, 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6800", 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.430 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6801", 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6803", 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.431 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6802", 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6800/3593670527", 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6801/3593670527", 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6803/3593670527", 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6802/3593670527", 2024-09-13T22:43:48.432 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8", 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.433 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 45, 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6804", 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.434 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6805", 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.435 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6807", 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6806", 2024-09-13T22:43:48.436 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6804/3271425338", 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6805/3271425338", 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6807/3271425338", 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6806/3271425338", 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.437 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98", 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 57, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.438 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6808", 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.439 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6809", 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6811", 2024-09-13T22:43:48.440 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6810", 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.441 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6808/684205162", 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6809/684205162", 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6811/684205162", 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6810/684205162", 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52", 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:43:48.442 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 67, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.443 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6812", 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6813", 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.444 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6815", 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:43:48.445 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6814", 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6812/1303971738", 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6813/1303971738", 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1303971738", 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6814/1303971738", 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:43:48.446 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_xinfo": [ 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:21.712932+0000", 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:43:48.447 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:42:38.222982+0000", 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.448 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:42:53.803761+0000", 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:42.507720+0000", 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:38.108862+0000", 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:43:48.450 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:42:42.638925+0000", 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.451 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:40.681119+0000", 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:25.862934+0000", 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:43:48.452 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap": [], 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_items": [ 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "2.0", 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 7, 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 2 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:43:48.453 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_primaries": [], 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_temp": [], 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_temp": [], 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "blocklist": { 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/268210503": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2039160385": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/669877409": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/129655740": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3632788039": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/388635774": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3265105147": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2959920382": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2494442689": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:43:48.454 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2676020773": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2675812306": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/1834442338": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2835351236": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3836928220": "2024-09-14T22:38:32.396576+0000" 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "range_blocklist": {}, 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profiles": { 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "default": { 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "crush-failure-domain": "osd", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "k": "2", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "m": "1", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "plugin": "jerasure", 2024-09-13T22:43:48.455 INFO:teuthology.orchestra.run.smithi178.stdout: "technique": "reed_sol_van" 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps_queue": [], 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "new_removed_snaps": [], 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "new_purged_snaps": [], 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_node_flags": {}, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "device_class_flags": {}, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode": { 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_enabled": false, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_bucket_count": 0, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "degraded_stretch_mode": 0, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "recovering_stretch_mode": 0, 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_bucket": 0 2024-09-13T22:43:48.456 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:43:48.457 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:43:48.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:48.475 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:48.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:48.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:48.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:48.735 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:48.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:48.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:48.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: cluster 2024-09-13T22:43:46.966319+0000 osd.6 (osd.6) 99 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: cluster 2024-09-13T22:43:46.968948+0000 osd.6 (osd.6) 100 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: audit 2024-09-13T22:43:48.024197+0000 mon.b (mon.1) 121 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: cluster 2024-09-13T22:43:48.313928+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: cluster 2024-09-13T22:43:48.316563+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: audit 2024-09-13T22:43:48.359494+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.178:0/3101595859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:43:48.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:48 smithi192 bash[24608]: cluster 2024-09-13T22:43:48.538177+0000 mgr.y (mgr.14613) 230 : cluster [DBG] pgmap v223: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-09-13T22:43:48.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:48.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:49.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:49.108 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: cluster 2024-09-13T22:43:46.966319+0000 osd.6 (osd.6) 99 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:43:49.108 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: cluster 2024-09-13T22:43:46.968948+0000 osd.6 (osd.6) 100 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:43:49.108 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: audit 2024-09-13T22:43:48.024197+0000 mon.b (mon.1) 121 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: cluster 2024-09-13T22:43:48.313928+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: cluster 2024-09-13T22:43:48.316563+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: audit 2024-09-13T22:43:48.359494+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.178:0/3101595859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[29889]: cluster 2024-09-13T22:43:48.538177+0000 mgr.y (mgr.14613) 230 : cluster [DBG] pgmap v223: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: cluster 2024-09-13T22:43:46.966319+0000 osd.6 (osd.6) 99 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: cluster 2024-09-13T22:43:46.968948+0000 osd.6 (osd.6) 100 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: audit 2024-09-13T22:43:48.024197+0000 mon.b (mon.1) 121 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:43:49.109 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: cluster 2024-09-13T22:43:48.313928+0000 osd.0 (osd.0) 141 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:43:49.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: cluster 2024-09-13T22:43:48.316563+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:43:49.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: audit 2024-09-13T22:43:48.359494+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.178:0/3101595859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:43:49.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:48 smithi178 bash[22840]: cluster 2024-09-13T22:43:48.538177+0000 mgr.y (mgr.14613) 230 : cluster [DBG] pgmap v223: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-09-13T22:43:49.123 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:49.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:49.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:49.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:49.382 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:49.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:49.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:49.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:49.637 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:49.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:49.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:49.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:49.893 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:49.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:49.927 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:43:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:43:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:43:49.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:49 smithi178 bash[29889]: cluster 2024-09-13T22:43:49.152853+0000 osd.4 (osd.4) 115 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:43:49.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:49 smithi178 bash[29889]: cluster 2024-09-13T22:43:49.155632+0000 osd.4 (osd.4) 116 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:43:49.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:49 smithi178 bash[22840]: cluster 2024-09-13T22:43:49.152853+0000 osd.4 (osd.4) 115 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:43:49.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:49 smithi178 bash[22840]: cluster 2024-09-13T22:43:49.155632+0000 osd.4 (osd.4) 116 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:43:49.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:50.014 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:50.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:50.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:49 smithi192 bash[24608]: cluster 2024-09-13T22:43:49.152853+0000 osd.4 (osd.4) 115 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:43:50.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:49 smithi192 bash[24608]: cluster 2024-09-13T22:43:49.155632+0000 osd.4 (osd.4) 116 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:43:50.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:50.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:50.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:50.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:50.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:50.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:50.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:50.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:50.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:50.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:50.786 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:50.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T22:43:50.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:50.941 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:50 smithi192 bash[24608]: cluster 2024-09-13T22:43:49.662157+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.b scrub starts 2024-09-13T22:43:50.941 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:50 smithi192 bash[24608]: cluster 2024-09-13T22:43:49.663027+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.b scrub ok 2024-09-13T22:43:50.941 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:50 smithi192 bash[24608]: cluster 2024-09-13T22:43:49.991900+0000 osd.5 (osd.5) 134 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:43:50.941 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:50 smithi192 bash[24608]: cluster 2024-09-13T22:43:50.053237+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:43:50.941 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:50 smithi192 bash[24608]: cluster 2024-09-13T22:43:50.538876+0000 mgr.y (mgr.14613) 231 : cluster [DBG] pgmap v224: 148 pgs: 148 active+clean; 3.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 958 op/s 2024-09-13T22:43:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:50.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:51.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.064 INFO:tasks.thrashosds.thrasher:j is {'epoch': 157, 'fsid': '03ae43e2-721f-11ef-bceb-c7b262605968', 'created': '2024-09-13T22:27:31.500176+0000', 'modified': '2024-09-13T22:43:42.641081+0000', 'last_up_change': '2024-09-13T22:37:07.372146+0000', 'last_in_change': '2024-09-13T22:36:38.469358+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T22:32:40.780155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T22:37:39.227580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T22:37:41.229251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-09-13T22:37:41.624329+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 85, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T22:37:43.199974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T22:37:45.423695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T22:40:53.509919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '394f2664-bba1-4b15-ace8-78f70f970cac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6801', 'nonce': 1489719354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6802', 'nonce': 1489719354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6804', 'nonce': 1489719354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6803', 'nonce': 1489719354}]}, 'public_addr': '172.21.15.178:6801/1489719354', 'cluster_addr': '172.21.15.178:6802/1489719354', 'heartbeat_back_addr': '172.21.15.178:6804/1489719354', 'heartbeat_front_addr': '172.21.15.178:6803/1489719354', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e65c08b-6431-4849-a6af-2da9cb1efdfa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6805', 'nonce': 4187084287}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6806', 'nonce': 4187084287}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6808', 'nonce': 4187084287}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6807', 'nonce': 4187084287}]}, 'public_addr': '172.21.15.178:6805/4187084287', 'cluster_addr': '172.21.15.178:6806/4187084287', 'heartbeat_back_addr': '172.21.15.178:6808/4187084287', 'heartbeat_front_addr': '172.21.15.178:6807/4187084287', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '40a79075-5102-4f6c-b41a-255102064624', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6809', 'nonce': 2303802730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6810', 'nonce': 2303802730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6812', 'nonce': 2303802730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6811', 'nonce': 2303802730}]}, 'public_addr': '172.21.15.178:6809/2303802730', 'cluster_addr': '172.21.15.178:6810/2303802730', 'heartbeat_back_addr': '172.21.15.178:6812/2303802730', 'heartbeat_front_addr': '172.21.15.178:6811/2303802730', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3962c796-6f97-4cc6-a186-4b6ef58d67ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6813', 'nonce': 320675660}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6814', 'nonce': 320675660}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6816', 'nonce': 320675660}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6815', 'nonce': 320675660}]}, 'public_addr': '172.21.15.178:6813/320675660', 'cluster_addr': '172.21.15.178:6814/320675660', 'heartbeat_back_addr': '172.21.15.178:6816/320675660', 'heartbeat_front_addr': '172.21.15.178:6815/320675660', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b89db2f-9e81-40b1-89bd-8de1dbff133b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 34, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 3593670527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 3593670527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 3593670527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 3593670527}]}, 'public_addr': '172.21.15.192:6800/3593670527', 'cluster_addr': '172.21.15.192:6801/3593670527', 'heartbeat_back_addr': '172.21.15.192:6803/3593670527', 'heartbeat_front_addr': '172.21.15.192:6802/3593670527', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'efa849a2-d789-4fdd-acc7-04f2c1bebfa8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 3271425338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 3271425338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 3271425338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 3271425338}]}, 'public_addr': '172.21.15.192:6804/3271425338', 'cluster_addr': '172.21.15.192:6805/3271425338', 'heartbeat_back_addr': '172.21.15.192:6807/3271425338', 'heartbeat_front_addr': '172.21.15.192:6806/3271425338', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '54a26ae9-bf31-49cf-a924-901eed4a0c98', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 684205162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 684205162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 684205162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 684205162}]}, 'public_addr': '172.21.15.192:6808/684205162', 'cluster_addr': '172.21.15.192:6809/684205162', 'heartbeat_back_addr': '172.21.15.192:6811/684205162', 'heartbeat_front_addr': '172.21.15.192:6810/684205162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '13e146db-61ac-4fe6-8f26-cbf8d1623a52', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 1303971738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1303971738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1303971738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 1303971738}]}, 'public_addr': '172.21.15.192:6812/1303971738', 'cluster_addr': '172.21.15.192:6813/1303971738', 'heartbeat_back_addr': '172.21.15.192:6815/1303971738', 'heartbeat_front_addr': '172.21.15.192:6814/1303971738', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:21.712932+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:42:38.222982+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:42:53.803761+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:42.507720+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:38.108862+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:42:42.638925+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:40.681119+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:25.862934+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.178:0/268210503': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:6800/2039160385': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/669877409': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/129655740': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/3632788039': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:6800/388635774': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3265105147': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2959920382': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/2494442689': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:6800/2676020773': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2675812306': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/1834442338': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2835351236': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3836928220': '2024-09-14T22:38:32.396576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T22:43:51.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:43:51.084 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:51.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[29889]: cluster 2024-09-13T22:43:49.662157+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.b scrub starts 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[29889]: cluster 2024-09-13T22:43:49.663027+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.b scrub ok 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[29889]: cluster 2024-09-13T22:43:49.991900+0000 osd.5 (osd.5) 134 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[29889]: cluster 2024-09-13T22:43:50.053237+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[29889]: cluster 2024-09-13T22:43:50.538876+0000 mgr.y (mgr.14613) 231 : cluster [DBG] pgmap v224: 148 pgs: 148 active+clean; 3.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 958 op/s 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[22840]: cluster 2024-09-13T22:43:49.662157+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.b scrub starts 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[22840]: cluster 2024-09-13T22:43:49.663027+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.b scrub ok 2024-09-13T22:43:51.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[22840]: cluster 2024-09-13T22:43:49.991900+0000 osd.5 (osd.5) 134 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:43:51.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[22840]: cluster 2024-09-13T22:43:50.053237+0000 osd.5 (osd.5) 135 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:43:51.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:50 smithi178 bash[22840]: cluster 2024-09-13T22:43:50.538876+0000 mgr.y (mgr.14613) 231 : cluster [DBG] pgmap v224: 148 pgs: 148 active+clean; 3.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 958 op/s 2024-09-13T22:43:51.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:51.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:51.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.358 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:51.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:51.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:51.530 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:43:51.530 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:43:51.561 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:43:51.561 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:43:51.589 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:43:51.589 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:43:51.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.627 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:43:51.627 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:43:51.629 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:51.659 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:43:51.659 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:43:51.683 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:43:51.683 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:43:51.718 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:43:51.718 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:43:51.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.756 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:51.757 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:43:51.757 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:43:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:51.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:51.887 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:51.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:51.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:51 smithi192 bash[24608]: cluster 2024-09-13T22:43:50.721554+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:43:51.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:51 smithi192 bash[24608]: cluster 2024-09-13T22:43:50.727108+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:43:51.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:51 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.338388+0000 osd.0 (osd.0) 143 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:43:51.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:51 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.349854+0000 osd.0 (osd.0) 144 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:43:51.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:51 smithi192 bash[24608]: audit 2024-09-13T22:43:51.504744+0000 mon.a (mon.0) 1026 : audit [DBG] from='client.? v1:172.21.15.178:0/1648574193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:51.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:52.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:52.113 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[29889]: cluster 2024-09-13T22:43:50.721554+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[29889]: cluster 2024-09-13T22:43:50.727108+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.338388+0000 osd.0 (osd.0) 143 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.349854+0000 osd.0 (osd.0) 144 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[29889]: audit 2024-09-13T22:43:51.504744+0000 mon.a (mon.0) 1026 : audit [DBG] from='client.? v1:172.21.15.178:0/1648574193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[22840]: cluster 2024-09-13T22:43:50.721554+0000 osd.3 (osd.3) 154 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:43:52.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[22840]: cluster 2024-09-13T22:43:50.727108+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:43:52.124 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.338388+0000 osd.0 (osd.0) 143 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:43:52.124 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.349854+0000 osd.0 (osd.0) 144 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:43:52.124 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:51 smithi178 bash[22840]: audit 2024-09-13T22:43:51.504744+0000 mon.a (mon.0) 1026 : audit [DBG] from='client.? v1:172.21.15.178:0/1648574193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:43:52.141 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:52.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:52.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:52.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:52.399 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:52.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:52.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:52.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:52.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:52.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:52.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:52.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:52.885 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:43:52.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:52.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:53.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:50.939956+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.7 scrub starts 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.125221+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.138645+0000 osd.1 (osd.1) 131 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.497496+0000 osd.1 (osd.1) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.497565+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.624707+0000 osd.7 (osd.7) 131 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.625637+0000 osd.7 (osd.7) 132 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.772025+0000 osd.3 (osd.3) 156 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.778543+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.803810+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.7 scrub ok 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:51.864927+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-13T22:43:53.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: audit 2024-09-13T22:43:51.955840+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.178:0/1497917127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:53.036 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: audit 2024-09-13T22:43:51.956512+0000 mon.a (mon.0) 1028 : audit [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:53.036 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:52 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.539766+0000 mgr.y (mgr.14613) 232 : cluster [DBG] pgmap v226: 148 pgs: 148 active+clean; 3.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 863 op/s 2024-09-13T22:43:53.047 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:53.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:50.939956+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.7 scrub starts 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.125221+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.138645+0000 osd.1 (osd.1) 131 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.497496+0000 osd.1 (osd.1) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.497565+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.624707+0000 osd.7 (osd.7) 131 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.625637+0000 osd.7 (osd.7) 132 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.772025+0000 osd.3 (osd.3) 156 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.778543+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:43:53.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.803810+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.7 scrub ok 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:51.864927+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: audit 2024-09-13T22:43:51.955840+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.178:0/1497917127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: audit 2024-09-13T22:43:51.956512+0000 mon.a (mon.0) 1028 : audit [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.539766+0000 mgr.y (mgr.14613) 232 : cluster [DBG] pgmap v226: 148 pgs: 148 active+clean; 3.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 863 op/s 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:50.939956+0000 osd.6 (osd.6) 101 : cluster [DBG] 7.7 scrub starts 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.125221+0000 osd.1 (osd.1) 130 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.138645+0000 osd.1 (osd.1) 131 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.497496+0000 osd.1 (osd.1) 132 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.497565+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.624707+0000 osd.7 (osd.7) 131 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:43:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.625637+0000 osd.7 (osd.7) 132 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.772025+0000 osd.3 (osd.3) 156 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.778543+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.803810+0000 osd.6 (osd.6) 102 : cluster [DBG] 7.7 scrub ok 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:51.864927+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: audit 2024-09-13T22:43:51.955840+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.178:0/1497917127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: audit 2024-09-13T22:43:51.956512+0000 mon.a (mon.0) 1028 : audit [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:43:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:52 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.539766+0000 mgr.y (mgr.14613) 232 : cluster [DBG] pgmap v226: 148 pgs: 148 active+clean; 3.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 863 op/s 2024-09-13T22:43:53.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:53.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:53.322 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:53.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:53.452 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:53.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:53.603 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:53.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:53.746 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:53.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:53.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:53.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:53.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:54.003 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:54.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:54.137 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:54.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.167626+0000 osd.1 (osd.1) 134 : cluster [DBG] 2.d scrub starts 2024-09-13T22:43:54.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.173500+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub ok 2024-09-13T22:43:54.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.849647+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.c scrub starts 2024-09-13T22:43:54.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.851694+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.c scrub ok 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.856739+0000 mon.a (mon.0) 1029 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.856782+0000 mon.a (mon.0) 1030 : cluster [INF] Cluster is now healthy 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: audit 2024-09-13T22:43:52.872667+0000 mon.a (mon.0) 1031 : audit [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.880329+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.994894+0000 osd.5 (osd.5) 136 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.994940+0000 osd.5 (osd.5) 137 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:52.998354+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.003464+0000 osd.5 (osd.5) 139 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.324646+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.335561+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.476940+0000 osd.2 (osd.2) 105 : cluster [DBG] 2.e scrub starts 2024-09-13T22:43:54.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.485394+0000 osd.2 (osd.2) 106 : cluster [DBG] 2.e scrub ok 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.167626+0000 osd.1 (osd.1) 134 : cluster [DBG] 2.d scrub starts 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.173500+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub ok 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.849647+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.c scrub starts 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.851694+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.c scrub ok 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.856739+0000 mon.a (mon.0) 1029 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.856782+0000 mon.a (mon.0) 1030 : cluster [INF] Cluster is now healthy 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: audit 2024-09-13T22:43:52.872667+0000 mon.a (mon.0) 1031 : audit [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.880329+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.994894+0000 osd.5 (osd.5) 136 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.994940+0000 osd.5 (osd.5) 137 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:52.998354+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.003464+0000 osd.5 (osd.5) 139 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T22:43:54.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.324646+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:43:54.141 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.335561+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:43:54.141 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.476940+0000 osd.2 (osd.2) 105 : cluster [DBG] 2.e scrub starts 2024-09-13T22:43:54.141 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:53 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.485394+0000 osd.2 (osd.2) 106 : cluster [DBG] 2.e scrub ok 2024-09-13T22:43:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.167626+0000 osd.1 (osd.1) 134 : cluster [DBG] 2.d scrub starts 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.173500+0000 osd.1 (osd.1) 135 : cluster [DBG] 2.d scrub ok 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.849647+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.c scrub starts 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.851694+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.c scrub ok 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.856739+0000 mon.a (mon.0) 1029 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.856782+0000 mon.a (mon.0) 1030 : cluster [INF] Cluster is now healthy 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: audit 2024-09-13T22:43:52.872667+0000 mon.a (mon.0) 1031 : audit [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.880329+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.994894+0000 osd.5 (osd.5) 136 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:43:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.994940+0000 osd.5 (osd.5) 137 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:43:54.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:52.998354+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T22:43:54.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.003464+0000 osd.5 (osd.5) 139 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T22:43:54.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.324646+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:43:54.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.335561+0000 osd.0 (osd.0) 146 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:43:54.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.476940+0000 osd.2 (osd.2) 105 : cluster [DBG] 2.e scrub starts 2024-09-13T22:43:54.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:53 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.485394+0000 osd.2 (osd.2) 106 : cluster [DBG] 2.e scrub ok 2024-09-13T22:43:54.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:54.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:54.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:54.388 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:54.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:54.519 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:54.622 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:54.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:54.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:54.786 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:54.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:54.911 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:54.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:55.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:55.042 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:55.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:54 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.887355+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-13T22:43:55.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:54 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.970113+0000 osd.5 (osd.5) 140 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:43:55.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:54 smithi192 bash[24608]: cluster 2024-09-13T22:43:53.971317+0000 osd.5 (osd.5) 141 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:43:55.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:54 smithi192 bash[24608]: cluster 2024-09-13T22:43:54.540538+0000 mgr.y (mgr.14613) 233 : cluster [DBG] pgmap v229: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-13T22:43:55.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:55.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:43:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:43:55.156 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.887355+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.970113+0000 osd.5 (osd.5) 140 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[29889]: cluster 2024-09-13T22:43:53.971317+0000 osd.5 (osd.5) 141 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[29889]: cluster 2024-09-13T22:43:54.540538+0000 mgr.y (mgr.14613) 233 : cluster [DBG] pgmap v229: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.887355+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.970113+0000 osd.5 (osd.5) 140 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[22840]: cluster 2024-09-13T22:43:53.971317+0000 osd.5 (osd.5) 141 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:43:55.157 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:54 smithi178 bash[22840]: cluster 2024-09-13T22:43:54.540538+0000 mgr.y (mgr.14613) 233 : cluster [DBG] pgmap v229: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-13T22:43:55.175 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:55.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:55.308 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:55.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:55.449 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:55.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:43:55.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:55.577 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:55.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:55.699 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:55.711 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:43:55.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:55.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:55.932 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:55.960 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:55.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:56.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:56.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:55 smithi192 bash[24608]: cluster 2024-09-13T22:43:54.640896+0000 osd.7 (osd.7) 133 : cluster [DBG] 7.5 scrub starts 2024-09-13T22:43:56.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:55 smithi192 bash[24608]: audit 2024-09-13T22:43:55.082426+0000 mgr.y (mgr.14613) 234 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:56.093 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:56.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:55 smithi178 bash[29889]: cluster 2024-09-13T22:43:54.640896+0000 osd.7 (osd.7) 133 : cluster [DBG] 7.5 scrub starts 2024-09-13T22:43:56.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:55 smithi178 bash[29889]: audit 2024-09-13T22:43:55.082426+0000 mgr.y (mgr.14613) 234 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:56.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:55 smithi178 bash[22840]: cluster 2024-09-13T22:43:54.640896+0000 osd.7 (osd.7) 133 : cluster [DBG] 7.5 scrub starts 2024-09-13T22:43:56.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:55 smithi178 bash[22840]: audit 2024-09-13T22:43:55.082426+0000 mgr.y (mgr.14613) 234 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:43:56.198 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:56.224 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:56.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:43:56.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:56.353 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:56.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:56.498 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:56.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:56.631 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:56.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:56.758 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:56.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:56.884 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:56.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:56.971 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:43:56.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:57.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:43:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:43:57.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:57.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:56 smithi192 bash[24608]: cluster 2024-09-13T22:43:55.279578+0000 osd.7 (osd.7) 134 : cluster [DBG] 7.5 scrub ok 2024-09-13T22:43:57.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:56 smithi192 bash[24608]: cluster 2024-09-13T22:43:55.906226+0000 osd.6 (osd.6) 103 : cluster [DBG] 7.d scrub starts 2024-09-13T22:43:57.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:56 smithi192 bash[24608]: cluster 2024-09-13T22:43:56.541492+0000 mgr.y (mgr.14613) 235 : cluster [DBG] pgmap v230: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s 2024-09-13T22:43:57.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:56 smithi178 bash[29889]: cluster 2024-09-13T22:43:55.279578+0000 osd.7 (osd.7) 134 : cluster [DBG] 7.5 scrub ok 2024-09-13T22:43:57.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:56 smithi178 bash[29889]: cluster 2024-09-13T22:43:55.906226+0000 osd.6 (osd.6) 103 : cluster [DBG] 7.d scrub starts 2024-09-13T22:43:57.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:56 smithi178 bash[29889]: cluster 2024-09-13T22:43:56.541492+0000 mgr.y (mgr.14613) 235 : cluster [DBG] pgmap v230: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s 2024-09-13T22:43:57.178 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:57.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:56 smithi178 bash[22840]: cluster 2024-09-13T22:43:55.279578+0000 osd.7 (osd.7) 134 : cluster [DBG] 7.5 scrub ok 2024-09-13T22:43:57.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:56 smithi178 bash[22840]: cluster 2024-09-13T22:43:55.906226+0000 osd.6 (osd.6) 103 : cluster [DBG] 7.d scrub starts 2024-09-13T22:43:57.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:56 smithi178 bash[22840]: cluster 2024-09-13T22:43:56.541492+0000 mgr.y (mgr.14613) 235 : cluster [DBG] pgmap v230: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s 2024-09-13T22:43:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:57.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:57.307 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:57.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:57.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:57.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:57.561 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:57.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:57.687 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:57.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:57.823 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:57.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:56.191124+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:56.198525+0000 osd.1 (osd.1) 137 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:56.734596+0000 osd.6 (osd.6) 104 : cluster [DBG] 7.d scrub ok 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.004794+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.009746+0000 osd.5 (osd.5) 143 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.415256+0000 osd.0 (osd.0) 147 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.430394+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.436429+0000 osd.0 (osd.0) 148 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:43:57.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:57 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.436429+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:43:57.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:43:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:57.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:43:58.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:58.076 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:58.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:56.191124+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:43:58.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:56.198525+0000 osd.1 (osd.1) 137 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:56.734596+0000 osd.6 (osd.6) 104 : cluster [DBG] 7.d scrub ok 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.004794+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.009746+0000 osd.5 (osd.5) 143 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.415256+0000 osd.0 (osd.0) 147 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.430394+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.436429+0000 osd.0 (osd.0) 148 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.436429+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:56.191124+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:43:58.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:56.198525+0000 osd.1 (osd.1) 137 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:56.734596+0000 osd.6 (osd.6) 104 : cluster [DBG] 7.d scrub ok 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.004794+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.009746+0000 osd.5 (osd.5) 143 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.415256+0000 osd.0 (osd.0) 147 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.430394+0000 osd.2 (osd.2) 107 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.436429+0000 osd.0 (osd.0) 148 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:43:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:57 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.436429+0000 osd.2 (osd.2) 108 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:43:58.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:58.198 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:58.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:58.323 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:58.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:58.450 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:58.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:58.579 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:58.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:58.720 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:58.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:58.856 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:58.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:58.981 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:43:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:58.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:43:58.986 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:58 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.811960+0000 osd.3 (osd.3) 160 : cluster [DBG] 3.b scrub starts 2024-09-13T22:43:58.986 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:58 smithi178 bash[22840]: cluster 2024-09-13T22:43:57.825396+0000 osd.3 (osd.3) 161 : cluster [DBG] 3.b scrub ok 2024-09-13T22:43:58.986 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:58 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.545084+0000 mgr.y (mgr.14613) 236 : cluster [DBG] pgmap v231: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 765 B/s rd, 0 B/s wr, 945 op/s 2024-09-13T22:43:58.986 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:58 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.811960+0000 osd.3 (osd.3) 160 : cluster [DBG] 3.b scrub starts 2024-09-13T22:43:58.986 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:58 smithi178 bash[29889]: cluster 2024-09-13T22:43:57.825396+0000 osd.3 (osd.3) 161 : cluster [DBG] 3.b scrub ok 2024-09-13T22:43:58.987 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:58 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.545084+0000 mgr.y (mgr.14613) 236 : cluster [DBG] pgmap v231: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 765 B/s rd, 0 B/s wr, 945 op/s 2024-09-13T22:43:59.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:59.108 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:59.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:59.244 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:43:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:43:59.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:59.378 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:59.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:58 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.811960+0000 osd.3 (osd.3) 160 : cluster [DBG] 3.b scrub starts 2024-09-13T22:43:59.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:58 smithi192 bash[24608]: cluster 2024-09-13T22:43:57.825396+0000 osd.3 (osd.3) 161 : cluster [DBG] 3.b scrub ok 2024-09-13T22:43:59.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:58 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.545084+0000 mgr.y (mgr.14613) 236 : cluster [DBG] pgmap v231: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 765 B/s rd, 0 B/s wr, 945 op/s 2024-09-13T22:43:59.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:59.502 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:43:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:43:59.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:59.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:43:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:43:59.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:59.761 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:43:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:43:59.867 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:59.893 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:43:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:43:59.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:43:59.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:00.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:00.052 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:00.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:00.130 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.129449+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:44:00.130 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.133002+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.777617+0000 osd.3 (osd.3) 162 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.778830+0000 osd.3 (osd.3) 163 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.886618+0000 osd.6 (osd.6) 105 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[22840]: cluster 2024-09-13T22:43:58.889782+0000 osd.6 (osd.6) 106 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:43:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.129449+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.133002+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:44:00.131 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.777617+0000 osd.3 (osd.3) 162 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:44:00.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.778830+0000 osd.3 (osd.3) 163 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:44:00.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.886618+0000 osd.6 (osd.6) 105 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:44:00.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:43:59 smithi178 bash[29889]: cluster 2024-09-13T22:43:58.889782+0000 osd.6 (osd.6) 106 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:44:00.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:00.206 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:59 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.129449+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:44:00.206 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:59 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.133002+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:44:00.207 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:59 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.777617+0000 osd.3 (osd.3) 162 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:44:00.207 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:59 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.778830+0000 osd.3 (osd.3) 163 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:44:00.207 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:59 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.886618+0000 osd.6 (osd.6) 105 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:44:00.207 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:43:59 smithi192 bash[24608]: cluster 2024-09-13T22:43:58.889782+0000 osd.6 (osd.6) 106 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:44:00.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:00.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:172673 7.1 7:83537c54:::benchmark_data_smithi178_64772_object66920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:00.363283+0000", 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.010250277, 2024-09-13T22:44:00.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010271967999999999, 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172673 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363283+0000", 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363283+0000", 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363285+0000", 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.951e-06 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363289+0000", 2024-09-13T22:44:00.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0879999999999997e-06 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363298+0000", 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4299999999999995e-06 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363303+0000", 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1070000000000004e-06 2024-09-13T22:44:00.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363352+0000", 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8414000000000001e-05 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363470+0000", 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118424 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:44:00.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.363598+0000", 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012735900000000001 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.365628+0000", 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002030441 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.366519+0000", 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00089117199999999997 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.375 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:172675 7.1 7:8268b089:::benchmark_data_smithi178_64772_object66922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:00.369862+0000", 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0036707369999999999, 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037731790000000002, 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:44:00.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172675 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369862+0000", 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369862+0000", 2024-09-13T22:44:00.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369864+0000", 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.821e-06 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369870+0000", 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369880+0000", 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0390999999999999e-05 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369887+0000", 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.861e-06 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.369953+0000", 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6655000000000005e-05 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.370056+0000", 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010223200000000001 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.370130+0000", 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4752000000000005e-05 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.372074+0000", 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019432690000000001 2024-09-13T22:44:00.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.372849+0000", 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00077499199999999996 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:172666 7.1 7:8dd8548a:::benchmark_data_smithi178_64772_object66913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:00.358273+0000", 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015260409000000001, 2024-09-13T22:44:00.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015430900000000001, 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172666 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358273+0000", 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:44:00.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358273+0000", 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358275+0000", 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3029999999999998e-06 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358281+0000", 2024-09-13T22:44:00.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2999999999999998e-06 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358293+0000", 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1228e-05 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358300+0000", 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8800000000000008e-06 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358353+0000", 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2077000000000003e-05 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358472+0000", 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119258 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.358593+0000", 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012135500000000001 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.360710+0000", 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021170579999999998 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.360898+0000", 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018787099999999999 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.386 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:44:00.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:44:00.387 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-09-13T22:44:00.387 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 3 2024-09-13T22:44:00.387 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:44:00.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:00.424 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:00.451 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:44:00.452 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:44:00.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:00.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:00.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:00.671 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:00.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:00.797 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:00.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:00.940 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:00.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:01.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:01.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[29889]: cluster 2024-09-13T22:43:59.657060+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.8 deep-scrub starts 2024-09-13T22:44:01.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[29889]: cluster 2024-09-13T22:43:59.658167+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.8 deep-scrub ok 2024-09-13T22:44:01.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[29889]: audit 2024-09-13T22:44:00.437666+0000 mgr.y (mgr.14613) 237 : audit [DBG] from='client.24907 v1:172.21.15.178:0/62737253' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:44:01.055 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[29889]: cluster 2024-09-13T22:44:00.545892+0000 mgr.y (mgr.14613) 238 : cluster [DBG] pgmap v232: 148 pgs: 148 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-13T22:44:01.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[22840]: cluster 2024-09-13T22:43:59.657060+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.8 deep-scrub starts 2024-09-13T22:44:01.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[22840]: cluster 2024-09-13T22:43:59.658167+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.8 deep-scrub ok 2024-09-13T22:44:01.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[22840]: audit 2024-09-13T22:44:00.437666+0000 mgr.y (mgr.14613) 237 : audit [DBG] from='client.24907 v1:172.21.15.178:0/62737253' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:44:01.056 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:00 smithi178 bash[22840]: cluster 2024-09-13T22:44:00.545892+0000 mgr.y (mgr.14613) 238 : cluster [DBG] pgmap v232: 148 pgs: 148 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-13T22:44:01.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:01.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:01.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:00 smithi192 bash[24608]: cluster 2024-09-13T22:43:59.657060+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.8 deep-scrub starts 2024-09-13T22:44:01.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:00 smithi192 bash[24608]: cluster 2024-09-13T22:43:59.658167+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.8 deep-scrub ok 2024-09-13T22:44:01.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:00 smithi192 bash[24608]: audit 2024-09-13T22:44:00.437666+0000 mgr.y (mgr.14613) 237 : audit [DBG] from='client.24907 v1:172.21.15.178:0/62737253' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:44:01.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:00.545892+0000 mgr.y (mgr.14613) 238 : cluster [DBG] pgmap v232: 148 pgs: 148 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-13T22:44:01.200 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:01.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T22:44:01.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:01.338 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:01.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:01.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:01.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:01.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:01.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:01.726 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:01.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:01.858 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:01.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:01.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:01.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:02.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:02.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:00.302499+0000 osd.4 (osd.4) 117 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:00.305488+0000 osd.4 (osd.4) 118 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:00.690508+0000 osd.7 (osd.7) 137 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:00.697330+0000 osd.7 (osd.7) 138 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:01.004102+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:01.009748+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:01.359458+0000 osd.0 (osd.0) 149 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:01.380162+0000 osd.0 (osd.0) 150 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:00.302499+0000 osd.4 (osd.4) 117 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:00.305488+0000 osd.4 (osd.4) 118 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:44:02.096 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:00.690508+0000 osd.7 (osd.7) 137 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:44:02.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:00.697330+0000 osd.7 (osd.7) 138 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:44:02.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:01.004102+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:44:02.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:01.009748+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:44:02.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:01.359458+0000 osd.0 (osd.0) 149 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:44:02.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:01.380162+0000 osd.0 (osd.0) 150 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:44:02.106 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:02.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:02.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:00.302499+0000 osd.4 (osd.4) 117 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:44:02.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:00.305488+0000 osd.4 (osd.4) 118 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:44:02.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:00.690508+0000 osd.7 (osd.7) 137 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:44:02.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:00.697330+0000 osd.7 (osd.7) 138 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:44:02.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:01.004102+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:44:02.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:01.009748+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:44:02.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:01.359458+0000 osd.0 (osd.0) 149 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:44:02.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:01 smithi192 bash[24608]: cluster 2024-09-13T22:44:01.380162+0000 osd.0 (osd.0) 150 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:44:02.232 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:02.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:02.356 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:02.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:02.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:02.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:02.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:02.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:02.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:02.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:02.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:02.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:02.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:03.008 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:03.010 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:02 smithi192 bash[24608]: cluster 2024-09-13T22:44:01.731180+0000 osd.2 (osd.2) 109 : cluster [DBG] 2.0 deep-scrub starts 2024-09-13T22:44:03.010 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:02 smithi192 bash[24608]: cluster 2024-09-13T22:44:01.750354+0000 osd.2 (osd.2) 110 : cluster [DBG] 2.0 deep-scrub ok 2024-09-13T22:44:03.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:02 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.040279+0000 osd.5 (osd.5) 146 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:44:03.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:02 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.041769+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:44:03.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:02 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.353625+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.9 deep-scrub starts 2024-09-13T22:44:03.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:02 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.546242+0000 mgr.y (mgr.14613) 239 : cluster [DBG] pgmap v233: 148 pgs: 148 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s 2024-09-13T22:44:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:03.032 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-13T22:43:58.544480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2823100979,"num_objects":43268,"num_object_clones":0,"num_object_copies":129804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43268,"num_whiteouts":0,"num_read":1304,"num_read_kb":1150,"num_write":169048,"num_write_kb":6770462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90434,"ondisk_log_size":90434,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9113544,"kb_used_data":8341944,"kb_used_omap":12,"kb_used_meta":771315,"kb_avail":740847672,"statfs":{"total":767960285184,"available":758628016128,"internally_reserved":0,"allocated":8542150656,"data_stored":8538801209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":789827120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,5,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-414056448,"num_objects":-6318,"num_object_clones":0,"num_object_copies":-18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6318,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.687932"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899550+0000","last_change":"2024-09-13T22:43:43.560398+0000","last_active":"2024-09-13T22:43:53.899550+0000","last_peered":"2024-09-13T22:43:53.899550+0000","last_clean":"2024-09-13T22:43:53.899550+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:43:53.899550+0000","last_undegraded":"2024-09-13T22:43:53.899550+0000","last_fullsized":"2024-09-13T22:43:53.899550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:43.560294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:43.560294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:51.769340+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895513+0000","last_change":"2024-09-13T22:43:39.277776+0000","last_active":"2024-09-13T22:43:53.895513+0000","last_peered":"2024-09-13T22:43:53.895513+0000","last_clean":"2024-09-13T22:43:53.895513+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:43:53.895513+0000","last_undegraded":"2024-09-13T22:43:53.895513+0000","last_fullsized":"2024-09-13T22:43:53.895513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:39.277676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:43:39.277676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:51.466151+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.901907+0000","last_change":"2024-09-13T22:43:51.778660+0000","last_active":"2024-09-13T22:43:53.901907+0000","last_peered":"2024-09-13T22:43:53.901907+0000","last_clean":"2024-09-13T22:43:53.901907+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:43:53.901907+0000","last_undegraded":"2024-09-13T22:43:53.901907+0000","last_fullsized":"2024-09-13T22:43:53.901907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:43:51.778550+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.778550+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:12.279052+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892810+0000","last_change":"2024-09-13T22:41:55.780046+0000","last_active":"2024-09-13T22:43:53.892810+0000","last_peered":"2024-09-13T22:43:53.892810+0000","last_clean":"2024-09-13T22:43:53.892810+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:43:53.892810+0000","last_undegraded":"2024-09-13T22:43:53.892810+0000","last_fullsized":"2024-09-13T22:43:53.892810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:41:55.779975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.902000+0000","last_change":"2024-09-13T22:43:36.511625+0000","last_active":"2024-09-13T22:43:53.902000+0000","last_peered":"2024-09-13T22:43:53.902000+0000","last_clean":"2024-09-13T22:43:53.902000+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:43:53.902000+0000","last_undegraded":"2024-09-13T22:43:53.902000+0000","last_fullsized":"2024-09-13T22:43:53.902000+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:36.511492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:36.511492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:57.138999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":226,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895199+0000","last_change":"2024-09-13T22:41:57.526625+0000","last_active":"2024-09-13T22:43:53.895199+0000","last_peered":"2024-09-13T22:43:53.895199+0000","last_clean":"2024-09-13T22:43:53.895199+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:43:53.895199+0000","last_undegraded":"2024-09-13T22:43:53.895199+0000","last_fullsized":"2024-09-13T22:43:53.895199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:41:57.526506+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:41:57.526506+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":196,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877580+0000","last_change":"2024-09-13T22:42:59.307163+0000","last_active":"2024-09-13T22:43:52.877580+0000","last_peered":"2024-09-13T22:43:52.877580+0000","last_clean":"2024-09-13T22:43:52.877580+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:43:52.877580+0000","last_undegraded":"2024-09-13T22:43:52.877580+0000","last_fullsized":"2024-09-13T22:43:52.877580+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.307067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:18.937510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892360+0000","last_change":"2024-09-13T22:41:53.897309+0000","last_active":"2024-09-13T22:43:53.892360+0000","last_peered":"2024-09-13T22:43:53.892360+0000","last_clean":"2024-09-13T22:43:53.892360+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:43:53.892360+0000","last_undegraded":"2024-09-13T22:43:53.892360+0000","last_fullsized":"2024-09-13T22:43:53.892360+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_clean_scrub_stamp":"2024-09-13T22:41:53.894021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885434+0000","last_change":"2024-09-13T22:42:35.837037+0000","last_active":"2024-09-13T22:43:53.885434+0000","last_peered":"2024-09-13T22:43:53.885434+0000","last_clean":"2024-09-13T22:43:53.885434+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:43:53.885434+0000","last_undegraded":"2024-09-13T22:43:53.885434+0000","last_fullsized":"2024-09-13T22:43:53.885434+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:35.836936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:35.836936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894349+0000","last_change":"2024-09-13T22:42:39.154087+0000","last_active":"2024-09-13T22:43:53.894349+0000","last_peered":"2024-09-13T22:43:53.894349+0000","last_clean":"2024-09-13T22:43:53.894349+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:43:53.894349+0000","last_undegraded":"2024-09-13T22:43:53.894349+0000","last_fullsized":"2024-09-13T22:43:53.894349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:42:39.153983+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.153983+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.884015+0000","last_change":"2024-09-13T22:41:49.716084+0000","last_active":"2024-09-13T22:43:53.884015+0000","last_peered":"2024-09-13T22:43:53.884015+0000","last_clean":"2024-09-13T22:43:53.884015+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:43:53.884015+0000","last_undegraded":"2024-09-13T22:43:53.884015+0000","last_fullsized":"2024-09-13T22:43:53.884015+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:49.715971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:41:49.715971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894118+0000","last_change":"2024-09-13T22:43:45.042148+0000","last_active":"2024-09-13T22:43:53.894118+0000","last_peered":"2024-09-13T22:43:53.894118+0000","last_clean":"2024-09-13T22:43:53.894118+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:43:53.894118+0000","last_undegraded":"2024-09-13T22:43:53.894118+0000","last_fullsized":"2024-09-13T22:43:53.894118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:45.042042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:45.042042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:59.385632+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.884262+0000","last_change":"2024-09-13T22:41:57.761398+0000","last_active":"2024-09-13T22:43:53.884262+0000","last_peered":"2024-09-13T22:43:53.884262+0000","last_clean":"2024-09-13T22:43:53.884262+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:43:53.884262+0000","last_undegraded":"2024-09-13T22:43:53.884262+0000","last_fullsized":"2024-09-13T22:43:53.884262+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:57.761322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:57.761322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":224,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894731+0000","last_change":"2024-09-13T22:42:29.188915+0000","last_active":"2024-09-13T22:43:53.894731+0000","last_peered":"2024-09-13T22:43:53.894731+0000","last_clean":"2024-09-13T22:43:53.894731+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:43:53.894731+0000","last_undegraded":"2024-09-13T22:43:53.894731+0000","last_fullsized":"2024-09-13T22:43:53.894731+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:42:29.188836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:29.188836+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895171+0000","last_change":"2024-09-13T22:42:37.031507+0000","last_active":"2024-09-13T22:43:53.895171+0000","last_peered":"2024-09-13T22:43:53.895171+0000","last_clean":"2024-09-13T22:43:53.895171+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:43:53.895171+0000","last_undegraded":"2024-09-13T22:43:53.895171+0000","last_fullsized":"2024-09-13T22:43:53.895171+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:37.031329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.031329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.995792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891941+0000","last_change":"2024-09-13T22:43:53.335673+0000","last_active":"2024-09-13T22:43:53.891941+0000","last_peered":"2024-09-13T22:43:53.891941+0000","last_clean":"2024-09-13T22:43:53.891941+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:43:53.891941+0000","last_undegraded":"2024-09-13T22:43:53.891941+0000","last_fullsized":"2024-09-13T22:43:53.891941+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.335567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.335567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:12.682488+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"159'10335","reported_seq":10496,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.738423+0000","last_change":"2024-09-13T22:42:57.288156+0000","last_active":"2024-09-13T22:43:53.738423+0000","last_peered":"2024-09-13T22:43:53.738423+0000","last_clean":"2024-09-13T22:43:53.738423+0000","last_became_active":"2024-09-13T22:40:54.965149+0000","last_became_peered":"2024-09-13T22:40:54.965149+0000","last_unstale":"2024-09-13T22:43:53.738423+0000","last_undegraded":"2024-09-13T22:43:53.738423+0000","last_fullsized":"2024-09-13T22:43:53.738423+0000","mapping_epoch":116,"log_start":"126'3600","ondisk_log_start":"126'3600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7538","last_scrub_stamp":"2024-09-13T22:42:57.287895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.287895+0000","objects_scrubbed":5635,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.157500+0000","scrub_duration":2527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183173120,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"114'28","reported_seq":250,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.896951+0000","last_change":"2024-09-13T22:42:54.838293+0000","last_active":"2024-09-13T22:43:53.896951+0000","last_peered":"2024-09-13T22:43:53.896951+0000","last_clean":"2024-09-13T22:43:53.896951+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:43:53.896951+0000","last_undegraded":"2024-09-13T22:43:53.896951+0000","last_fullsized":"2024-09-13T22:43:53.896951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'28","last_scrub_stamp":"2024-09-13T22:42:54.838192+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.838192+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:00.553515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885442+0000","last_change":"2024-09-13T22:43:49.663119+0000","last_active":"2024-09-13T22:43:53.885442+0000","last_peered":"2024-09-13T22:43:53.885442+0000","last_clean":"2024-09-13T22:43:53.885442+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:43:53.885442+0000","last_undegraded":"2024-09-13T22:43:53.885442+0000","last_fullsized":"2024-09-13T22:43:53.885442+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:49.663031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:49.663031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:56.088957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":196,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877310+0000","last_change":"2024-09-13T22:42:28.929922+0000","last_active":"2024-09-13T22:43:52.877310+0000","last_peered":"2024-09-13T22:43:52.877310+0000","last_clean":"2024-09-13T22:43:52.877310+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:43:52.877310+0000","last_undegraded":"2024-09-13T22:43:52.877310+0000","last_fullsized":"2024-09-13T22:43:52.877310+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:28.929617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:28.929617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":200,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.883398+0000","last_change":"2024-09-13T22:43:45.608859+0000","last_active":"2024-09-13T22:43:52.883398+0000","last_peered":"2024-09-13T22:43:52.883398+0000","last_clean":"2024-09-13T22:43:52.883398+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:43:52.883398+0000","last_undegraded":"2024-09-13T22:43:52.883398+0000","last_fullsized":"2024-09-13T22:43:52.883398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:45.608685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:45.608685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:45.687315+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"160'10335","reported_seq":10504,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.993314+0000","last_change":"2024-09-13T22:42:48.154144+0000","last_active":"2024-09-13T22:43:53.993314+0000","last_peered":"2024-09-13T22:43:53.993314+0000","last_clean":"2024-09-13T22:43:53.993314+0000","last_became_active":"2024-09-13T22:40:54.957954+0000","last_became_peered":"2024-09-13T22:40:54.957954+0000","last_unstale":"2024-09-13T22:43:53.993314+0000","last_undegraded":"2024-09-13T22:43:53.993314+0000","last_fullsized":"2024-09-13T22:43:53.993314+0000","mapping_epoch":116,"log_start":"136'5100","ondisk_log_start":"136'5100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7054","last_scrub_stamp":"2024-09-13T22:42:48.154030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:48.154030+0000","objects_scrubbed":6079,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:00.734631+0000","scrub_duration":3823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177405952,"num_objects":2707,"num_object_clones":0,"num_object_copies":8121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":417344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"110'13","reported_seq":222,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899947+0000","last_change":"2024-09-13T22:41:56.802293+0000","last_active":"2024-09-13T22:43:53.899947+0000","last_peered":"2024-09-13T22:43:53.899947+0000","last_clean":"2024-09-13T22:43:53.899947+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:43:53.899947+0000","last_undegraded":"2024-09-13T22:43:53.899947+0000","last_fullsized":"2024-09-13T22:43:53.899947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-09-13T22:41:56.802209+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:41:56.802209+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.880435+0000","last_change":"2024-09-13T22:42:52.153298+0000","last_active":"2024-09-13T22:43:52.880435+0000","last_peered":"2024-09-13T22:43:52.880435+0000","last_clean":"2024-09-13T22:43:52.880435+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:43:52.880435+0000","last_undegraded":"2024-09-13T22:43:52.880435+0000","last_fullsized":"2024-09-13T22:43:52.880435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:52.153187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.153187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.767977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":281,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.883078+0000","last_change":"2024-09-13T22:42:47.690211+0000","last_active":"2024-09-13T22:43:52.883078+0000","last_peered":"2024-09-13T22:43:52.883078+0000","last_clean":"2024-09-13T22:43:52.883078+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:43:52.883078+0000","last_undegraded":"2024-09-13T22:43:52.883078+0000","last_fullsized":"2024-09-13T22:43:52.883078+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:47.690105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.690105+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.251912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892881+0000","last_change":"2024-09-13T22:43:40.080415+0000","last_active":"2024-09-13T22:43:53.892881+0000","last_peered":"2024-09-13T22:43:53.892881+0000","last_clean":"2024-09-13T22:43:53.892881+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:43:53.892881+0000","last_undegraded":"2024-09-13T22:43:53.892881+0000","last_fullsized":"2024-09-13T22:43:53.892881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.080319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.080319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:45.156525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"160'10736","reported_seq":10909,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:58.141493+0000","last_change":"2024-09-13T22:42:37.860665+0000","last_active":"2024-09-13T22:43:58.141493+0000","last_peered":"2024-09-13T22:43:58.141493+0000","last_clean":"2024-09-13T22:43:58.141493+0000","last_became_active":"2024-09-13T22:40:54.952746+0000","last_became_peered":"2024-09-13T22:40:54.952746+0000","last_unstale":"2024-09-13T22:43:58.141493+0000","last_undegraded":"2024-09-13T22:43:58.141493+0000","last_fullsized":"2024-09-13T22:43:58.141493+0000","mapping_epoch":116,"log_start":"130'4300","ondisk_log_start":"130'4300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'6674","last_scrub_stamp":"2024-09-13T22:42:37.860298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.860298+0000","objects_scrubbed":6611,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:07.575624+0000","scrub_duration":2031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10736,"num_write_kb":425088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899798+0000","last_change":"2024-09-13T22:42:28.345281+0000","last_active":"2024-09-13T22:43:53.899798+0000","last_peered":"2024-09-13T22:43:53.899798+0000","last_clean":"2024-09-13T22:43:53.899798+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:43:53.899798+0000","last_undegraded":"2024-09-13T22:43:53.899798+0000","last_fullsized":"2024-09-13T22:43:53.899798+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:42:28.345153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:28.345153+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877828+0000","last_change":"2024-09-13T22:41:52.638569+0000","last_active":"2024-09-13T22:43:52.877828+0000","last_peered":"2024-09-13T22:43:52.877828+0000","last_clean":"2024-09-13T22:43:52.877828+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:43:52.877828+0000","last_undegraded":"2024-09-13T22:43:52.877828+0000","last_fullsized":"2024-09-13T22:43:52.877828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:52.638224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:52.638224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":285,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892998+0000","last_change":"2024-09-13T22:42:50.045223+0000","last_active":"2024-09-13T22:43:53.892998+0000","last_peered":"2024-09-13T22:43:53.892998+0000","last_clean":"2024-09-13T22:43:53.892998+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:43:53.892998+0000","last_undegraded":"2024-09-13T22:43:53.892998+0000","last_fullsized":"2024-09-13T22:43:53.892998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:42:50.045100+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.045100+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.086555+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894380+0000","last_change":"2024-09-13T22:43:53.003544+0000","last_active":"2024-09-13T22:43:53.894380+0000","last_peered":"2024-09-13T22:43:53.894380+0000","last_clean":"2024-09-13T22:43:53.894380+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:43:53.894380+0000","last_undegraded":"2024-09-13T22:43:53.894380+0000","last_fullsized":"2024-09-13T22:43:53.894380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.003468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:22.190124+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"160'10766","reported_seq":10974,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:58.354270+0000","last_change":"2024-09-13T22:43:56.734827+0000","last_active":"2024-09-13T22:43:58.354270+0000","last_peered":"2024-09-13T22:43:58.354270+0000","last_clean":"2024-09-13T22:43:58.354270+0000","last_became_active":"2024-09-13T22:40:54.959205+0000","last_became_peered":"2024-09-13T22:40:54.959205+0000","last_unstale":"2024-09-13T22:43:58.354270+0000","last_undegraded":"2024-09-13T22:43:58.354270+0000","last_fullsized":"2024-09-13T22:43:58.354270+0000","mapping_epoch":116,"log_start":"130'4300","ondisk_log_start":"130'4300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10665","last_scrub_stamp":"2024-09-13T22:43:56.734606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:43:56.734606+0000","objects_scrubbed":2759,"log_size":6466,"log_dups_size":0,"ondisk_log_size":6466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:07.180934+0000","scrub_duration":839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172032024,"num_objects":2626,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10766,"num_write_kb":428481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892582+0000","last_change":"2024-09-13T22:41:54.888290+0000","last_active":"2024-09-13T22:43:53.892582+0000","last_peered":"2024-09-13T22:43:53.892582+0000","last_clean":"2024-09-13T22:43:53.892582+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:43:53.892582+0000","last_undegraded":"2024-09-13T22:43:53.892582+0000","last_fullsized":"2024-09-13T22:43:53.892582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:41:54.887968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:54.887968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885653+0000","last_change":"2024-09-13T22:41:47.671619+0000","last_active":"2024-09-13T22:43:53.885653+0000","last_peered":"2024-09-13T22:43:53.885653+0000","last_clean":"2024-09-13T22:43:53.885653+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:43:53.885653+0000","last_undegraded":"2024-09-13T22:43:53.885653+0000","last_fullsized":"2024-09-13T22:43:53.885653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:47.671515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:47.671515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894992+0000","last_change":"2024-09-13T22:43:40.973506+0000","last_active":"2024-09-13T22:43:53.894992+0000","last_peered":"2024-09-13T22:43:53.894992+0000","last_clean":"2024-09-13T22:43:53.894992+0000","last_became_active":"2024-09-13T22:37:44.888267+0000","last_became_peered":"2024-09-13T22:37:44.888267+0000","last_unstale":"2024-09-13T22:43:53.894992+0000","last_undegraded":"2024-09-13T22:43:53.894992+0000","last_fullsized":"2024-09-13T22:43:53.894992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.973397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.973397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:07.382987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.902448+0000","last_change":"2024-09-13T22:43:52.851848+0000","last_active":"2024-09-13T22:43:53.902448+0000","last_peered":"2024-09-13T22:43:53.902448+0000","last_clean":"2024-09-13T22:43:53.902448+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:43:53.902448+0000","last_undegraded":"2024-09-13T22:43:53.902448+0000","last_fullsized":"2024-09-13T22:43:53.902448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:52.851702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:52.851702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:53.282822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"160'10343","reported_seq":10513,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:54.450347+0000","last_change":"2024-09-13T22:42:42.864731+0000","last_active":"2024-09-13T22:43:54.450347+0000","last_peered":"2024-09-13T22:43:54.450347+0000","last_clean":"2024-09-13T22:43:54.450347+0000","last_became_active":"2024-09-13T22:40:54.961774+0000","last_became_peered":"2024-09-13T22:40:54.961774+0000","last_unstale":"2024-09-13T22:43:54.450347+0000","last_undegraded":"2024-09-13T22:43:54.450347+0000","last_fullsized":"2024-09-13T22:43:54.450347+0000","mapping_epoch":116,"log_start":"136'5200","ondisk_log_start":"136'5200","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6785","last_scrub_stamp":"2024-09-13T22:42:42.864469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:42.864469+0000","objects_scrubbed":6365,"log_size":5143,"log_dups_size":0,"ondisk_log_size":5143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.071905+0000","scrub_duration":3367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":8205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.893253+0000","last_change":"2024-09-13T22:42:40.311926+0000","last_active":"2024-09-13T22:43:53.893253+0000","last_peered":"2024-09-13T22:43:53.893253+0000","last_clean":"2024-09-13T22:43:53.893253+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:43:53.893253+0000","last_undegraded":"2024-09-13T22:43:53.893253+0000","last_fullsized":"2024-09-13T22:43:53.893253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:42:40.311810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.311810+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.898242+0000","last_change":"2024-09-13T22:42:49.867175+0000","last_active":"2024-09-13T22:43:53.898242+0000","last_peered":"2024-09-13T22:43:53.898242+0000","last_clean":"2024-09-13T22:43:53.898242+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:43:53.898242+0000","last_undegraded":"2024-09-13T22:43:53.898242+0000","last_fullsized":"2024-09-13T22:43:53.898242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:49.867081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.867081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.457345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":211,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892530+0000","last_change":"2024-09-13T22:42:38.889988+0000","last_active":"2024-09-13T22:43:53.892530+0000","last_peered":"2024-09-13T22:43:53.892530+0000","last_clean":"2024-09-13T22:43:53.892530+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:43:53.892530+0000","last_undegraded":"2024-09-13T22:43:53.892530+0000","last_fullsized":"2024-09-13T22:43:53.892530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:42:38.889867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:38.889867+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.145041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":284,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899423+0000","last_change":"2024-09-13T22:43:40.553900+0000","last_active":"2024-09-13T22:43:53.899423+0000","last_peered":"2024-09-13T22:43:53.899423+0000","last_clean":"2024-09-13T22:43:53.899423+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:43:53.899423+0000","last_undegraded":"2024-09-13T22:43:53.899423+0000","last_fullsized":"2024-09-13T22:43:53.899423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:40.553646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.553646+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:59.915475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":185,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.884261+0000","last_change":"2024-09-13T22:41:35.694198+0000","last_active":"2024-09-13T22:43:53.884261+0000","last_peered":"2024-09-13T22:43:53.884261+0000","last_clean":"2024-09-13T22:43:53.884261+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:43:53.884261+0000","last_undegraded":"2024-09-13T22:43:53.884261+0000","last_fullsized":"2024-09-13T22:43:53.884261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:35.694064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:35.694064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"160'10424","reported_seq":10615,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:54.409783+0000","last_change":"2024-09-13T22:43:35.741430+0000","last_active":"2024-09-13T22:43:54.409783+0000","last_peered":"2024-09-13T22:43:54.409783+0000","last_clean":"2024-09-13T22:43:54.409783+0000","last_became_active":"2024-09-13T22:40:54.961695+0000","last_became_peered":"2024-09-13T22:40:54.961695+0000","last_unstale":"2024-09-13T22:43:54.409783+0000","last_undegraded":"2024-09-13T22:43:54.409783+0000","last_fullsized":"2024-09-13T22:43:54.409783+0000","mapping_epoch":116,"log_start":"136'5300","ondisk_log_start":"136'5300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9400","last_scrub_stamp":"2024-09-13T22:43:35.741145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:43:35.741145+0000","objects_scrubbed":3820,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:37.152954+0000","scrub_duration":1359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180486144,"num_objects":2754,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":421696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":234,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.908196+0000","last_change":"2024-09-13T22:42:53.788869+0000","last_active":"2024-09-13T22:43:53.908196+0000","last_peered":"2024-09-13T22:43:53.908196+0000","last_clean":"2024-09-13T22:43:53.908196+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:43:53.908196+0000","last_undegraded":"2024-09-13T22:43:53.908196+0000","last_fullsized":"2024-09-13T22:43:53.908196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:42:53.788721+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.788721+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.693042+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":205,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877983+0000","last_change":"2024-09-13T22:42:46.200128+0000","last_active":"2024-09-13T22:43:52.877983+0000","last_peered":"2024-09-13T22:43:52.877983+0000","last_clean":"2024-09-13T22:43:52.877983+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:43:52.877983+0000","last_undegraded":"2024-09-13T22:43:52.877983+0000","last_fullsized":"2024-09-13T22:43:52.877983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:46.200054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.200054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":566,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899572+0000","last_change":"2024-09-13T22:43:38.560388+0000","last_active":"2024-09-13T22:43:53.899572+0000","last_peered":"2024-09-13T22:43:53.899572+0000","last_clean":"2024-09-13T22:43:53.899572+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:43:53.899572+0000","last_undegraded":"2024-09-13T22:43:53.899572+0000","last_fullsized":"2024-09-13T22:43:53.899572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:43:38.560255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:38.560255+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:06.695495+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":199,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892290+0000","last_change":"2024-09-13T22:41:36.600260+0000","last_active":"2024-09-13T22:43:53.892290+0000","last_peered":"2024-09-13T22:43:53.892290+0000","last_clean":"2024-09-13T22:43:53.892290+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:43:53.892290+0000","last_undegraded":"2024-09-13T22:43:53.892290+0000","last_fullsized":"2024-09-13T22:43:53.892290+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:36.600156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:36.600156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892205+0000","last_change":"2024-09-13T22:42:54.080648+0000","last_active":"2024-09-13T22:43:53.892205+0000","last_peered":"2024-09-13T22:43:53.892205+0000","last_clean":"2024-09-13T22:43:53.892205+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:43:53.892205+0000","last_undegraded":"2024-09-13T22:43:53.892205+0000","last_fullsized":"2024-09-13T22:43:53.892205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.080532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.080532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:06.210400+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"160'14121","reported_seq":14409,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:58.366320+0000","last_change":"2024-09-13T22:43:39.950595+0000","last_active":"2024-09-13T22:43:58.366320+0000","last_peered":"2024-09-13T22:43:58.366320+0000","last_clean":"2024-09-13T22:43:58.366320+0000","last_became_active":"2024-09-13T22:40:54.952389+0000","last_became_peered":"2024-09-13T22:40:54.952389+0000","last_unstale":"2024-09-13T22:43:58.366320+0000","last_undegraded":"2024-09-13T22:43:58.366320+0000","last_fullsized":"2024-09-13T22:43:58.366320+0000","mapping_epoch":116,"log_start":"150'8802","ondisk_log_start":"150'8802","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'13062","last_scrub_stamp":"2024-09-13T22:43:39.950405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:43:39.950405+0000","objects_scrubbed":3652,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:42.479017+0000","scrub_duration":1019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10677,"num_write_kb":423744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":325,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877510+0000","last_change":"2024-09-13T22:42:51.196536+0000","last_active":"2024-09-13T22:43:52.877510+0000","last_peered":"2024-09-13T22:43:52.877510+0000","last_clean":"2024-09-13T22:43:52.877510+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:43:52.877510+0000","last_undegraded":"2024-09-13T22:43:52.877510+0000","last_fullsized":"2024-09-13T22:43:52.877510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:42:51.196417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:51.196417+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:10.030113+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885509+0000","last_change":"2024-09-13T22:43:37.803726+0000","last_active":"2024-09-13T22:43:53.885509+0000","last_peered":"2024-09-13T22:43:53.885509+0000","last_clean":"2024-09-13T22:43:53.885509+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:43:53.885509+0000","last_undegraded":"2024-09-13T22:43:53.885509+0000","last_fullsized":"2024-09-13T22:43:53.885509+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:37.803588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:37.803588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:45.720989+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.896879+0000","last_change":"2024-09-13T22:42:51.845782+0000","last_active":"2024-09-13T22:43:53.896879+0000","last_peered":"2024-09-13T22:43:53.896879+0000","last_clean":"2024-09-13T22:43:53.896879+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:43:53.896879+0000","last_undegraded":"2024-09-13T22:43:53.896879+0000","last_fullsized":"2024-09-13T22:43:53.896879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:51.845704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:51.845704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:09.906670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":192,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877484+0000","last_change":"2024-09-13T22:42:44.246269+0000","last_active":"2024-09-13T22:43:52.877484+0000","last_peered":"2024-09-13T22:43:52.877484+0000","last_clean":"2024-09-13T22:43:52.877484+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:43:52.877484+0000","last_undegraded":"2024-09-13T22:43:52.877484+0000","last_fullsized":"2024-09-13T22:43:52.877484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:44.246165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.246165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:58.274315+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"160'10702","reported_seq":10871,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:58.117414+0000","last_change":"2024-09-13T22:42:49.489354+0000","last_active":"2024-09-13T22:43:58.117414+0000","last_peered":"2024-09-13T22:43:58.117414+0000","last_clean":"2024-09-13T22:43:58.117414+0000","last_became_active":"2024-09-13T22:40:54.952306+0000","last_became_peered":"2024-09-13T22:40:54.952306+0000","last_unstale":"2024-09-13T22:43:58.117414+0000","last_undegraded":"2024-09-13T22:43:58.117414+0000","last_fullsized":"2024-09-13T22:43:58.117414+0000","mapping_epoch":116,"log_start":"139'5700","ondisk_log_start":"139'5700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7216","last_scrub_stamp":"2024-09-13T22:42:49.489064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.489064+0000","objects_scrubbed":6088,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167641088,"num_objects":2558,"num_object_clones":0,"num_object_copies":7674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":424320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":307,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894829+0000","last_change":"2024-09-13T22:43:39.932429+0000","last_active":"2024-09-13T22:43:53.894829+0000","last_peered":"2024-09-13T22:43:53.894829+0000","last_clean":"2024-09-13T22:43:53.894829+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:43:53.894829+0000","last_undegraded":"2024-09-13T22:43:53.894829+0000","last_fullsized":"2024-09-13T22:43:53.894829+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:43:39.932323+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:43:39.932323+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:41.023647+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":220,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877655+0000","last_change":"2024-09-13T22:43:36.147077+0000","last_active":"2024-09-13T22:43:52.877655+0000","last_peered":"2024-09-13T22:43:52.877655+0000","last_clean":"2024-09-13T22:43:52.877655+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:43:52.877655+0000","last_undegraded":"2024-09-13T22:43:52.877655+0000","last_fullsized":"2024-09-13T22:43:52.877655+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:43:36.146966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:36.146966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:04.027105+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":209,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.880251+0000","last_change":"2024-09-13T22:42:34.994631+0000","last_active":"2024-09-13T22:43:52.880251+0000","last_peered":"2024-09-13T22:43:52.880251+0000","last_clean":"2024-09-13T22:43:52.880251+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:43:52.880251+0000","last_undegraded":"2024-09-13T22:43:52.880251+0000","last_fullsized":"2024-09-13T22:43:52.880251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:42:34.994416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.994416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":292,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891435+0000","last_change":"2024-09-13T22:42:53.286723+0000","last_active":"2024-09-13T22:43:53.891435+0000","last_peered":"2024-09-13T22:43:53.891435+0000","last_clean":"2024-09-13T22:43:53.891435+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:43:53.891435+0000","last_undegraded":"2024-09-13T22:43:53.891435+0000","last_fullsized":"2024-09-13T22:43:53.891435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:42:53.286646+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.286646+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:16.002562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892129+0000","last_change":"2024-09-13T22:42:39.282264+0000","last_active":"2024-09-13T22:43:53.892129+0000","last_peered":"2024-09-13T22:43:53.892129+0000","last_clean":"2024-09-13T22:43:53.892129+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:43:53.892129+0000","last_undegraded":"2024-09-13T22:43:53.892129+0000","last_fullsized":"2024-09-13T22:43:53.892129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:39.282139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.282139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.104358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"160'10983","reported_seq":11192,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:58.134944+0000","last_change":"2024-09-13T22:43:51.803988+0000","last_active":"2024-09-13T22:43:58.134944+0000","last_peered":"2024-09-13T22:43:58.134944+0000","last_clean":"2024-09-13T22:43:58.134944+0000","last_became_active":"2024-09-13T22:40:54.952085+0000","last_became_peered":"2024-09-13T22:40:54.952085+0000","last_unstale":"2024-09-13T22:43:58.134944+0000","last_undegraded":"2024-09-13T22:43:58.134944+0000","last_fullsized":"2024-09-13T22:43:58.134944+0000","mapping_epoch":116,"log_start":"131'4600","ondisk_log_start":"131'4600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'10613","last_scrub_stamp":"2024-09-13T22:43:51.803814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.803814+0000","objects_scrubbed":2980,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:11.973054+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169672704,"num_objects":2589,"num_object_clones":0,"num_object_copies":7767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10983,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":247,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892457+0000","last_change":"2024-09-13T22:43:49.155754+0000","last_active":"2024-09-13T22:43:53.892457+0000","last_peered":"2024-09-13T22:43:53.892457+0000","last_clean":"2024-09-13T22:43:53.892457+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:43:53.892457+0000","last_undegraded":"2024-09-13T22:43:53.892457+0000","last_fullsized":"2024-09-13T22:43:53.892457+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:43:49.155639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:49.155639+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.833978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894396+0000","last_change":"2024-09-13T22:42:31.100216+0000","last_active":"2024-09-13T22:43:53.894396+0000","last_peered":"2024-09-13T22:43:53.894396+0000","last_clean":"2024-09-13T22:43:53.894396+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:43:53.894396+0000","last_undegraded":"2024-09-13T22:43:53.894396+0000","last_fullsized":"2024-09-13T22:43:53.894396+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.100088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.100088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.897734+0000","last_change":"2024-09-13T22:43:48.316803+0000","last_active":"2024-09-13T22:43:53.897734+0000","last_peered":"2024-09-13T22:43:53.897734+0000","last_clean":"2024-09-13T22:43:53.897734+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:43:53.897734+0000","last_undegraded":"2024-09-13T22:43:53.897734+0000","last_fullsized":"2024-09-13T22:43:53.897734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:48.316570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:43:48.316570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:50.219456+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.900015+0000","last_change":"2024-09-13T22:43:37.549939+0000","last_active":"2024-09-13T22:43:53.900015+0000","last_peered":"2024-09-13T22:43:53.900015+0000","last_clean":"2024-09-13T22:43:53.900015+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:43:53.900015+0000","last_undegraded":"2024-09-13T22:43:53.900015+0000","last_fullsized":"2024-09-13T22:43:53.900015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:37.549864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:37.549864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:00.170066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"160'10185","reported_seq":10356,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.947613+0000","last_change":"2024-09-13T22:42:40.742692+0000","last_active":"2024-09-13T22:43:53.947613+0000","last_peered":"2024-09-13T22:43:53.947613+0000","last_clean":"2024-09-13T22:43:53.947613+0000","last_became_active":"2024-09-13T22:40:54.959564+0000","last_became_peered":"2024-09-13T22:40:54.959564+0000","last_unstale":"2024-09-13T22:43:53.947613+0000","last_undegraded":"2024-09-13T22:43:53.947613+0000","last_fullsized":"2024-09-13T22:43:53.947613+0000","mapping_epoch":116,"log_start":"136'5000","ondisk_log_start":"136'5000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6596","last_scrub_stamp":"2024-09-13T22:42:40.742539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.742539+0000","objects_scrubbed":6364,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178716672,"num_objects":2727,"num_object_clones":0,"num_object_copies":8181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":241,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.876997+0000","last_change":"2024-09-13T22:42:32.975550+0000","last_active":"2024-09-13T22:43:52.876997+0000","last_peered":"2024-09-13T22:43:52.876997+0000","last_clean":"2024-09-13T22:43:52.876997+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:43:52.876997+0000","last_undegraded":"2024-09-13T22:43:52.876997+0000","last_fullsized":"2024-09-13T22:43:52.876997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-09-13T22:42:32.975430+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:42:32.975430+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.970807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.883007+0000","last_change":"2024-09-13T22:42:30.787370+0000","last_active":"2024-09-13T22:43:52.883007+0000","last_peered":"2024-09-13T22:43:52.883007+0000","last_clean":"2024-09-13T22:43:52.883007+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:43:52.883007+0000","last_undegraded":"2024-09-13T22:43:52.883007+0000","last_fullsized":"2024-09-13T22:43:52.883007+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:30.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:30.787265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:55.200968+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.883506+0000","last_change":"2024-09-13T22:43:34.541258+0000","last_active":"2024-09-13T22:43:52.883506+0000","last_peered":"2024-09-13T22:43:52.883506+0000","last_clean":"2024-09-13T22:43:52.883506+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:43:52.883506+0000","last_undegraded":"2024-09-13T22:43:52.883506+0000","last_fullsized":"2024-09-13T22:43:52.883506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:34.541085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:43:34.541085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:35.710054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885236+0000","last_change":"2024-09-13T22:42:45.743372+0000","last_active":"2024-09-13T22:43:53.885236+0000","last_peered":"2024-09-13T22:43:53.885236+0000","last_clean":"2024-09-13T22:43:53.885236+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:43:53.885236+0000","last_undegraded":"2024-09-13T22:43:53.885236+0000","last_fullsized":"2024-09-13T22:43:53.885236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.743269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.743269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.530343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"160'10689","reported_seq":10823,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:58.514815+0000","last_change":"2024-09-13T22:43:55.279660+0000","last_active":"2024-09-13T22:43:58.514815+0000","last_peered":"2024-09-13T22:43:58.514815+0000","last_clean":"2024-09-13T22:43:58.514815+0000","last_became_active":"2024-09-13T22:40:54.960739+0000","last_became_peered":"2024-09-13T22:40:54.960739+0000","last_unstale":"2024-09-13T22:43:58.514815+0000","last_undegraded":"2024-09-13T22:43:58.514815+0000","last_fullsized":"2024-09-13T22:43:58.514815+0000","mapping_epoch":116,"log_start":"140'6000","ondisk_log_start":"140'6000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10529","last_scrub_stamp":"2024-09-13T22:43:55.279583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:43:55.279583+0000","objects_scrubbed":2756,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:55.866294+0000","scrub_duration":639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":7737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":424576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891163+0000","last_change":"2024-09-13T22:43:51.350004+0000","last_active":"2024-09-13T22:43:53.891163+0000","last_peered":"2024-09-13T22:43:53.891163+0000","last_clean":"2024-09-13T22:43:53.891163+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:43:53.891163+0000","last_undegraded":"2024-09-13T22:43:53.891163+0000","last_fullsized":"2024-09-13T22:43:53.891163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:43:51.349861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.349861+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:05.847047+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.884426+0000","last_change":"2024-09-13T22:42:45.684106+0000","last_active":"2024-09-13T22:43:52.884426+0000","last_peered":"2024-09-13T22:43:52.884426+0000","last_clean":"2024-09-13T22:43:52.884426+0000","last_became_active":"2024-09-13T22:39:34.702361+0000","last_became_peered":"2024-09-13T22:39:34.702361+0000","last_unstale":"2024-09-13T22:43:52.884426+0000","last_undegraded":"2024-09-13T22:43:52.884426+0000","last_fullsized":"2024-09-13T22:43:52.884426+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.683896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.683896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:57.709636+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895054+0000","last_change":"2024-09-13T22:42:49.271307+0000","last_active":"2024-09-13T22:43:53.895054+0000","last_peered":"2024-09-13T22:43:53.895054+0000","last_clean":"2024-09-13T22:43:53.895054+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:43:53.895054+0000","last_undegraded":"2024-09-13T22:43:53.895054+0000","last_fullsized":"2024-09-13T22:43:53.895054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:42:49.271179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:05.043890+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":192,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877187+0000","last_change":"2024-09-13T22:41:50.524433+0000","last_active":"2024-09-13T22:43:52.877187+0000","last_peered":"2024-09-13T22:43:52.877187+0000","last_clean":"2024-09-13T22:43:52.877187+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:43:52.877187+0000","last_undegraded":"2024-09-13T22:43:52.877187+0000","last_fullsized":"2024-09-13T22:43:52.877187+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:50.524098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.524098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"160'10257","reported_seq":10425,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.990557+0000","last_change":"2024-09-13T22:42:44.078796+0000","last_active":"2024-09-13T22:43:53.990557+0000","last_peered":"2024-09-13T22:43:53.990557+0000","last_clean":"2024-09-13T22:43:53.990557+0000","last_became_active":"2024-09-13T22:40:54.955979+0000","last_became_peered":"2024-09-13T22:40:54.955979+0000","last_unstale":"2024-09-13T22:43:53.990557+0000","last_undegraded":"2024-09-13T22:43:53.990557+0000","last_fullsized":"2024-09-13T22:43:53.990557+0000","mapping_epoch":116,"log_start":"135'5000","ondisk_log_start":"135'5000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'6814","last_scrub_stamp":"2024-09-13T22:42:44.078550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.078550+0000","objects_scrubbed":6262,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:10.325784+0000","scrub_duration":2783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180289536,"num_objects":2751,"num_object_clones":0,"num_object_copies":8253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10257,"num_write_kb":416256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899847+0000","last_change":"2024-09-13T22:42:43.552263+0000","last_active":"2024-09-13T22:43:53.899847+0000","last_peered":"2024-09-13T22:43:53.899847+0000","last_clean":"2024-09-13T22:43:53.899847+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:43:53.899847+0000","last_undegraded":"2024-09-13T22:43:53.899847+0000","last_fullsized":"2024-09-13T22:43:53.899847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:42:43.551997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:43.551997+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.760390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895045+0000","last_change":"2024-09-13T22:42:36.032459+0000","last_active":"2024-09-13T22:43:53.895045+0000","last_peered":"2024-09-13T22:43:53.895045+0000","last_clean":"2024-09-13T22:43:53.895045+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:43:53.895045+0000","last_undegraded":"2024-09-13T22:43:53.895045+0000","last_fullsized":"2024-09-13T22:43:53.895045+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:36.032356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.032356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:02.001671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":459,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885708+0000","last_change":"2024-09-13T22:41:45.619812+0000","last_active":"2024-09-13T22:43:53.885708+0000","last_peered":"2024-09-13T22:43:53.885708+0000","last_clean":"2024-09-13T22:43:53.885708+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:43:53.885708+0000","last_undegraded":"2024-09-13T22:43:53.885708+0000","last_fullsized":"2024-09-13T22:43:53.885708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:41:45.619735+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:41:45.619735+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.884007+0000","last_change":"2024-09-13T22:41:55.748722+0000","last_active":"2024-09-13T22:43:53.884007+0000","last_peered":"2024-09-13T22:43:53.884007+0000","last_clean":"2024-09-13T22:43:53.884007+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:43:53.884007+0000","last_undegraded":"2024-09-13T22:43:53.884007+0000","last_fullsized":"2024-09-13T22:43:53.884007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:55.748627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:41:55.748627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:57.009829+0000","last_change":"2024-09-13T22:43:57.009877+0000","last_active":"2024-09-13T22:43:57.009829+0000","last_peered":"2024-09-13T22:43:57.009829+0000","last_clean":"2024-09-13T22:43:57.009829+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:43:57.009829+0000","last_undegraded":"2024-09-13T22:43:57.009829+0000","last_fullsized":"2024-09-13T22:43:57.009829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:57.009754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.009754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:08.636298+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"160'10440","reported_seq":10609,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.989995+0000","last_change":"2024-09-13T22:42:31.664279+0000","last_active":"2024-09-13T22:43:53.989995+0000","last_peered":"2024-09-13T22:43:53.989995+0000","last_clean":"2024-09-13T22:43:53.989995+0000","last_became_active":"2024-09-13T22:40:54.959549+0000","last_became_peered":"2024-09-13T22:40:54.959549+0000","last_unstale":"2024-09-13T22:43:53.989995+0000","last_undegraded":"2024-09-13T22:43:53.989995+0000","last_fullsized":"2024-09-13T22:43:53.989995+0000","mapping_epoch":116,"log_start":"136'5200","ondisk_log_start":"136'5200","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'6299","last_scrub_stamp":"2024-09-13T22:42:31.664041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.664041+0000","objects_scrubbed":6215,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10440,"num_write_kb":423232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885372+0000","last_change":"2024-09-13T22:41:33.652663+0000","last_active":"2024-09-13T22:43:53.885372+0000","last_peered":"2024-09-13T22:43:53.885372+0000","last_clean":"2024-09-13T22:43:53.885372+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:43:53.885372+0000","last_undegraded":"2024-09-13T22:43:53.885372+0000","last_fullsized":"2024-09-13T22:43:53.885372+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:41:33.652584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:33.652584+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.885473+0000","last_change":"2024-09-13T22:42:58.827975+0000","last_active":"2024-09-13T22:43:52.885473+0000","last_peered":"2024-09-13T22:43:52.885473+0000","last_clean":"2024-09-13T22:43:52.885473+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:43:52.885473+0000","last_undegraded":"2024-09-13T22:43:52.885473+0000","last_fullsized":"2024-09-13T22:43:52.885473+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.827870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.827870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.240808+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":196,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.885759+0000","last_change":"2024-09-13T22:42:46.715235+0000","last_active":"2024-09-13T22:43:52.885759+0000","last_peered":"2024-09-13T22:43:52.885759+0000","last_clean":"2024-09-13T22:43:52.885759+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:43:52.885759+0000","last_undegraded":"2024-09-13T22:43:52.885759+0000","last_fullsized":"2024-09-13T22:43:52.885759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:46.715135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.715135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.458531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":185,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885720+0000","last_change":"2024-09-13T22:41:40.678879+0000","last_active":"2024-09-13T22:43:53.885720+0000","last_peered":"2024-09-13T22:43:53.885720+0000","last_clean":"2024-09-13T22:43:53.885720+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:43:53.885720+0000","last_undegraded":"2024-09-13T22:43:53.885720+0000","last_fullsized":"2024-09-13T22:43:53.885720+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:40.678635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:40.678635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"159'10328","reported_seq":10489,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.764764+0000","last_change":"2024-09-13T22:42:57.541592+0000","last_active":"2024-09-13T22:43:53.764764+0000","last_peered":"2024-09-13T22:43:53.764764+0000","last_clean":"2024-09-13T22:43:53.764764+0000","last_became_active":"2024-09-13T22:40:54.960207+0000","last_became_peered":"2024-09-13T22:40:54.960207+0000","last_unstale":"2024-09-13T22:43:53.764764+0000","last_undegraded":"2024-09-13T22:43:53.764764+0000","last_fullsized":"2024-09-13T22:43:53.764764+0000","mapping_epoch":116,"log_start":"126'3600","ondisk_log_start":"126'3600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7482","last_scrub_stamp":"2024-09-13T22:42:57.541325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.541325+0000","objects_scrubbed":5583,"log_size":6728,"log_dups_size":0,"ondisk_log_size":6728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.613669+0000","scrub_duration":1715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":216,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894468+0000","last_change":"2024-09-13T22:42:38.043048+0000","last_active":"2024-09-13T22:43:53.894468+0000","last_peered":"2024-09-13T22:43:53.894468+0000","last_clean":"2024-09-13T22:43:53.894468+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:43:53.894468+0000","last_undegraded":"2024-09-13T22:43:53.894468+0000","last_fullsized":"2024-09-13T22:43:53.894468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:42:38.042796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:38.042796+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:53.974317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.884087+0000","last_change":"2024-09-13T22:43:52.173697+0000","last_active":"2024-09-13T22:43:52.884087+0000","last_peered":"2024-09-13T22:43:52.884087+0000","last_clean":"2024-09-13T22:43:52.884087+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:43:52.884087+0000","last_undegraded":"2024-09-13T22:43:52.884087+0000","last_fullsized":"2024-09-13T22:43:52.884087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:52.173506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:52.173506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:52.330981+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.885787+0000","last_change":"2024-09-13T22:43:37.537514+0000","last_active":"2024-09-13T22:43:52.885787+0000","last_peered":"2024-09-13T22:43:52.885787+0000","last_clean":"2024-09-13T22:43:52.885787+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:43:52.885787+0000","last_undegraded":"2024-09-13T22:43:52.885787+0000","last_fullsized":"2024-09-13T22:43:52.885787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:37.536972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:37.536972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:52.406074+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891064+0000","last_change":"2024-09-13T22:42:54.275527+0000","last_active":"2024-09-13T22:43:53.891064+0000","last_peered":"2024-09-13T22:43:53.891064+0000","last_clean":"2024-09-13T22:43:53.891064+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:43:53.891064+0000","last_undegraded":"2024-09-13T22:43:53.891064+0000","last_fullsized":"2024-09-13T22:43:53.891064+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.275361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.824268+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"159'14262","reported_seq":14560,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.759617+0000","last_change":"2024-09-13T22:43:45.561678+0000","last_active":"2024-09-13T22:43:53.759617+0000","last_peered":"2024-09-13T22:43:53.759617+0000","last_clean":"2024-09-13T22:43:53.759617+0000","last_became_active":"2024-09-13T22:40:54.949821+0000","last_became_peered":"2024-09-13T22:40:54.949821+0000","last_unstale":"2024-09-13T22:43:53.759617+0000","last_undegraded":"2024-09-13T22:43:53.759617+0000","last_fullsized":"2024-09-13T22:43:53.759617+0000","mapping_epoch":116,"log_start":"149'8601","ondisk_log_start":"149'8601","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'13782","last_scrub_stamp":"2024-09-13T22:43:45.561492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:43:45.561492+0000","objects_scrubbed":3367,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:58.332720+0000","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188284928,"num_objects":2873,"num_object_clones":0,"num_object_copies":8619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10507,"num_write_kb":428160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.884054+0000","last_change":"2024-09-13T22:41:32.631943+0000","last_active":"2024-09-13T22:43:53.884054+0000","last_peered":"2024-09-13T22:43:53.884054+0000","last_clean":"2024-09-13T22:43:53.884054+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:43:53.884054+0000","last_undegraded":"2024-09-13T22:43:53.884054+0000","last_fullsized":"2024-09-13T22:43:53.884054+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:41:32.631815+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:41:32.631815+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.485460+0000","last_change":"2024-09-13T22:43:53.485493+0000","last_active":"2024-09-13T22:43:53.485460+0000","last_peered":"2024-09-13T22:43:53.485460+0000","last_clean":"2024-09-13T22:43:53.485460+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:43:53.485460+0000","last_undegraded":"2024-09-13T22:43:53.485460+0000","last_fullsized":"2024-09-13T22:43:53.485460+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.485398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.485398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:14.238379+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":283,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.883812+0000","last_change":"2024-09-13T22:41:50.679387+0000","last_active":"2024-09-13T22:43:53.883812+0000","last_peered":"2024-09-13T22:43:53.883812+0000","last_clean":"2024-09-13T22:43:53.883812+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:43:53.883812+0000","last_undegraded":"2024-09-13T22:43:53.883812+0000","last_fullsized":"2024-09-13T22:43:53.883812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:50.679300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.679300+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894062+0000","last_change":"2024-09-13T22:42:46.267477+0000","last_active":"2024-09-13T22:43:53.894062+0000","last_peered":"2024-09-13T22:43:53.894062+0000","last_clean":"2024-09-13T22:43:53.894062+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:43:53.894062+0000","last_undegraded":"2024-09-13T22:43:53.894062+0000","last_fullsized":"2024-09-13T22:43:53.894062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:46.267344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:46.267344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.234794+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"160'10726","reported_seq":10888,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:54.425159+0000","last_change":"2024-09-13T22:42:58.835575+0000","last_active":"2024-09-13T22:43:54.425159+0000","last_peered":"2024-09-13T22:43:54.425159+0000","last_clean":"2024-09-13T22:43:54.425159+0000","last_became_active":"2024-09-13T22:40:54.964195+0000","last_became_peered":"2024-09-13T22:40:54.964195+0000","last_unstale":"2024-09-13T22:43:54.425159+0000","last_undegraded":"2024-09-13T22:43:54.425159+0000","last_fullsized":"2024-09-13T22:43:54.425159+0000","mapping_epoch":116,"log_start":"138'5600","ondisk_log_start":"138'5600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'7811","last_scrub_stamp":"2024-09-13T22:42:58.835009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:53.917678+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.835009+0000","objects_scrubbed":5759,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:22.402928+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10726,"num_write_kb":432384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885572+0000","last_change":"2024-09-13T22:41:53.752721+0000","last_active":"2024-09-13T22:43:53.885572+0000","last_peered":"2024-09-13T22:43:53.885572+0000","last_clean":"2024-09-13T22:43:53.885572+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:43:53.885572+0000","last_undegraded":"2024-09-13T22:43:53.885572+0000","last_fullsized":"2024-09-13T22:43:53.885572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:41:53.752600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:53.752600+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":224,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.893210+0000","last_change":"2024-09-13T22:42:45.983095+0000","last_active":"2024-09-13T22:43:53.893210+0000","last_peered":"2024-09-13T22:43:53.893210+0000","last_clean":"2024-09-13T22:43:53.893210+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:43:53.893210+0000","last_undegraded":"2024-09-13T22:43:53.893210+0000","last_fullsized":"2024-09-13T22:43:53.893210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:42:45.982969+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.982969+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":196,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.884604+0000","last_change":"2024-09-13T22:42:57.858562+0000","last_active":"2024-09-13T22:43:52.884604+0000","last_peered":"2024-09-13T22:43:52.884604+0000","last_clean":"2024-09-13T22:43:52.884604+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:43:52.884604+0000","last_undegraded":"2024-09-13T22:43:52.884604+0000","last_fullsized":"2024-09-13T22:43:52.884604+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:57.858139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.858139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:27.371302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899652+0000","last_change":"2024-09-13T22:42:31.280678+0000","last_active":"2024-09-13T22:43:53.899652+0000","last_peered":"2024-09-13T22:43:53.899652+0000","last_clean":"2024-09-13T22:43:53.899652+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:43:53.899652+0000","last_undegraded":"2024-09-13T22:43:53.899652+0000","last_fullsized":"2024-09-13T22:43:53.899652+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:31.280433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:18.998416+0000","last_clean_scrub_stamp":"2024-09-13T22:42:31.280433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"112'16","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.883831+0000","last_change":"2024-09-13T22:42:36.814689+0000","last_active":"2024-09-13T22:43:53.883831+0000","last_peered":"2024-09-13T22:43:53.883831+0000","last_clean":"2024-09-13T22:43:53.883831+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:43:53.883831+0000","last_undegraded":"2024-09-13T22:43:53.883831+0000","last_fullsized":"2024-09-13T22:43:53.883831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'16","last_scrub_stamp":"2024-09-13T22:42:36.814607+0000","last_deep_scrub":"112'16","last_deep_scrub_stamp":"2024-09-13T22:40:26.503359+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.814607+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.577055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.884731+0000","last_change":"2024-09-13T22:43:40.564638+0000","last_active":"2024-09-13T22:43:52.884731+0000","last_peered":"2024-09-13T22:43:52.884731+0000","last_clean":"2024-09-13T22:43:52.884731+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:43:52.884731+0000","last_undegraded":"2024-09-13T22:43:52.884731+0000","last_fullsized":"2024-09-13T22:43:52.884731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.564538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.564538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:51.075996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.908072+0000","last_change":"2024-09-13T22:42:29.335168+0000","last_active":"2024-09-13T22:43:53.908072+0000","last_peered":"2024-09-13T22:43:53.908072+0000","last_clean":"2024-09-13T22:43:53.908072+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:43:53.908072+0000","last_undegraded":"2024-09-13T22:43:53.908072+0000","last_fullsized":"2024-09-13T22:43:53.908072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:29.335020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:29.335020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":200,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.885000+0000","last_change":"2024-09-13T22:42:59.874545+0000","last_active":"2024-09-13T22:43:52.885000+0000","last_peered":"2024-09-13T22:43:52.885000+0000","last_clean":"2024-09-13T22:43:52.885000+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:43:52.885000+0000","last_undegraded":"2024-09-13T22:43:52.885000+0000","last_fullsized":"2024-09-13T22:43:52.885000+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.874442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.874442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:20.548793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":216,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894684+0000","last_change":"2024-09-13T22:42:53.857519+0000","last_active":"2024-09-13T22:43:53.894684+0000","last_peered":"2024-09-13T22:43:53.894684+0000","last_clean":"2024-09-13T22:43:53.894684+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:43:53.894684+0000","last_undegraded":"2024-09-13T22:43:53.894684+0000","last_fullsized":"2024-09-13T22:43:53.894684+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:42:53.857434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.857434+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.762599+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899330+0000","last_change":"2024-09-13T22:42:39.863272+0000","last_active":"2024-09-13T22:43:53.899330+0000","last_peered":"2024-09-13T22:43:53.899330+0000","last_clean":"2024-09-13T22:43:53.899330+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:43:53.899330+0000","last_undegraded":"2024-09-13T22:43:53.899330+0000","last_fullsized":"2024-09-13T22:43:53.899330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:39.863156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:39.863156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:54.360048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.896681+0000","last_change":"2024-09-13T22:42:41.074516+0000","last_active":"2024-09-13T22:43:53.896681+0000","last_peered":"2024-09-13T22:43:53.896681+0000","last_clean":"2024-09-13T22:43:53.896681+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:43:53.896681+0000","last_undegraded":"2024-09-13T22:43:53.896681+0000","last_fullsized":"2024-09-13T22:43:53.896681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:41.074413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:42:41.074413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:06.392680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885142+0000","last_change":"2024-09-13T22:43:40.670309+0000","last_active":"2024-09-13T22:43:53.885142+0000","last_peered":"2024-09-13T22:43:53.885142+0000","last_clean":"2024-09-13T22:43:53.885142+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:43:53.885142+0000","last_undegraded":"2024-09-13T22:43:53.885142+0000","last_fullsized":"2024-09-13T22:43:53.885142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.670209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.670209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:47.299713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885233+0000","last_change":"2024-09-13T22:42:34.859780+0000","last_active":"2024-09-13T22:43:53.885233+0000","last_peered":"2024-09-13T22:43:53.885233+0000","last_clean":"2024-09-13T22:43:53.885233+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:43:53.885233+0000","last_undegraded":"2024-09-13T22:43:53.885233+0000","last_fullsized":"2024-09-13T22:43:53.885233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:34.859649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.859649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.462988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.971389+0000","last_change":"2024-09-13T22:43:53.971419+0000","last_active":"2024-09-13T22:43:53.971389+0000","last_peered":"2024-09-13T22:43:53.971389+0000","last_clean":"2024-09-13T22:43:53.971389+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:43:53.971389+0000","last_undegraded":"2024-09-13T22:43:53.971389+0000","last_fullsized":"2024-09-13T22:43:53.971389+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.971324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.971324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:16.928312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":294,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894646+0000","last_change":"2024-09-13T22:43:50.053369+0000","last_active":"2024-09-13T22:43:53.894646+0000","last_peered":"2024-09-13T22:43:53.894646+0000","last_clean":"2024-09-13T22:43:53.894646+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:43:53.894646+0000","last_undegraded":"2024-09-13T22:43:53.894646+0000","last_fullsized":"2024-09-13T22:43:53.894646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:50.053246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:50.053246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:03.467190+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892181+0000","last_change":"2024-09-13T22:42:35.206528+0000","last_active":"2024-09-13T22:43:53.892181+0000","last_peered":"2024-09-13T22:43:53.892181+0000","last_clean":"2024-09-13T22:43:53.892181+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:43:53.892181+0000","last_undegraded":"2024-09-13T22:43:53.892181+0000","last_fullsized":"2024-09-13T22:43:53.892181+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:35.206294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:35.206294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":218,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891218+0000","last_change":"2024-09-13T22:41:50.549051+0000","last_active":"2024-09-13T22:43:53.891218+0000","last_peered":"2024-09-13T22:43:53.891218+0000","last_clean":"2024-09-13T22:43:53.891218+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:43:53.891218+0000","last_undegraded":"2024-09-13T22:43:53.891218+0000","last_fullsized":"2024-09-13T22:43:53.891218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:41:50.548929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:41:50.548929+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891828+0000","last_change":"2024-09-13T22:42:56.298096+0000","last_active":"2024-09-13T22:43:53.891828+0000","last_peered":"2024-09-13T22:43:53.891828+0000","last_clean":"2024-09-13T22:43:53.891828+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:43:53.891828+0000","last_undegraded":"2024-09-13T22:43:53.891828+0000","last_fullsized":"2024-09-13T22:43:53.891828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:56.297990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:56.297990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.122881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":291,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.902578+0000","last_change":"2024-09-13T22:43:42.603416+0000","last_active":"2024-09-13T22:43:53.902578+0000","last_peered":"2024-09-13T22:43:53.902578+0000","last_clean":"2024-09-13T22:43:53.902578+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:43:53.902578+0000","last_undegraded":"2024-09-13T22:43:53.902578+0000","last_fullsized":"2024-09-13T22:43:53.902578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:42.603288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:42.603288+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:01.185909+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.893070+0000","last_change":"2024-09-13T22:41:58.751646+0000","last_active":"2024-09-13T22:43:53.893070+0000","last_peered":"2024-09-13T22:43:53.893070+0000","last_clean":"2024-09-13T22:43:53.893070+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:43:53.893070+0000","last_undegraded":"2024-09-13T22:43:53.893070+0000","last_fullsized":"2024-09-13T22:43:53.893070+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:58.751579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:58.751579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885335+0000","last_change":"2024-09-13T22:42:52.779439+0000","last_active":"2024-09-13T22:43:53.885335+0000","last_peered":"2024-09-13T22:43:53.885335+0000","last_clean":"2024-09-13T22:43:53.885335+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:43:53.885335+0000","last_undegraded":"2024-09-13T22:43:53.885335+0000","last_fullsized":"2024-09-13T22:43:53.885335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.779356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.779356+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.029073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.897375+0000","last_change":"2024-09-13T22:43:38.824445+0000","last_active":"2024-09-13T22:43:53.897375+0000","last_peered":"2024-09-13T22:43:53.897375+0000","last_clean":"2024-09-13T22:43:53.897375+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:43:53.897375+0000","last_undegraded":"2024-09-13T22:43:53.897375+0000","last_fullsized":"2024-09-13T22:43:53.897375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:38.824331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:38.824331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:39.700519+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899736+0000","last_change":"2024-09-13T22:42:50.643442+0000","last_active":"2024-09-13T22:43:53.899736+0000","last_peered":"2024-09-13T22:43:53.899736+0000","last_clean":"2024-09-13T22:43:53.899736+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:43:53.899736+0000","last_undegraded":"2024-09-13T22:43:53.899736+0000","last_fullsized":"2024-09-13T22:43:53.899736+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:50.643281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.643281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.030417+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.890011+0000","last_change":"2024-09-13T22:42:58.240077+0000","last_active":"2024-09-13T22:43:53.890011+0000","last_peered":"2024-09-13T22:43:53.890011+0000","last_clean":"2024-09-13T22:43:53.890011+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:43:53.890011+0000","last_undegraded":"2024-09-13T22:43:53.890011+0000","last_fullsized":"2024-09-13T22:43:53.890011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.239969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.239969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:18.532690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892674+0000","last_change":"2024-09-13T22:43:36.040902+0000","last_active":"2024-09-13T22:43:53.892674+0000","last_peered":"2024-09-13T22:43:53.892674+0000","last_clean":"2024-09-13T22:43:53.892674+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:43:53.892674+0000","last_undegraded":"2024-09-13T22:43:53.892674+0000","last_fullsized":"2024-09-13T22:43:53.892674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:43:36.040693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:36.040693+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:55.904492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.884010+0000","last_change":"2024-09-13T22:41:44.491145+0000","last_active":"2024-09-13T22:43:52.884010+0000","last_peered":"2024-09-13T22:43:52.884010+0000","last_clean":"2024-09-13T22:43:52.884010+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:43:52.884010+0000","last_undegraded":"2024-09-13T22:43:52.884010+0000","last_fullsized":"2024-09-13T22:43:52.884010+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.490926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.490926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":196,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.879995+0000","last_change":"2024-09-13T22:41:56.720968+0000","last_active":"2024-09-13T22:43:52.879995+0000","last_peered":"2024-09-13T22:43:52.879995+0000","last_clean":"2024-09-13T22:43:52.879995+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:43:52.879995+0000","last_undegraded":"2024-09-13T22:43:52.879995+0000","last_fullsized":"2024-09-13T22:43:52.879995+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:56.720495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:41:56.720495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.904589+0000","last_change":"2024-09-13T22:42:34.299357+0000","last_active":"2024-09-13T22:43:53.904589+0000","last_peered":"2024-09-13T22:43:53.904589+0000","last_clean":"2024-09-13T22:43:53.904589+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:43:53.904589+0000","last_undegraded":"2024-09-13T22:43:53.904589+0000","last_fullsized":"2024-09-13T22:43:53.904589+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:34.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":219,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895282+0000","last_change":"2024-09-13T22:43:46.254609+0000","last_active":"2024-09-13T22:43:53.895282+0000","last_peered":"2024-09-13T22:43:53.895282+0000","last_clean":"2024-09-13T22:43:53.895282+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:43:53.895282+0000","last_undegraded":"2024-09-13T22:43:53.895282+0000","last_fullsized":"2024-09-13T22:43:53.895282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:43:46.254539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:46.254539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:57.250545+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.896846+0000","last_change":"2024-09-13T22:42:40.026232+0000","last_active":"2024-09-13T22:43:53.896846+0000","last_peered":"2024-09-13T22:43:53.896846+0000","last_clean":"2024-09-13T22:43:53.896846+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:43:53.896846+0000","last_undegraded":"2024-09-13T22:43:53.896846+0000","last_fullsized":"2024-09-13T22:43:53.896846+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:40.026160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:42:40.026160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:03.829266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.896655+0000","last_change":"2024-09-13T22:43:39.856677+0000","last_active":"2024-09-13T22:43:53.896655+0000","last_peered":"2024-09-13T22:43:53.896655+0000","last_clean":"2024-09-13T22:43:53.896655+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:43:53.896655+0000","last_undegraded":"2024-09-13T22:43:53.896655+0000","last_fullsized":"2024-09-13T22:43:53.896655+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:39.856566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:43:39.856566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:58.872191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":185,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.883925+0000","last_change":"2024-09-13T22:41:51.724840+0000","last_active":"2024-09-13T22:43:53.883925+0000","last_peered":"2024-09-13T22:43:53.883925+0000","last_clean":"2024-09-13T22:43:53.883925+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:43:53.883925+0000","last_undegraded":"2024-09-13T22:43:53.883925+0000","last_fullsized":"2024-09-13T22:43:53.883925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:51.724731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:51.724731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.896788+0000","last_change":"2024-09-13T22:42:44.126239+0000","last_active":"2024-09-13T22:43:53.896788+0000","last_peered":"2024-09-13T22:43:53.896788+0000","last_clean":"2024-09-13T22:43:53.896788+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:43:53.896788+0000","last_undegraded":"2024-09-13T22:43:53.896788+0000","last_fullsized":"2024-09-13T22:43:53.896788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:44.126118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:44.126118+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.929366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.898173+0000","last_change":"2024-09-13T22:42:55.792471+0000","last_active":"2024-09-13T22:43:53.898173+0000","last_peered":"2024-09-13T22:43:53.898173+0000","last_clean":"2024-09-13T22:43:53.898173+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:43:53.898173+0000","last_undegraded":"2024-09-13T22:43:53.898173+0000","last_fullsized":"2024-09-13T22:43:53.898173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:55.792364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:42:55.792364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:04.743688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.885291+0000","last_change":"2024-09-13T22:43:51.625709+0000","last_active":"2024-09-13T22:43:53.885291+0000","last_peered":"2024-09-13T22:43:53.885291+0000","last_clean":"2024-09-13T22:43:53.885291+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:43:53.885291+0000","last_undegraded":"2024-09-13T22:43:53.885291+0000","last_fullsized":"2024-09-13T22:43:53.885291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:51.625640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.625640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:52.491342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899882+0000","last_change":"2024-09-13T22:43:45.603477+0000","last_active":"2024-09-13T22:43:53.899882+0000","last_peered":"2024-09-13T22:43:53.899882+0000","last_clean":"2024-09-13T22:43:53.899882+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:43:53.899882+0000","last_undegraded":"2024-09-13T22:43:53.899882+0000","last_fullsized":"2024-09-13T22:43:53.899882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:45.603355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:43:45.603355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:59.939264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.883914+0000","last_change":"2024-09-13T22:42:54.751350+0000","last_active":"2024-09-13T22:43:53.883914+0000","last_peered":"2024-09-13T22:43:53.883914+0000","last_clean":"2024-09-13T22:43:53.883914+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:43:53.883914+0000","last_undegraded":"2024-09-13T22:43:53.883914+0000","last_fullsized":"2024-09-13T22:43:53.883914+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:42:54.751219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.751219+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:13.922863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"111'22","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.877677+0000","last_change":"2024-09-13T22:43:51.139213+0000","last_active":"2024-09-13T22:43:52.877677+0000","last_peered":"2024-09-13T22:43:52.877677+0000","last_clean":"2024-09-13T22:43:52.877677+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:43:52.877677+0000","last_undegraded":"2024-09-13T22:43:52.877677+0000","last_fullsized":"2024-09-13T22:43:52.877677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-13T22:43:51.138649+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.138649+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:02.115729+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.894039+0000","last_change":"2024-09-13T22:42:57.304510+0000","last_active":"2024-09-13T22:43:53.894039+0000","last_peered":"2024-09-13T22:43:53.894039+0000","last_clean":"2024-09-13T22:43:53.894039+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:43:53.894039+0000","last_undegraded":"2024-09-13T22:43:53.894039+0000","last_fullsized":"2024-09-13T22:43:53.894039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:57.304411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:42:57.304411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:15.973967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'11","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:56.098054+0000","last_change":"2024-09-13T22:43:46.969042+0000","last_active":"2024-09-13T22:43:56.098054+0000","last_peered":"2024-09-13T22:43:56.098054+0000","last_clean":"2024-09-13T22:43:56.098054+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:43:56.098054+0000","last_undegraded":"2024-09-13T22:43:56.098054+0000","last_fullsized":"2024-09-13T22:43:56.098054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:46.968953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:46.968953+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:53.220544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":192,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.882858+0000","last_change":"2024-09-13T22:41:41.532153+0000","last_active":"2024-09-13T22:43:52.882858+0000","last_peered":"2024-09-13T22:43:52.882858+0000","last_clean":"2024-09-13T22:43:52.882858+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:43:52.882858+0000","last_undegraded":"2024-09-13T22:43:52.882858+0000","last_fullsized":"2024-09-13T22:43:52.882858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:41.531837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:41.531837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":218,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.899691+0000","last_change":"2024-09-13T22:42:37.463998+0000","last_active":"2024-09-13T22:43:53.899691+0000","last_peered":"2024-09-13T22:43:53.899691+0000","last_clean":"2024-09-13T22:43:53.899691+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:43:53.899691+0000","last_undegraded":"2024-09-13T22:43:53.899691+0000","last_fullsized":"2024-09-13T22:43:53.899691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:37.463889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:37.463889+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.904943+0000","last_change":"2024-09-13T22:42:45.551596+0000","last_active":"2024-09-13T22:43:53.904943+0000","last_peered":"2024-09-13T22:43:53.904943+0000","last_clean":"2024-09-13T22:43:53.904943+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:43:53.904943+0000","last_undegraded":"2024-09-13T22:43:53.904943+0000","last_fullsized":"2024-09-13T22:43:53.904943+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:42:45.551192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.551192+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:07.564548+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.895077+0000","last_change":"2024-09-13T22:42:59.279764+0000","last_active":"2024-09-13T22:43:53.895077+0000","last_peered":"2024-09-13T22:43:53.895077+0000","last_clean":"2024-09-13T22:43:53.895077+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:43:53.895077+0000","last_undegraded":"2024-09-13T22:43:53.895077+0000","last_fullsized":"2024-09-13T22:43:53.895077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.279506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.279506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:12.337637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892839+0000","last_change":"2024-09-13T22:41:44.682932+0000","last_active":"2024-09-13T22:43:53.892839+0000","last_peered":"2024-09-13T22:43:53.892839+0000","last_clean":"2024-09-13T22:43:53.892839+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:43:53.892839+0000","last_undegraded":"2024-09-13T22:43:53.892839+0000","last_fullsized":"2024-09-13T22:43:53.892839+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:41:44.682861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:41:44.682861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.897024+0000","last_change":"2024-09-13T22:42:47.806567+0000","last_active":"2024-09-13T22:43:53.897024+0000","last_peered":"2024-09-13T22:43:53.897024+0000","last_clean":"2024-09-13T22:43:53.897024+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:43:53.897024+0000","last_undegraded":"2024-09-13T22:43:53.897024+0000","last_fullsized":"2024-09-13T22:43:53.897024+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:47.806485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:47.806485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:08.641266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":222,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.891689+0000","last_change":"2024-09-13T22:42:52.339140+0000","last_active":"2024-09-13T22:43:53.891689+0000","last_peered":"2024-09-13T22:43:53.891689+0000","last_clean":"2024-09-13T22:43:53.891689+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:43:53.891689+0000","last_undegraded":"2024-09-13T22:43:53.891689+0000","last_fullsized":"2024-09-13T22:43:53.891689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:42:52.339029+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.339029+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:57.801246+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":196,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T22:43:52.883808+0000","last_change":"2024-09-13T22:42:53.183255+0000","last_active":"2024-09-13T22:43:52.883808+0000","last_peered":"2024-09-13T22:43:52.883808+0000","last_clean":"2024-09-13T22:43:52.883808+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:43:52.883808+0000","last_undegraded":"2024-09-13T22:43:52.883808+0000","last_fullsized":"2024-09-13T22:43:52.883808+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:53.183134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.183134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:56.729375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.907984+0000","last_change":"2024-09-13T22:43:50.727253+0000","last_active":"2024-09-13T22:43:53.907984+0000","last_peered":"2024-09-13T22:43:53.907984+0000","last_clean":"2024-09-13T22:43:53.907984+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:43:53.907984+0000","last_undegraded":"2024-09-13T22:43:53.907984+0000","last_fullsized":"2024-09-13T22:43:53.907984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:43:50.727116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:50.727116+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:03.935709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.900275+0000","last_change":"2024-09-13T22:42:48.550930+0000","last_active":"2024-09-13T22:43:53.900275+0000","last_peered":"2024-09-13T22:43:53.900275+0000","last_clean":"2024-09-13T22:43:53.900275+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:43:53.900275+0000","last_undegraded":"2024-09-13T22:43:53.900275+0000","last_fullsized":"2024-09-13T22:43:53.900275+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:48.550845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:42:48.550845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:43:59.389475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":218,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892726+0000","last_change":"2024-09-13T22:42:52.059935+0000","last_active":"2024-09-13T22:43:53.892726+0000","last_peered":"2024-09-13T22:43:53.892726+0000","last_clean":"2024-09-13T22:43:53.892726+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:43:53.892726+0000","last_undegraded":"2024-09-13T22:43:53.892726+0000","last_fullsized":"2024-09-13T22:43:53.892726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.059600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.059600+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:01.524878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T22:43:53.892483+0000","last_change":"2024-09-13T22:43:38.061761+0000","last_active":"2024-09-13T22:43:53.892483+0000","last_peered":"2024-09-13T22:43:53.892483+0000","last_clean":"2024-09-13T22:43:53.892483+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:43:53.892483+0000","last_undegraded":"2024-09-13T22:43:53.892483+0000","last_fullsized":"2024-09-13T22:43:53.892483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:38.061690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:43:38.061690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:44:56.312683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2822504472,"num_objects":43069,"num_object_clones":0,"num_object_copies":129207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168433,"num_write_kb":6768001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8447143936,"data_stored":8447131720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89729,"ondisk_log_size":89729,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808916,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890324,"kb_used_data":810564,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":92854828,"statfs":{"total":95995035648,"available":95083343872,"internally_reserved":0,"allocated":830017536,"data_stored":829582315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":81656266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135967,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":932028,"kb_used_data":832812,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92813124,"statfs":{"total":95995035648,"available":95040638976,"internally_reserved":0,"allocated":852799488,"data_stored":852386545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528426,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725304,"kb_used_data":658024,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93019848,"statfs":{"total":95995035648,"available":95252324352,"internally_reserved":0,"allocated":673816576,"data_stored":673407563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888180,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865560,"kb_used_data":802440,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92879592,"statfs":{"total":95995035648,"available":95108702208,"internally_reserved":0,"allocated":821698560,"data_stored":821257309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084413,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1750196,"kb_used_data":1610716,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91994956,"statfs":{"total":95995035648,"available":94202834944,"internally_reserved":0,"allocated":1649373184,"data_stored":1648970064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378761,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188848,"kb_used_data":1077148,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92556304,"statfs":{"total":95995035648,"available":94777655296,"internally_reserved":0,"allocated":1102999552,"data_stored":1102596525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1211008,"kb_used_data":1099308,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92534144,"statfs":{"total":95995035648,"available":94754963456,"internally_reserved":0,"allocated":1125691392,"data_stored":1125272400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550276,"kb_used_data":1450932,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92194876,"statfs":{"total":95995035648,"available":94407553024,"internally_reserved":0,"allocated":1485754368,"data_stored":1485328488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1460342784,"data_stored":1460338712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1109331968,"data_stored":1109327896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1094582272,"data_stored":1094582272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1625096192,"data_stored":1625096192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":820314112,"data_stored":820314112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":670367744,"data_stored":670367744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":846401536,"data_stored":846397464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":820707328,"data_stored":820707328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:44:03.039 INFO:tasks.thrashosds.thrasher:Setting 5.f to [1, 3, 5, 7, 0, 2] 2024-09-13T22:44:03.039 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.f', '1', '3', '5', '7', '0', '2'] 2024-09-13T22:44:03.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 5.f 1 3 5 7 0 2 2024-09-13T22:44:03.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:03.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:03.209 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:01.731180+0000 osd.2 (osd.2) 109 : cluster [DBG] 2.0 deep-scrub starts 2024-09-13T22:44:03.209 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:01.750354+0000 osd.2 (osd.2) 110 : cluster [DBG] 2.0 deep-scrub ok 2024-09-13T22:44:03.209 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.040279+0000 osd.5 (osd.5) 146 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.041769+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.353625+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.9 deep-scrub starts 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.546242+0000 mgr.y (mgr.14613) 239 : cluster [DBG] pgmap v233: 148 pgs: 148 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[22840]: cluster 2024-09-13T22:44:01.731180+0000 osd.2 (osd.2) 109 : cluster [DBG] 2.0 deep-scrub starts 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[22840]: cluster 2024-09-13T22:44:01.750354+0000 osd.2 (osd.2) 110 : cluster [DBG] 2.0 deep-scrub ok 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.040279+0000 osd.5 (osd.5) 146 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.041769+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.353625+0000 osd.0 (osd.0) 151 : cluster [DBG] 7.9 deep-scrub starts 2024-09-13T22:44:03.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:02 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.546242+0000 mgr.y (mgr.14613) 239 : cluster [DBG] pgmap v233: 148 pgs: 148 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s 2024-09-13T22:44:03.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:03.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:03.387 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:03.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:03.438 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:44:03.438 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:44:03.452 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:03.464 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:44:03.464 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:44:03.494 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:44:03.494 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:44:03.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:44:03.569 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:44:03.572 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:03.602 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:44:03.602 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:44:03.638 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:44:03.638 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:44:03.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:03.683 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:44:03.683 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:44:03.700 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:03.718 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:44:03.719 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:44:03.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:03.821 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:03.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:03.947 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:03.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:04.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:04.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.285028+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:44:04.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.286329+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:44:04.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.640553+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:44:04.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.641724+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.877349+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:02.881477+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: audit 2024-09-13T22:44:03.049860+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: audit 2024-09-13T22:44:03.053451+0000 mon.b (mon.1) 123 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:03.080776+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: cluster 2024-09-13T22:44:03.084956+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:44:04.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:03 smithi192 bash[24608]: audit 2024-09-13T22:44:03.410342+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.178:0/33889109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:04.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:04.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:04.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:04.321 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:04.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.285028+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.286329+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.640553+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.641724+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.877349+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:02.881477+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: audit 2024-09-13T22:44:03.049860+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: audit 2024-09-13T22:44:03.053451+0000 mon.b (mon.1) 123 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:04.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:03.080776+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:03.084956+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: audit 2024-09-13T22:44:03.410342+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.178:0/33889109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.285028+0000 osd.4 (osd.4) 119 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.286329+0000 osd.4 (osd.4) 120 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.640553+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.641724+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.877349+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:02.881477+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: audit 2024-09-13T22:44:03.049860+0000 mon.a (mon.0) 1034 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:04.355 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: audit 2024-09-13T22:44:03.053451+0000 mon.b (mon.1) 123 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:04.356 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:03.080776+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:44:04.356 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:03.084956+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:44:04.356 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: audit 2024-09-13T22:44:03.410342+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? v1:172.21.15.178:0/33889109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:04.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:04.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:04.576 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:04.609 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:04.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:04.732 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:04.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:04.856 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:04.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:04.978 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:03.600576+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:03.606168+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:03.752467+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:04.095804+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.c scrub starts 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:04.097849+0000 osd.5 (osd.5) 151 : cluster [DBG] 3.c scrub ok 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:04.547219+0000 mgr.y (mgr.14613) 240 : cluster [DBG] pgmap v234: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 977 op/s 2024-09-13T22:44:04.979 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:04 smithi192 bash[24608]: cluster 2024-09-13T22:44:04.604807+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-13T22:44:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:04.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:05.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:05.104 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:05.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:05.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:44:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:44:05.232 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:05.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:03.600576+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:03.606168+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:03.752467+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:04.095804+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.c scrub starts 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:04.097849+0000 osd.5 (osd.5) 151 : cluster [DBG] 3.c scrub ok 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:04.547219+0000 mgr.y (mgr.14613) 240 : cluster [DBG] pgmap v234: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 977 op/s 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[29889]: cluster 2024-09-13T22:44:04.604807+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-13T22:44:05.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:03.600576+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:44:05.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:03.606168+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:44:05.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:03.752467+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub starts 2024-09-13T22:44:05.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:04.095804+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.c scrub starts 2024-09-13T22:44:05.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:04.097849+0000 osd.5 (osd.5) 151 : cluster [DBG] 3.c scrub ok 2024-09-13T22:44:05.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:04.547219+0000 mgr.y (mgr.14613) 240 : cluster [DBG] pgmap v234: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 977 op/s 2024-09-13T22:44:05.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:04 smithi178 bash[22840]: cluster 2024-09-13T22:44:04.604807+0000 osd.2 (osd.2) 112 : cluster [DBG] 7.8 scrub ok 2024-09-13T22:44:05.366 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:05.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:05.510 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:05.620 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:05.646 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:05.751 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:05.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:05.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:05.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:05.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:06.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:06.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:06.079 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:06.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:06.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:05 smithi178 bash[29889]: audit 2024-09-13T22:44:05.090622+0000 mgr.y (mgr.14613) 241 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:06.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:05 smithi178 bash[22840]: audit 2024-09-13T22:44:05.090622+0000 mgr.y (mgr.14613) 241 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:06.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:06.217 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:05 smithi192 bash[24608]: audit 2024-09-13T22:44:05.090622+0000 mgr.y (mgr.14613) 241 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:06.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:06.314 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:06.418 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:06.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:06.550 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:06.575 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:06.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:06.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:06.829 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:06.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:06.944 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:44:06.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:06.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:07.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:07.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:07.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:07.207 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:07.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:06 smithi192 bash[24608]: cluster 2024-09-13T22:44:05.578637+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.4 scrub starts 2024-09-13T22:44:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:06 smithi192 bash[24608]: cluster 2024-09-13T22:44:05.967272+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T22:44:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:06 smithi192 bash[24608]: cluster 2024-09-13T22:44:05.968759+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T22:44:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:06 smithi192 bash[24608]: audit 2024-09-13T22:44:06.547078+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? v1:172.21.15.178:0/2374312270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:44:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:06 smithi192 bash[24608]: cluster 2024-09-13T22:44:06.548036+0000 mgr.y (mgr.14613) 242 : cluster [DBG] pgmap v235: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 697 op/s 2024-09-13T22:44:07.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:07.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[29889]: cluster 2024-09-13T22:44:05.578637+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.4 scrub starts 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[29889]: cluster 2024-09-13T22:44:05.967272+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[29889]: cluster 2024-09-13T22:44:05.968759+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[29889]: audit 2024-09-13T22:44:06.547078+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? v1:172.21.15.178:0/2374312270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[29889]: cluster 2024-09-13T22:44:06.548036+0000 mgr.y (mgr.14613) 242 : cluster [DBG] pgmap v235: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 697 op/s 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[22840]: cluster 2024-09-13T22:44:05.578637+0000 osd.0 (osd.0) 152 : cluster [DBG] 7.4 scrub starts 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[22840]: cluster 2024-09-13T22:44:05.967272+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[22840]: cluster 2024-09-13T22:44:05.968759+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[22840]: audit 2024-09-13T22:44:06.547078+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? v1:172.21.15.178:0/2374312270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:44:07.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:06 smithi178 bash[22840]: cluster 2024-09-13T22:44:06.548036+0000 mgr.y (mgr.14613) 242 : cluster [DBG] pgmap v235: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 697 op/s 2024-09-13T22:44:07.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:07.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:07.591 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:07.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:07.720 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:07.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:44:07.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.845 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:07.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.969 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:07.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:07 smithi192 bash[24608]: cluster 2024-09-13T22:44:06.364783+0000 osd.0 (osd.0) 153 : cluster [DBG] 7.4 scrub ok 2024-09-13T22:44:07.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:07 smithi192 bash[24608]: cluster 2024-09-13T22:44:06.946265+0000 mon.a (mon.0) 1036 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:07.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:07 smithi192 bash[24608]: audit 2024-09-13T22:44:06.948069+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.178:0/2374312270' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:44:07.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:07 smithi192 bash[24608]: cluster 2024-09-13T22:44:06.958485+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-13T22:44:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:07.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:08.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:08.098 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:08.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:08.226 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:08.330 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:08.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[29889]: cluster 2024-09-13T22:44:06.364783+0000 osd.0 (osd.0) 153 : cluster [DBG] 7.4 scrub ok 2024-09-13T22:44:08.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[29889]: cluster 2024-09-13T22:44:06.946265+0000 mon.a (mon.0) 1036 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:08.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[29889]: audit 2024-09-13T22:44:06.948069+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.178:0/2374312270' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:44:08.344 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[29889]: cluster 2024-09-13T22:44:06.958485+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-13T22:44:08.344 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[22840]: cluster 2024-09-13T22:44:06.364783+0000 osd.0 (osd.0) 153 : cluster [DBG] 7.4 scrub ok 2024-09-13T22:44:08.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[22840]: cluster 2024-09-13T22:44:06.946265+0000 mon.a (mon.0) 1036 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:08.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[22840]: audit 2024-09-13T22:44:06.948069+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.178:0/2374312270' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:44:08.345 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:07 smithi178 bash[22840]: cluster 2024-09-13T22:44:06.958485+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-13T22:44:08.369 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:08.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:08.513 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:08.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:08.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:08.671 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:44:08.671 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:44:08.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:08.771 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:08.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:08.901 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:08.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:09.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:09.018 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:08 smithi192 bash[24608]: cluster 2024-09-13T22:44:08.549099+0000 mgr.y (mgr.14613) 243 : cluster [DBG] pgmap v237: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-09-13T22:44:09.032 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:09.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:09.158 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:09.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:09.284 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:09.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:09.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:08 smithi178 bash[29889]: cluster 2024-09-13T22:44:08.549099+0000 mgr.y (mgr.14613) 243 : cluster [DBG] pgmap v237: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-09-13T22:44:09.398 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:08 smithi178 bash[22840]: cluster 2024-09-13T22:44:08.549099+0000 mgr.y (mgr.14613) 243 : cluster [DBG] pgmap v237: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-09-13T22:44:09.420 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:09.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T22:44:09.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:09.547 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:09.652 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:09.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:09.784 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:09.817 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:09.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:09.927 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:44:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:44:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:09.944 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:09.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:10.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:10.073 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:10.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:10.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:10.332 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:10.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:10.463 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:10.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:10.586 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:10.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:10.712 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:10.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:44:10.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:10.874 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:10.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:10.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:11.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.135 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:11.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:11.267 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:11.289 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:11.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:11.391 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:11.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.520 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:11.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:11.652 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:11.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:11 smithi192 bash[24608]: cluster 2024-09-13T22:44:10.549974+0000 mgr.y (mgr.14613) 244 : cluster [DBG] pgmap v238: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 695 op/s 2024-09-13T22:44:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:11.736 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:11 smithi178 bash[29889]: cluster 2024-09-13T22:44:10.549974+0000 mgr.y (mgr.14613) 244 : cluster [DBG] pgmap v238: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 695 op/s 2024-09-13T22:44:11.736 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:11 smithi178 bash[22840]: cluster 2024-09-13T22:44:10.549974+0000 mgr.y (mgr.14613) 244 : cluster [DBG] pgmap v238: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 695 op/s 2024-09-13T22:44:11.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:11.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:11.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.927 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:11.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:12.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:12.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:12.168 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:12.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:12.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:12.330 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:12.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:12.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:12.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.585 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:12.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:12.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:12.707 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:12.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:12.832 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:12.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:12.957 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:12.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:13.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:13.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:13.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:13.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:13.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:13.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:13.334 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:13.374 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:13.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:13.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:13.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:13.664 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:13.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:13.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:13.802 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:13.891 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:13 smithi178 bash[29889]: cluster 2024-09-13T22:44:12.551005+0000 mgr.y (mgr.14613) 245 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 695 op/s 2024-09-13T22:44:13.891 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:13 smithi178 bash[22840]: cluster 2024-09-13T22:44:12.551005+0000 mgr.y (mgr.14613) 245 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 695 op/s 2024-09-13T22:44:13.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:13.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:13 smithi192 bash[24608]: cluster 2024-09-13T22:44:12.551005+0000 mgr.y (mgr.14613) 245 : cluster [DBG] pgmap v239: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 695 op/s 2024-09-13T22:44:13.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:13.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:14.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:14.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:14.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:14.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:14.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:14.303 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:14.317 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:14.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:14.439 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:14.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:14.565 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:14.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:14.690 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:14.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:14.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:14.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:14.954 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:14.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:15.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:15.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:15.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:15.201 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:15.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:15.326 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:15.368 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:15.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:44:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:44:15.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:15.447 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:15.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:15.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:15.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:15.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:15.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:15.840 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[29889]: cluster 2024-09-13T22:44:14.552579+0000 mgr.y (mgr.14613) 246 : cluster [DBG] pgmap v240: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 733 op/s 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[29889]: cluster 2024-09-13T22:44:14.571904+0000 osd.2 (osd.2) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[29889]: cluster 2024-09-13T22:44:14.571995+0000 osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[29889]: cluster 2024-09-13T22:44:14.623437+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[22840]: cluster 2024-09-13T22:44:14.552579+0000 mgr.y (mgr.14613) 246 : cluster [DBG] pgmap v240: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 733 op/s 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[22840]: cluster 2024-09-13T22:44:14.571904+0000 osd.2 (osd.2) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:15.843 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[22840]: cluster 2024-09-13T22:44:14.571995+0000 osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:15.844 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:15 smithi178 bash[22840]: cluster 2024-09-13T22:44:14.623437+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-13T22:44:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:15 smithi192 bash[24608]: cluster 2024-09-13T22:44:14.552579+0000 mgr.y (mgr.14613) 246 : cluster [DBG] pgmap v240: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 733 op/s 2024-09-13T22:44:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:15 smithi192 bash[24608]: cluster 2024-09-13T22:44:14.571904+0000 osd.2 (osd.2) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:15.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:15 smithi192 bash[24608]: cluster 2024-09-13T22:44:14.571995+0000 osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:15.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:15 smithi192 bash[24608]: cluster 2024-09-13T22:44:14.623437+0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-13T22:44:15.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:15.965 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:15.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:16.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:16.087 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:16.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:16.219 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:16.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:16.355 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:16.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:16.483 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:16.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:16.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:16.618 INFO:teuthology.orchestra.run.smithi178.stderr:set 5.f pg_upmap_items mapping to [1->3,5->7,0->2] 2024-09-13T22:44:16.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:16.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:16.801 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:44:16.802 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:44:16.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:16 smithi192 bash[24608]: audit 2024-09-13T22:44:15.098274+0000 mgr.y (mgr.14613) 247 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:16.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:16 smithi192 bash[24608]: audit 2024-09-13T22:44:16.174122+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? v1:172.21.15.178:0/2937115226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.f", "id": [1, 3, 5, 7, 0, 2]}]: dispatch 2024-09-13T22:44:16.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:16.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:16.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:16.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:16 smithi178 bash[29889]: audit 2024-09-13T22:44:15.098274+0000 mgr.y (mgr.14613) 247 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:16.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:16 smithi178 bash[29889]: audit 2024-09-13T22:44:16.174122+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? v1:172.21.15.178:0/2937115226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.f", "id": [1, 3, 5, 7, 0, 2]}]: dispatch 2024-09-13T22:44:16.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:16 smithi178 bash[22840]: audit 2024-09-13T22:44:15.098274+0000 mgr.y (mgr.14613) 247 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:16.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:16 smithi178 bash[22840]: audit 2024-09-13T22:44:16.174122+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? v1:172.21.15.178:0/2937115226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.f", "id": [1, 3, 5, 7, 0, 2]}]: dispatch 2024-09-13T22:44:16.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:17.005 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:17.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:17.134 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:17.241 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:17.274 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:17.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:17.410 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:17.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:17.534 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:17.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:17.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:17.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T22:44:17.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:17.784 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:17.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:17.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:17 smithi192 bash[24608]: cluster 2024-09-13T22:44:16.407897+0000 osd.0 (osd.0) 154 : cluster [DBG] 7.9 deep-scrub ok 2024-09-13T22:44:17.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:17 smithi192 bash[24608]: cluster 2024-09-13T22:44:16.553477+0000 mgr.y (mgr.14613) 248 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 763 op/s 2024-09-13T22:44:17.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:17 smithi192 bash[24608]: audit 2024-09-13T22:44:16.622023+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? v1:172.21.15.178:0/2937115226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.f", "id": [1, 3, 5, 7, 0, 2]}]': finished 2024-09-13T22:44:17.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:17 smithi192 bash[24608]: cluster 2024-09-13T22:44:16.631183+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-13T22:44:17.907 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[29889]: cluster 2024-09-13T22:44:16.407897+0000 osd.0 (osd.0) 154 : cluster [DBG] 7.9 deep-scrub ok 2024-09-13T22:44:17.907 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[29889]: cluster 2024-09-13T22:44:16.553477+0000 mgr.y (mgr.14613) 248 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 763 op/s 2024-09-13T22:44:17.908 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[29889]: audit 2024-09-13T22:44:16.622023+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? v1:172.21.15.178:0/2937115226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.f", "id": [1, 3, 5, 7, 0, 2]}]': finished 2024-09-13T22:44:17.908 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[29889]: cluster 2024-09-13T22:44:16.631183+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-13T22:44:17.908 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[22840]: cluster 2024-09-13T22:44:16.407897+0000 osd.0 (osd.0) 154 : cluster [DBG] 7.9 deep-scrub ok 2024-09-13T22:44:17.908 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[22840]: cluster 2024-09-13T22:44:16.553477+0000 mgr.y (mgr.14613) 248 : cluster [DBG] pgmap v242: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 763 op/s 2024-09-13T22:44:17.908 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[22840]: audit 2024-09-13T22:44:16.622023+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? v1:172.21.15.178:0/2937115226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.f", "id": [1, 3, 5, 7, 0, 2]}]': finished 2024-09-13T22:44:17.908 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:17 smithi178 bash[22840]: cluster 2024-09-13T22:44:16.631183+0000 mon.a (mon.0) 1042 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-13T22:44:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:17.910 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:17.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:18.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:18.037 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:18.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:18.166 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:18.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:18.300 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:18.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:18.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:18.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:18.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:18.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:18.774 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:18.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:18.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[29889]: cluster 2024-09-13T22:44:17.640667+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-13T22:44:18.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[29889]: audit 2024-09-13T22:44:18.049087+0000 mon.b (mon.1) 125 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:44:18.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[29889]: audit 2024-09-13T22:44:18.075322+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:18.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[29889]: audit 2024-09-13T22:44:18.080283+0000 mon.b (mon.1) 126 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:18.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[29889]: cluster 2024-09-13T22:44:18.554242+0000 mgr.y (mgr.14613) 249 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 933 op/s 2024-09-13T22:44:18.882 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[22840]: cluster 2024-09-13T22:44:17.640667+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-13T22:44:18.882 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[22840]: audit 2024-09-13T22:44:18.049087+0000 mon.b (mon.1) 125 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:44:18.882 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[22840]: audit 2024-09-13T22:44:18.075322+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:18.882 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[22840]: audit 2024-09-13T22:44:18.080283+0000 mon.b (mon.1) 126 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:18.882 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:18 smithi178 bash[22840]: cluster 2024-09-13T22:44:18.554242+0000 mgr.y (mgr.14613) 249 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 933 op/s 2024-09-13T22:44:18.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:18 smithi192 bash[24608]: cluster 2024-09-13T22:44:17.640667+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-13T22:44:18.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:18 smithi192 bash[24608]: audit 2024-09-13T22:44:18.049087+0000 mon.b (mon.1) 125 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:44:18.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:18 smithi192 bash[24608]: audit 2024-09-13T22:44:18.075322+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:18.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:18 smithi192 bash[24608]: audit 2024-09-13T22:44:18.080283+0000 mon.b (mon.1) 126 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:18.894 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:18 smithi192 bash[24608]: cluster 2024-09-13T22:44:18.554242+0000 mgr.y (mgr.14613) 249 : cluster [DBG] pgmap v245: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.7 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 933 op/s 2024-09-13T22:44:18.906 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:18.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:19.009 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:19.042 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:19.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:19.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:19.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:19.300 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:19.382 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:44:19.383 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:44:19.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:19.421 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:44:19.421 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:44:19.449 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:19.464 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:44:19.464 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:44:19.514 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:44:19.514 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:44:19.552 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:44:19.553 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:44:19.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:19.576 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:19.588 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:44:19.589 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:44:19.622 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:44:19.622 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:44:19.662 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:44:19.662 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:44:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[29889]: audit 2024-09-13T22:44:18.950312+0000 mon.b (mon.1) 127 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:44:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[29889]: audit 2024-09-13T22:44:18.954534+0000 mon.b (mon.1) 128 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:44:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[29889]: audit 2024-09-13T22:44:18.964416+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:19.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[29889]: audit 2024-09-13T22:44:19.329867+0000 mon.a (mon.0) 1046 : audit [DBG] from='client.? v1:172.21.15.178:0/2285420320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:19.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[22840]: audit 2024-09-13T22:44:18.950312+0000 mon.b (mon.1) 127 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:44:19.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[22840]: audit 2024-09-13T22:44:18.954534+0000 mon.b (mon.1) 128 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:44:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[22840]: audit 2024-09-13T22:44:18.964416+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:19.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[22840]: audit 2024-09-13T22:44:19.329867+0000 mon.a (mon.0) 1046 : audit [DBG] from='client.? v1:172.21.15.178:0/2285420320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:19.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:19.702 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:19.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:19.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:19 smithi192 bash[24608]: audit 2024-09-13T22:44:18.950312+0000 mon.b (mon.1) 127 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:44:19.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:19 smithi192 bash[24608]: audit 2024-09-13T22:44:18.954534+0000 mon.b (mon.1) 128 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:44:19.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:19 smithi192 bash[24608]: audit 2024-09-13T22:44:18.964416+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:44:19.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:19 smithi192 bash[24608]: audit 2024-09-13T22:44:19.329867+0000 mon.a (mon.0) 1046 : audit [DBG] from='client.? v1:172.21.15.178:0/2285420320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:19.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:19.966 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:19.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:20.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:20.094 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:20.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:20.217 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:20.221 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:44:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:44:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:20.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:20.338 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:20.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:20.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:20.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:20.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:20.653 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:44:20.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:20.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:20 smithi192 bash[24608]: audit 2024-09-13T22:44:20.049619+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.178:0/2487286929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:20.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:20 smithi192 bash[24608]: audit 2024-09-13T22:44:20.050291+0000 mon.a (mon.0) 1047 : audit [INF] from='client.24931 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:20.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:20 smithi192 bash[24608]: cluster 2024-09-13T22:44:20.555382+0000 mgr.y (mgr.14613) 250 : cluster [DBG] pgmap v246: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 589 op/s 2024-09-13T22:44:20.731 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:20.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:20.854 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:20.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:20 smithi178 bash[29889]: audit 2024-09-13T22:44:20.049619+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.178:0/2487286929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:20.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:20 smithi178 bash[29889]: audit 2024-09-13T22:44:20.050291+0000 mon.a (mon.0) 1047 : audit [INF] from='client.24931 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:20.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:20 smithi178 bash[29889]: cluster 2024-09-13T22:44:20.555382+0000 mgr.y (mgr.14613) 250 : cluster [DBG] pgmap v246: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 589 op/s 2024-09-13T22:44:20.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:20 smithi178 bash[22840]: audit 2024-09-13T22:44:20.049619+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.178:0/2487286929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:20.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:20 smithi178 bash[22840]: audit 2024-09-13T22:44:20.050291+0000 mon.a (mon.0) 1047 : audit [INF] from='client.24931 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:20.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:20 smithi178 bash[22840]: cluster 2024-09-13T22:44:20.555382+0000 mgr.y (mgr.14613) 250 : cluster [DBG] pgmap v246: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 589 op/s 2024-09-13T22:44:20.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:20.985 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:20.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:21.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:21.116 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:21.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:21.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:21.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:21.375 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:21.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:21.501 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:21.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:21.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:21.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:21.758 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:21.759 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:21 smithi192 bash[24608]: cluster 2024-09-13T22:44:20.649092+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:21.759 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:21 smithi192 bash[24608]: audit 2024-09-13T22:44:20.650445+0000 mon.a (mon.0) 1049 : audit [INF] from='client.24931 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:44:21.759 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:21 smithi192 bash[24608]: cluster 2024-09-13T22:44:20.654168+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-13T22:44:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:21.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:21.881 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:21.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:21 smithi178 bash[29889]: cluster 2024-09-13T22:44:20.649092+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:21 smithi178 bash[29889]: audit 2024-09-13T22:44:20.650445+0000 mon.a (mon.0) 1049 : audit [INF] from='client.24931 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:44:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:21 smithi178 bash[29889]: cluster 2024-09-13T22:44:20.654168+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-13T22:44:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:21 smithi178 bash[22840]: cluster 2024-09-13T22:44:20.649092+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:21 smithi178 bash[22840]: audit 2024-09-13T22:44:20.650445+0000 mon.a (mon.0) 1049 : audit [INF] from='client.24931 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:44:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:21 smithi178 bash[22840]: cluster 2024-09-13T22:44:20.654168+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-13T22:44:21.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:22.016 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:22.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:22.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:22.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:22.300 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:22.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:22.427 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:22.517 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:22.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:22.554 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:22.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:22.669 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:22 smithi192 bash[24608]: cluster 2024-09-13T22:44:22.556041+0000 mgr.y (mgr.14613) 251 : cluster [DBG] pgmap v248: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 589 op/s 2024-09-13T22:44:22.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:22.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:22.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:22.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:22.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:22 smithi178 bash[29889]: cluster 2024-09-13T22:44:22.556041+0000 mgr.y (mgr.14613) 251 : cluster [DBG] pgmap v248: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 589 op/s 2024-09-13T22:44:22.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:22 smithi178 bash[22840]: cluster 2024-09-13T22:44:22.556041+0000 mgr.y (mgr.14613) 251 : cluster [DBG] pgmap v248: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 589 op/s 2024-09-13T22:44:22.943 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:22.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:23.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:23.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:23.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:23.195 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:23.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:44:23.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:23.325 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:23.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:23.461 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:23.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:23.590 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:23.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:23.723 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:23.828 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:23.854 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:23.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:44:23.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:23.983 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:23.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:24.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:24.131 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:24.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:24.269 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:24.373 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:24.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:24.430 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:44:24.430 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:44:24.430 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T22:44:24.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T22:44:24.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:24.551 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:24.655 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:24.696 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:24.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:24.837 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:24.874 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:44:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:44:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-09-13T22:44:24.875 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:44:24.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:24.966 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:24.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:25.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:25.106 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:25.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:25.238 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:25.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:25.367 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:25.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:44:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:44:25.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:25.506 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:25.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:25.633 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:25.735 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:25.762 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:25.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T22:44:25.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:25.878 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:25 smithi178 bash[29889]: cluster 2024-09-13T22:44:24.556954+0000 mgr.y (mgr.14613) 252 : cluster [DBG] pgmap v249: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-13T22:44:25.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:25 smithi178 bash[22840]: cluster 2024-09-13T22:44:24.556954+0000 mgr.y (mgr.14613) 252 : cluster [DBG] pgmap v249: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-13T22:44:25.900 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:25.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:25.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:25 smithi192 bash[24608]: cluster 2024-09-13T22:44:24.556954+0000 mgr.y (mgr.14613) 252 : cluster [DBG] pgmap v249: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-13T22:44:26.006 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:26.047 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:26.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:26.178 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:26.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:26.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:26.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:26.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:26.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:26.557 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:26.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:26.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:26.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:26.799 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:26 smithi192 bash[24608]: audit 2024-09-13T22:44:25.106189+0000 mgr.y (mgr.14613) 253 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:26.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:26.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:26.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:26 smithi178 bash[29889]: audit 2024-09-13T22:44:25.106189+0000 mgr.y (mgr.14613) 253 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:26.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:26 smithi178 bash[22840]: audit 2024-09-13T22:44:25.106189+0000 mgr.y (mgr.14613) 253 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:26.943 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:26.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:27.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:27.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:27.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:27.206 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:27.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:27.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:27.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:27.457 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:27.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:27.585 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:27.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:27.715 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:27.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:27.862 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:27.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:27.867 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:27 smithi178 bash[29889]: cluster 2024-09-13T22:44:26.557356+0000 mgr.y (mgr.14613) 254 : cluster [DBG] pgmap v250: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 850 op/s 2024-09-13T22:44:27.867 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:27 smithi178 bash[22840]: cluster 2024-09-13T22:44:26.557356+0000 mgr.y (mgr.14613) 254 : cluster [DBG] pgmap v250: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 850 op/s 2024-09-13T22:44:27.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:27 smithi192 bash[24608]: cluster 2024-09-13T22:44:26.557356+0000 mgr.y (mgr.14613) 254 : cluster [DBG] pgmap v250: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 850 op/s 2024-09-13T22:44:27.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:27.978 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:27.978 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:28.009 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:28.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:28.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:28.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:28.278 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:28.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:28.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:28.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:28.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:28.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:28.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:28.803 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:28.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:28 smithi192 bash[24608]: cluster 2024-09-13T22:44:28.557830+0000 mgr.y (mgr.14613) 255 : cluster [DBG] pgmap v251: 148 pgs: 148 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 648 op/s 2024-09-13T22:44:28.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:28.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:28.996 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:29.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:29.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:29.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:29.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:29.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:29.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:29.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:29.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:29.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:29.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:29.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:29.714 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:28 smithi178 bash[22840]: cluster 2024-09-13T22:44:28.557830+0000 mgr.y (mgr.14613) 255 : cluster [DBG] pgmap v251: 148 pgs: 148 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 648 op/s 2024-09-13T22:44:29.715 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:28 smithi178 bash[29889]: cluster 2024-09-13T22:44:28.557830+0000 mgr.y (mgr.14613) 255 : cluster [DBG] pgmap v251: 148 pgs: 148 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 648 op/s 2024-09-13T22:44:29.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:29.838 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:29.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:29.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:30.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:30.087 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:29 smithi192 bash[24608]: cluster 2024-09-13T22:44:28.744396+0000 osd.7 (osd.7) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:30.087 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:29 smithi192 bash[24608]: cluster 2024-09-13T22:44:28.744464+0000 osd.7 (osd.7) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:30.088 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:29 smithi192 bash[24608]: cluster 2024-09-13T22:44:28.847131+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-13T22:44:30.107 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:30.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:30.229 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:44:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:30.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[29889]: cluster 2024-09-13T22:44:28.744396+0000 osd.7 (osd.7) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:30.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[29889]: cluster 2024-09-13T22:44:28.744464+0000 osd.7 (osd.7) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:30.230 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[29889]: cluster 2024-09-13T22:44:28.847131+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-13T22:44:30.230 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[22840]: cluster 2024-09-13T22:44:28.744396+0000 osd.7 (osd.7) 143 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:30.230 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[22840]: cluster 2024-09-13T22:44:28.744464+0000 osd.7 (osd.7) 144 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:30.230 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:29 smithi178 bash[22840]: cluster 2024-09-13T22:44:28.847131+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-13T22:44:30.231 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:30.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:30.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:30.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:30.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:30.571 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:30.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:30.618 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:30.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:30.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:30.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:30.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:30.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:31.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:31.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:31.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:30 smithi192 bash[24608]: cluster 2024-09-13T22:44:30.558355+0000 mgr.y (mgr.14613) 256 : cluster [DBG] pgmap v253: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 892 op/s 2024-09-13T22:44:31.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:31.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:31.293 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:31.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:31.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:31 smithi178 bash[22840]: cluster 2024-09-13T22:44:30.558355+0000 mgr.y (mgr.14613) 256 : cluster [DBG] pgmap v253: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 892 op/s 2024-09-13T22:44:31.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:31 smithi178 bash[29889]: cluster 2024-09-13T22:44:30.558355+0000 mgr.y (mgr.14613) 256 : cluster [DBG] pgmap v253: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 892 op/s 2024-09-13T22:44:31.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:31.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:31.542 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:31.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:31.670 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:31.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:31.804 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:31.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:31.943 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:31.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:32.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:32.078 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:32.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:32.216 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:32.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:32.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.359 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:32.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:32.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:32.480 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:32.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:32.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:32.713 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:32.749 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:32.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:32.880 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:32.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:32.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:33.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:33.048 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:44:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:44:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:44:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:44:33.048 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:44:33.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:33.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:33.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:33.255 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:33.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:33.392 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:33.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:33.518 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:33.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:33.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:33.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:33.756 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:33 smithi192 bash[24608]: cluster 2024-09-13T22:44:32.558685+0000 mgr.y (mgr.14613) 257 : cluster [DBG] pgmap v254: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 883 op/s 2024-09-13T22:44:33.757 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:33 smithi192 bash[24608]: audit 2024-09-13T22:44:33.099386+0000 mon.b (mon.1) 130 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:33.772 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:33.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T22:44:33.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:33 smithi178 bash[22840]: cluster 2024-09-13T22:44:32.558685+0000 mgr.y (mgr.14613) 257 : cluster [DBG] pgmap v254: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 883 op/s 2024-09-13T22:44:33.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:33 smithi178 bash[22840]: audit 2024-09-13T22:44:33.099386+0000 mon.b (mon.1) 130 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:33.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:33 smithi178 bash[29889]: cluster 2024-09-13T22:44:32.558685+0000 mgr.y (mgr.14613) 257 : cluster [DBG] pgmap v254: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 883 op/s 2024-09-13T22:44:33.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:33 smithi178 bash[29889]: audit 2024-09-13T22:44:33.099386+0000 mon.b (mon.1) 130 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:33.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:33.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:34.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:34.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:34.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:34.192 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:34.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:34.319 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:34.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:34.460 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:34.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:34.586 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:34.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:34.696 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:44:34.696 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:44:34.710 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:34.736 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:44:34.736 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:44:34.782 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:44:34.783 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:44:34.812 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:34.822 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:34 smithi178 bash[29889]: audit 2024-09-13T22:44:34.600255+0000 mon.a (mon.0) 1052 : audit [DBG] from='client.? v1:172.21.15.178:0/483479355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:34.823 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:34 smithi178 bash[22840]: audit 2024-09-13T22:44:34.600255+0000 mon.a (mon.0) 1052 : audit [DBG] from='client.? v1:172.21.15.178:0/483479355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:34.843 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:44:34.844 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:44:34.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:34.874 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:44:34.874 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:44:34.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:34 smithi192 bash[24608]: audit 2024-09-13T22:44:34.600255+0000 mon.a (mon.0) 1052 : audit [DBG] from='client.? v1:172.21.15.178:0/483479355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:34.922 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:44:34.922 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:44:34.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:34.976 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:44:34.976 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:44:34.986 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:35.015 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:44:35.015 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:44:35.081 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:44:35.081 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:44:35.081 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 166, 2024-09-13T22:44:35.081 INFO:teuthology.orchestra.run.smithi178.stdout: "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "created": "2024-09-13T22:27:31.500176+0000", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "modified": "2024-09-13T22:44:28.827590+0000", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "last_up_change": "2024-09-13T22:37:07.372146+0000", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "last_in_change": "2024-09-13T22:36:38.469358+0000", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_num": 5806080, 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_set": [ 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "nodeep-scrub", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "noscrub", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "pglog_hardlimit", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "purged_snapdirs", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_deletes", 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: "sortbitwise" 2024-09-13T22:44:35.082 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_version": 18, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_snapshot": "", 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_max": 7, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "max_osd": 8, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "require_min_compat_client": "luminous", 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "min_compat_client": "luminous", 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "require_osd_release": "squid", 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "allow_crimson": false, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "pools": [ 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 1, 2024-09-13T22:44:35.083 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".mgr", 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:32:40.780155+0000", 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.084 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 1, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 1, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 1, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 1, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 1, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.085 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "22", 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.086 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.087 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.088 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_max": 32, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_min": 1 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr": {} 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 7.8899998664855957, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 7.8899998664855957, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T22:44:35.089 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 3, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 3, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 2, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".rgw.root", 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:39.227580+0000", 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:44:35.090 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:44:35.091 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "76", 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.092 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.093 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.094 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.095 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 3, 2024-09-13T22:44:35.096 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.log", 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.229251+0000", 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.097 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.098 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "78", 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.099 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.100 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 4, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "datapool", 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.624329+0000", 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 8193, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 3, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 3, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 3, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 3, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 3, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "85", 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 2, 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 85, 2024-09-13T22:44:35.105 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.106 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.107 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd": {} 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.6500000953674316, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.6500000953674316, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T22:44:35.108 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 5, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.control", 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:43.199974+0000", 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.109 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:44:35.110 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "80", 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.111 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.112 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.113 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.114 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.25, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.25, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.25, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.25, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 6, 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.meta", 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:45.423695+0000", 2024-09-13T22:44:35.115 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.116 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "82", 2024-09-13T22:44:35.117 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.118 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.119 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:44:35.120 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_bias": 4 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.75, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.75, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.75, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.75, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.121 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 7, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "unique_pool_0", 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:40:53.509919+0000", 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:44:35.122 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 16, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 16, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 16, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 16, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 16, 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:44:35.123 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "119", 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:44:35.124 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:44:35.125 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:44:35.126 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "rados": {} 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:44:35.127 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac", 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.128 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 8, 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6801", 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.129 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.130 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.130 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.130 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.130 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.130 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6802", 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6804", 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:44:35.132 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6803", 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6801/1489719354", 2024-09-13T22:44:35.133 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6802/1489719354", 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6804/1489719354", 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6803/1489719354", 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa", 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.134 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 13, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6805", 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:44:35.135 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6806", 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6808", 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6807", 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:44:35.137 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6805/4187084287", 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6806/4187084287", 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6808/4187084287", 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6807/4187084287", 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:44:35.138 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "40a79075-5102-4f6c-b41a-255102064624", 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 19, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.139 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6809", 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6810", 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6812", 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.141 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6811", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6809/2303802730", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6810/2303802730", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6812/2303802730", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6811/2303802730", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.142 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad", 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 28, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.143 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6813", 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.144 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6814", 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6816", 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.145 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6815", 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6813/320675660", 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6814/320675660", 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6816/320675660", 2024-09-13T22:44:35.146 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6815/320675660", 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b", 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.147 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 34, 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6800", 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.148 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6801", 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.149 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6803", 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6802", 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.150 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6800/3593670527", 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6801/3593670527", 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6803/3593670527", 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6802/3593670527", 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8", 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.151 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 45, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6804", 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:44:35.152 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6805", 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.153 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6807", 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6806", 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:44:35.154 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6804/3271425338", 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6805/3271425338", 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6807/3271425338", 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6806/3271425338", 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:44:35.155 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98", 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 57, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.156 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6808", 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6809", 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.157 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6811", 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.158 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6810", 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6808/684205162", 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6809/684205162", 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6811/684205162", 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6810/684205162", 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.159 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52", 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 67, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 163, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:44:35.160 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6812", 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.161 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6813", 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6815", 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:44:35.162 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6814", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6812/1303971738", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6813/1303971738", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1303971738", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6814/1303971738", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:44:35.163 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_xinfo": [ 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:21.712932+0000", 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:51.497567+0000", 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:44:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:44:14.572000+0000", 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.166 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:42.507720+0000", 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:38.108862+0000", 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.167 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:52.994941+0000", 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.168 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:43:40.681119+0000", 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:44:28.744467+0000", 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:44:35.169 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap": [], 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_items": [ 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "2.0", 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 7, 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 2 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.170 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "5.f", 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 5, 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 7 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_primaries": [], 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_temp": [], 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_temp": [], 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "blocklist": { 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/268210503": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:44:35.171 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2039160385": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/669877409": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/129655740": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3632788039": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/388635774": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3265105147": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2959920382": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2494442689": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2676020773": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2675812306": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/1834442338": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2835351236": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3836928220": "2024-09-14T22:38:32.396576+0000" 2024-09-13T22:44:35.172 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "range_blocklist": {}, 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profiles": { 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "default": { 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "crush-failure-domain": "osd", 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "k": "2", 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "m": "1", 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "plugin": "jerasure", 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "technique": "reed_sol_van" 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps_queue": [], 2024-09-13T22:44:35.173 INFO:teuthology.orchestra.run.smithi178.stdout: "new_removed_snaps": [], 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "new_purged_snaps": [], 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_node_flags": {}, 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "device_class_flags": {}, 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode": { 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_enabled": false, 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_bucket_count": 0, 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "degraded_stretch_mode": 0, 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "recovering_stretch_mode": 0, 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_bucket": 0 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:44:35.174 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:44:35.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:35.200 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:35.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:35.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:35.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:35.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:35.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:44:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:44:35.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:35.465 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:35.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:35.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:35.592 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:35.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:35.635 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:44:35.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:35.728 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:35.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:35.834 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:35.843 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[22840]: cluster 2024-09-13T22:44:34.563185+0000 mgr.y (mgr.14613) 258 : cluster [DBG] pgmap v255: 148 pgs: 148 active+clean; 811 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 853 op/s 2024-09-13T22:44:35.843 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[22840]: audit 2024-09-13T22:44:35.078993+0000 mon.c (mon.2) 32 : audit [DBG] from='client.? v1:172.21.15.178:0/3883270356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:44:35.844 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[22840]: audit 2024-09-13T22:44:35.114288+0000 mgr.y (mgr.14613) 259 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:35.844 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[22840]: audit 2024-09-13T22:44:35.124891+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? v1:172.21.15.178:0/2190006629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:44:35.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[29889]: cluster 2024-09-13T22:44:34.563185+0000 mgr.y (mgr.14613) 258 : cluster [DBG] pgmap v255: 148 pgs: 148 active+clean; 811 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 853 op/s 2024-09-13T22:44:35.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[29889]: audit 2024-09-13T22:44:35.078993+0000 mon.c (mon.2) 32 : audit [DBG] from='client.? v1:172.21.15.178:0/3883270356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:44:35.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[29889]: audit 2024-09-13T22:44:35.114288+0000 mgr.y (mgr.14613) 259 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:35.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:35 smithi178 bash[29889]: audit 2024-09-13T22:44:35.124891+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? v1:172.21.15.178:0/2190006629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:44:35.860 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:35.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:35.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:35 smithi192 bash[24608]: cluster 2024-09-13T22:44:34.563185+0000 mgr.y (mgr.14613) 258 : cluster [DBG] pgmap v255: 148 pgs: 148 active+clean; 811 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 853 op/s 2024-09-13T22:44:35.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:35 smithi192 bash[24608]: audit 2024-09-13T22:44:35.078993+0000 mon.c (mon.2) 32 : audit [DBG] from='client.? v1:172.21.15.178:0/3883270356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:44:35.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:35 smithi192 bash[24608]: audit 2024-09-13T22:44:35.114288+0000 mgr.y (mgr.14613) 259 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:35.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:35 smithi192 bash[24608]: audit 2024-09-13T22:44:35.124891+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? v1:172.21.15.178:0/2190006629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:44:35.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:36.001 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:36.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:36.142 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:36.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:36.279 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:36.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:36.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:36.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:36.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:36.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:36.665 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[29889]: cluster 2024-09-13T22:44:35.624644+0000 mon.a (mon.0) 1054 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[29889]: audit 2024-09-13T22:44:35.635161+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? v1:172.21.15.178:0/2190006629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[29889]: cluster 2024-09-13T22:44:35.643518+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[29889]: cluster 2024-09-13T22:44:36.563786+0000 mgr.y (mgr.14613) 260 : cluster [DBG] pgmap v257: 148 pgs: 148 active+clean; 811 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[22840]: cluster 2024-09-13T22:44:35.624644+0000 mon.a (mon.0) 1054 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[22840]: audit 2024-09-13T22:44:35.635161+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? v1:172.21.15.178:0/2190006629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[22840]: cluster 2024-09-13T22:44:35.643518+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-13T22:44:36.678 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:36 smithi178 bash[22840]: cluster 2024-09-13T22:44:36.563786+0000 mgr.y (mgr.14613) 260 : cluster [DBG] pgmap v257: 148 pgs: 148 active+clean; 811 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T22:44:36.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:36.778 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:36 smithi192 bash[24608]: cluster 2024-09-13T22:44:35.624644+0000 mon.a (mon.0) 1054 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:44:36.778 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:36 smithi192 bash[24608]: audit 2024-09-13T22:44:35.635161+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? v1:172.21.15.178:0/2190006629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:44:36.778 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:36 smithi192 bash[24608]: cluster 2024-09-13T22:44:35.643518+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-13T22:44:36.778 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:36 smithi192 bash[24608]: cluster 2024-09-13T22:44:36.563786+0000 mgr.y (mgr.14613) 260 : cluster [DBG] pgmap v257: 148 pgs: 148 active+clean; 811 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T22:44:36.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:36.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:36.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:36.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:37.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:37.052 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:37.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.195 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:37.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:37.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.323 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:37.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:37.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:37.458 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:37.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:37.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:37.711 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:37 smithi178 bash[29889]: cluster 2024-09-13T22:44:36.741424+0000 osd.0 (osd.0) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:37.711 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:37 smithi178 bash[29889]: cluster 2024-09-13T22:44:36.741486+0000 osd.0 (osd.0) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:37.711 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:37 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.333934+0000 osd.3 (osd.3) 166 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:44:37.711 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:37 smithi178 bash[22840]: cluster 2024-09-13T22:44:36.741424+0000 osd.0 (osd.0) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:37.711 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:37 smithi178 bash[22840]: cluster 2024-09-13T22:44:36.741486+0000 osd.0 (osd.0) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:37.711 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:37 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.333934+0000 osd.3 (osd.3) 166 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:44:37.730 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:37.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:37.856 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:37.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:37 smithi192 bash[24608]: cluster 2024-09-13T22:44:36.741424+0000 osd.0 (osd.0) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:37.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:37 smithi192 bash[24608]: cluster 2024-09-13T22:44:36.741486+0000 osd.0 (osd.0) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:37.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:37 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.333934+0000 osd.3 (osd.3) 166 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:44:37.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:37.985 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:37.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:38.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:38.107 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:38.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:38.240 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:38.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:38.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:38.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:38.504 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:38.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:38.628 INFO:tasks.thrashosds.thrasher:j is {'epoch': 166, 'fsid': '03ae43e2-721f-11ef-bceb-c7b262605968', 'created': '2024-09-13T22:27:31.500176+0000', 'modified': '2024-09-13T22:44:28.827590+0000', 'last_up_change': '2024-09-13T22:37:07.372146+0000', 'last_in_change': '2024-09-13T22:36:38.469358+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T22:32:40.780155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T22:37:39.227580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T22:37:41.229251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-09-13T22:37:41.624329+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 85, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T22:37:43.199974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T22:37:45.423695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T22:40:53.509919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '394f2664-bba1-4b15-ace8-78f70f970cac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6801', 'nonce': 1489719354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6802', 'nonce': 1489719354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6804', 'nonce': 1489719354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6803', 'nonce': 1489719354}]}, 'public_addr': '172.21.15.178:6801/1489719354', 'cluster_addr': '172.21.15.178:6802/1489719354', 'heartbeat_back_addr': '172.21.15.178:6804/1489719354', 'heartbeat_front_addr': '172.21.15.178:6803/1489719354', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e65c08b-6431-4849-a6af-2da9cb1efdfa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6805', 'nonce': 4187084287}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6806', 'nonce': 4187084287}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6808', 'nonce': 4187084287}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6807', 'nonce': 4187084287}]}, 'public_addr': '172.21.15.178:6805/4187084287', 'cluster_addr': '172.21.15.178:6806/4187084287', 'heartbeat_back_addr': '172.21.15.178:6808/4187084287', 'heartbeat_front_addr': '172.21.15.178:6807/4187084287', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '40a79075-5102-4f6c-b41a-255102064624', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6809', 'nonce': 2303802730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6810', 'nonce': 2303802730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6812', 'nonce': 2303802730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6811', 'nonce': 2303802730}]}, 'public_addr': '172.21.15.178:6809/2303802730', 'cluster_addr': '172.21.15.178:6810/2303802730', 'heartbeat_back_addr': '172.21.15.178:6812/2303802730', 'heartbeat_front_addr': '172.21.15.178:6811/2303802730', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3962c796-6f97-4cc6-a186-4b6ef58d67ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6813', 'nonce': 320675660}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6814', 'nonce': 320675660}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6816', 'nonce': 320675660}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6815', 'nonce': 320675660}]}, 'public_addr': '172.21.15.178:6813/320675660', 'cluster_addr': '172.21.15.178:6814/320675660', 'heartbeat_back_addr': '172.21.15.178:6816/320675660', 'heartbeat_front_addr': '172.21.15.178:6815/320675660', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b89db2f-9e81-40b1-89bd-8de1dbff133b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 34, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 3593670527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 3593670527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 3593670527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 3593670527}]}, 'public_addr': '172.21.15.192:6800/3593670527', 'cluster_addr': '172.21.15.192:6801/3593670527', 'heartbeat_back_addr': '172.21.15.192:6803/3593670527', 'heartbeat_front_addr': '172.21.15.192:6802/3593670527', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'efa849a2-d789-4fdd-acc7-04f2c1bebfa8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 3271425338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 3271425338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 3271425338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 3271425338}]}, 'public_addr': '172.21.15.192:6804/3271425338', 'cluster_addr': '172.21.15.192:6805/3271425338', 'heartbeat_back_addr': '172.21.15.192:6807/3271425338', 'heartbeat_front_addr': '172.21.15.192:6806/3271425338', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '54a26ae9-bf31-49cf-a924-901eed4a0c98', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 684205162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 684205162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 684205162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 684205162}]}, 'public_addr': '172.21.15.192:6808/684205162', 'cluster_addr': '172.21.15.192:6809/684205162', 'heartbeat_back_addr': '172.21.15.192:6811/684205162', 'heartbeat_front_addr': '172.21.15.192:6810/684205162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '13e146db-61ac-4fe6-8f26-cbf8d1623a52', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 1303971738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1303971738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1303971738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 1303971738}]}, 'public_addr': '172.21.15.192:6812/1303971738', 'cluster_addr': '172.21.15.192:6813/1303971738', 'heartbeat_back_addr': '172.21.15.192:6815/1303971738', 'heartbeat_front_addr': '172.21.15.192:6814/1303971738', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:21.712932+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:51.497567+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:44:14.572000+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:42.507720+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:38.108862+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:52.994941+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:43:40.681119+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:44:28.744467+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '5.f', 'mappings': [{'from': 5, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.178:0/268210503': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:6800/2039160385': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/669877409': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/129655740': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/3632788039': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:6800/388635774': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3265105147': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2959920382': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/2494442689': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:6800/2676020773': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2675812306': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/1834442338': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2835351236': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3836928220': '2024-09-14T22:38:32.396576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T22:44:38.628 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.0 2024-09-13T22:44:38.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap-items 2.0 2024-09-13T22:44:38.632 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:38.673 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:38.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:36.733533+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:36.735584+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.057582+0000 osd.5 (osd.5) 152 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.060654+0000 osd.5 (osd.5) 153 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.351111+0000 osd.3 (osd.3) 167 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.689178+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.698666+0000 osd.0 (osd.0) 157 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:44:38.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:37.772329+0000 osd.0 (osd.0) 158 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:44:38.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.045536+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:44:38.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.049674+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:44:38.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.362134+0000 osd.1 (osd.1) 140 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:44:38.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.367560+0000 osd.1 (osd.1) 141 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:44:38.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:38 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.564328+0000 mgr.y (mgr.14613) 261 : cluster [DBG] pgmap v259: 148 pgs: 148 active+clean; 794 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 500 op/s 2024-09-13T22:44:38.761 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:38.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:38.884 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:38.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:36.733533+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:36.735584+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.057582+0000 osd.5 (osd.5) 152 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.060654+0000 osd.5 (osd.5) 153 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.351111+0000 osd.3 (osd.3) 167 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.689178+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.698666+0000 osd.0 (osd.0) 157 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:44:38.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:37.772329+0000 osd.0 (osd.0) 158 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.045536+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.049674+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.362134+0000 osd.1 (osd.1) 140 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.367560+0000 osd.1 (osd.1) 141 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.564328+0000 mgr.y (mgr.14613) 261 : cluster [DBG] pgmap v259: 148 pgs: 148 active+clean; 794 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 500 op/s 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:36.733533+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:36.735584+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.057582+0000 osd.5 (osd.5) 152 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.060654+0000 osd.5 (osd.5) 153 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:44:38.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.351111+0000 osd.3 (osd.3) 167 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.689178+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.698666+0000 osd.0 (osd.0) 157 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:37.772329+0000 osd.0 (osd.0) 158 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.045536+0000 osd.4 (osd.4) 121 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.049674+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.362134+0000 osd.1 (osd.1) 140 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.367560+0000 osd.1 (osd.1) 141 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:44:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:38 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.564328+0000 mgr.y (mgr.14613) 261 : cluster [DBG] pgmap v259: 148 pgs: 148 active+clean; 794 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 500 op/s 2024-09-13T22:44:38.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:39.020 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:39.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:39.149 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:39.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:39.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:39.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:39.410 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:39.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:39.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:39.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:39.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:39.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:39.791 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:39.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:39.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:39 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.377382+0000 osd.3 (osd.3) 168 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:44:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:39 smithi192 bash[24608]: cluster 2024-09-13T22:44:38.382110+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:44:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:39 smithi192 bash[24608]: cluster 2024-09-13T22:44:39.002177+0000 osd.4 (osd.4) 123 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:44:39.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:39 smithi192 bash[24608]: cluster 2024-09-13T22:44:39.003062+0000 osd.4 (osd.4) 124 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:44:39.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:39.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:40.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:40.062 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:40.166 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:40.174 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:44:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.377382+0000 osd.3 (osd.3) 168 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[29889]: cluster 2024-09-13T22:44:38.382110+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[29889]: cluster 2024-09-13T22:44:39.002177+0000 osd.4 (osd.4) 123 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[29889]: cluster 2024-09-13T22:44:39.003062+0000 osd.4 (osd.4) 124 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.377382+0000 osd.3 (osd.3) 168 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[22840]: cluster 2024-09-13T22:44:38.382110+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[22840]: cluster 2024-09-13T22:44:39.002177+0000 osd.4 (osd.4) 123 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:44:40.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:39 smithi178 bash[22840]: cluster 2024-09-13T22:44:39.003062+0000 osd.4 (osd.4) 124 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:44:40.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:40.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:40.320 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:40.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:40.449 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:40.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:44:40.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:40.575 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:40.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:40.702 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:40.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:40.830 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:44:40.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:40.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:40.944 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:40 smithi192 bash[24608]: cluster 2024-09-13T22:44:39.037479+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:44:40.944 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:40 smithi192 bash[24608]: cluster 2024-09-13T22:44:39.038604+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:44:40.944 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:40 smithi192 bash[24608]: cluster 2024-09-13T22:44:40.565195+0000 mgr.y (mgr.14613) 262 : cluster [DBG] pgmap v260: 148 pgs: 148 active+clean; 541 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T22:44:40.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:40.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:41.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:41.074 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:40 smithi178 bash[29889]: cluster 2024-09-13T22:44:39.037479+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:44:41.074 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:40 smithi178 bash[29889]: cluster 2024-09-13T22:44:39.038604+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:44:41.074 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:40 smithi178 bash[29889]: cluster 2024-09-13T22:44:40.565195+0000 mgr.y (mgr.14613) 262 : cluster [DBG] pgmap v260: 148 pgs: 148 active+clean; 541 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T22:44:41.074 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:40 smithi178 bash[22840]: cluster 2024-09-13T22:44:39.037479+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:44:41.074 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:40 smithi178 bash[22840]: cluster 2024-09-13T22:44:39.038604+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:44:41.074 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:40 smithi178 bash[22840]: cluster 2024-09-13T22:44:40.565195+0000 mgr.y (mgr.14613) 262 : cluster [DBG] pgmap v260: 148 pgs: 148 active+clean; 541 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T22:44:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:41.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:41.232 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:41.292 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:44:41.293 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:44:41.336 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:41.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:41.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:41.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:41.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:41.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:41.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:41.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:41.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:41.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:41 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.471568+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:44:41.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:41 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.472950+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:44:41.904 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:41.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:42.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:42.032 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:42.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:42.148 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:41 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.471568+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:44:42.148 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:41 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.472950+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:44:42.149 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:41 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.471568+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:44:42.149 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:41 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.472950+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:44:42.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:42.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:42.296 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:42.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T22:44:42.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:42.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:42.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:42.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:42.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:42.700 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:42.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:42.836 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:42.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:42.963 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:42.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:43.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:43.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.006135+0000 osd.6 (osd.6) 109 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:44:43.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.009604+0000 osd.6 (osd.6) 110 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:44:43.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.708776+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.721707+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.801059+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:41.802323+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.026356+0000 osd.4 (osd.4) 125 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.028839+0000 osd.4 (osd.4) 126 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.505200+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.511171+0000 osd.1 (osd.1) 145 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:44:43.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:42 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.565531+0000 mgr.y (mgr.14613) 263 : cluster [DBG] pgmap v261: 148 pgs: 148 active+clean; 541 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s 2024-09-13T22:44:43.089 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.006135+0000 osd.6 (osd.6) 109 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.009604+0000 osd.6 (osd.6) 110 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.708776+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.721707+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.801059+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:41.802323+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.026356+0000 osd.4 (osd.4) 125 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.028839+0000 osd.4 (osd.4) 126 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:44:43.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.505200+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.511171+0000 osd.1 (osd.1) 145 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.565531+0000 mgr.y (mgr.14613) 263 : cluster [DBG] pgmap v261: 148 pgs: 148 active+clean; 541 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.006135+0000 osd.6 (osd.6) 109 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.009604+0000 osd.6 (osd.6) 110 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.708776+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.721707+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.801059+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:41.802323+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.026356+0000 osd.4 (osd.4) 125 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.028839+0000 osd.4 (osd.4) 126 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.505200+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:44:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.511171+0000 osd.1 (osd.1) 145 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:44:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:42 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.565531+0000 mgr.y (mgr.14613) 263 : cluster [DBG] pgmap v261: 148 pgs: 148 active+clean; 541 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s 2024-09-13T22:44:43.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:43.212 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:43.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:43.335 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:43.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:43.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:43.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:43.592 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:43.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:43.723 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:43.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:43.850 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[29889]: cluster 2024-09-13T22:44:43.043058+0000 osd.4 (osd.4) 127 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[29889]: cluster 2024-09-13T22:44:43.044024+0000 osd.4 (osd.4) 128 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[29889]: cluster 2024-09-13T22:44:43.525869+0000 osd.1 (osd.1) 146 : cluster [DBG] 5.c scrub starts 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[29889]: cluster 2024-09-13T22:44:43.533797+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.c scrub ok 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[22840]: cluster 2024-09-13T22:44:43.043058+0000 osd.4 (osd.4) 127 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[22840]: cluster 2024-09-13T22:44:43.044024+0000 osd.4 (osd.4) 128 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[22840]: cluster 2024-09-13T22:44:43.525869+0000 osd.1 (osd.1) 146 : cluster [DBG] 5.c scrub starts 2024-09-13T22:44:43.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:43 smithi178 bash[22840]: cluster 2024-09-13T22:44:43.533797+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.c scrub ok 2024-09-13T22:44:43.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:43.981 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:43.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:44.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:44.123 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:44.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:43 smithi192 bash[24608]: cluster 2024-09-13T22:44:43.043058+0000 osd.4 (osd.4) 127 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:44:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:43 smithi192 bash[24608]: cluster 2024-09-13T22:44:43.044024+0000 osd.4 (osd.4) 128 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:44:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:43 smithi192 bash[24608]: cluster 2024-09-13T22:44:43.525869+0000 osd.1 (osd.1) 146 : cluster [DBG] 5.c scrub starts 2024-09-13T22:44:44.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:43 smithi192 bash[24608]: cluster 2024-09-13T22:44:43.533797+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.c scrub ok 2024-09-13T22:44:44.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:44.257 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:44.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:44.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:44.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:44.528 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:44.537 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:44.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:44.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:44.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:44.777 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:44.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:44.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.912712+0000 osd.5 (osd.5) 156 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:44:44.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:42.914153+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:44:44.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.069202+0000 osd.4 (osd.4) 129 : cluster [DBG] 2.f scrub starts 2024-09-13T22:44:44.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.075456+0000 osd.4 (osd.4) 130 : cluster [DBG] 2.f scrub ok 2024-09-13T22:44:44.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.192517+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:44.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.192578+0000 osd.4 (osd.4) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:44.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:44 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.566254+0000 mgr.y (mgr.14613) 264 : cluster [DBG] pgmap v262: 148 pgs: 148 active+clean; 257 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 992 op/s 2024-09-13T22:44:44.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:44.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:45.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:45.036 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:45.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:45.147 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:44:45.157 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:45.197 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.912712+0000 osd.5 (osd.5) 156 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:44:45.197 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:42.914153+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:44:45.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.069202+0000 osd.4 (osd.4) 129 : cluster [DBG] 2.f scrub starts 2024-09-13T22:44:45.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.075456+0000 osd.4 (osd.4) 130 : cluster [DBG] 2.f scrub ok 2024-09-13T22:44:45.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.192517+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:45.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.192578+0000 osd.4 (osd.4) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:45.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.566254+0000 mgr.y (mgr.14613) 264 : cluster [DBG] pgmap v262: 148 pgs: 148 active+clean; 257 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 992 op/s 2024-09-13T22:44:45.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.912712+0000 osd.5 (osd.5) 156 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:44:45.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:42.914153+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:44:45.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.069202+0000 osd.4 (osd.4) 129 : cluster [DBG] 2.f scrub starts 2024-09-13T22:44:45.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.075456+0000 osd.4 (osd.4) 130 : cluster [DBG] 2.f scrub ok 2024-09-13T22:44:45.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.192517+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:45.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.192578+0000 osd.4 (osd.4) 132 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:45.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:44 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.566254+0000 mgr.y (mgr.14613) 264 : cluster [DBG] pgmap v262: 148 pgs: 148 active+clean; 257 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 992 op/s 2024-09-13T22:44:45.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:45.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:45.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:45.411 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:45.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:45.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:45.553 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:45.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:45.656 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:45.759 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:45.786 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:45.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:45.922 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:45.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:46.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:46.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:43.904225+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:43.908942+0000 osd.5 (osd.5) 159 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.787423+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.c scrub starts 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.794459+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.795793+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.c scrub ok 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.796878+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.883839+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.903657+0000 osd.5 (osd.5) 160 : cluster [DBG] 6.a scrub starts 2024-09-13T22:44:46.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: cluster 2024-09-13T22:44:44.904773+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.a scrub ok 2024-09-13T22:44:46.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:45 smithi192 bash[24608]: audit 2024-09-13T22:44:45.122079+0000 mgr.y (mgr.14613) 265 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:46.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:46.176 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:46.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:43.904225+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:44:46.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:43.908942+0000 osd.5 (osd.5) 159 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:44:46.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.787423+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.c scrub starts 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.794459+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.795793+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.c scrub ok 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.796878+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.883839+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.903657+0000 osd.5 (osd.5) 160 : cluster [DBG] 6.a scrub starts 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: cluster 2024-09-13T22:44:44.904773+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.a scrub ok 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[29889]: audit 2024-09-13T22:44:45.122079+0000 mgr.y (mgr.14613) 265 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:43.904225+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:44:46.208 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:43.908942+0000 osd.5 (osd.5) 159 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.787423+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.c scrub starts 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.794459+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.795793+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.c scrub ok 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.796878+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.883839+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.903657+0000 osd.5 (osd.5) 160 : cluster [DBG] 6.a scrub starts 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: cluster 2024-09-13T22:44:44.904773+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.a scrub ok 2024-09-13T22:44:46.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:45 smithi178 bash[22840]: audit 2024-09-13T22:44:45.122079+0000 mgr.y (mgr.14613) 265 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:46.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:46.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:46.416 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:46.444 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:46.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:46.577 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:46.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:46.705 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:46.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:46.834 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:46.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:46.966 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:46.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:47.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:47.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[29889]: cluster 2024-09-13T22:44:45.055595+0000 osd.6 (osd.6) 111 : cluster [DBG] 7.c scrub starts 2024-09-13T22:44:47.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[29889]: cluster 2024-09-13T22:44:45.214744+0000 osd.6 (osd.6) 112 : cluster [DBG] 7.c scrub ok 2024-09-13T22:44:47.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[29889]: cluster 2024-09-13T22:44:45.457854+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:44:47.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[29889]: cluster 2024-09-13T22:44:45.461712+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:44:47.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[29889]: cluster 2024-09-13T22:44:46.566655+0000 mgr.y (mgr.14613) 266 : cluster [DBG] pgmap v264: 148 pgs: 148 active+clean; 257 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 997 op/s 2024-09-13T22:44:47.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[22840]: cluster 2024-09-13T22:44:45.055595+0000 osd.6 (osd.6) 111 : cluster [DBG] 7.c scrub starts 2024-09-13T22:44:47.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[22840]: cluster 2024-09-13T22:44:45.214744+0000 osd.6 (osd.6) 112 : cluster [DBG] 7.c scrub ok 2024-09-13T22:44:47.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[22840]: cluster 2024-09-13T22:44:45.457854+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:44:47.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[22840]: cluster 2024-09-13T22:44:45.461712+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:44:47.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:46 smithi178 bash[22840]: cluster 2024-09-13T22:44:46.566655+0000 mgr.y (mgr.14613) 266 : cluster [DBG] pgmap v264: 148 pgs: 148 active+clean; 257 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 997 op/s 2024-09-13T22:44:47.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:47.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:46 smithi192 bash[24608]: cluster 2024-09-13T22:44:45.055595+0000 osd.6 (osd.6) 111 : cluster [DBG] 7.c scrub starts 2024-09-13T22:44:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:46 smithi192 bash[24608]: cluster 2024-09-13T22:44:45.214744+0000 osd.6 (osd.6) 112 : cluster [DBG] 7.c scrub ok 2024-09-13T22:44:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:46 smithi192 bash[24608]: cluster 2024-09-13T22:44:45.457854+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:44:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:46 smithi192 bash[24608]: cluster 2024-09-13T22:44:45.461712+0000 osd.3 (osd.3) 171 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:44:47.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:46 smithi192 bash[24608]: cluster 2024-09-13T22:44:46.566655+0000 mgr.y (mgr.14613) 266 : cluster [DBG] pgmap v264: 148 pgs: 148 active+clean; 257 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 997 op/s 2024-09-13T22:44:47.172 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:47.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:47.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:47.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:47.358 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:47.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:47.486 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:47.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:47.612 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:47.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:47.742 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:47.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:47.879 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:47.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:47.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:48.009 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:48.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:48.139 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[22840]: cluster 2024-09-13T22:44:46.043247+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[22840]: cluster 2024-09-13T22:44:46.047605+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[22840]: cluster 2024-09-13T22:44:46.836787+0000 osd.2 (osd.2) 119 : cluster [DBG] 7.e scrub starts 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.044639+0000 osd.2 (osd.2) 120 : cluster [DBG] 7.e scrub ok 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.482493+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.484516+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[29889]: cluster 2024-09-13T22:44:46.043247+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[29889]: cluster 2024-09-13T22:44:46.047605+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:44:48.146 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[29889]: cluster 2024-09-13T22:44:46.836787+0000 osd.2 (osd.2) 119 : cluster [DBG] 7.e scrub starts 2024-09-13T22:44:48.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.044639+0000 osd.2 (osd.2) 120 : cluster [DBG] 7.e scrub ok 2024-09-13T22:44:48.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.482493+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:44:48.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:47 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.484516+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:44:48.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:47 smithi192 bash[24608]: cluster 2024-09-13T22:44:46.043247+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:44:48.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:47 smithi192 bash[24608]: cluster 2024-09-13T22:44:46.047605+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:44:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:47 smithi192 bash[24608]: cluster 2024-09-13T22:44:46.836787+0000 osd.2 (osd.2) 119 : cluster [DBG] 7.e scrub starts 2024-09-13T22:44:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:47 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.044639+0000 osd.2 (osd.2) 120 : cluster [DBG] 7.e scrub ok 2024-09-13T22:44:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:47 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.482493+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:44:48.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:47 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.484516+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:44:48.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:48.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:48.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:48.390 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:48.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:48.528 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:48.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:48.657 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:48.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:48.779 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:48.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:48.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:48.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:49.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:49.032 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:49.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:48 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.022549+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:44:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:48 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.023630+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:44:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:48 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.853375+0000 osd.7 (osd.7) 147 : cluster [DBG] 3.f scrub starts 2024-09-13T22:44:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:48 smithi192 bash[24608]: cluster 2024-09-13T22:44:47.859153+0000 osd.7 (osd.7) 148 : cluster [DBG] 3.f scrub ok 2024-09-13T22:44:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:48 smithi192 bash[24608]: audit 2024-09-13T22:44:48.124762+0000 mon.b (mon.1) 131 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:48 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.567184+0000 mgr.y (mgr.14613) 267 : cluster [DBG] pgmap v265: 148 pgs: 148 active+clean; 243 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 882 op/s 2024-09-13T22:44:49.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:49.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:49.285 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:49.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:49.420 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.022549+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:44:49.420 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.023630+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.853375+0000 osd.7 (osd.7) 147 : cluster [DBG] 3.f scrub starts 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:47.859153+0000 osd.7 (osd.7) 148 : cluster [DBG] 3.f scrub ok 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: audit 2024-09-13T22:44:48.124762+0000 mon.b (mon.1) 131 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.567184+0000 mgr.y (mgr.14613) 267 : cluster [DBG] pgmap v265: 148 pgs: 148 active+clean; 243 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 882 op/s 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.022549+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.023630+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.853375+0000 osd.7 (osd.7) 147 : cluster [DBG] 3.f scrub starts 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:47.859153+0000 osd.7 (osd.7) 148 : cluster [DBG] 3.f scrub ok 2024-09-13T22:44:49.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: audit 2024-09-13T22:44:48.124762+0000 mon.b (mon.1) 131 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:44:49.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.567184+0000 mgr.y (mgr.14613) 267 : cluster [DBG] pgmap v265: 148 pgs: 148 active+clean; 243 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 882 op/s 2024-09-13T22:44:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.422 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:49.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:49.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:49.564 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:49.611 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Removed 105750 objects 2024-09-13T22:44:49.611 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Clean up completed and total clean up time :132.381 2024-09-13T22:44:49.655 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:44:49.655 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:44:49.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:49.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:49.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:49.813 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:49.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:49.928 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.032404+0000 osd.6 (osd.6) 117 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.034315+0000 osd.6 (osd.6) 118 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.856677+0000 osd.7 (osd.7) 149 : cluster [DBG] 3.d scrub starts 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.862641+0000 osd.7 (osd.7) 150 : cluster [DBG] 3.d scrub ok 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.932748+0000 osd.5 (osd.5) 162 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:48.934697+0000 osd.5 (osd.5) 163 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:49.486775+0000 osd.1 (osd.1) 150 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:44:49.929 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:49 smithi192 bash[24608]: cluster 2024-09-13T22:44:49.488850+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:44:49.940 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:49.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:50.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:50.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:50.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:50.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:44:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:50.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.032404+0000 osd.6 (osd.6) 117 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.034315+0000 osd.6 (osd.6) 118 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.856677+0000 osd.7 (osd.7) 149 : cluster [DBG] 3.d scrub starts 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.862641+0000 osd.7 (osd.7) 150 : cluster [DBG] 3.d scrub ok 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.932748+0000 osd.5 (osd.5) 162 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:48.934697+0000 osd.5 (osd.5) 163 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:49.486775+0000 osd.1 (osd.1) 150 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[29889]: cluster 2024-09-13T22:44:49.488850+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.032404+0000 osd.6 (osd.6) 117 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:44:50.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.034315+0000 osd.6 (osd.6) 118 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:44:50.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.856677+0000 osd.7 (osd.7) 149 : cluster [DBG] 3.d scrub starts 2024-09-13T22:44:50.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.862641+0000 osd.7 (osd.7) 150 : cluster [DBG] 3.d scrub ok 2024-09-13T22:44:50.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.932748+0000 osd.5 (osd.5) 162 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:44:50.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:48.934697+0000 osd.5 (osd.5) 163 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:44:50.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:49.486775+0000 osd.1 (osd.1) 150 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:44:50.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:49 smithi178 bash[22840]: cluster 2024-09-13T22:44:49.488850+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:44:50.193 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:50.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:50.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:50.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:50.448 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:50.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T22:44:50.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:50.571 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:50.672 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:44:50.672 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:44:50.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:50.689 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:44:50.690 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:44:50.694 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:50.706 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:44:50.706 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:44:50.723 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:44:50.723 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:44:50.740 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:44:50.740 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:44:50.757 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:44:50.757 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:44:50.773 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:44:50.773 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:44:50.790 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:44:50.790 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:44:50.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:50.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:50.914 INFO:teuthology.orchestra.run.smithi178.stderr:clear 2.0 pg_upmap_items mapping 2024-09-13T22:44:50.914 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:44:50.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:50.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:50.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:51.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: cluster 2024-09-13T22:44:49.476846+0000 osd.3 (osd.3) 172 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: cluster 2024-09-13T22:44:49.489410+0000 osd.3 (osd.3) 173 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: audit 2024-09-13T22:44:50.301916+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.178:0/600296866' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]: dispatch 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: audit 2024-09-13T22:44:50.302496+0000 mon.a (mon.0) 1059 : audit [INF] from='client.24974 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]: dispatch 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: cluster 2024-09-13T22:44:50.487630+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: cluster 2024-09-13T22:44:50.489631+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: cluster 2024-09-13T22:44:50.567916+0000 mgr.y (mgr.14613) 268 : cluster [DBG] pgmap v266: 148 pgs: 148 active+clean; 28 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: audit 2024-09-13T22:44:50.657991+0000 mon.c (mon.2) 34 : audit [DBG] from='client.? v1:172.21.15.178:0/3846905702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: audit 2024-09-13T22:44:50.848952+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.178:0/3439372942' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[22840]: audit 2024-09-13T22:44:50.849287+0000 mon.a (mon.0) 1060 : audit [INF] from='client.24986 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:51.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: cluster 2024-09-13T22:44:49.476846+0000 osd.3 (osd.3) 172 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: cluster 2024-09-13T22:44:49.489410+0000 osd.3 (osd.3) 173 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: audit 2024-09-13T22:44:50.301916+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.178:0/600296866' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]: dispatch 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: audit 2024-09-13T22:44:50.302496+0000 mon.a (mon.0) 1059 : audit [INF] from='client.24974 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]: dispatch 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: cluster 2024-09-13T22:44:50.487630+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: cluster 2024-09-13T22:44:50.489631+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: cluster 2024-09-13T22:44:50.567916+0000 mgr.y (mgr.14613) 268 : cluster [DBG] pgmap v266: 148 pgs: 148 active+clean; 28 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: audit 2024-09-13T22:44:50.657991+0000 mon.c (mon.2) 34 : audit [DBG] from='client.? v1:172.21.15.178:0/3846905702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: audit 2024-09-13T22:44:50.848952+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.178:0/3439372942' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:51.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:50 smithi178 bash[29889]: audit 2024-09-13T22:44:50.849287+0000 mon.a (mon.0) 1060 : audit [INF] from='client.24986 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:51.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:51.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: cluster 2024-09-13T22:44:49.476846+0000 osd.3 (osd.3) 172 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: cluster 2024-09-13T22:44:49.489410+0000 osd.3 (osd.3) 173 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: audit 2024-09-13T22:44:50.301916+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.178:0/600296866' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]: dispatch 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: audit 2024-09-13T22:44:50.302496+0000 mon.a (mon.0) 1059 : audit [INF] from='client.24974 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]: dispatch 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: cluster 2024-09-13T22:44:50.487630+0000 osd.1 (osd.1) 152 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: cluster 2024-09-13T22:44:50.489631+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: cluster 2024-09-13T22:44:50.567916+0000 mgr.y (mgr.14613) 268 : cluster [DBG] pgmap v266: 148 pgs: 148 active+clean; 28 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s 2024-09-13T22:44:51.186 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: audit 2024-09-13T22:44:50.657991+0000 mon.c (mon.2) 34 : audit [DBG] from='client.? v1:172.21.15.178:0/3846905702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:44:51.187 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: audit 2024-09-13T22:44:50.848952+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.178:0/3439372942' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:51.187 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:50 smithi192 bash[24608]: audit 2024-09-13T22:44:50.849287+0000 mon.a (mon.0) 1060 : audit [INF] from='client.24986 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:44:51.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:51.304 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:51.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:51.461 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:51.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:51.591 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:51.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:51.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:51.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:51.853 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:51.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:51.984 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:51.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:52.013 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[22840]: cluster 2024-09-13T22:44:50.906643+0000 mon.a (mon.0) 1061 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:44:52.013 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[22840]: cluster 2024-09-13T22:44:50.906664+0000 mon.a (mon.0) 1062 : cluster [INF] Cluster is now healthy 2024-09-13T22:44:52.013 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[22840]: audit 2024-09-13T22:44:50.913679+0000 mon.a (mon.0) 1063 : audit [INF] from='client.24974 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]': finished 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[22840]: audit 2024-09-13T22:44:50.913748+0000 mon.a (mon.0) 1064 : audit [INF] from='client.24986 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[22840]: cluster 2024-09-13T22:44:50.917417+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[29889]: cluster 2024-09-13T22:44:50.906643+0000 mon.a (mon.0) 1061 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[29889]: cluster 2024-09-13T22:44:50.906664+0000 mon.a (mon.0) 1062 : cluster [INF] Cluster is now healthy 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[29889]: audit 2024-09-13T22:44:50.913679+0000 mon.a (mon.0) 1063 : audit [INF] from='client.24974 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]': finished 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[29889]: audit 2024-09-13T22:44:50.913748+0000 mon.a (mon.0) 1064 : audit [INF] from='client.24986 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:44:52.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:51 smithi178 bash[29889]: cluster 2024-09-13T22:44:50.917417+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-13T22:44:52.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:52.098 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:51 smithi192 bash[24608]: cluster 2024-09-13T22:44:50.906643+0000 mon.a (mon.0) 1061 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:44:52.099 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:51 smithi192 bash[24608]: cluster 2024-09-13T22:44:50.906664+0000 mon.a (mon.0) 1062 : cluster [INF] Cluster is now healthy 2024-09-13T22:44:52.099 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:51 smithi192 bash[24608]: audit 2024-09-13T22:44:50.913679+0000 mon.a (mon.0) 1063 : audit [INF] from='client.24974 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.0"}]': finished 2024-09-13T22:44:52.099 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:51 smithi192 bash[24608]: audit 2024-09-13T22:44:50.913748+0000 mon.a (mon.0) 1064 : audit [INF] from='client.24986 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:44:52.099 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:51 smithi192 bash[24608]: cluster 2024-09-13T22:44:50.917417+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-13T22:44:52.113 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:52.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:52.244 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:52.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:52.372 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:52.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:52.503 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:52.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:52.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:52.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:52.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:52.759 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:52.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:52.886 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:52.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:52.908 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T22:44:52.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T22:44:52.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:53.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[22840]: cluster 2024-09-13T22:44:51.055347+0000 osd.6 (osd.6) 119 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[22840]: cluster 2024-09-13T22:44:51.058576+0000 osd.6 (osd.6) 120 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[22840]: cluster 2024-09-13T22:44:51.780396+0000 osd.7 (osd.7) 151 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[22840]: cluster 2024-09-13T22:44:51.782318+0000 osd.7 (osd.7) 152 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[22840]: cluster 2024-09-13T22:44:51.929942+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[22840]: cluster 2024-09-13T22:44:52.568569+0000 mgr.y (mgr.14613) 269 : cluster [DBG] pgmap v269: 148 pgs: 148 active+clean; 28 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 478 op/s 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[29889]: cluster 2024-09-13T22:44:51.055347+0000 osd.6 (osd.6) 119 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[29889]: cluster 2024-09-13T22:44:51.058576+0000 osd.6 (osd.6) 120 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:44:53.100 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[29889]: cluster 2024-09-13T22:44:51.780396+0000 osd.7 (osd.7) 151 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:44:53.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[29889]: cluster 2024-09-13T22:44:51.782318+0000 osd.7 (osd.7) 152 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:44:53.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[29889]: cluster 2024-09-13T22:44:51.929942+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-13T22:44:53.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:52 smithi178 bash[29889]: cluster 2024-09-13T22:44:52.568569+0000 mgr.y (mgr.14613) 269 : cluster [DBG] pgmap v269: 148 pgs: 148 active+clean; 28 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 478 op/s 2024-09-13T22:44:53.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:53.146 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:53.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:53.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:52 smithi192 bash[24608]: cluster 2024-09-13T22:44:51.055347+0000 osd.6 (osd.6) 119 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:44:53.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:52 smithi192 bash[24608]: cluster 2024-09-13T22:44:51.058576+0000 osd.6 (osd.6) 120 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:44:53.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:52 smithi192 bash[24608]: cluster 2024-09-13T22:44:51.780396+0000 osd.7 (osd.7) 151 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:44:53.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:52 smithi192 bash[24608]: cluster 2024-09-13T22:44:51.782318+0000 osd.7 (osd.7) 152 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:44:53.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:52 smithi192 bash[24608]: cluster 2024-09-13T22:44:51.929942+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-13T22:44:53.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:52 smithi192 bash[24608]: cluster 2024-09-13T22:44:52.568569+0000 mgr.y (mgr.14613) 269 : cluster [DBG] pgmap v269: 148 pgs: 148 active+clean; 28 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 478 op/s 2024-09-13T22:44:53.274 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:53.377 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:53.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:53.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:53.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:53.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:53.662 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:53.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:53.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:53.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:53.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:53.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:54.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:54.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:53 smithi192 bash[24608]: cluster 2024-09-13T22:44:52.081824+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:44:54.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:53 smithi192 bash[24608]: cluster 2024-09-13T22:44:52.084184+0000 osd.6 (osd.6) 122 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:44:54.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:53 smithi192 bash[24608]: cluster 2024-09-13T22:44:52.537340+0000 osd.3 (osd.3) 174 : cluster [DBG] 7.2 deep-scrub starts 2024-09-13T22:44:54.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:53 smithi192 bash[24608]: cluster 2024-09-13T22:44:52.550969+0000 osd.3 (osd.3) 175 : cluster [DBG] 7.2 deep-scrub ok 2024-09-13T22:44:54.053 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:54.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:54.181 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:54.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[29889]: cluster 2024-09-13T22:44:52.081824+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[29889]: cluster 2024-09-13T22:44:52.084184+0000 osd.6 (osd.6) 122 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[29889]: cluster 2024-09-13T22:44:52.537340+0000 osd.3 (osd.3) 174 : cluster [DBG] 7.2 deep-scrub starts 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[29889]: cluster 2024-09-13T22:44:52.550969+0000 osd.3 (osd.3) 175 : cluster [DBG] 7.2 deep-scrub ok 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[22840]: cluster 2024-09-13T22:44:52.081824+0000 osd.6 (osd.6) 121 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[22840]: cluster 2024-09-13T22:44:52.084184+0000 osd.6 (osd.6) 122 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[22840]: cluster 2024-09-13T22:44:52.537340+0000 osd.3 (osd.3) 174 : cluster [DBG] 7.2 deep-scrub starts 2024-09-13T22:44:54.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:53 smithi178 bash[22840]: cluster 2024-09-13T22:44:52.550969+0000 osd.3 (osd.3) 175 : cluster [DBG] 7.2 deep-scrub ok 2024-09-13T22:44:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:54.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:54.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:54.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:54.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:54.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:54.572 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:54.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:54.704 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:54.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:54.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:54.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:54.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:54.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:54.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:44:55.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:55.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:55.133 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.038872+0000 osd.4 (osd.4) 133 : cluster [DBG] 7.1 deep-scrub starts 2024-09-13T22:44:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.051424+0000 osd.4 (osd.4) 134 : cluster [DBG] 7.1 deep-scrub ok 2024-09-13T22:44:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.570146+0000 mgr.y (mgr.14613) 270 : cluster [DBG] pgmap v270: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 514 op/s 2024-09-13T22:44:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.585765+0000 osd.1 (osd.1) 154 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:44:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.588252+0000 osd.1 (osd.1) 155 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:44:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.769460+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.b scrub starts 2024-09-13T22:44:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:54 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.771543+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.b scrub ok 2024-09-13T22:44:55.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:55.204 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.038872+0000 osd.4 (osd.4) 133 : cluster [DBG] 7.1 deep-scrub starts 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.051424+0000 osd.4 (osd.4) 134 : cluster [DBG] 7.1 deep-scrub ok 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.570146+0000 mgr.y (mgr.14613) 270 : cluster [DBG] pgmap v270: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 514 op/s 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.585765+0000 osd.1 (osd.1) 154 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.588252+0000 osd.1 (osd.1) 155 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.769460+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.b scrub starts 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.771543+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.b scrub ok 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.038872+0000 osd.4 (osd.4) 133 : cluster [DBG] 7.1 deep-scrub starts 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.051424+0000 osd.4 (osd.4) 134 : cluster [DBG] 7.1 deep-scrub ok 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.570146+0000 mgr.y (mgr.14613) 270 : cluster [DBG] pgmap v270: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 514 op/s 2024-09-13T22:44:55.205 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.585765+0000 osd.1 (osd.1) 154 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:44:55.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.588252+0000 osd.1 (osd.1) 155 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:44:55.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.769460+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.b scrub starts 2024-09-13T22:44:55.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:54 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.771543+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.b scrub ok 2024-09-13T22:44:55.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:55.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:55.328 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:44:55.342 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:55.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:44:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:44:55.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:55.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:55.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:55.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:55.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:44:55.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:55.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:55.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:55.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:55.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:55.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:55.977 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:55 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.868992+0000 osd.7 (osd.7) 153 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:44:55.977 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:55 smithi192 bash[24608]: cluster 2024-09-13T22:44:54.870615+0000 osd.7 (osd.7) 154 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:44:55.977 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:55 smithi192 bash[24608]: audit 2024-09-13T22:44:55.136381+0000 mgr.y (mgr.14613) 271 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:55.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:55 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.597677+0000 osd.1 (osd.1) 156 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:44:55.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:55 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.600794+0000 osd.1 (osd.1) 157 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:44:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:55.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:56.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:56.120 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:56.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.868992+0000 osd.7 (osd.7) 153 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[29889]: cluster 2024-09-13T22:44:54.870615+0000 osd.7 (osd.7) 154 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[29889]: audit 2024-09-13T22:44:55.136381+0000 mgr.y (mgr.14613) 271 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.597677+0000 osd.1 (osd.1) 156 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.600794+0000 osd.1 (osd.1) 157 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.868992+0000 osd.7 (osd.7) 153 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[22840]: cluster 2024-09-13T22:44:54.870615+0000 osd.7 (osd.7) 154 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:44:56.234 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[22840]: audit 2024-09-13T22:44:55.136381+0000 mgr.y (mgr.14613) 271 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:44:56.235 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.597677+0000 osd.1 (osd.1) 156 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:44:56.235 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:55 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.600794+0000 osd.1 (osd.1) 157 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:44:56.249 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:56.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:56.380 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:56.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:56.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:56.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:56.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:56.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:56.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:56.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:56.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:56.892 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:56.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:56.993 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:44:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:44:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:44:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:44:56.994 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:44:56.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.559337+0000 osd.3 (osd.3) 176 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.563583+0000 osd.3 (osd.3) 177 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.853768+0000 osd.7 (osd.7) 155 : cluster [DBG] 3.e scrub starts 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.856062+0000 osd.7 (osd.7) 156 : cluster [DBG] 3.e scrub ok 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.858808+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:55.859840+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:44:57.006 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.570875+0000 mgr.y (mgr.14613) 272 : cluster [DBG] pgmap v271: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 485 op/s 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.621213+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.a scrub starts 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.623446+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.a scrub ok 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.559337+0000 osd.3 (osd.3) 176 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.563583+0000 osd.3 (osd.3) 177 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.853768+0000 osd.7 (osd.7) 155 : cluster [DBG] 3.e scrub starts 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.856062+0000 osd.7 (osd.7) 156 : cluster [DBG] 3.e scrub ok 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.858808+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:55.859840+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:44:57.007 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.570875+0000 mgr.y (mgr.14613) 272 : cluster [DBG] pgmap v271: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 485 op/s 2024-09-13T22:44:57.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.621213+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.a scrub starts 2024-09-13T22:44:57.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:56 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.623446+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.a scrub ok 2024-09-13T22:44:57.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:57.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:57.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.559337+0000 osd.3 (osd.3) 176 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:44:57.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.563583+0000 osd.3 (osd.3) 177 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:44:57.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.853768+0000 osd.7 (osd.7) 155 : cluster [DBG] 3.e scrub starts 2024-09-13T22:44:57.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.856062+0000 osd.7 (osd.7) 156 : cluster [DBG] 3.e scrub ok 2024-09-13T22:44:57.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.858808+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:44:57.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:55.859840+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:44:57.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.570875+0000 mgr.y (mgr.14613) 272 : cluster [DBG] pgmap v271: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 485 op/s 2024-09-13T22:44:57.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.621213+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.a scrub starts 2024-09-13T22:44:57.135 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:56 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.623446+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.a scrub ok 2024-09-13T22:44:57.146 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:57.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:57.272 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:57.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:57.403 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:57.506 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:57.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:57.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:57.662 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:57.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T22:44:57.726 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:57.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:57.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:44:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:44:57.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:57.916 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:57.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:58.019 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:58.046 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:58.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:58.176 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:58.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.802780+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.d scrub starts 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.805390+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.d scrub ok 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.940824+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[29889]: cluster 2024-09-13T22:44:56.942204+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.453913+0000 osd.5 (osd.5) 166 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.454024+0000 osd.5 (osd.5) 167 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.802780+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.d scrub starts 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.805390+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.d scrub ok 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.940824+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[22840]: cluster 2024-09-13T22:44:56.942204+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.453913+0000 osd.5 (osd.5) 166 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:57 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.454024+0000 osd.5 (osd.5) 167 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:58.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:58.290 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:57 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.802780+0000 osd.2 (osd.2) 123 : cluster [DBG] 5.d scrub starts 2024-09-13T22:44:58.290 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:57 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.805390+0000 osd.2 (osd.2) 124 : cluster [DBG] 5.d scrub ok 2024-09-13T22:44:58.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:57 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.940824+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:44:58.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:57 smithi192 bash[24608]: cluster 2024-09-13T22:44:56.942204+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:44:58.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:57 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.453913+0000 osd.5 (osd.5) 166 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:44:58.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:57 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.454024+0000 osd.5 (osd.5) 167 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:44:58.307 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:58.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:58.436 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:58.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:58.566 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:58.584 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:44:58.585 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:44:58.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:58.698 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:58.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:58.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:58.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:58.957 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:44:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:58.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:44:59.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:59.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:44:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:44:59.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:59.202 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.927376+0000 osd.0 (osd.0) 165 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:44:59.202 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.928748+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.932949+0000 osd.5 (osd.5) 168 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.933938+0000 osd.5 (osd.5) 169 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:57.963558+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.000234+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.003170+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.571707+0000 mgr.y (mgr.14613) 273 : cluster [DBG] pgmap v273: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 669 B/s rd, 0 B/s wr, 57 op/s 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.740728+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.743128+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.927376+0000 osd.0 (osd.0) 165 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.928748+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:44:59.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.932949+0000 osd.5 (osd.5) 168 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.933938+0000 osd.5 (osd.5) 169 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:57.963558+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.000234+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.003170+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.571707+0000 mgr.y (mgr.14613) 273 : cluster [DBG] pgmap v273: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 669 B/s rd, 0 B/s wr, 57 op/s 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.740728+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:44:59.204 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:58 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.743128+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:44:59.216 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:59.319 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:59.341 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:44:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:44:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.927376+0000 osd.0 (osd.0) 165 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:44:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.928748+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:44:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.932949+0000 osd.5 (osd.5) 168 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:44:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.933938+0000 osd.5 (osd.5) 169 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:44:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:57.963558+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-13T22:44:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.000234+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:44:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.003170+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:44:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.571707+0000 mgr.y (mgr.14613) 273 : cluster [DBG] pgmap v273: 148 pgs: 148 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 669 B/s rd, 0 B/s wr, 57 op/s 2024-09-13T22:44:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.740728+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:44:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:58 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.743128+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:44:59.444 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:59.463 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:44:59.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:59.595 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:44:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:44:59.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:59.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:44:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:44:59.763 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:59.763 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:44:59.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:59.855 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:44:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:44:59.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:44:59.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:44:59.965 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_0' removed 2024-09-13T22:44:59.973 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:44:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:44:59.994 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.539913+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.542770+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.797116+0000 osd.7 (osd.7) 157 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.799229+0000 osd.7 (osd.7) 158 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.927368+0000 osd.5 (osd.5) 170 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.932594+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:45:00.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.997525+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:58.997606+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.012314+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f deep-scrub starts 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.017604+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f deep-scrub ok 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[29889]: audit 2024-09-13T22:44:59.631691+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? v1:172.21.15.178:0/849578357' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.539913+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.542770+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:45:00.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.797116+0000 osd.7 (osd.7) 157 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.799229+0000 osd.7 (osd.7) 158 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.927368+0000 osd.5 (osd.5) 170 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.932594+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.997525+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:58.997606+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.012314+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f deep-scrub starts 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.017604+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f deep-scrub ok 2024-09-13T22:45:00.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:44:59 smithi178 bash[22840]: audit 2024-09-13T22:44:59.631691+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? v1:172.21.15.178:0/849578357' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:45:00.103 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:00.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:00.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.539913+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:45:00.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.542770+0000 osd.3 (osd.3) 179 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:45:00.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.797116+0000 osd.7 (osd.7) 157 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:45:00.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.799229+0000 osd.7 (osd.7) 158 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:45:00.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.927368+0000 osd.5 (osd.5) 170 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:45:00.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.932594+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:45:00.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.997525+0000 osd.1 (osd.1) 160 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:00.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:58.997606+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:00.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.012314+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f deep-scrub starts 2024-09-13T22:45:00.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.017604+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f deep-scrub ok 2024-09-13T22:45:00.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:44:59 smithi192 bash[24608]: audit 2024-09-13T22:44:59.631691+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? v1:172.21.15.178:0/849578357' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:45:00.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:00.252 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:00.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:00.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:00.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:00.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:00.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:00.646 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:00.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:00.772 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:00.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:00.897 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:00.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:01.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:01.027 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:01.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:01.159 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:01.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.818917+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.820961+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.915662+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.f deep-scrub starts 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.927516+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.f deep-scrub ok 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: audit 2024-09-13T22:44:59.969071+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? v1:172.21.15.178:0/849578357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.974357+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:44:59.978726+0000 osd.5 (osd.5) 172 : cluster [DBG] 7.0 scrub starts 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:45:00.001085+0000 osd.5 (osd.5) 173 : cluster [DBG] 7.0 scrub ok 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:45:00.575224+0000 mgr.y (mgr.14613) 274 : cluster [DBG] pgmap v275: 132 pgs: 132 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:45:00.603148+0000 osd.1 (osd.1) 162 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:45:01.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:00 smithi192 bash[24608]: cluster 2024-09-13T22:45:00.620462+0000 osd.1 (osd.1) 163 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:45:01.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:01.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.818917+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:45:01.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.820961+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:45:01.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.915662+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.f deep-scrub starts 2024-09-13T22:45:01.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.927516+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.f deep-scrub ok 2024-09-13T22:45:01.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: audit 2024-09-13T22:44:59.969071+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? v1:172.21.15.178:0/849578357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:45:01.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.974357+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:44:59.978726+0000 osd.5 (osd.5) 172 : cluster [DBG] 7.0 scrub starts 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:00.001085+0000 osd.5 (osd.5) 173 : cluster [DBG] 7.0 scrub ok 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:00.575224+0000 mgr.y (mgr.14613) 274 : cluster [DBG] pgmap v275: 132 pgs: 132 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:00.603148+0000 osd.1 (osd.1) 162 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:00.620462+0000 osd.1 (osd.1) 163 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.818917+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.820961+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.915662+0000 osd.0 (osd.0) 167 : cluster [DBG] 7.f deep-scrub starts 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.927516+0000 osd.0 (osd.0) 168 : cluster [DBG] 7.f deep-scrub ok 2024-09-13T22:45:01.314 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: audit 2024-09-13T22:44:59.969071+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? v1:172.21.15.178:0/849578357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:45:01.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.974357+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-13T22:45:01.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:44:59.978726+0000 osd.5 (osd.5) 172 : cluster [DBG] 7.0 scrub starts 2024-09-13T22:45:01.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:00.001085+0000 osd.5 (osd.5) 173 : cluster [DBG] 7.0 scrub ok 2024-09-13T22:45:01.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:00.575224+0000 mgr.y (mgr.14613) 274 : cluster [DBG] pgmap v275: 132 pgs: 132 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:01.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:00.603148+0000 osd.1 (osd.1) 162 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:45:01.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:00.620462+0000 osd.1 (osd.1) 163 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:45:01.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:01.420 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:01.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:01.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:01.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:01.681 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:01.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:01.811 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:01.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:01.941 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:01.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:02.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:02.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:02.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:01 smithi192 bash[24608]: cluster 2024-09-13T22:45:00.533364+0000 osd.3 (osd.3) 180 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:45:02.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:01 smithi192 bash[24608]: cluster 2024-09-13T22:45:00.534592+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:45:02.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:01 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.717631+0000 osd.2 (osd.2) 127 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T22:45:02.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:01 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.719660+0000 osd.2 (osd.2) 128 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T22:45:02.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:02.203 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:02.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:02.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:00.533364+0000 osd.3 (osd.3) 180 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:45:02.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:00.534592+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:45:02.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.717631+0000 osd.2 (osd.2) 127 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T22:45:02.318 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.719660+0000 osd.2 (osd.2) 128 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T22:45:02.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:00.533364+0000 osd.3 (osd.3) 180 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:45:02.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:00.534592+0000 osd.3 (osd.3) 181 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:45:02.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.717631+0000 osd.2 (osd.2) 127 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T22:45:02.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:01 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.719660+0000 osd.2 (osd.2) 128 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T22:45:02.335 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:02.438 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:02.464 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:02.485 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T22:45:02.486 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T22:45:02.486 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-13T22:45:02.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-13T22:45:02.540 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:45:02.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:02.597 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:02.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:02.722 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:02.825 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:02.855 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:02.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:02.978 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:02.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:03.032 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:45:03.032 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:45:03.050 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:45:03.050 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:45:03.066 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:45:03.067 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:45:03.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:03.084 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:45:03.084 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:45:03.106 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:45:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:45:03.107 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:03.124 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:45:03.125 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:45:03.143 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:45:03.143 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:45:03.172 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:45:03.173 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.546316+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.550393+0000 osd.3 (osd.3) 183 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.824714+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.826192+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.886949+0000 osd.5 (osd.5) 174 : cluster [DBG] 6.19 deep-scrub starts 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:01.888820+0000 osd.5 (osd.5) 175 : cluster [DBG] 6.19 deep-scrub ok 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.575653+0000 mgr.y (mgr.14613) 275 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.746884+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.f scrub starts 2024-09-13T22:45:03.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.748325+0000 osd.2 (osd.2) 130 : cluster [DBG] 6.f scrub ok 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.546316+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.550393+0000 osd.3 (osd.3) 183 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.824714+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.826192+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.886949+0000 osd.5 (osd.5) 174 : cluster [DBG] 6.19 deep-scrub starts 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:01.888820+0000 osd.5 (osd.5) 175 : cluster [DBG] 6.19 deep-scrub ok 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.575653+0000 mgr.y (mgr.14613) 275 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.746884+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.f scrub starts 2024-09-13T22:45:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:03 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.748325+0000 osd.2 (osd.2) 130 : cluster [DBG] 6.f scrub ok 2024-09-13T22:45:03.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:03.236 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:03.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:03.363 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.546316+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.550393+0000 osd.3 (osd.3) 183 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.824714+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.826192+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.886949+0000 osd.5 (osd.5) 174 : cluster [DBG] 6.19 deep-scrub starts 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:01.888820+0000 osd.5 (osd.5) 175 : cluster [DBG] 6.19 deep-scrub ok 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.575653+0000 mgr.y (mgr.14613) 275 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 583 KiB data, 930 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.746884+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.f scrub starts 2024-09-13T22:45:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:03 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.748325+0000 osd.2 (osd.2) 130 : cluster [DBG] 6.f scrub ok 2024-09-13T22:45:03.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:03.494 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:03.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:03.624 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:03.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:03.753 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:03.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:03.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:03.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:03.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:04.014 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:04.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:04.145 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:45:04.147 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:45:04.147 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:45:04.151 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:04.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.512358+0000 osd.3 (osd.3) 184 : cluster [DBG] 6.1b deep-scrub starts 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.514738+0000 osd.3 (osd.3) 185 : cluster [DBG] 6.1b deep-scrub ok 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.964892+0000 osd.4 (osd.4) 137 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.968145+0000 osd.4 (osd.4) 138 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.987492+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:02.989786+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: audit 2024-09-13T22:45:03.011392+0000 mon.c (mon.2) 36 : audit [DBG] from='client.? v1:172.21.15.178:0/3273498483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: audit 2024-09-13T22:45:03.145062+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: audit 2024-09-13T22:45:03.150103+0000 mon.b (mon.1) 132 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: audit 2024-09-13T22:45:03.241169+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.178:0/1311586390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: audit 2024-09-13T22:45:03.241479+0000 mon.a (mon.0) 1072 : audit [INF] from='client.25004 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:03.699764+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:45:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:04 smithi192 bash[24608]: cluster 2024-09-13T22:45:03.702170+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:45:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:04.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:04.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.512358+0000 osd.3 (osd.3) 184 : cluster [DBG] 6.1b deep-scrub starts 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.514738+0000 osd.3 (osd.3) 185 : cluster [DBG] 6.1b deep-scrub ok 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.964892+0000 osd.4 (osd.4) 137 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.968145+0000 osd.4 (osd.4) 138 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.987492+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:02.989786+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: audit 2024-09-13T22:45:03.011392+0000 mon.c (mon.2) 36 : audit [DBG] from='client.? v1:172.21.15.178:0/3273498483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: audit 2024-09-13T22:45:03.145062+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: audit 2024-09-13T22:45:03.150103+0000 mon.b (mon.1) 132 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: audit 2024-09-13T22:45:03.241169+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.178:0/1311586390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: audit 2024-09-13T22:45:03.241479+0000 mon.a (mon.0) 1072 : audit [INF] from='client.25004 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:03.699764+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:45:04.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[29889]: cluster 2024-09-13T22:45:03.702170+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.512358+0000 osd.3 (osd.3) 184 : cluster [DBG] 6.1b deep-scrub starts 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.514738+0000 osd.3 (osd.3) 185 : cluster [DBG] 6.1b deep-scrub ok 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.964892+0000 osd.4 (osd.4) 137 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.968145+0000 osd.4 (osd.4) 138 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.987492+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:02.989786+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: audit 2024-09-13T22:45:03.011392+0000 mon.c (mon.2) 36 : audit [DBG] from='client.? v1:172.21.15.178:0/3273498483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: audit 2024-09-13T22:45:03.145062+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: audit 2024-09-13T22:45:03.150103+0000 mon.b (mon.1) 132 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: audit 2024-09-13T22:45:03.241169+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.178:0/1311586390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: audit 2024-09-13T22:45:03.241479+0000 mon.a (mon.0) 1072 : audit [INF] from='client.25004 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:45:04.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:03.699764+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:45:04.271 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:04 smithi178 bash[22840]: cluster 2024-09-13T22:45:03.702170+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:45:04.280 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:04.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:04.412 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:04.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:04.542 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:04.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:04.672 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:04.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:04.803 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:04.905 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:04.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:04.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:04.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T22:45:05.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:05.061 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:05.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:05.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:03.475700+0000 osd.3 (osd.3) 186 : cluster [DBG] 6.c scrub starts 2024-09-13T22:45:05.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:03.477923+0000 osd.3 (osd.3) 187 : cluster [DBG] 6.c scrub ok 2024-09-13T22:45:05.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:03.918300+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:45:05.177 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:03.920304+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.033770+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.11 deep-scrub starts 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.034986+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.11 deep-scrub ok 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.141917+0000 mon.a (mon.0) 1073 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: audit 2024-09-13T22:45:04.143624+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25004 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.148487+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.576486+0000 mgr.y (mgr.14613) 276 : cluster [DBG] pgmap v278: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:03.475700+0000 osd.3 (osd.3) 186 : cluster [DBG] 6.c scrub starts 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:03.477923+0000 osd.3 (osd.3) 187 : cluster [DBG] 6.c scrub ok 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:03.918300+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:45:05.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:03.920304+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:45:05.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.033770+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.11 deep-scrub starts 2024-09-13T22:45:05.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.034986+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.11 deep-scrub ok 2024-09-13T22:45:05.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.141917+0000 mon.a (mon.0) 1073 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:05.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: audit 2024-09-13T22:45:04.143624+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25004 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:45:05.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.148487+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-13T22:45:05.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:05 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.576486+0000 mgr.y (mgr.14613) 276 : cluster [DBG] pgmap v278: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:45:05.192 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:05.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:05.327 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:05.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:03.475700+0000 osd.3 (osd.3) 186 : cluster [DBG] 6.c scrub starts 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:03.477923+0000 osd.3 (osd.3) 187 : cluster [DBG] 6.c scrub ok 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:03.918300+0000 osd.0 (osd.0) 169 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:03.920304+0000 osd.0 (osd.0) 170 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.033770+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.11 deep-scrub starts 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.034986+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.11 deep-scrub ok 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.141917+0000 mon.a (mon.0) 1073 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: audit 2024-09-13T22:45:04.143624+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25004 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:45:05.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.148487+0000 mon.a (mon.0) 1075 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-13T22:45:05.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:05 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.576486+0000 mgr.y (mgr.14613) 276 : cluster [DBG] pgmap v278: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:45:05.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:05.457 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:05.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:05.587 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:05.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:05.713 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:05.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:05.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:05.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:05.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:05.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:06.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:06.103 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:06.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:45:06.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:06.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:06 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.478161+0000 osd.3 (osd.3) 188 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:45:06.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:06 smithi192 bash[24608]: cluster 2024-09-13T22:45:04.482204+0000 osd.3 (osd.3) 189 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:45:06.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:06 smithi192 bash[24608]: audit 2024-09-13T22:45:05.146023+0000 mgr.y (mgr.14613) 277 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:06.235 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:06.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:06.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:06 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.478161+0000 osd.3 (osd.3) 188 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:45:06.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:06 smithi178 bash[29889]: cluster 2024-09-13T22:45:04.482204+0000 osd.3 (osd.3) 189 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:45:06.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:06 smithi178 bash[29889]: audit 2024-09-13T22:45:05.146023+0000 mgr.y (mgr.14613) 277 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:06.350 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:06 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.478161+0000 osd.3 (osd.3) 188 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:45:06.350 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:06 smithi178 bash[22840]: cluster 2024-09-13T22:45:04.482204+0000 osd.3 (osd.3) 189 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:45:06.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:06 smithi178 bash[22840]: audit 2024-09-13T22:45:05.146023+0000 mgr.y (mgr.14613) 277 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:06.363 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:06.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:06.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:06.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:06.620 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:06.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:06.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:06.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:06.880 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:06.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:06.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:45:06.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:07.011 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:07.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:07.137 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:07.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:07.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:07 smithi178 bash[29889]: cluster 2024-09-13T22:45:06.577158+0000 mgr.y (mgr.14613) 278 : cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:07.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:07 smithi178 bash[22840]: cluster 2024-09-13T22:45:06.577158+0000 mgr.y (mgr.14613) 278 : cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:07.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:07.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:07.390 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:07.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:07 smithi192 bash[24608]: cluster 2024-09-13T22:45:06.577158+0000 mgr.y (mgr.14613) 278 : cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:07.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:07.522 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:07.625 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:07.651 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:07.709 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:07.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:07.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:07.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:07.906 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:07.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:08.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:08.036 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:08.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:08.165 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:08.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:08.295 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:08.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:08.427 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:08.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:08.557 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:08.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:08.688 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:08.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:08.812 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:08.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:08.940 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:08.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:09.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.051 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:08 smithi178 bash[29889]: cluster 2024-09-13T22:45:08.577759+0000 mgr.y (mgr.14613) 279 : cluster [DBG] pgmap v280: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-09-13T22:45:09.051 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:08 smithi178 bash[22840]: cluster 2024-09-13T22:45:08.577759+0000 mgr.y (mgr.14613) 279 : cluster [DBG] pgmap v280: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-09-13T22:45:09.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:09.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:08 smithi192 bash[24608]: cluster 2024-09-13T22:45:08.577759+0000 mgr.y (mgr.14613) 279 : cluster [DBG] pgmap v280: 132 pgs: 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-09-13T22:45:09.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.185 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:09.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:09.315 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:09.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.440 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:09.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:09.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:09.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:09.696 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:09.766 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:45:09.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:09.823 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:09.842 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_1' created 2024-09-13T22:45:09.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:09.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:10.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:10.070 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:10.071 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:45:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:45:10.072 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[29889]: cluster 2024-09-13T22:45:09.211869+0000 osd.3 (osd.3) 190 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:10.072 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[29889]: cluster 2024-09-13T22:45:09.211930+0000 osd.3 (osd.3) 191 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:10.072 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[29889]: audit 2024-09-13T22:45:09.360677+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? v1:172.21.15.178:0/101040456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-13T22:45:10.072 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[22840]: cluster 2024-09-13T22:45:09.211869+0000 osd.3 (osd.3) 190 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:10.072 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[22840]: cluster 2024-09-13T22:45:09.211930+0000 osd.3 (osd.3) 191 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:10.072 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:09 smithi178 bash[22840]: audit 2024-09-13T22:45:09.360677+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? v1:172.21.15.178:0/101040456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-13T22:45:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:09 smithi192 bash[24608]: cluster 2024-09-13T22:45:09.211869+0000 osd.3 (osd.3) 190 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:09 smithi192 bash[24608]: cluster 2024-09-13T22:45:09.211930+0000 osd.3 (osd.3) 191 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:09 smithi192 bash[24608]: audit 2024-09-13T22:45:09.360677+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? v1:172.21.15.178:0/101040456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-13T22:45:10.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:10.197 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:10.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:10.327 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:10.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:10.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:10.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:10.588 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:10.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:10.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:10.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:10.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-13T22:45:10.847 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:10.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:10.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:10.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:11.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:11.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:10 smithi192 bash[24608]: cluster 2024-09-13T22:45:08.979852+0000 osd.6 (osd.6) 129 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:11.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:10 smithi192 bash[24608]: cluster 2024-09-13T22:45:08.979965+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:11.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:10 smithi192 bash[24608]: audit 2024-09-13T22:45:09.845948+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? v1:172.21.15.178:0/101040456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T22:45:11.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:10 smithi192 bash[24608]: cluster 2024-09-13T22:45:09.849987+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-13T22:45:11.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:10 smithi192 bash[24608]: cluster 2024-09-13T22:45:10.578457+0000 mgr.y (mgr.14613) 280 : cluster [DBG] pgmap v282: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:45:11.098 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:11.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[29889]: cluster 2024-09-13T22:45:08.979852+0000 osd.6 (osd.6) 129 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[29889]: cluster 2024-09-13T22:45:08.979965+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[29889]: audit 2024-09-13T22:45:09.845948+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? v1:172.21.15.178:0/101040456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[29889]: cluster 2024-09-13T22:45:09.849987+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[29889]: cluster 2024-09-13T22:45:10.578457+0000 mgr.y (mgr.14613) 280 : cluster [DBG] pgmap v282: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[22840]: cluster 2024-09-13T22:45:08.979852+0000 osd.6 (osd.6) 129 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[22840]: cluster 2024-09-13T22:45:08.979965+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[22840]: audit 2024-09-13T22:45:09.845948+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? v1:172.21.15.178:0/101040456' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[22840]: cluster 2024-09-13T22:45:09.849987+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-13T22:45:11.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:10 smithi178 bash[22840]: cluster 2024-09-13T22:45:10.578457+0000 mgr.y (mgr.14613) 280 : cluster [DBG] pgmap v282: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:45:11.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:11.228 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:11.244 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:45:11.244 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:45:11.306 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:11.306 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:11.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:11.353 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:11.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:11.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:11.612 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:11.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:11.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:11.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:11.867 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:11 smithi178 bash[22840]: cluster 2024-09-13T22:45:10.851938+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:45:11.867 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:11 smithi178 bash[22840]: cluster 2024-09-13T22:45:10.865114+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-13T22:45:11.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:11.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:11.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.004 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:12.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T22:45:12.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:12.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:11 smithi178 bash[29889]: cluster 2024-09-13T22:45:10.851938+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:45:12.117 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:11 smithi178 bash[29889]: cluster 2024-09-13T22:45:10.865114+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-13T22:45:12.132 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:12.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:11 smithi192 bash[24608]: cluster 2024-09-13T22:45:10.851938+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:45:12.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:11 smithi192 bash[24608]: cluster 2024-09-13T22:45:10.865114+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-13T22:45:12.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:12.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:12.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:12.385 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:12.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:12.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:12.647 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:12.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:12.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:12 smithi192 bash[24608]: cluster 2024-09-13T22:45:11.875249+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-13T22:45:12.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:12 smithi192 bash[24608]: cluster 2024-09-13T22:45:12.579237+0000 mgr.y (mgr.14613) 281 : cluster [DBG] pgmap v285: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:45:12.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:12.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:13.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:13.038 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:13.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:13.167 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:12 smithi178 bash[29889]: cluster 2024-09-13T22:45:11.875249+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-13T22:45:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:12 smithi178 bash[29889]: cluster 2024-09-13T22:45:12.579237+0000 mgr.y (mgr.14613) 281 : cluster [DBG] pgmap v285: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:45:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:12 smithi178 bash[22840]: cluster 2024-09-13T22:45:11.875249+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-13T22:45:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:12 smithi178 bash[22840]: cluster 2024-09-13T22:45:12.579237+0000 mgr.y (mgr.14613) 281 : cluster [DBG] pgmap v285: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:45:13.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:13.296 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:13.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:13.422 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:13.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:13.548 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:13.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:13.679 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:13.723 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:45:13.723 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:45:13.740 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:45:13.741 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:45:13.757 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:45:13.758 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:45:13.776 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:45:13.777 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:45:13.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:13.798 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:45:13.798 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:45:13.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:13.819 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:45:13.819 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:45:13.841 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:45:13.841 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:45:13.862 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:45:13.862 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:45:13.885 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:45:13.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:13.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:13 smithi192 bash[24608]: audit 2024-09-13T22:45:13.693159+0000 mon.a (mon.0) 1082 : audit [DBG] from='client.? v1:172.21.15.178:0/1833324462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:13.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:13 smithi192 bash[24608]: audit 2024-09-13T22:45:13.875026+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.178:0/1350694786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:13.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:13 smithi192 bash[24608]: audit 2024-09-13T22:45:13.875359+0000 mon.a (mon.0) 1083 : audit [INF] from='client.25016 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:13.934 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:13.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:14.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:14.047 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:13 smithi178 bash[22840]: audit 2024-09-13T22:45:13.693159+0000 mon.a (mon.0) 1082 : audit [DBG] from='client.? v1:172.21.15.178:0/1833324462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:14.047 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:13 smithi178 bash[22840]: audit 2024-09-13T22:45:13.875026+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.178:0/1350694786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:14.048 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:13 smithi178 bash[22840]: audit 2024-09-13T22:45:13.875359+0000 mon.a (mon.0) 1083 : audit [INF] from='client.25016 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:14.048 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:13 smithi178 bash[29889]: audit 2024-09-13T22:45:13.693159+0000 mon.a (mon.0) 1082 : audit [DBG] from='client.? v1:172.21.15.178:0/1833324462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:14.048 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:13 smithi178 bash[29889]: audit 2024-09-13T22:45:13.875026+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.178:0/1350694786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:14.048 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:13 smithi178 bash[29889]: audit 2024-09-13T22:45:13.875359+0000 mon.a (mon.0) 1083 : audit [INF] from='client.25016 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:14.063 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:14.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:14.196 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:14.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:14.325 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:14.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:14.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:14.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:14.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:14.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:14.716 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:14.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:14.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:14.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:14.977 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:14.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:15.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:15.108 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:15.154 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:14 smithi192 bash[24608]: cluster 2024-09-13T22:45:13.877184+0000 mon.a (mon.0) 1084 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:14 smithi192 bash[24608]: audit 2024-09-13T22:45:13.884810+0000 mon.a (mon.0) 1085 : audit [INF] from='client.25016 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:45:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:14 smithi192 bash[24608]: cluster 2024-09-13T22:45:13.888497+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-13T22:45:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:14 smithi192 bash[24608]: cluster 2024-09-13T22:45:14.580232+0000 mgr.y (mgr.14613) 282 : cluster [DBG] pgmap v287: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:15.189 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[29889]: cluster 2024-09-13T22:45:13.877184+0000 mon.a (mon.0) 1084 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[29889]: audit 2024-09-13T22:45:13.884810+0000 mon.a (mon.0) 1085 : audit [INF] from='client.25016 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[29889]: cluster 2024-09-13T22:45:13.888497+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[29889]: cluster 2024-09-13T22:45:14.580232+0000 mgr.y (mgr.14613) 282 : cluster [DBG] pgmap v287: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[22840]: cluster 2024-09-13T22:45:13.877184+0000 mon.a (mon.0) 1084 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[22840]: audit 2024-09-13T22:45:13.884810+0000 mon.a (mon.0) 1085 : audit [INF] from='client.25016 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[22840]: cluster 2024-09-13T22:45:13.888497+0000 mon.a (mon.0) 1086 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-13T22:45:15.190 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:14 smithi178 bash[22840]: cluster 2024-09-13T22:45:14.580232+0000 mgr.y (mgr.14613) 282 : cluster [DBG] pgmap v287: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:45:15.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:15.239 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:15.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:15.368 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:15.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:45:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:45:15.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:15.498 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:15.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:15.628 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:15.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:15.759 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:15.862 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:15.889 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:15.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:15.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:16.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:16.114 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:16.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:16.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:15 smithi192 bash[24608]: audit 2024-09-13T22:45:15.157000+0000 mgr.y (mgr.14613) 283 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:16.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:16.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:15 smithi178 bash[22840]: audit 2024-09-13T22:45:15.157000+0000 mgr.y (mgr.14613) 283 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:16.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:15 smithi178 bash[29889]: audit 2024-09-13T22:45:15.157000+0000 mgr.y (mgr.14613) 283 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:16.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:16.282 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:16.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:16.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:16.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:16.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:16.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:16.667 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:16.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:16.798 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:16.846 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:45:16.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:16 smithi192 bash[24608]: cluster 2024-09-13T22:45:16.581009+0000 mgr.y (mgr.14613) 284 : cluster [DBG] pgmap v288: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:45:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:16 smithi192 bash[24608]: cluster 2024-09-13T22:45:16.668204+0000 osd.2 (osd.2) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:16.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:16 smithi192 bash[24608]: cluster 2024-09-13T22:45:16.668309+0000 osd.2 (osd.2) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:16.924 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:16.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:17.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:17.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:17.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:17.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:16 smithi178 bash[29889]: cluster 2024-09-13T22:45:16.581009+0000 mgr.y (mgr.14613) 284 : cluster [DBG] pgmap v288: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:45:17.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:16 smithi178 bash[29889]: cluster 2024-09-13T22:45:16.668204+0000 osd.2 (osd.2) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:17.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:16 smithi178 bash[29889]: cluster 2024-09-13T22:45:16.668309+0000 osd.2 (osd.2) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:17.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:16 smithi178 bash[22840]: cluster 2024-09-13T22:45:16.581009+0000 mgr.y (mgr.14613) 284 : cluster [DBG] pgmap v288: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:45:17.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:16 smithi178 bash[22840]: cluster 2024-09-13T22:45:16.668204+0000 osd.2 (osd.2) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:17.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:16 smithi178 bash[22840]: cluster 2024-09-13T22:45:16.668309+0000 osd.2 (osd.2) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:17.184 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:17.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:17.308 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:17.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:17.432 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:17.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:17.561 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:17.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:17.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:45:17.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:17.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:45:17.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:17.819 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:17.898 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-13T22:45:17.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:17.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:17.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:18.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:18.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:17 smithi178 bash[22840]: cluster 2024-09-13T22:45:16.895753+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-13T22:45:18.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:17 smithi178 bash[22840]: audit 2024-09-13T22:45:17.560129+0000 mon.c (mon.2) 39 : audit [INF] from='client.? v1:172.21.15.178:0/2121476281' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:45:18.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:17 smithi178 bash[22840]: audit 2024-09-13T22:45:17.560835+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25028 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:45:18.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:17 smithi178 bash[29889]: cluster 2024-09-13T22:45:16.895753+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-13T22:45:18.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:17 smithi178 bash[29889]: audit 2024-09-13T22:45:17.560129+0000 mon.c (mon.2) 39 : audit [INF] from='client.? v1:172.21.15.178:0/2121476281' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:45:18.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:17 smithi178 bash[29889]: audit 2024-09-13T22:45:17.560835+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25028 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:45:18.074 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:18.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:17 smithi192 bash[24608]: cluster 2024-09-13T22:45:16.895753+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-13T22:45:18.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:17 smithi192 bash[24608]: audit 2024-09-13T22:45:17.560129+0000 mon.c (mon.2) 39 : audit [INF] from='client.? v1:172.21.15.178:0/2121476281' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:45:18.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:17 smithi192 bash[24608]: audit 2024-09-13T22:45:17.560835+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25028 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:45:18.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:18.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:18.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:18.326 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:18.411 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:45:18.411 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:45:18.411 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:45:18.411 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:45:18.411 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:45:18.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:18.448 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:18.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:18.580 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:18.683 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:18.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:18.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:18.836 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:18.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:18.962 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:18.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:19.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:19.094 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:19.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T22:45:19.170 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:18 smithi192 bash[24608]: audit 2024-09-13T22:45:17.899321+0000 mon.a (mon.0) 1089 : audit [INF] from='client.25028 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:45:19.171 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:18 smithi192 bash[24608]: cluster 2024-09-13T22:45:17.903228+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-13T22:45:19.171 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:18 smithi192 bash[24608]: audit 2024-09-13T22:45:18.170478+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:19.171 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:18 smithi192 bash[24608]: audit 2024-09-13T22:45:18.173244+0000 mon.b (mon.1) 133 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:19.171 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:18 smithi192 bash[24608]: cluster 2024-09-13T22:45:18.581628+0000 mgr.y (mgr.14613) 285 : cluster [DBG] pgmap v291: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:45:19.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[29889]: audit 2024-09-13T22:45:17.899321+0000 mon.a (mon.0) 1089 : audit [INF] from='client.25028 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:45:19.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[29889]: cluster 2024-09-13T22:45:17.903228+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-13T22:45:19.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[29889]: audit 2024-09-13T22:45:18.170478+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[29889]: audit 2024-09-13T22:45:18.173244+0000 mon.b (mon.1) 133 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[29889]: cluster 2024-09-13T22:45:18.581628+0000 mgr.y (mgr.14613) 285 : cluster [DBG] pgmap v291: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[22840]: audit 2024-09-13T22:45:17.899321+0000 mon.a (mon.0) 1089 : audit [INF] from='client.25028 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[22840]: cluster 2024-09-13T22:45:17.903228+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[22840]: audit 2024-09-13T22:45:18.170478+0000 mon.a (mon.0) 1091 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[22840]: audit 2024-09-13T22:45:18.173244+0000 mon.b (mon.1) 133 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:19.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:18 smithi178 bash[22840]: cluster 2024-09-13T22:45:18.581628+0000 mgr.y (mgr.14613) 285 : cluster [DBG] pgmap v291: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:45:19.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:19.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:19.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:19.350 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:19.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:19.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:19.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:19.603 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:19.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:19.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:19.760 DEBUG:teuthology.orchestra.run.smithi178:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-13T22:45:19.761 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T22:45:19.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:45:19.772 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T22:45:19.772 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T22:45:19.809 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:hints = 1 2024-09-13T22:45:19.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T22:45:19.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_80607 2024-09-13T22:45:19.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:45:19.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-13T22:45:19.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:19.858 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:19.909 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:45:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:45:19.909 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[29889]: audit 2024-09-13T22:45:19.090511+0000 mon.b (mon.1) 134 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:45:19.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:19.988 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:19.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:20.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:20.105 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[22840]: audit 2024-09-13T22:45:19.090511+0000 mon.b (mon.1) 134 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:45:20.105 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[22840]: cluster 2024-09-13T22:45:19.166089+0000 mon.a (mon.0) 1092 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:45:20.105 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[22840]: audit 2024-09-13T22:45:19.838283+0000 mon.b (mon.1) 135 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:45:20.105 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[22840]: audit 2024-09-13T22:45:19.842267+0000 mon.b (mon.1) 136 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:45:20.105 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[22840]: audit 2024-09-13T22:45:19.855333+0000 mon.a (mon.0) 1093 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:20.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[29889]: cluster 2024-09-13T22:45:19.166089+0000 mon.a (mon.0) 1092 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:45:20.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[29889]: audit 2024-09-13T22:45:19.838283+0000 mon.b (mon.1) 135 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:45:20.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[29889]: audit 2024-09-13T22:45:19.842267+0000 mon.b (mon.1) 136 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:45:20.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:19 smithi178 bash[29889]: audit 2024-09-13T22:45:19.855333+0000 mon.a (mon.0) 1093 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:20.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:20.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:19 smithi192 bash[24608]: audit 2024-09-13T22:45:19.090511+0000 mon.b (mon.1) 134 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:45:20.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:19 smithi192 bash[24608]: cluster 2024-09-13T22:45:19.166089+0000 mon.a (mon.0) 1092 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:45:20.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:19 smithi192 bash[24608]: audit 2024-09-13T22:45:19.838283+0000 mon.b (mon.1) 135 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:45:20.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:19 smithi192 bash[24608]: audit 2024-09-13T22:45:19.842267+0000 mon.b (mon.1) 136 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:45:20.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:19 smithi192 bash[24608]: audit 2024-09-13T22:45:19.855333+0000 mon.a (mon.0) 1093 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:20.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:20.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:20.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:20.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:20.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:20.513 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:20.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:20.653 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:20.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:20.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:20.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 1 16 3015 2999 187.427 187.438 0.00210522 0.00531142 2024-09-13T22:45:20.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:20.924 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:20.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:21.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:21.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:21.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:20 smithi192 bash[24608]: cluster 2024-09-13T22:45:20.582537+0000 mgr.y (mgr.14613) 286 : cluster [DBG] pgmap v292: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:45:21.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:21.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:20 smithi178 bash[29889]: cluster 2024-09-13T22:45:20.582537+0000 mgr.y (mgr.14613) 286 : cluster [DBG] pgmap v292: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:45:21.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:20 smithi178 bash[22840]: cluster 2024-09-13T22:45:20.582537+0000 mgr.y (mgr.14613) 286 : cluster [DBG] pgmap v292: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:45:21.193 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:21.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:21.320 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:21.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:21.449 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:21.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:21.578 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:21.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:21.710 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:21.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 2 16 6120 6104 190.737 194.062 0.00646498 0.005233 2024-09-13T22:45:21.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:21.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:21.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:21.971 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:21.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:22.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:22.101 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:22.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:22.229 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:22.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:22.355 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:22.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:22.466 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:22.466 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:22.491 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:22.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:22.623 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:22.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:22.748 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:22.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 3 16 8912 8896 185.32 174.5 0.0105469 0.00538558 2024-09-13T22:45:22.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:22.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:22.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:22.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:22 smithi178 bash[29889]: cluster 2024-09-13T22:45:22.582981+0000 mgr.y (mgr.14613) 287 : cluster [DBG] pgmap v293: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:45:22.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:22 smithi178 bash[22840]: cluster 2024-09-13T22:45:22.582981+0000 mgr.y (mgr.14613) 287 : cluster [DBG] pgmap v293: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:45:22.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:23.014 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:23.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:23.128 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:22 smithi192 bash[24608]: cluster 2024-09-13T22:45:22.582981+0000 mgr.y (mgr.14613) 287 : cluster [DBG] pgmap v293: 148 pgs: 148 active+clean; 583 KiB data, 929 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:45:23.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:23.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:23.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:23.373 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:23.397 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:23.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:23.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:23.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:23.653 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:23.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:23.781 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:23.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 4 16 11612 11596 181.174 168.75 0.00487926 0.0055106 2024-09-13T22:45:23.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:23.911 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:23.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:23.939 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:45:24.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:24.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:24.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:24.167 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:24.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:24.294 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:24.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:24.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:24.501 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:24.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:24.552 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:24.655 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:24.677 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:24.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:24.805 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:24.807 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:24 smithi178 bash[29889]: cluster 2024-09-13T22:45:24.583664+0000 mgr.y (mgr.14613) 288 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 454 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T22:45:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:24.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:24 smithi178 bash[22840]: cluster 2024-09-13T22:45:24.583664+0000 mgr.y (mgr.14613) 288 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 454 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T22:45:24.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 5 16 13623 13607 170.074 125.688 0.00372748 0.00587202 2024-09-13T22:45:24.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:24.932 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:24.934 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:24 smithi192 bash[24608]: cluster 2024-09-13T22:45:24.583664+0000 mgr.y (mgr.14613) 288 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 454 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T22:45:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:24.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:25.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:25.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:25.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:25.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:45:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:45:25.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:25.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:25.314 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:25.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:25.435 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:25.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:25.563 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:25.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:25.684 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:25.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:25.806 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:25.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 6 16 15615 15599 162.475 124.5 0.00141701 0.00612164 2024-09-13T22:45:25.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:25.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:25.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:25.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:25 smithi178 bash[29889]: audit 2024-09-13T22:45:25.165884+0000 mgr.y (mgr.14613) 289 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:25.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:25 smithi178 bash[22840]: audit 2024-09-13T22:45:25.165884+0000 mgr.y (mgr.14613) 289 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:26.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:26.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:26.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:25 smithi192 bash[24608]: audit 2024-09-13T22:45:25.165884+0000 mgr.y (mgr.14613) 289 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:26.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:26.181 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:26.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:26.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:26.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:26.438 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:26.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:26.568 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:26.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:26.699 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:26.804 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:26.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 7 16 16222 16206 144.683 37.9375 0.00634187 0.00690205 2024-09-13T22:45:26.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:26 smithi192 bash[24608]: cluster 2024-09-13T22:45:26.584586+0000 mgr.y (mgr.14613) 290 : cluster [DBG] pgmap v295: 148 pgs: 148 active+clean; 764 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 950 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T22:45:26.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:26.948 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:26 smithi178 bash[29889]: cluster 2024-09-13T22:45:26.584586+0000 mgr.y (mgr.14613) 290 : cluster [DBG] pgmap v295: 148 pgs: 148 active+clean; 764 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 950 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T22:45:26.948 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:26 smithi178 bash[22840]: cluster 2024-09-13T22:45:26.584586+0000 mgr.y (mgr.14613) 290 : cluster [DBG] pgmap v295: 148 pgs: 148 active+clean; 764 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 950 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T22:45:26.968 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:26.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:27.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:27.102 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:27.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:27.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:27.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:27.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:27.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:27.485 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:27.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:27.607 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:27.710 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:27.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:27.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 8 16 17948 17932 140.081 107.875 0.00441289 0.00713113 2024-09-13T22:45:27.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:27.873 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:27.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:27.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:28.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:28.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:28.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:28.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:28.261 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:28.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:28.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:28.489 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:62593 7.5 7:a8c10539:::benchmark_data_smithi178_64772_object62592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:48.808981+0000", 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 219.68215684, 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36854543299999998, 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 62593 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.808981+0000", 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.808981+0000", 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.808984+0000", 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0340000000000001e-06 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.809026+0000", 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2092999999999998e-05 2024-09-13T22:45:28.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.809040+0000", 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.466e-05 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.809048+0000", 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1740000000000003e-06 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.809116+0000", 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8369999999999998e-05 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.809222+0000", 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010614 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.809308+0000", 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5717000000000006e-05 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.810893+0000", 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001584685 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.811585+0000", 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000692543 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:49.177356+0000", 2024-09-13T22:45:28.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36577066000000003 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:49.177452+0000", 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6574000000000001e-05 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:49.177526+0000", 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3782999999999995e-05 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.496 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:64799 7.5 7:ae124af4:::benchmark_data_smithi178_64772_object64798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:51.147580+0000", 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 217.34355722000001, 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21721337600000001, 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 64799 2024-09-13T22:45:28.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147580+0000", 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147580+0000", 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147583+0000", 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3939999999999999e-06 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147660+0000", 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6978999999999996e-05 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147671+0000", 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.147e-05 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147680+0000", 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1749999999999994e-06 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147746+0000", 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5996999999999998e-05 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147881+0000", 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134439 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147979+0000", 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8039000000000001e-05 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.151654+0000", 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036749230000000001 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.153437+0000", 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017829250000000001 2024-09-13T22:45:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.364483+0000", 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21104667599999999 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.364667+0000", 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018377500000000001 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.364794+0000", 2024-09-13T22:45:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012658400000000001 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:64800 7.5 7:ad8f593c:::benchmark_data_smithi178_64772_object64799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:51.147714+0000", 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 217.34342351500001, 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.217598603, 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 64800 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147714+0000", 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147714+0000", 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147715+0000", 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.173e-06 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147770+0000", 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4874999999999999e-05 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147778+0000", 2024-09-13T22:45:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7740000000000004e-06 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.147785+0000", 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8249999999999999e-06 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.149515+0000", 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001730475 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.149651+0000", 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013552700000000001 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.149789+0000", 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013794799999999999 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.152917+0000", 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031280969999999998 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.153705+0000", 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000788023 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.364937+0000", 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211232376 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.365193+0000", 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025584199999999999 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:51.365313+0000", 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011966799999999999 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:69031 7.5 7:a2de6756:::benchmark_data_smithi178_64772_object69030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:55.901682+0000", 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 212.58945537100001, 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21065900700000001, 2024-09-13T22:45:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 69031 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901682+0000", 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901682+0000", 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901685+0000", 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8260000000000001e-06 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901727+0000", 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2268999999999998e-05 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901743+0000", 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5665e-05 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901751+0000", 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3960000000000006e-06 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901807+0000", 2024-09-13T22:45:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5977000000000003e-05 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.901908+0000", 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010103 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.902007+0000", 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9147000000000005e-05 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.904077+0000", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020697409999999999 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.904922+0000", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00084473999999999997 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:56.112206+0000", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20728445700000001 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:56.112280+0000", 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3164000000000006e-05 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:56.112341+0000", 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1594999999999996e-05 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:69037 7.5 7:ac547ba2:::benchmark_data_smithi178_64772_object69036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:55.907166+0000", 2024-09-13T22:45:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 212.583971953, 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.214441348, 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 69037 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907166+0000", 2024-09-13T22:45:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907166+0000", 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907168+0000", 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5950000000000001e-06 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907213+0000", 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4934e-05 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907227+0000", 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4368e-05 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907235+0000", 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9670000000000001e-06 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907293+0000", 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7154999999999997e-05 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907417+0000", 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012454499999999999 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.907587+0000", 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016988999999999999 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.909859+0000", 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022720190000000001 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:55.910662+0000", 2024-09-13T22:45:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080243999999999997 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:56.121495+0000", 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21083384399999999 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:56.121546+0000", 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0939000000000001e-05 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:56.121607+0000", 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0652000000000001e-05 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:109855 7.5 7:a63f1e65:::benchmark_data_smithi178_64772_object4102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:42:42.975119+0000", 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 165.51601877600001, 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211391721, 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 109855 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975119+0000", 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975119+0000", 2024-09-13T22:45:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975122+0000", 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7369999999999998e-06 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975128+0000", 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8369999999999999e-06 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975143+0000", 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4609e-05 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975150+0000", 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8390000000000001e-06 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975215+0000", 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5096000000000002e-05 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975353+0000", 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013772799999999999 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.975492+0000", 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000138975 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.982982+0000", 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00749089 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:42.983006+0000", 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3550999999999999e-05 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:43.186231+0000", 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20322526799999999 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:43.186420+0000", 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018856299999999999 2024-09-13T22:45:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:43.186510+0000", 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0628e-05 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:111104 7.5 7:a32481cf:::benchmark_data_smithi178_64772_object5351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:42:44.878307+0000", 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 163.61283016199999, 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35306960599999998, 2024-09-13T22:45:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111104 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878307+0000", 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878307+0000", 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878312+0000", 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3649999999999997e-06 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878323+0000", 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.149e-05 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878352+0000", 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8487e-05 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878370+0000", 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7842e-05 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878500+0000", 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000130135 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878698+0000", 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000197877 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.878834+0000", 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013635000000000001 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.886964+0000", 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081299849999999993 2024-09-13T22:45:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.887114+0000", 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014992599999999999 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:45.231214+0000", 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34409996399999998 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:45.231303+0000", 2024-09-13T22:45:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8737000000000001e-05 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:45.231377+0000", 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4448000000000003e-05 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:111107 7.5 7:af9f7ca6:::benchmark_data_smithi178_64772_object5354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:42:44.881013+0000", 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 163.610124509, 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36280696200000001, 2024-09-13T22:45:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111107 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881013+0000", 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881013+0000", 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881016+0000", 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5900000000000002e-06 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881025+0000", 2024-09-13T22:45:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7000000000000003e-06 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881045+0000", 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0081999999999998e-05 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881057+0000", 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1355000000000001e-05 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.881683+0000", 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00062601400000000004 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.882074+0000", 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039113100000000002 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.882687+0000", 2024-09-13T22:45:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00061283099999999999 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.887079+0000", 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043923360000000002 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:44.887317+0000", 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023772100000000001 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:45.243569+0000", 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35625215399999999 2024-09-13T22:45:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:45.243712+0000", 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000143237 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:42:45.243820+0000", 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107811 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:170908 7.5 7:ab149750:::benchmark_data_smithi178_64772_object65155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:43:58.154058+0000", 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 90.337079825000004, 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36090060200000001, 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 170908 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154058+0000", 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154058+0000", 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154060+0000", 2024-09-13T22:45:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9269999999999999e-06 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154065+0000", 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154074+0000", 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2359999999999993e-06 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154079+0000", 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9629999999999997e-06 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154134+0000", 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4391000000000002e-05 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154257+0000", 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012363700000000001 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.154607+0000", 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034965700000000002 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.157123+0000", 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025163749999999999 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.164821+0000", 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0076975860000000002 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.514611+0000", 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34979036200000002 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.514811+0000", 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019975900000000001 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.514958+0000", 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014739700000000001 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:170918 7.5 7:a26ee1d0:::benchmark_data_smithi178_64772_object65165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:43:58.162709+0000", 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 90.328428376000005, 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35266528200000002, 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 170918 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162709+0000", 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162709+0000", 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162711+0000", 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2740000000000002e-06 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162719+0000", 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0870000000000001e-06 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162734+0000", 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5396e-05 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162744+0000", 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0465999999999999e-05 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162798+0000", 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3177000000000003e-05 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162877+0000", 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9374000000000003e-05 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.162948+0000", 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0851000000000003e-05 2024-09-13T22:45:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.164924+0000", 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019766100000000002 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.167471+0000", 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025470319999999999 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.515092+0000", 2024-09-13T22:45:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34762091099999998 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.515249+0000", 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015691300000000001 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:58.515374+0000", 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125191 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:172743 7.5 7:a8310eda:::benchmark_data_smithi178_64772_object66990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:00.448383+0000", 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.042754993000003, 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26114089499999998, 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172743 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448383+0000", 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448383+0000", 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448384+0000", 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5999999999999999e-06 2024-09-13T22:45:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448389+0000", 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3719999999999998e-06 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448398+0000", 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7990000000000005e-06 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448404+0000", 2024-09-13T22:45:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2739999999999998e-06 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448454+0000", 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0538000000000003e-05 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448545+0000", 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0638999999999997e-05 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.448629+0000", 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4302999999999996e-05 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.450536+0000", 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019065600000000001 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.450624+0000", 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8664000000000004e-05 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.709342+0000", 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25871737 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.709447+0000", 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105472 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.709523+0000", 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6304000000000007e-05 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:173978 7.5 7:ab71518e:::benchmark_data_smithi178_64772_object68225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:02.056434+0000", 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.434703780000007, 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.247607041, 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 173978 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056434+0000", 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056434+0000", 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056436+0000", 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3039999999999999e-06 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056442+0000", 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7939999999999999e-06 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056455+0000", 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3083e-05 2024-09-13T22:45:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056462+0000", 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4279999999999999e-06 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056520+0000", 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7352999999999998e-05 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056609+0000", 2024-09-13T22:45:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9598999999999994e-05 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.056687+0000", 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8059000000000006e-05 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.058858+0000", 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021705959999999999 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.062595+0000", 2024-09-13T22:45:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037374169999999998 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.303958+0000", 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24136232199999999 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.304003+0000", 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5225999999999997e-05 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:02.304041+0000", 2024-09-13T22:45:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.786e-05 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:187131 7.5 7:a8cc48b4:::benchmark_data_smithi178_64772_object81378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:20.319525+0000", 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.171612936000002, 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.249313272, 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 187131 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319525+0000", 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319525+0000", 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319527+0000", 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2409999999999998e-06 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319533+0000", 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7010000000000001e-06 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319546+0000", 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3342e-05 2024-09-13T22:45:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319552+0000", 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5230000000000001e-06 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319613+0000", 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0081999999999998e-05 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319692+0000", 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9708000000000003e-05 2024-09-13T22:45:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.319760+0000", 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8249000000000003e-05 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.321707+0000", 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019463029999999999 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.322110+0000", 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040330299999999998 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.568770+0000", 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.246660304 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.568805+0000", 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5039000000000001e-05 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.568838+0000", 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2477000000000001e-05 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:187134 7.5 7:a98c072f:::benchmark_data_smithi178_64772_object81381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:20.324629+0000", 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.166508338, 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.244354404, 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 187134 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324629+0000", 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324629+0000", 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324631+0000", 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3289999999999999e-06 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324636+0000", 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7690000000000002e-06 2024-09-13T22:45:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324647+0000", 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0855e-05 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324652+0000", 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4129999999999999e-06 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324686+0000", 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4483000000000002e-05 2024-09-13T22:45:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324757+0000", 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1205999999999996e-05 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.324878+0000", 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120271 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.326723+0000", 2024-09-13T22:45:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018453600000000001 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.328336+0000", 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001613151 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.568926+0000", 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24058958799999999 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.568958+0000", 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2475000000000003e-05 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:20.568984+0000", 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5503999999999998e-05 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:197577 7.5 7:a88d8481:::benchmark_data_smithi178_64772_object91824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:32.907009+0000", 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.584128925000002, 2024-09-13T22:45:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23423081200000001, 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 197577 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907009+0000", 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907009+0000", 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907011+0000", 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1040000000000001e-06 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907016+0000", 2024-09-13T22:45:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5720000000000002e-06 2024-09-13T22:45:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907029+0000", 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2418e-05 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907036+0000", 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2049999999999996e-06 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907084+0000", 2024-09-13T22:45:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7936999999999999e-05 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907176+0000", 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1974999999999999e-05 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.907257+0000", 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1533999999999996e-05 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.909791+0000", 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025335240000000001 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.910215+0000", 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000423862 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:33.141151+0000", 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23093593300000001 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:33.141190+0000", 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236999999999997e-05 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:33.141239+0000", 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9511000000000001e-05 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:197596 7.5 7:a94c32c3:::benchmark_data_smithi178_64772_object91843:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T22:45:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:32.933674+0000", 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.557463482000003, 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20782568100000001, 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 197596 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933674+0000", 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933674+0000", 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933676+0000", 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933683+0000", 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4180000000000002e-06 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933699+0000", 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6392000000000002e-05 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933710+0000", 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1106e-05 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933760+0000", 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9904999999999997e-05 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.933885+0000", 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012515900000000001 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.934013+0000", 2024-09-13T22:45:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012735399999999999 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.936142+0000", 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021288919999999999 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:32.936318+0000", 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000176262 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:45:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:33.141394+0000", 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.205075969 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:33.141455+0000", 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1464000000000006e-05 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:33.141500+0000", 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4490000000000003e-05 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25037.0:15720 8.d e180/175 8:bad8a164:::benchmark_data_smithi178_80607_object15719:head v 180'1021, mlcod=180'1019)", 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.899668+0000", 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.5914696990000001, 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69480107199999996, 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899668+0000", 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899668+0000", 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899670+0000", 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7719999999999999e-06 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899700+0000", 2024-09-13T22:45:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9998e-05 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899710+0000", 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.03e-05 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899716+0000", 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5679999999999996e-06 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899774+0000", 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7802e-05 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899799+0000", 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5017000000000001e-05 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.594409+0000", 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69460982599999999 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.594469+0000", 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9789e-05 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25037.0:15727 8.c e180/175 8:304e4f43:::benchmark_data_smithi178_80607_object15726:head v 180'1019, mlcod=180'1017)", 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.899986+0000", 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.5911517430000002, 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69562781200000001, 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899986+0000", 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899986+0000", 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899987+0000", 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.446e-06 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900110+0000", 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012305500000000001 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900145+0000", 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5008000000000001e-05 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900162+0000", 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6682999999999999e-05 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900254+0000", 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1856999999999998e-05 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.594032+0000", 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69377838800000002 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.595534+0000", 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001501339 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.595614+0000", 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0036000000000004e-05 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25037.0:15726 8.6 e180/175 8:6f764505:::benchmark_data_smithi178_80607_object15725:head v 180'961, mlcod=180'959)", 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.900192+0000", 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.5909450999999999, 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69606694099999999, 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900192+0000", 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900192+0000", 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900194+0000", 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1680000000000001e-06 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900247+0000", 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3875000000000002e-05 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.593912+0000", 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69366410099999998 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.593924+0000", 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2397e-05 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.593993+0000", 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8773e-05 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.594028+0000", 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5139999999999999e-05 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596197+0000", 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021693709999999998 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596259+0000", 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2116e-05 2024-09-13T22:45:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25037.0:15731 8.5 e180/175 8:ac380b75:::benchmark_data_smithi178_80607_object15730:head v 180'987, mlcod=180'985)", 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.900739+0000", 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.5903981869999999, 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69564527700000001, 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:45:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900739+0000", 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900739+0000", 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900741+0000", 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.855e-06 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900770+0000", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8731999999999999e-05 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.593819+0000", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69304866300000001 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.593850+0000", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0926000000000002e-05 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:45:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.594410+0000", 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056028700000000002 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.594432+0000", 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2031999999999998e-05 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596333+0000", 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019011010000000001 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:45:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596385+0000", 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1681000000000002e-05 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:45:28.608 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:45:28.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:28.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:28.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:28.753 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:28.810 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 9 16 19564 19548 135.738 101 0.0090909 0.00736042 2024-09-13T22:45:28.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:28.878 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:28.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:28.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:28.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:28 smithi192 bash[24608]: cluster 2024-09-13T22:45:28.585096+0000 mgr.y (mgr.14613) 291 : cluster [DBG] pgmap v296: 148 pgs: 148 active+clean; 796 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 862 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-13T22:45:28.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:29.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:29.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:28 smithi178 bash[29889]: cluster 2024-09-13T22:45:28.585096+0000 mgr.y (mgr.14613) 291 : cluster [DBG] pgmap v296: 148 pgs: 148 active+clean; 796 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 862 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-13T22:45:29.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:28 smithi178 bash[22840]: cluster 2024-09-13T22:45:28.585096+0000 mgr.y (mgr.14613) 291 : cluster [DBG] pgmap v296: 148 pgs: 148 active+clean; 796 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 862 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-13T22:45:29.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:29.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:29.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:29.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:29.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:29.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T22:45:29.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:29.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:29.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:29.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 10 16 20870 20854 130.321 81.625 0.0086898 0.00766837 2024-09-13T22:45:29.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:29.903 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:45:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:45:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:45:29.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:29.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:30.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:30.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:30.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:30.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:30.284 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:30.301 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:30.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:30.432 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:30.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:30.563 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:30.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:30.690 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:30.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:30.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 11 16 23004 22988 130.597 133.375 0.00462431 0.00765089 2024-09-13T22:45:30.816 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:30.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:30.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:30 smithi178 bash[29889]: cluster 2024-09-13T22:45:30.585787+0000 mgr.y (mgr.14613) 292 : cluster [DBG] pgmap v297: 148 pgs: 148 active+clean; 1.2 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-13T22:45:30.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:30 smithi178 bash[22840]: cluster 2024-09-13T22:45:30.585787+0000 mgr.y (mgr.14613) 292 : cluster [DBG] pgmap v297: 148 pgs: 148 active+clean; 1.2 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-13T22:45:30.953 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:30.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:31.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:31.090 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:30 smithi192 bash[24608]: cluster 2024-09-13T22:45:30.585787+0000 mgr.y (mgr.14613) 292 : cluster [DBG] pgmap v297: 148 pgs: 148 active+clean; 1.2 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-13T22:45:31.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:31.318 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:45:31.319 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:45:31.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:31.342 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:31.342 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:45:31.343 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:45:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:31.369 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:45:31.369 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:45:31.400 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:45:31.400 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:45:31.423 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:45:31.424 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:45:31.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:31.454 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:45:31.454 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:45:31.471 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:31.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:31.490 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:45:31.490 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:45:31.520 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:45:31.522 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:45:31.526 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:45:31.526 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:45:31.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:31.602 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:31.695 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:45:31.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:31.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:31.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 12 15 24639 24624 128.234 102.25 0.00654702 0.00779224 2024-09-13T22:45:31.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:31.850 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:31 smithi178 bash[29889]: audit 2024-09-13T22:45:31.292700+0000 mon.a (mon.0) 1094 : audit [DBG] from='client.? v1:172.21.15.178:0/1273248383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:31.850 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:31 smithi178 bash[29889]: audit 2024-09-13T22:45:31.412445+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? v1:172.21.15.178:0/677325163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:45:31.850 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:31 smithi178 bash[29889]: audit 2024-09-13T22:45:31.501291+0000 mgr.y (mgr.14613) 293 : audit [DBG] from='client.25021 v1:172.21.15.178:0/1891890278' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:45:31.851 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:31 smithi178 bash[22840]: audit 2024-09-13T22:45:31.292700+0000 mon.a (mon.0) 1094 : audit [DBG] from='client.? v1:172.21.15.178:0/1273248383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:31.851 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:31 smithi178 bash[22840]: audit 2024-09-13T22:45:31.412445+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? v1:172.21.15.178:0/677325163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:45:31.851 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:31 smithi178 bash[22840]: audit 2024-09-13T22:45:31.501291+0000 mgr.y (mgr.14613) 293 : audit [DBG] from='client.25021 v1:172.21.15.178:0/1891890278' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:45:31.867 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:31.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:31 smithi192 bash[24608]: audit 2024-09-13T22:45:31.292700+0000 mon.a (mon.0) 1094 : audit [DBG] from='client.? v1:172.21.15.178:0/1273248383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:31.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:31 smithi192 bash[24608]: audit 2024-09-13T22:45:31.412445+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? v1:172.21.15.178:0/677325163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:45:31.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:31 smithi192 bash[24608]: audit 2024-09-13T22:45:31.501291+0000 mgr.y (mgr.14613) 293 : audit [DBG] from='client.25021 v1:172.21.15.178:0/1891890278' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:45:31.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:31.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:31.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:32.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:32.123 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:32.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:32.250 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:32.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:32.378 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:32.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:32.506 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:32.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:32.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:32.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:32.769 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:32.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 13 16 26716 26700 128.35 129.75 0.00610503 0.00778598 2024-09-13T22:45:32.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[29889]: cluster 2024-09-13T22:45:31.684409+0000 mon.a (mon.0) 1096 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[29889]: audit 2024-09-13T22:45:31.698149+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? v1:172.21.15.178:0/677325163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[29889]: cluster 2024-09-13T22:45:31.702687+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[29889]: cluster 2024-09-13T22:45:31.764551+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[29889]: cluster 2024-09-13T22:45:31.769044+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[29889]: cluster 2024-09-13T22:45:32.586128+0000 mgr.y (mgr.14613) 294 : cluster [DBG] pgmap v299: 148 pgs: 148 active+clean; 1.2 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[22840]: cluster 2024-09-13T22:45:31.684409+0000 mon.a (mon.0) 1096 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:32.885 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[22840]: audit 2024-09-13T22:45:31.698149+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? v1:172.21.15.178:0/677325163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:45:32.886 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[22840]: cluster 2024-09-13T22:45:31.702687+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-13T22:45:32.886 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[22840]: cluster 2024-09-13T22:45:31.764551+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:45:32.886 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[22840]: cluster 2024-09-13T22:45:31.769044+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:45:32.886 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:32 smithi178 bash[22840]: cluster 2024-09-13T22:45:32.586128+0000 mgr.y (mgr.14613) 294 : cluster [DBG] pgmap v299: 148 pgs: 148 active+clean; 1.2 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-13T22:45:32.900 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:32.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:33.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:33.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:32 smithi192 bash[24608]: cluster 2024-09-13T22:45:31.684409+0000 mon.a (mon.0) 1096 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:45:33.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:32 smithi192 bash[24608]: audit 2024-09-13T22:45:31.698149+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? v1:172.21.15.178:0/677325163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:45:33.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:32 smithi192 bash[24608]: cluster 2024-09-13T22:45:31.702687+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-13T22:45:33.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:32 smithi192 bash[24608]: cluster 2024-09-13T22:45:31.764551+0000 osd.0 (osd.0) 171 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:45:33.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:32 smithi192 bash[24608]: cluster 2024-09-13T22:45:31.769044+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:45:33.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:32 smithi192 bash[24608]: cluster 2024-09-13T22:45:32.586128+0000 mgr.y (mgr.14613) 294 : cluster [DBG] pgmap v299: 148 pgs: 148 active+clean; 1.2 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-13T22:45:33.027 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:33.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:33.172 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:33.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:33.303 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:33.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:33.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:33.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:33.562 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:33.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:33.698 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:33.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:33.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 14 16 28724 28708 128.145 125.5 0.00349149 0.007797 2024-09-13T22:45:33.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:33.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:33.959 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:33.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:34.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:34.084 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: cluster 2024-09-13T22:45:32.777100+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: cluster 2024-09-13T22:45:32.781038+0000 osd.0 (osd.0) 174 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: audit 2024-09-13T22:45:32.855310+0000 mon.b (mon.1) 137 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: audit 2024-09-13T22:45:32.855590+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: audit 2024-09-13T22:45:33.203954+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: audit 2024-09-13T22:45:33.206460+0000 mon.b (mon.1) 138 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: cluster 2024-09-13T22:45:33.696973+0000 osd.1 (osd.1) 164 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:45:34.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:33 smithi192 bash[24608]: cluster 2024-09-13T22:45:33.700473+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: cluster 2024-09-13T22:45:32.777100+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: cluster 2024-09-13T22:45:32.781038+0000 osd.0 (osd.0) 174 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: audit 2024-09-13T22:45:32.855310+0000 mon.b (mon.1) 137 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: audit 2024-09-13T22:45:32.855590+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: audit 2024-09-13T22:45:33.203954+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: audit 2024-09-13T22:45:33.206460+0000 mon.b (mon.1) 138 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: cluster 2024-09-13T22:45:33.696973+0000 osd.1 (osd.1) 164 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[22840]: cluster 2024-09-13T22:45:33.700473+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: cluster 2024-09-13T22:45:32.777100+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:45:34.186 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: cluster 2024-09-13T22:45:32.781038+0000 osd.0 (osd.0) 174 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:45:34.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: audit 2024-09-13T22:45:32.855310+0000 mon.b (mon.1) 137 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-13T22:45:34.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: audit 2024-09-13T22:45:32.855590+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-13T22:45:34.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: audit 2024-09-13T22:45:33.203954+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:45:34.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: audit 2024-09-13T22:45:33.206460+0000 mon.b (mon.1) 138 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:34.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: cluster 2024-09-13T22:45:33.696973+0000 osd.1 (osd.1) 164 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:45:34.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:33 smithi178 bash[29889]: cluster 2024-09-13T22:45:33.700473+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:45:34.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:34.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:34.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:34.351 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:34.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:34.479 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:34.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:34.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:34.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:34.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:34.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 15 16 29918 29902 124.577 74.625 0.00278735 0.00802153 2024-09-13T22:45:34.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:34.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:34.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:34.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:35.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:35.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:35.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:35.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:34 smithi192 bash[24608]: cluster 2024-09-13T22:45:33.669790+0000 osd.7 (osd.7) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:35.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:34 smithi192 bash[24608]: cluster 2024-09-13T22:45:33.669868+0000 osd.7 (osd.7) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:35.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:34 smithi192 bash[24608]: audit 2024-09-13T22:45:33.857782+0000 mon.a (mon.0) 1101 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-09-13T22:45:35.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:34 smithi192 bash[24608]: cluster 2024-09-13T22:45:33.862372+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-13T22:45:35.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:34 smithi192 bash[24608]: cluster 2024-09-13T22:45:34.586973+0000 mgr.y (mgr.14613) 295 : cluster [DBG] pgmap v301: 148 pgs: 148 active+clean; 1.6 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:45:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:35.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[29889]: cluster 2024-09-13T22:45:33.669790+0000 osd.7 (osd.7) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:35.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[29889]: cluster 2024-09-13T22:45:33.669868+0000 osd.7 (osd.7) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:35.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[29889]: audit 2024-09-13T22:45:33.857782+0000 mon.a (mon.0) 1101 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[29889]: cluster 2024-09-13T22:45:33.862372+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[29889]: cluster 2024-09-13T22:45:34.586973+0000 mgr.y (mgr.14613) 295 : cluster [DBG] pgmap v301: 148 pgs: 148 active+clean; 1.6 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[22840]: cluster 2024-09-13T22:45:33.669790+0000 osd.7 (osd.7) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[22840]: cluster 2024-09-13T22:45:33.669868+0000 osd.7 (osd.7) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[22840]: audit 2024-09-13T22:45:33.857782+0000 mon.a (mon.0) 1101 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[22840]: cluster 2024-09-13T22:45:33.862372+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-13T22:45:35.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:34 smithi178 bash[22840]: cluster 2024-09-13T22:45:34.586973+0000 mgr.y (mgr.14613) 295 : cluster [DBG] pgmap v301: 148 pgs: 148 active+clean; 1.6 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:45:35.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:35.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:35.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:35.384 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:35.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:45:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:45:35.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:35.509 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:35.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:35.639 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:35.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:35.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:35.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 16 16 31208 31192 121.83 80.625 0.00359308 0.00820183 2024-09-13T22:45:35.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:35.897 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:35.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:36.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:36.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:36.122 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:36.135 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[29889]: cluster 2024-09-13T22:45:34.864003+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-13T22:45:36.135 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[29889]: audit 2024-09-13T22:45:35.173910+0000 mgr.y (mgr.14613) 296 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:36.136 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:35 smithi178 bash[22840]: cluster 2024-09-13T22:45:34.864003+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-13T22:45:36.136 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:35 smithi178 bash[22840]: audit 2024-09-13T22:45:35.173910+0000 mgr.y (mgr.14613) 296 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:36.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:36.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:35 smithi192 bash[24608]: cluster 2024-09-13T22:45:34.864003+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-13T22:45:36.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:35 smithi192 bash[24608]: audit 2024-09-13T22:45:35.173910+0000 mgr.y (mgr.14613) 296 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:36.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:36.279 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:36.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:36.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:36.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:36.537 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:36.584 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-13T22:45:30.585281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1310399003,"num_objects":20185,"num_object_clones":0,"num_object_copies":60555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20185,"num_whiteouts":0,"num_read":1398,"num_read_kb":1244,"num_write":20601,"num_write_kb":1281565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20692,"ondisk_log_size":20692,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4701052,"kb_used_data":3696380,"kb_used_omap":12,"kb_used_meta":1004531,"kb_avail":745260164,"statfs":{"total":767960285184,"available":763146407936,"internally_reserved":0,"allocated":3785093120,"data_stored":3781836945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1028640304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,3,0,0,0,2,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1309802496,"num_objects":19986,"num_object_clones":0,"num_object_copies":59958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19986,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":19986,"num_write_kb":1279104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004022"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901412+0000","last_change":"2024-09-13T22:45:02.514835+0000","last_active":"2024-09-13T22:45:17.901412+0000","last_peered":"2024-09-13T22:45:17.901412+0000","last_clean":"2024-09-13T22:45:17.901412+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:45:17.901412+0000","last_undegraded":"2024-09-13T22:45:17.901412+0000","last_fullsized":"2024-09-13T22:45:17.901412+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.514742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:10.408175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608729+0000","last_change":"2024-09-13T22:43:39.277776+0000","last_active":"2024-09-13T22:45:18.608729+0000","last_peered":"2024-09-13T22:45:18.608729+0000","last_clean":"2024-09-13T22:45:18.608729+0000","last_became_active":"2024-09-13T22:37:40.743787+0000","last_became_peered":"2024-09-13T22:37:40.743787+0000","last_unstale":"2024-09-13T22:45:18.608729+0000","last_undegraded":"2024-09-13T22:45:18.608729+0000","last_fullsized":"2024-09-13T22:45:18.608729+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:39.277676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:43:39.277676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":271,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901833+0000","last_change":"2024-09-13T22:43:51.778660+0000","last_active":"2024-09-13T22:45:17.901833+0000","last_peered":"2024-09-13T22:45:17.901833+0000","last_clean":"2024-09-13T22:45:17.901833+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:45:17.901833+0000","last_undegraded":"2024-09-13T22:45:17.901833+0000","last_fullsized":"2024-09-13T22:45:17.901833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:43:51.778550+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.778550+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.478018+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904259+0000","last_change":"2024-09-13T22:44:38.049786+0000","last_active":"2024-09-13T22:45:17.904259+0000","last_peered":"2024-09-13T22:45:17.904259+0000","last_clean":"2024-09-13T22:45:17.904259+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:45:17.904259+0000","last_undegraded":"2024-09-13T22:45:17.904259+0000","last_fullsized":"2024-09-13T22:45:17.904259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:38.049680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:44:38.049680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:50.994770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901918+0000","last_change":"2024-09-13T22:45:04.482417+0000","last_active":"2024-09-13T22:45:17.901918+0000","last_peered":"2024-09-13T22:45:17.901918+0000","last_clean":"2024-09-13T22:45:17.901918+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:45:17.901918+0000","last_undegraded":"2024-09-13T22:45:17.901918+0000","last_fullsized":"2024-09-13T22:45:17.901918+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:04.482214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:04.482214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:05.376050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":275,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608530+0000","last_change":"2024-09-13T22:44:01.380285+0000","last_active":"2024-09-13T22:45:18.608530+0000","last_peered":"2024-09-13T22:45:18.608530+0000","last_clean":"2024-09-13T22:45:18.608530+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:45:18.608530+0000","last_undegraded":"2024-09-13T22:45:18.608530+0000","last_fullsized":"2024-09-13T22:45:18.608530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:44:01.380169+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:44:01.380169+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":239,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442861+0000","last_change":"2024-09-13T22:42:59.307163+0000","last_active":"2024-09-13T22:45:18.442861+0000","last_peered":"2024-09-13T22:45:18.442861+0000","last_clean":"2024-09-13T22:45:18.442861+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:45:18.442861+0000","last_undegraded":"2024-09-13T22:45:18.442861+0000","last_fullsized":"2024-09-13T22:45:18.442861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.307067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.665139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.905012+0000","last_change":"2024-09-13T22:44:02.286413+0000","last_active":"2024-09-13T22:45:17.905012+0000","last_peered":"2024-09-13T22:45:17.905012+0000","last_clean":"2024-09-13T22:45:17.905012+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:45:17.905012+0000","last_undegraded":"2024-09-13T22:45:17.905012+0000","last_fullsized":"2024-09-13T22:45:17.905012+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:02.286333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:53.894021+0000","last_clean_scrub_stamp":"2024-09-13T22:44:02.286333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:28.350370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.612946+0000","last_change":"2024-09-13T22:45:01.826287+0000","last_active":"2024-09-13T22:45:18.612946+0000","last_peered":"2024-09-13T22:45:18.612946+0000","last_clean":"2024-09-13T22:45:18.612946+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:45:18.612946+0000","last_undegraded":"2024-09-13T22:45:18.612946+0000","last_fullsized":"2024-09-13T22:45:18.612946+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:01.826196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.826196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:17.681243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":282,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484761+0000","last_change":"2024-09-13T22:44:03.085096+0000","last_active":"2024-09-13T22:45:18.484761+0000","last_peered":"2024-09-13T22:45:18.484761+0000","last_clean":"2024-09-13T22:45:18.484761+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:45:18.484761+0000","last_undegraded":"2024-09-13T22:45:18.484761+0000","last_fullsized":"2024-09-13T22:45:18.484761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:44:03.084964+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:44:03.084964+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:28.100476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":238,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609907+0000","last_change":"2024-09-13T22:44:58.799450+0000","last_active":"2024-09-13T22:45:18.609907+0000","last_peered":"2024-09-13T22:45:18.609907+0000","last_clean":"2024-09-13T22:45:18.609907+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:45:18.609907+0000","last_undegraded":"2024-09-13T22:45:18.609907+0000","last_fullsized":"2024-09-13T22:45:18.609907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:58.799239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.799239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:12.622034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":242,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484416+0000","last_change":"2024-09-13T22:45:01.888912+0000","last_active":"2024-09-13T22:45:18.484416+0000","last_peered":"2024-09-13T22:45:18.484416+0000","last_clean":"2024-09-13T22:45:18.484416+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:45:18.484416+0000","last_undegraded":"2024-09-13T22:45:18.484416+0000","last_fullsized":"2024-09-13T22:45:18.484416+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:30.066021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.612891+0000","last_change":"2024-09-13T22:44:41.802443+0000","last_active":"2024-09-13T22:45:18.612891+0000","last_peered":"2024-09-13T22:45:18.612891+0000","last_clean":"2024-09-13T22:45:18.612891+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:45:18.612891+0000","last_undegraded":"2024-09-13T22:45:18.612891+0000","last_fullsized":"2024-09-13T22:45:18.612891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:41.802331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:41.802331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:01.414960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":273,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.485236+0000","last_change":"2024-09-13T22:44:02.041895+0000","last_active":"2024-09-13T22:45:18.485236+0000","last_peered":"2024-09-13T22:45:18.485236+0000","last_clean":"2024-09-13T22:45:18.485236+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:45:18.485236+0000","last_undegraded":"2024-09-13T22:45:18.485236+0000","last_fullsized":"2024-09-13T22:45:18.485236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:44:02.041773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:02.041773+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:20.560116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484558+0000","last_change":"2024-09-13T22:44:42.914244+0000","last_active":"2024-09-13T22:45:18.484558+0000","last_peered":"2024-09-13T22:45:18.484558+0000","last_clean":"2024-09-13T22:45:18.484558+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:45:18.484558+0000","last_undegraded":"2024-09-13T22:45:18.484558+0000","last_fullsized":"2024-09-13T22:45:18.484558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:42.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:42.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:07.329166+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.607633+0000","last_change":"2024-09-13T22:43:53.335673+0000","last_active":"2024-09-13T22:45:18.607633+0000","last_peered":"2024-09-13T22:45:18.607633+0000","last_clean":"2024-09-13T22:45:18.607633+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:45:18.607633+0000","last_undegraded":"2024-09-13T22:45:18.607633+0000","last_fullsized":"2024-09-13T22:45:18.607633+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.335567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.335567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"180'1267","reported_seq":1287,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.816883+0000","last_change":"2024-09-13T22:45:10.877405+0000","last_active":"2024-09-13T22:45:28.816883+0000","last_peered":"2024-09-13T22:45:28.816883+0000","last_clean":"2024-09-13T22:45:28.816883+0000","last_became_active":"2024-09-13T22:45:10.877045+0000","last_became_peered":"2024-09-13T22:45:10.877045+0000","last_unstale":"2024-09-13T22:45:28.816883+0000","last_undegraded":"2024-09-13T22:45:28.816883+0000","last_fullsized":"2024-09-13T22:45:28.816883+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:27.160558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83034112,"num_objects":1267,"num_object_clones":0,"num_object_copies":3801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1267,"num_write_kb":81088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"114'28","reported_seq":299,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610070+0000","last_change":"2024-09-13T22:44:02.881565+0000","last_active":"2024-09-13T22:45:18.610070+0000","last_peered":"2024-09-13T22:45:18.610070+0000","last_clean":"2024-09-13T22:45:18.610070+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:45:18.610070+0000","last_undegraded":"2024-09-13T22:45:18.610070+0000","last_fullsized":"2024-09-13T22:45:18.610070+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'28","last_scrub_stamp":"2024-09-13T22:44:02.881481+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:44:02.881481+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.612145+0000","last_change":"2024-09-13T22:43:49.663119+0000","last_active":"2024-09-13T22:45:18.612145+0000","last_peered":"2024-09-13T22:45:18.612145+0000","last_clean":"2024-09-13T22:45:18.612145+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:45:18.612145+0000","last_undegraded":"2024-09-13T22:45:18.612145+0000","last_fullsized":"2024-09-13T22:45:18.612145+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:49.663031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:49.663031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.625060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443983+0000","last_change":"2024-09-13T22:44:43.533920+0000","last_active":"2024-09-13T22:45:18.443983+0000","last_peered":"2024-09-13T22:45:18.443983+0000","last_clean":"2024-09-13T22:45:18.443983+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:45:18.443983+0000","last_undegraded":"2024-09-13T22:45:18.443983+0000","last_fullsized":"2024-09-13T22:45:18.443983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:43.533803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:43.533803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:11.881291+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":251,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444512+0000","last_change":"2024-09-13T22:45:02.748405+0000","last_active":"2024-09-13T22:45:18.444512+0000","last_peered":"2024-09-13T22:45:18.444512+0000","last_clean":"2024-09-13T22:45:18.444512+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:45:18.444512+0000","last_undegraded":"2024-09-13T22:45:18.444512+0000","last_fullsized":"2024-09-13T22:45:18.444512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.748330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.748330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:29.239492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"180'1255","reported_seq":1275,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:29.590128+0000","last_change":"2024-09-13T22:45:10.877879+0000","last_active":"2024-09-13T22:45:29.590128+0000","last_peered":"2024-09-13T22:45:29.590128+0000","last_clean":"2024-09-13T22:45:29.590128+0000","last_became_active":"2024-09-13T22:45:10.877303+0000","last_became_peered":"2024-09-13T22:45:10.877303+0000","last_unstale":"2024-09-13T22:45:29.590128+0000","last_undegraded":"2024-09-13T22:45:29.590128+0000","last_fullsized":"2024-09-13T22:45:29.590128+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:17.487998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82247680,"num_objects":1255,"num_object_clones":0,"num_object_copies":3765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1255,"num_write_kb":80320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"110'13","reported_seq":271,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901710+0000","last_change":"2024-09-13T22:43:57.825534+0000","last_active":"2024-09-13T22:45:17.901710+0000","last_peered":"2024-09-13T22:45:17.901710+0000","last_clean":"2024-09-13T22:45:17.901710+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:45:17.901710+0000","last_undegraded":"2024-09-13T22:45:17.901710+0000","last_fullsized":"2024-09-13T22:45:17.901710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-09-13T22:43:57.825403+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.825403+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.824443+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443810+0000","last_change":"2024-09-13T22:44:56.623755+0000","last_active":"2024-09-13T22:45:18.443810+0000","last_peered":"2024-09-13T22:45:18.443810+0000","last_clean":"2024-09-13T22:45:18.443810+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:45:18.443810+0000","last_undegraded":"2024-09-13T22:45:18.443810+0000","last_fullsized":"2024-09-13T22:45:18.443810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:56.623456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:56.623456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:08.138673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":351,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.159631+0000","last_change":"2024-09-13T22:44:56.805492+0000","last_active":"2024-09-13T22:45:21.159631+0000","last_peered":"2024-09-13T22:45:21.159631+0000","last_clean":"2024-09-13T22:45:21.159631+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:45:21.159631+0000","last_undegraded":"2024-09-13T22:45:21.159631+0000","last_fullsized":"2024-09-13T22:45:21.159631+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:44:56.805394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:56.805394+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:06.273134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.905079+0000","last_change":"2024-09-13T22:43:40.080415+0000","last_active":"2024-09-13T22:45:17.905079+0000","last_peered":"2024-09-13T22:45:17.905079+0000","last_clean":"2024-09-13T22:45:17.905079+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:45:17.905079+0000","last_undegraded":"2024-09-13T22:45:17.905079+0000","last_fullsized":"2024-09-13T22:45:17.905079+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.080319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.080319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.979259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"180'1236","reported_seq":1256,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.811232+0000","last_change":"2024-09-13T22:45:10.877854+0000","last_active":"2024-09-13T22:45:28.811232+0000","last_peered":"2024-09-13T22:45:28.811232+0000","last_clean":"2024-09-13T22:45:28.811232+0000","last_became_active":"2024-09-13T22:45:10.877601+0000","last_became_peered":"2024-09-13T22:45:10.877601+0000","last_unstale":"2024-09-13T22:45:28.811232+0000","last_undegraded":"2024-09-13T22:45:28.811232+0000","last_fullsized":"2024-09-13T22:45:28.811232+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:13.913598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81002496,"num_objects":1236,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":79104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":276,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901119+0000","last_change":"2024-09-13T22:44:37.351243+0000","last_active":"2024-09-13T22:45:17.901119+0000","last_peered":"2024-09-13T22:45:17.901119+0000","last_clean":"2024-09-13T22:45:17.901119+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:45:17.901119+0000","last_undegraded":"2024-09-13T22:45:17.901119+0000","last_fullsized":"2024-09-13T22:45:17.901119+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:44:37.351117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:37.351117+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:52.942768+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443357+0000","last_change":"2024-09-13T22:43:58.133084+0000","last_active":"2024-09-13T22:45:18.443357+0000","last_peered":"2024-09-13T22:45:18.443357+0000","last_clean":"2024-09-13T22:45:18.443357+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:45:18.443357+0000","last_undegraded":"2024-09-13T22:45:18.443357+0000","last_fullsized":"2024-09-13T22:45:18.443357+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:58.133006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:58.133006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.665173+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":344,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.159890+0000","last_change":"2024-09-13T22:42:50.045223+0000","last_active":"2024-09-13T22:45:21.159890+0000","last_peered":"2024-09-13T22:45:21.159890+0000","last_clean":"2024-09-13T22:45:21.159890+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:45:21.159890+0000","last_undegraded":"2024-09-13T22:45:21.159890+0000","last_fullsized":"2024-09-13T22:45:21.159890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:42:50.045100+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:42:50.045100+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:29.893491+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":234,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484949+0000","last_change":"2024-09-13T22:43:53.003544+0000","last_active":"2024-09-13T22:45:18.484949+0000","last_peered":"2024-09-13T22:45:18.484949+0000","last_clean":"2024-09-13T22:45:18.484949+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:45:18.484949+0000","last_undegraded":"2024-09-13T22:45:18.484949+0000","last_fullsized":"2024-09-13T22:45:18.484949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.003468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:22.190124+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"180'1193","reported_seq":1213,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.554586+0000","last_change":"2024-09-13T22:45:10.874698+0000","last_active":"2024-09-13T22:45:28.554586+0000","last_peered":"2024-09-13T22:45:28.554586+0000","last_clean":"2024-09-13T22:45:28.554586+0000","last_became_active":"2024-09-13T22:45:10.874312+0000","last_became_peered":"2024-09-13T22:45:10.874312+0000","last_unstale":"2024-09-13T22:45:28.554586+0000","last_undegraded":"2024-09-13T22:45:28.554586+0000","last_fullsized":"2024-09-13T22:45:28.554586+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1193,"log_dups_size":0,"ondisk_log_size":1193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:32.487521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78184448,"num_objects":1193,"num_object_clones":0,"num_object_copies":3579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1193,"num_write_kb":76352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":274,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904809+0000","last_change":"2024-09-13T22:44:42.028974+0000","last_active":"2024-09-13T22:45:17.904809+0000","last_peered":"2024-09-13T22:45:17.904809+0000","last_clean":"2024-09-13T22:45:17.904809+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:45:17.904809+0000","last_undegraded":"2024-09-13T22:45:17.904809+0000","last_fullsized":"2024-09-13T22:45:17.904809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:44:42.028847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:42.028847+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:56.732701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610446+0000","last_change":"2024-09-13T22:44:51.782458+0000","last_active":"2024-09-13T22:45:18.610446+0000","last_peered":"2024-09-13T22:45:18.610446+0000","last_clean":"2024-09-13T22:45:18.610446+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:45:18.610446+0000","last_undegraded":"2024-09-13T22:45:18.610446+0000","last_fullsized":"2024-09-13T22:45:18.610446+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:51.782326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:51.782326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:58.977149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610835+0000","last_change":"2024-09-13T22:44:17.652685+0000","last_active":"2024-09-13T22:45:18.610835+0000","last_peered":"2024-09-13T22:45:18.610835+0000","last_clean":"2024-09-13T22:45:18.610835+0000","last_became_active":"2024-09-13T22:44:17.652323+0000","last_became_peered":"2024-09-13T22:44:17.652323+0000","last_unstale":"2024-09-13T22:45:18.610835+0000","last_undegraded":"2024-09-13T22:45:18.610835+0000","last_fullsized":"2024-09-13T22:45:18.610835+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.973397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.973397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.900879+0000","last_change":"2024-09-13T22:45:03.478116+0000","last_active":"2024-09-13T22:45:17.900879+0000","last_peered":"2024-09-13T22:45:17.900879+0000","last_clean":"2024-09-13T22:45:17.900879+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:45:17.900879+0000","last_undegraded":"2024-09-13T22:45:17.900879+0000","last_fullsized":"2024-09-13T22:45:17.900879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.477932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.477932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:17.287804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"180'1272","reported_seq":1292,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.581377+0000","last_change":"2024-09-13T22:45:11.075836+0000","last_active":"2024-09-13T22:45:28.581377+0000","last_peered":"2024-09-13T22:45:28.581377+0000","last_clean":"2024-09-13T22:45:28.581377+0000","last_became_active":"2024-09-13T22:45:11.075425+0000","last_became_peered":"2024-09-13T22:45:11.075425+0000","last_unstale":"2024-09-13T22:45:28.581377+0000","last_undegraded":"2024-09-13T22:45:28.581377+0000","last_fullsized":"2024-09-13T22:45:28.581377+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1272,"log_dups_size":0,"ondisk_log_size":1272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:14.196855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83361792,"num_objects":1272,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1272,"num_write_kb":81408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":269,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608707+0000","last_change":"2024-09-13T22:44:37.772426+0000","last_active":"2024-09-13T22:45:18.608707+0000","last_peered":"2024-09-13T22:45:18.608707+0000","last_clean":"2024-09-13T22:45:18.608707+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:45:18.608707+0000","last_undegraded":"2024-09-13T22:45:18.608707+0000","last_fullsized":"2024-09-13T22:45:18.608707+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:44:37.772333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:37.772333+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:05.969653+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.613368+0000","last_change":"2024-09-13T22:44:46.049984+0000","last_active":"2024-09-13T22:45:18.613368+0000","last_peered":"2024-09-13T22:45:18.613368+0000","last_clean":"2024-09-13T22:45:18.613368+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:45:18.613368+0000","last_undegraded":"2024-09-13T22:45:18.613368+0000","last_fullsized":"2024-09-13T22:45:18.613368+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:46.047614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:46.047614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:04.101230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"83'1","reported_seq":259,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904689+0000","last_change":"2024-09-13T22:44:58.003380+0000","last_active":"2024-09-13T22:45:17.904689+0000","last_peered":"2024-09-13T22:45:17.904689+0000","last_clean":"2024-09-13T22:45:17.904689+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:45:17.904689+0000","last_undegraded":"2024-09-13T22:45:17.904689+0000","last_fullsized":"2024-09-13T22:45:17.904689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'1","last_scrub_stamp":"2024-09-13T22:44:58.003182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.003182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:59.766560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":343,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.160473+0000","last_change":"2024-09-13T22:43:40.553900+0000","last_active":"2024-09-13T22:45:21.160473+0000","last_peered":"2024-09-13T22:45:21.160473+0000","last_clean":"2024-09-13T22:45:21.160473+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:45:21.160473+0000","last_undegraded":"2024-09-13T22:45:21.160473+0000","last_fullsized":"2024-09-13T22:45:21.160473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:40.553646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.553646+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:27.468012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":234,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609804+0000","last_change":"2024-09-13T22:44:03.606280+0000","last_active":"2024-09-13T22:45:18.609804+0000","last_peered":"2024-09-13T22:45:18.609804+0000","last_clean":"2024-09-13T22:45:18.609804+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:45:18.609804+0000","last_undegraded":"2024-09-13T22:45:18.609804+0000","last_fullsized":"2024-09-13T22:45:18.609804+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:03.606176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:03.606176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:22.574079+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"180'1319","reported_seq":1339,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:29.590226+0000","last_change":"2024-09-13T22:45:11.071616+0000","last_active":"2024-09-13T22:45:29.590226+0000","last_peered":"2024-09-13T22:45:29.590226+0000","last_clean":"2024-09-13T22:45:29.590226+0000","last_became_active":"2024-09-13T22:45:11.071318+0000","last_became_peered":"2024-09-13T22:45:11.071318+0000","last_unstale":"2024-09-13T22:45:29.590226+0000","last_undegraded":"2024-09-13T22:45:29.590226+0000","last_fullsized":"2024-09-13T22:45:29.590226+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:34.473688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":84416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":283,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.902236+0000","last_change":"2024-09-13T22:45:01.550550+0000","last_active":"2024-09-13T22:45:17.902236+0000","last_peered":"2024-09-13T22:45:17.902236+0000","last_clean":"2024-09-13T22:45:17.902236+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:45:17.902236+0000","last_undegraded":"2024-09-13T22:45:17.902236+0000","last_fullsized":"2024-09-13T22:45:17.902236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:45:01.550400+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.550400+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:10.068936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":256,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.445298+0000","last_change":"2024-09-13T22:44:49.488944+0000","last_active":"2024-09-13T22:45:18.445298+0000","last_peered":"2024-09-13T22:45:18.445298+0000","last_clean":"2024-09-13T22:45:18.445298+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:45:18.445298+0000","last_undegraded":"2024-09-13T22:45:18.445298+0000","last_fullsized":"2024-09-13T22:45:18.445298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:49.488854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:49.488854+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:07.437214+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":701,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.709718+0000","last_change":"2024-09-13T22:43:38.560388+0000","last_active":"2024-09-13T22:45:28.709718+0000","last_peered":"2024-09-13T22:45:28.709718+0000","last_clean":"2024-09-13T22:45:28.709718+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:45:28.709718+0000","last_undegraded":"2024-09-13T22:45:28.709718+0000","last_fullsized":"2024-09-13T22:45:28.709718+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:43:38.560255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:38.560255+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904365+0000","last_change":"2024-09-13T22:44:00.305703+0000","last_active":"2024-09-13T22:45:17.904365+0000","last_peered":"2024-09-13T22:45:17.904365+0000","last_clean":"2024-09-13T22:45:17.904365+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:45:17.904365+0000","last_undegraded":"2024-09-13T22:45:17.904365+0000","last_fullsized":"2024-09-13T22:45:17.904365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:00.305494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:00.305494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.979311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":235,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904534+0000","last_change":"2024-09-13T22:42:54.080648+0000","last_active":"2024-09-13T22:45:17.904534+0000","last_peered":"2024-09-13T22:45:17.904534+0000","last_clean":"2024-09-13T22:45:17.904534+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:45:17.904534+0000","last_undegraded":"2024-09-13T22:45:17.904534+0000","last_fullsized":"2024-09-13T22:45:17.904534+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:54.080532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.080532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.979241+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"180'1250","reported_seq":1270,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:29.763504+0000","last_change":"2024-09-13T22:45:10.879441+0000","last_active":"2024-09-13T22:45:29.763504+0000","last_peered":"2024-09-13T22:45:29.763504+0000","last_clean":"2024-09-13T22:45:29.763504+0000","last_became_active":"2024-09-13T22:45:10.879109+0000","last_became_peered":"2024-09-13T22:45:10.879109+0000","last_unstale":"2024-09-13T22:45:29.763504+0000","last_undegraded":"2024-09-13T22:45:29.763504+0000","last_fullsized":"2024-09-13T22:45:29.763504+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:11.754112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":395,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.159530+0000","last_change":"2024-09-13T22:45:00.620616+0000","last_active":"2024-09-13T22:45:21.159530+0000","last_peered":"2024-09-13T22:45:21.159530+0000","last_clean":"2024-09-13T22:45:21.159530+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:45:21.159530+0000","last_undegraded":"2024-09-13T22:45:21.159530+0000","last_fullsized":"2024-09-13T22:45:21.159530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:45:00.620468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:00.620468+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:20.457082+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.613194+0000","last_change":"2024-09-13T22:43:37.803726+0000","last_active":"2024-09-13T22:45:18.613194+0000","last_peered":"2024-09-13T22:45:18.613194+0000","last_clean":"2024-09-13T22:45:18.613194+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:45:18.613194+0000","last_undegraded":"2024-09-13T22:45:18.613194+0000","last_fullsized":"2024-09-13T22:45:18.613194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:37.803588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:37.803588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624941+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609611+0000","last_change":"2024-09-13T22:44:51.058720+0000","last_active":"2024-09-13T22:45:18.609611+0000","last_peered":"2024-09-13T22:45:18.609611+0000","last_clean":"2024-09-13T22:45:18.609611+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:45:18.609611+0000","last_undegraded":"2024-09-13T22:45:18.609611+0000","last_fullsized":"2024-09-13T22:45:18.609611+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:51.058582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:51.058582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:03.255766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442712+0000","last_change":"2024-09-13T22:44:54.588476+0000","last_active":"2024-09-13T22:45:18.442712+0000","last_peered":"2024-09-13T22:45:18.442712+0000","last_clean":"2024-09-13T22:45:18.442712+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:45:18.442712+0000","last_undegraded":"2024-09-13T22:45:18.442712+0000","last_fullsized":"2024-09-13T22:45:18.442712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:54.588262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:54.588262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:14.802700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"180'1367","reported_seq":1386,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:29.597954+0000","last_change":"2024-09-13T22:45:10.878616+0000","last_active":"2024-09-13T22:45:29.597954+0000","last_peered":"2024-09-13T22:45:29.597954+0000","last_clean":"2024-09-13T22:45:29.597954+0000","last_became_active":"2024-09-13T22:45:10.877979+0000","last_became_peered":"2024-09-13T22:45:10.877979+0000","last_unstale":"2024-09-13T22:45:29.597954+0000","last_undegraded":"2024-09-13T22:45:29.597954+0000","last_fullsized":"2024-09-13T22:45:29.597954+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1367,"log_dups_size":0,"ondisk_log_size":1367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:38.861678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":87488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":374,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.160209+0000","last_change":"2024-09-13T22:44:58.932861+0000","last_active":"2024-09-13T22:45:21.160209+0000","last_peered":"2024-09-13T22:45:21.160209+0000","last_clean":"2024-09-13T22:45:21.160209+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:45:21.160209+0000","last_undegraded":"2024-09-13T22:45:21.160209+0000","last_fullsized":"2024-09-13T22:45:21.160209+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:44:58.932607+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.932607+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:28.776680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444239+0000","last_change":"2024-09-13T22:43:36.147077+0000","last_active":"2024-09-13T22:45:18.444239+0000","last_peered":"2024-09-13T22:45:18.444239+0000","last_clean":"2024-09-13T22:45:18.444239+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:45:18.444239+0000","last_undegraded":"2024-09-13T22:45:18.444239+0000","last_fullsized":"2024-09-13T22:45:18.444239+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:43:36.146966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:36.146966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.665209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":260,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443529+0000","last_change":"2024-09-13T22:44:47.484690+0000","last_active":"2024-09-13T22:45:18.443529+0000","last_peered":"2024-09-13T22:45:18.443529+0000","last_clean":"2024-09-13T22:45:18.443529+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:45:18.443529+0000","last_undegraded":"2024-09-13T22:45:18.443529+0000","last_fullsized":"2024-09-13T22:45:18.443529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:44:47.484525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:47.484525+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:58.603242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":352,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.161401+0000","last_change":"2024-09-13T22:42:53.286723+0000","last_active":"2024-09-13T22:45:21.161401+0000","last_peered":"2024-09-13T22:45:21.161401+0000","last_clean":"2024-09-13T22:45:21.161401+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:45:21.161401+0000","last_undegraded":"2024-09-13T22:45:21.161401+0000","last_fullsized":"2024-09-13T22:45:21.161401+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:42:53.286646+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:42:53.286646+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:30.661085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.607768+0000","last_change":"2024-09-13T22:44:55.859959+0000","last_active":"2024-09-13T22:45:18.607768+0000","last_peered":"2024-09-13T22:45:18.607768+0000","last_clean":"2024-09-13T22:45:18.607768+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:45:18.607768+0000","last_undegraded":"2024-09-13T22:45:18.607768+0000","last_fullsized":"2024-09-13T22:45:18.607768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:55.859847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.859847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:03.217888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"180'1248","reported_seq":1268,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.581199+0000","last_change":"2024-09-13T22:45:11.072418+0000","last_active":"2024-09-13T22:45:28.581199+0000","last_peered":"2024-09-13T22:45:28.581199+0000","last_clean":"2024-09-13T22:45:28.581199+0000","last_became_active":"2024-09-13T22:45:11.071983+0000","last_became_peered":"2024-09-13T22:45:11.071983+0000","last_unstale":"2024-09-13T22:45:28.581199+0000","last_undegraded":"2024-09-13T22:45:28.581199+0000","last_fullsized":"2024-09-13T22:45:28.581199+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1248,"log_dups_size":0,"ondisk_log_size":1248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:29.204118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1248,"num_write_kb":79872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":287,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904981+0000","last_change":"2024-09-13T22:43:49.155754+0000","last_active":"2024-09-13T22:45:17.904981+0000","last_peered":"2024-09-13T22:45:17.904981+0000","last_clean":"2024-09-13T22:45:17.904981+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:45:17.904981+0000","last_undegraded":"2024-09-13T22:45:17.904981+0000","last_fullsized":"2024-09-13T22:45:17.904981+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:43:49.155639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:49.155639+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.833978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.485057+0000","last_change":"2024-09-13T22:44:01.009857+0000","last_active":"2024-09-13T22:45:18.485057+0000","last_peered":"2024-09-13T22:45:18.485057+0000","last_clean":"2024-09-13T22:45:18.485057+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:45:18.485057+0000","last_undegraded":"2024-09-13T22:45:18.485057+0000","last_fullsized":"2024-09-13T22:45:18.485057+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:01.009752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:44:01.009752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.081929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608462+0000","last_change":"2024-09-13T22:43:48.316803+0000","last_active":"2024-09-13T22:45:18.608462+0000","last_peered":"2024-09-13T22:45:18.608462+0000","last_clean":"2024-09-13T22:45:18.608462+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:45:18.608462+0000","last_undegraded":"2024-09-13T22:45:18.608462+0000","last_fullsized":"2024-09-13T22:45:18.608462+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:48.316570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:43:48.316570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":235,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901649+0000","last_change":"2024-09-13T22:43:37.549939+0000","last_active":"2024-09-13T22:45:17.901649+0000","last_peered":"2024-09-13T22:45:17.901649+0000","last_clean":"2024-09-13T22:45:17.901649+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:45:17.901649+0000","last_undegraded":"2024-09-13T22:45:17.901649+0000","last_fullsized":"2024-09-13T22:45:17.901649+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:37.549864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:37.549864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.511948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"180'1253","reported_seq":1273,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:29.599690+0000","last_change":"2024-09-13T22:45:10.881296+0000","last_active":"2024-09-13T22:45:29.599690+0000","last_peered":"2024-09-13T22:45:29.599690+0000","last_clean":"2024-09-13T22:45:29.599690+0000","last_became_active":"2024-09-13T22:45:10.880563+0000","last_became_peered":"2024-09-13T22:45:10.880563+0000","last_unstale":"2024-09-13T22:45:29.599690+0000","last_undegraded":"2024-09-13T22:45:29.599690+0000","last_fullsized":"2024-09-13T22:45:29.599690+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1253,"log_dups_size":0,"ondisk_log_size":1253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:11.994028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82116608,"num_objects":1253,"num_object_clones":0,"num_object_copies":3759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1253,"num_write_kb":80192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":291,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442450+0000","last_change":"2024-09-13T22:44:55.600907+0000","last_active":"2024-09-13T22:45:18.442450+0000","last_peered":"2024-09-13T22:45:18.442450+0000","last_clean":"2024-09-13T22:45:18.442450+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:45:18.442450+0000","last_undegraded":"2024-09-13T22:45:18.442450+0000","last_fullsized":"2024-09-13T22:45:18.442450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-09-13T22:44:55.600799+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.600799+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:22.514411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442237+0000","last_change":"2024-09-13T22:43:57.436549+0000","last_active":"2024-09-13T22:45:18.442237+0000","last_peered":"2024-09-13T22:45:18.442237+0000","last_clean":"2024-09-13T22:45:18.442237+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:45:18.442237+0000","last_undegraded":"2024-09-13T22:45:18.442237+0000","last_fullsized":"2024-09-13T22:45:18.442237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:43:57.436436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.436436+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.598122+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.445440+0000","last_change":"2024-09-13T22:44:36.735848+0000","last_active":"2024-09-13T22:45:18.445440+0000","last_peered":"2024-09-13T22:45:18.445440+0000","last_clean":"2024-09-13T22:45:18.445440+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:45:18.445440+0000","last_undegraded":"2024-09-13T22:45:18.445440+0000","last_fullsized":"2024-09-13T22:45:18.445440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:36.735593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:44:36.735593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:54.453077+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611055+0000","last_change":"2024-09-13T22:42:45.743372+0000","last_active":"2024-09-13T22:45:18.611055+0000","last_peered":"2024-09-13T22:45:18.611055+0000","last_clean":"2024-09-13T22:45:18.611055+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:45:18.611055+0000","last_undegraded":"2024-09-13T22:45:18.611055+0000","last_fullsized":"2024-09-13T22:45:18.611055+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:45.743269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:45.743269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"180'1207","reported_seq":1227,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.832943+0000","last_change":"2024-09-13T22:45:10.878791+0000","last_active":"2024-09-13T22:45:28.832943+0000","last_peered":"2024-09-13T22:45:28.832943+0000","last_clean":"2024-09-13T22:45:28.832943+0000","last_became_active":"2024-09-13T22:45:10.878487+0000","last_became_peered":"2024-09-13T22:45:10.878487+0000","last_unstale":"2024-09-13T22:45:28.832943+0000","last_undegraded":"2024-09-13T22:45:28.832943+0000","last_fullsized":"2024-09-13T22:45:28.832943+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79101952,"num_objects":1207,"num_object_clones":0,"num_object_copies":3621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1207,"num_write_kb":77248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":272,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.607789+0000","last_change":"2024-09-13T22:43:51.350004+0000","last_active":"2024-09-13T22:45:18.607789+0000","last_peered":"2024-09-13T22:45:18.607789+0000","last_clean":"2024-09-13T22:45:18.607789+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:45:18.607789+0000","last_undegraded":"2024-09-13T22:45:18.607789+0000","last_fullsized":"2024-09-13T22:45:18.607789+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:43:51.349861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.349861+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609178+0000","last_change":"2024-09-13T22:44:52.527561+0000","last_active":"2024-09-13T22:45:18.609178+0000","last_peered":"2024-09-13T22:45:18.609178+0000","last_clean":"2024-09-13T22:45:18.609178+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:45:18.609178+0000","last_undegraded":"2024-09-13T22:45:18.609178+0000","last_fullsized":"2024-09-13T22:45:18.609178+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:44:01.750360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:22.574058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.485104+0000","last_change":"2024-09-13T22:44:56.942285+0000","last_active":"2024-09-13T22:45:18.485104+0000","last_peered":"2024-09-13T22:45:18.485104+0000","last_clean":"2024-09-13T22:45:18.485104+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:45:18.485104+0000","last_undegraded":"2024-09-13T22:45:18.485104+0000","last_fullsized":"2024-09-13T22:45:18.485104+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:56.942208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:44:56.942208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:57.344451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443725+0000","last_change":"2024-09-13T22:44:42.511563+0000","last_active":"2024-09-13T22:45:18.443725+0000","last_peered":"2024-09-13T22:45:18.443725+0000","last_clean":"2024-09-13T22:45:18.443725+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:45:18.443725+0000","last_undegraded":"2024-09-13T22:45:18.443725+0000","last_fullsized":"2024-09-13T22:45:18.443725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:42.511177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:44:42.511177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:07.146172+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"180'1160","reported_seq":1180,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.573596+0000","last_change":"2024-09-13T22:45:10.876077+0000","last_active":"2024-09-13T22:45:28.573596+0000","last_peered":"2024-09-13T22:45:28.573596+0000","last_clean":"2024-09-13T22:45:28.573596+0000","last_became_active":"2024-09-13T22:45:10.875657+0000","last_became_peered":"2024-09-13T22:45:10.875657+0000","last_unstale":"2024-09-13T22:45:28.573596+0000","last_undegraded":"2024-09-13T22:45:28.573596+0000","last_fullsized":"2024-09-13T22:45:28.573596+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1160,"log_dups_size":0,"ondisk_log_size":1160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.835256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1160,"num_write_kb":74240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":266,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901028+0000","last_change":"2024-09-13T22:44:55.563811+0000","last_active":"2024-09-13T22:45:17.901028+0000","last_peered":"2024-09-13T22:45:17.901028+0000","last_clean":"2024-09-13T22:45:17.901028+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:45:17.901028+0000","last_undegraded":"2024-09-13T22:45:17.901028+0000","last_fullsized":"2024-09-13T22:45:17.901028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:44:55.563593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.563593+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:10.172554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484706+0000","last_change":"2024-09-13T22:44:39.038942+0000","last_active":"2024-09-13T22:45:18.484706+0000","last_peered":"2024-09-13T22:45:18.484706+0000","last_clean":"2024-09-13T22:45:18.484706+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:45:18.484706+0000","last_undegraded":"2024-09-13T22:45:18.484706+0000","last_fullsized":"2024-09-13T22:45:18.484706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:39.038611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:39.038611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:49.648824+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":508,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610898+0000","last_change":"2024-09-13T22:44:00.697623+0000","last_active":"2024-09-13T22:45:18.610898+0000","last_peered":"2024-09-13T22:45:18.610898+0000","last_clean":"2024-09-13T22:45:18.610898+0000","last_became_active":"2024-09-13T22:37:10.741989+0000","last_became_peered":"2024-09-13T22:37:10.741989+0000","last_unstale":"2024-09-13T22:45:18.610898+0000","last_undegraded":"2024-09-13T22:45:18.610898+0000","last_fullsized":"2024-09-13T22:45:18.610898+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:44:00.697344+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:44:00.697344+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:22.932693+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611414+0000","last_change":"2024-09-13T22:44:59.821248+0000","last_active":"2024-09-13T22:45:18.611414+0000","last_peered":"2024-09-13T22:45:18.611414+0000","last_clean":"2024-09-13T22:45:18.611414+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:45:18.611414+0000","last_undegraded":"2024-09-13T22:45:18.611414+0000","last_fullsized":"2024-09-13T22:45:18.611414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:59.820971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:44:59.820971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.857639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":242,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484977+0000","last_change":"2024-09-13T22:43:57.009877+0000","last_active":"2024-09-13T22:45:18.484977+0000","last_peered":"2024-09-13T22:45:18.484977+0000","last_clean":"2024-09-13T22:45:18.484977+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:45:18.484977+0000","last_undegraded":"2024-09-13T22:45:18.484977+0000","last_fullsized":"2024-09-13T22:45:18.484977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:57.009754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.009754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.133316+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"180'1313","reported_seq":1333,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:29.603349+0000","last_change":"2024-09-13T22:45:11.071425+0000","last_active":"2024-09-13T22:45:29.603349+0000","last_peered":"2024-09-13T22:45:29.603349+0000","last_clean":"2024-09-13T22:45:29.603349+0000","last_became_active":"2024-09-13T22:45:11.071153+0000","last_became_peered":"2024-09-13T22:45:11.071153+0000","last_unstale":"2024-09-13T22:45:29.603349+0000","last_undegraded":"2024-09-13T22:45:29.603349+0000","last_fullsized":"2024-09-13T22:45:29.603349+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.432805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86048768,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1313,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":270,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611436+0000","last_change":"2024-09-13T22:44:48.862741+0000","last_active":"2024-09-13T22:45:18.611436+0000","last_peered":"2024-09-13T22:45:18.611436+0000","last_clean":"2024-09-13T22:45:18.611436+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:45:18.611436+0000","last_undegraded":"2024-09-13T22:45:18.611436+0000","last_fullsized":"2024-09-13T22:45:18.611436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:44:48.862647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:48.862647+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:16.033063+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444931+0000","last_change":"2024-09-13T22:44:44.795957+0000","last_active":"2024-09-13T22:45:18.444931+0000","last_peered":"2024-09-13T22:45:18.444931+0000","last_clean":"2024-09-13T22:45:18.444931+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:45:18.444931+0000","last_undegraded":"2024-09-13T22:45:18.444931+0000","last_fullsized":"2024-09-13T22:45:18.444931+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:44.795803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:44.795803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:03.056737+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.445092+0000","last_change":"2024-09-13T22:44:54.771739+0000","last_active":"2024-09-13T22:45:18.445092+0000","last_peered":"2024-09-13T22:45:18.445092+0000","last_clean":"2024-09-13T22:45:18.445092+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:45:18.445092+0000","last_undegraded":"2024-09-13T22:45:18.445092+0000","last_fullsized":"2024-09-13T22:45:18.445092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:54.771552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:54.771552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:19.451322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":234,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610772+0000","last_change":"2024-09-13T22:43:59.658346+0000","last_active":"2024-09-13T22:45:18.610772+0000","last_peered":"2024-09-13T22:45:18.610772+0000","last_clean":"2024-09-13T22:45:18.610772+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:45:18.610772+0000","last_undegraded":"2024-09-13T22:45:18.610772+0000","last_fullsized":"2024-09-13T22:45:18.610772+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:43:59.658172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.625100+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"180'1212","reported_seq":1232,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.799437+0000","last_change":"2024-09-13T22:45:10.879726+0000","last_active":"2024-09-13T22:45:28.799437+0000","last_peered":"2024-09-13T22:45:28.799437+0000","last_clean":"2024-09-13T22:45:28.799437+0000","last_became_active":"2024-09-13T22:45:10.879140+0000","last_became_peered":"2024-09-13T22:45:10.879140+0000","last_unstale":"2024-09-13T22:45:28.799437+0000","last_undegraded":"2024-09-13T22:45:28.799437+0000","last_fullsized":"2024-09-13T22:45:28.799437+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:37.379391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1212,"num_write_kb":77568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":265,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484828+0000","last_change":"2024-09-13T22:44:04.097947+0000","last_active":"2024-09-13T22:45:18.484828+0000","last_peered":"2024-09-13T22:45:18.484828+0000","last_clean":"2024-09-13T22:45:18.484828+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:45:18.484828+0000","last_undegraded":"2024-09-13T22:45:18.484828+0000","last_fullsized":"2024-09-13T22:45:18.484828+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:44:04.097854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:04.097854+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:20.978972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444588+0000","last_change":"2024-09-13T22:43:52.173697+0000","last_active":"2024-09-13T22:45:18.444588+0000","last_peered":"2024-09-13T22:45:18.444588+0000","last_clean":"2024-09-13T22:45:18.444588+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:45:18.444588+0000","last_undegraded":"2024-09-13T22:45:18.444588+0000","last_fullsized":"2024-09-13T22:45:18.444588+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:52.173506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:43:52.173506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.665157+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442876+0000","last_change":"2024-09-13T22:43:37.537514+0000","last_active":"2024-09-13T22:45:18.442876+0000","last_peered":"2024-09-13T22:45:18.442876+0000","last_clean":"2024-09-13T22:45:18.442876+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:45:18.442876+0000","last_undegraded":"2024-09-13T22:45:18.442876+0000","last_fullsized":"2024-09-13T22:45:18.442876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:37.536972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:37.536972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":251,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.607735+0000","last_change":"2024-09-13T22:44:44.797246+0000","last_active":"2024-09-13T22:45:18.607735+0000","last_peered":"2024-09-13T22:45:18.607735+0000","last_clean":"2024-09-13T22:45:18.607735+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:45:18.607735+0000","last_undegraded":"2024-09-13T22:45:18.607735+0000","last_fullsized":"2024-09-13T22:45:18.607735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:44.796884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:44:44.796884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:13.794031+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"180'1229","reported_seq":1249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.578656+0000","last_change":"2024-09-13T22:45:10.873543+0000","last_active":"2024-09-13T22:45:28.578656+0000","last_peered":"2024-09-13T22:45:28.578656+0000","last_clean":"2024-09-13T22:45:28.578656+0000","last_became_active":"2024-09-13T22:45:10.873170+0000","last_became_peered":"2024-09-13T22:45:10.873170+0000","last_unstale":"2024-09-13T22:45:28.578656+0000","last_undegraded":"2024-09-13T22:45:28.578656+0000","last_fullsized":"2024-09-13T22:45:28.578656+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1229,"log_dups_size":0,"ondisk_log_size":1229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:27.152804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80543744,"num_objects":1229,"num_object_clones":0,"num_object_copies":3687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1229,"num_write_kb":78656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":274,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611337+0000","last_change":"2024-09-13T22:44:47.859269+0000","last_active":"2024-09-13T22:45:18.611337+0000","last_peered":"2024-09-13T22:45:18.611337+0000","last_clean":"2024-09-13T22:45:18.611337+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:45:18.611337+0000","last_undegraded":"2024-09-13T22:45:18.611337+0000","last_fullsized":"2024-09-13T22:45:18.611337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:44:47.859159+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:44:47.859159+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:59.540819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.445669+0000","last_change":"2024-09-13T22:43:53.485493+0000","last_active":"2024-09-13T22:45:18.445669+0000","last_peered":"2024-09-13T22:45:18.445669+0000","last_clean":"2024-09-13T22:45:18.445669+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:45:18.445669+0000","last_undegraded":"2024-09-13T22:45:18.445669+0000","last_fullsized":"2024-09-13T22:45:18.445669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.485398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.485398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.584015+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":350,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.160568+0000","last_change":"2024-09-13T22:44:02.641797+0000","last_active":"2024-09-13T22:45:21.160568+0000","last_peered":"2024-09-13T22:45:21.160568+0000","last_clean":"2024-09-13T22:45:21.160568+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:45:21.160568+0000","last_undegraded":"2024-09-13T22:45:21.160568+0000","last_fullsized":"2024-09-13T22:45:21.160568+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:44:02.641728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:02.641728+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:30.648601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":242,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484472+0000","last_change":"2024-09-13T22:44:44.904917+0000","last_active":"2024-09-13T22:45:18.484472+0000","last_peered":"2024-09-13T22:45:18.484472+0000","last_clean":"2024-09-13T22:45:18.484472+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:45:18.484472+0000","last_undegraded":"2024-09-13T22:45:18.484472+0000","last_fullsized":"2024-09-13T22:45:18.484472+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:44.904781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:44.904781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:52.815260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"180'1205","reported_seq":1225,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:28.585957+0000","last_change":"2024-09-13T22:45:11.076685+0000","last_active":"2024-09-13T22:45:28.585957+0000","last_peered":"2024-09-13T22:45:28.585957+0000","last_clean":"2024-09-13T22:45:28.585957+0000","last_became_active":"2024-09-13T22:45:11.075580+0000","last_became_peered":"2024-09-13T22:45:11.075580+0000","last_unstale":"2024-09-13T22:45:28.585957+0000","last_undegraded":"2024-09-13T22:45:28.585957+0000","last_fullsized":"2024-09-13T22:45:28.585957+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":1205,"log_dups_size":0,"ondisk_log_size":1205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:30.512256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78970880,"num_objects":1205,"num_object_clones":0,"num_object_copies":3615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":77120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":269,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.612529+0000","last_change":"2024-09-13T22:44:55.856200+0000","last_active":"2024-09-13T22:45:18.612529+0000","last_peered":"2024-09-13T22:45:18.612529+0000","last_clean":"2024-09-13T22:45:18.612529+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:45:18.612529+0000","last_undegraded":"2024-09-13T22:45:18.612529+0000","last_fullsized":"2024-09-13T22:45:18.612529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:44:55.856068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.856068+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:58.271871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":273,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.905233+0000","last_change":"2024-09-13T22:44:44.075576+0000","last_active":"2024-09-13T22:45:17.905233+0000","last_peered":"2024-09-13T22:45:17.905233+0000","last_clean":"2024-09-13T22:45:17.905233+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:45:17.905233+0000","last_undegraded":"2024-09-13T22:45:17.905233+0000","last_fullsized":"2024-09-13T22:45:17.905233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:44:44.075463+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:44:44.075463+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:53.777711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.441767+0000","last_change":"2024-09-13T22:45:01.719814+0000","last_active":"2024-09-13T22:45:18.441767+0000","last_peered":"2024-09-13T22:45:18.441767+0000","last_clean":"2024-09-13T22:45:18.441767+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:45:18.441767+0000","last_undegraded":"2024-09-13T22:45:18.441767+0000","last_fullsized":"2024-09-13T22:45:18.441767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:16.601356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901460+0000","last_change":"2024-09-13T22:44:05.968876+0000","last_active":"2024-09-13T22:45:17.901460+0000","last_peered":"2024-09-13T22:45:17.901460+0000","last_clean":"2024-09-13T22:45:17.901460+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:45:17.901460+0000","last_undegraded":"2024-09-13T22:45:17.901460+0000","last_fullsized":"2024-09-13T22:45:17.901460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:44:05.968767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:22.516012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"112'16","reported_seq":266,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610576+0000","last_change":"2024-09-13T22:42:36.814689+0000","last_active":"2024-09-13T22:45:18.610576+0000","last_peered":"2024-09-13T22:45:18.610576+0000","last_clean":"2024-09-13T22:45:18.610576+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:45:18.610576+0000","last_undegraded":"2024-09-13T22:45:18.610576+0000","last_fullsized":"2024-09-13T22:45:18.610576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'16","last_scrub_stamp":"2024-09-13T22:42:36.814607+0000","last_deep_scrub":"112'16","last_deep_scrub_stamp":"2024-09-13T22:40:26.503359+0000","last_clean_scrub_stamp":"2024-09-13T22:42:36.814607+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442041+0000","last_change":"2024-09-13T22:45:03.702368+0000","last_active":"2024-09-13T22:45:18.442041+0000","last_peered":"2024-09-13T22:45:18.442041+0000","last_clean":"2024-09-13T22:45:18.442041+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:45:18.442041+0000","last_undegraded":"2024-09-13T22:45:18.442041+0000","last_fullsized":"2024-09-13T22:45:18.442041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.702179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.702179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:28.781648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.902148+0000","last_change":"2024-09-13T22:43:58.778941+0000","last_active":"2024-09-13T22:45:17.902148+0000","last_peered":"2024-09-13T22:45:17.902148+0000","last_clean":"2024-09-13T22:45:17.902148+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:45:17.902148+0000","last_undegraded":"2024-09-13T22:45:17.902148+0000","last_fullsized":"2024-09-13T22:45:17.902148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:58.778835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:58.778835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":251,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444135+0000","last_change":"2024-09-13T22:44:58.743225+0000","last_active":"2024-09-13T22:45:18.444135+0000","last_peered":"2024-09-13T22:45:18.444135+0000","last_clean":"2024-09-13T22:45:18.444135+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:45:18.444135+0000","last_undegraded":"2024-09-13T22:45:18.444135+0000","last_fullsized":"2024-09-13T22:45:18.444135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:58.743133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.743133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:02.477009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":265,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609265+0000","last_change":"2024-09-13T22:44:48.034436+0000","last_active":"2024-09-13T22:45:18.609265+0000","last_peered":"2024-09-13T22:45:18.609265+0000","last_clean":"2024-09-13T22:45:18.609265+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:45:18.609265+0000","last_undegraded":"2024-09-13T22:45:18.609265+0000","last_fullsized":"2024-09-13T22:45:18.609265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:44:48.034322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:48.034322+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:04.740947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610168+0000","last_change":"2024-09-13T22:45:04.035137+0000","last_active":"2024-09-13T22:45:18.610168+0000","last_peered":"2024-09-13T22:45:18.610168+0000","last_clean":"2024-09-13T22:45:18.610168+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:45:18.610168+0000","last_undegraded":"2024-09-13T22:45:18.610168+0000","last_fullsized":"2024-09-13T22:45:18.610168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_clean_scrub_stamp":"2024-09-13T22:45:04.034995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:08.753673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":246,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484961+0000","last_change":"2024-09-13T22:44:57.934018+0000","last_active":"2024-09-13T22:45:18.484961+0000","last_peered":"2024-09-13T22:45:18.484961+0000","last_clean":"2024-09-13T22:45:18.484961+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:45:18.484961+0000","last_undegraded":"2024-09-13T22:45:18.484961+0000","last_fullsized":"2024-09-13T22:45:18.484961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:57.933941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:57.933941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:05.732701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":234,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611023+0000","last_change":"2024-09-13T22:43:40.670309+0000","last_active":"2024-09-13T22:45:18.611023+0000","last_peered":"2024-09-13T22:45:18.611023+0000","last_clean":"2024-09-13T22:45:18.611023+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:45:18.611023+0000","last_undegraded":"2024-09-13T22:45:18.611023+0000","last_fullsized":"2024-09-13T22:45:18.611023+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:40.670209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:40.670209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611087+0000","last_change":"2024-09-13T22:42:34.859780+0000","last_active":"2024-09-13T22:45:18.611087+0000","last_peered":"2024-09-13T22:45:18.611087+0000","last_clean":"2024-09-13T22:45:18.611087+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:45:18.611087+0000","last_undegraded":"2024-09-13T22:45:18.611087+0000","last_fullsized":"2024-09-13T22:45:18.611087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:42:34.859649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:34.859649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624725+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484978+0000","last_change":"2024-09-13T22:43:53.971419+0000","last_active":"2024-09-13T22:45:18.484978+0000","last_peered":"2024-09-13T22:45:18.484978+0000","last_clean":"2024-09-13T22:45:18.484978+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:45:18.484978+0000","last_undegraded":"2024-09-13T22:45:18.484978+0000","last_fullsized":"2024-09-13T22:45:18.484978+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:53.971324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:43:53.971324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:23.091918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":353,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.160471+0000","last_change":"2024-09-13T22:43:50.053369+0000","last_active":"2024-09-13T22:45:21.160471+0000","last_peered":"2024-09-13T22:45:21.160471+0000","last_clean":"2024-09-13T22:45:21.160471+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:45:21.160471+0000","last_undegraded":"2024-09-13T22:45:21.160471+0000","last_fullsized":"2024-09-13T22:45:21.160471+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:50.053246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:50.053246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:28.059666+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608229+0000","last_change":"2024-09-13T22:44:41.721931+0000","last_active":"2024-09-13T22:45:18.608229+0000","last_peered":"2024-09-13T22:45:18.608229+0000","last_clean":"2024-09-13T22:45:18.608229+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:45:18.608229+0000","last_undegraded":"2024-09-13T22:45:18.608229+0000","last_fullsized":"2024-09-13T22:45:18.608229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:41.721714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:41.721714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:55.502472+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":267,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608076+0000","last_change":"2024-09-13T22:43:57.436556+0000","last_active":"2024-09-13T22:45:18.608076+0000","last_peered":"2024-09-13T22:45:18.608076+0000","last_clean":"2024-09-13T22:45:18.608076+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:45:18.608076+0000","last_undegraded":"2024-09-13T22:45:18.608076+0000","last_fullsized":"2024-09-13T22:45:18.608076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:43:57.436436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.436436+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:24.698419+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608962+0000","last_change":"2024-09-13T22:45:03.920511+0000","last_active":"2024-09-13T22:45:18.608962+0000","last_peered":"2024-09-13T22:45:18.608962+0000","last_clean":"2024-09-13T22:45:18.608962+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:45:18.608962+0000","last_undegraded":"2024-09-13T22:45:18.608962+0000","last_fullsized":"2024-09-13T22:45:18.608962+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.920319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.920319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.344069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":351,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.160298+0000","last_change":"2024-09-13T22:43:42.603416+0000","last_active":"2024-09-13T22:45:21.160298+0000","last_peered":"2024-09-13T22:45:21.160298+0000","last_clean":"2024-09-13T22:45:21.160298+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:45:21.160298+0000","last_undegraded":"2024-09-13T22:45:21.160298+0000","last_fullsized":"2024-09-13T22:45:21.160298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:43:42.603288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:42.603288+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:27.468097+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904901+0000","last_change":"2024-09-13T22:44:43.044133+0000","last_active":"2024-09-13T22:45:17.904901+0000","last_peered":"2024-09-13T22:45:17.904901+0000","last_clean":"2024-09-13T22:45:17.904901+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:45:17.904901+0000","last_undegraded":"2024-09-13T22:45:17.904901+0000","last_fullsized":"2024-09-13T22:45:17.904901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:43.044031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:43.044031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:08.606139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":258,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611329+0000","last_change":"2024-09-13T22:42:52.779439+0000","last_active":"2024-09-13T22:45:18.611329+0000","last_peered":"2024-09-13T22:45:18.611329+0000","last_clean":"2024-09-13T22:45:18.611329+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:45:18.611329+0000","last_undegraded":"2024-09-13T22:45:18.611329+0000","last_fullsized":"2024-09-13T22:45:18.611329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:42:52.779356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:42:52.779356+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.612132+0000","last_change":"2024-09-13T22:44:52.084394+0000","last_active":"2024-09-13T22:45:18.612132+0000","last_peered":"2024-09-13T22:45:18.612132+0000","last_clean":"2024-09-13T22:45:18.612132+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:45:18.612132+0000","last_undegraded":"2024-09-13T22:45:18.612132+0000","last_fullsized":"2024-09-13T22:45:18.612132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:52.084196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:52.084196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:12.969502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901545+0000","last_change":"2024-09-13T22:45:00.534688+0000","last_active":"2024-09-13T22:45:17.901545+0000","last_peered":"2024-09-13T22:45:17.901545+0000","last_clean":"2024-09-13T22:45:17.901545+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:45:17.901545+0000","last_undegraded":"2024-09-13T22:45:17.901545+0000","last_fullsized":"2024-09-13T22:45:17.901545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:00.534596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:45:00.534596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:08.991640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":235,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.607548+0000","last_change":"2024-09-13T22:42:58.240077+0000","last_active":"2024-09-13T22:45:18.607548+0000","last_peered":"2024-09-13T22:45:18.607548+0000","last_clean":"2024-09-13T22:45:18.607548+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:45:18.607548+0000","last_undegraded":"2024-09-13T22:45:18.607548+0000","last_fullsized":"2024-09-13T22:45:18.607548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:58.239969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:58.239969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":266,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.905333+0000","last_change":"2024-09-13T22:43:36.040902+0000","last_active":"2024-09-13T22:45:17.905333+0000","last_peered":"2024-09-13T22:45:17.905333+0000","last_clean":"2024-09-13T22:45:17.905333+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:45:17.905333+0000","last_undegraded":"2024-09-13T22:45:17.905333+0000","last_fullsized":"2024-09-13T22:45:17.905333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:43:36.040693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:36.040693+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.979278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444570+0000","last_change":"2024-09-13T22:44:41.473196+0000","last_active":"2024-09-13T22:45:18.444570+0000","last_peered":"2024-09-13T22:45:18.444570+0000","last_clean":"2024-09-13T22:45:18.444570+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:45:18.444570+0000","last_undegraded":"2024-09-13T22:45:18.444570+0000","last_fullsized":"2024-09-13T22:45:18.444570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:41.472956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:41.472956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:51.081445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443270+0000","last_change":"2024-09-13T22:44:38.367826+0000","last_active":"2024-09-13T22:45:18.443270+0000","last_peered":"2024-09-13T22:45:18.443270+0000","last_clean":"2024-09-13T22:45:18.443270+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:45:18.443270+0000","last_undegraded":"2024-09-13T22:45:18.443270+0000","last_fullsized":"2024-09-13T22:45:18.443270+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:38.367570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:38.367570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:47.054212+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.900722+0000","last_change":"2024-09-13T22:44:45.461864+0000","last_active":"2024-09-13T22:45:17.900722+0000","last_peered":"2024-09-13T22:45:17.900722+0000","last_clean":"2024-09-13T22:45:17.900722+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:45:17.900722+0000","last_undegraded":"2024-09-13T22:45:17.900722+0000","last_fullsized":"2024-09-13T22:45:17.900722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:45.461720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:44:45.461720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:09.262417+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":260,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608570+0000","last_change":"2024-09-13T22:43:46.254609+0000","last_active":"2024-09-13T22:45:18.608570+0000","last_peered":"2024-09-13T22:45:18.608570+0000","last_clean":"2024-09-13T22:45:18.608570+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:45:18.608570+0000","last_undegraded":"2024-09-13T22:45:18.608570+0000","last_fullsized":"2024-09-13T22:45:18.608570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:43:46.254539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:46.254539+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.485127+0000","last_change":"2024-09-13T22:44:43.909015+0000","last_active":"2024-09-13T22:45:18.485127+0000","last_peered":"2024-09-13T22:45:18.485127+0000","last_clean":"2024-09-13T22:45:18.485127+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:45:18.485127+0000","last_undegraded":"2024-09-13T22:45:18.485127+0000","last_fullsized":"2024-09-13T22:45:18.485127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:43.908946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:44:43.908946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:47.476146+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609528+0000","last_change":"2024-09-13T22:45:02.990024+0000","last_active":"2024-09-13T22:45:18.609528+0000","last_peered":"2024-09-13T22:45:18.609528+0000","last_clean":"2024-09-13T22:45:18.609528+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:45:18.609528+0000","last_undegraded":"2024-09-13T22:45:18.609528+0000","last_fullsized":"2024-09-13T22:45:18.609528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.989795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.989795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:24.890892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":234,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609618+0000","last_change":"2024-09-13T22:44:54.870749+0000","last_active":"2024-09-13T22:45:18.609618+0000","last_peered":"2024-09-13T22:45:18.609618+0000","last_clean":"2024-09-13T22:45:18.609618+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:45:18.609618+0000","last_undegraded":"2024-09-13T22:45:18.609618+0000","last_fullsized":"2024-09-13T22:45:18.609618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:54.870619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:54.870619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:05.051074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":266,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484993+0000","last_change":"2024-09-13T22:44:48.935410+0000","last_active":"2024-09-13T22:45:18.484993+0000","last_peered":"2024-09-13T22:45:18.484993+0000","last_clean":"2024-09-13T22:45:18.484993+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:45:18.484993+0000","last_undegraded":"2024-09-13T22:45:18.484993+0000","last_fullsized":"2024-09-13T22:45:18.484993+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:44:48.934703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:48.934703+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:54.403791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.613251+0000","last_change":"2024-09-13T22:44:41.009736+0000","last_active":"2024-09-13T22:45:18.613251+0000","last_peered":"2024-09-13T22:45:18.613251+0000","last_clean":"2024-09-13T22:45:18.613251+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:45:18.613251+0000","last_undegraded":"2024-09-13T22:45:18.613251+0000","last_fullsized":"2024-09-13T22:45:18.613251+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:41.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:44:41.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:43.539723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":238,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.609448+0000","last_change":"2024-09-13T22:43:51.625709+0000","last_active":"2024-09-13T22:45:18.609448+0000","last_peered":"2024-09-13T22:45:18.609448+0000","last_clean":"2024-09-13T22:45:18.609448+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:45:18.609448+0000","last_undegraded":"2024-09-13T22:45:18.609448+0000","last_fullsized":"2024-09-13T22:45:18.609448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:51.625640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.625640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.625021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901627+0000","last_change":"2024-09-13T22:43:45.603477+0000","last_active":"2024-09-13T22:45:17.901627+0000","last_peered":"2024-09-13T22:45:17.901627+0000","last_clean":"2024-09-13T22:45:17.901627+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:45:17.901627+0000","last_undegraded":"2024-09-13T22:45:17.901627+0000","last_fullsized":"2024-09-13T22:45:17.901627+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:45.603355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:43:45.603355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.511907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":236,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.611258+0000","last_change":"2024-09-13T22:42:54.751350+0000","last_active":"2024-09-13T22:45:18.611258+0000","last_peered":"2024-09-13T22:45:18.611258+0000","last_clean":"2024-09-13T22:45:18.611258+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:45:18.611258+0000","last_undegraded":"2024-09-13T22:45:18.611258+0000","last_fullsized":"2024-09-13T22:45:18.611258+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:42:54.751219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:42:54.751219+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:19.624860+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"111'22","reported_seq":283,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.443588+0000","last_change":"2024-09-13T22:43:51.139213+0000","last_active":"2024-09-13T22:45:18.443588+0000","last_peered":"2024-09-13T22:45:18.443588+0000","last_clean":"2024-09-13T22:45:18.443588+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:45:18.443588+0000","last_undegraded":"2024-09-13T22:45:18.443588+0000","last_fullsized":"2024-09-13T22:45:18.443588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-13T22:43:51.138649+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.138649+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:18.665191+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.484467+0000","last_change":"2024-09-13T22:44:37.060754+0000","last_active":"2024-09-13T22:45:18.484467+0000","last_peered":"2024-09-13T22:45:18.484467+0000","last_clean":"2024-09-13T22:45:18.484467+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:45:18.484467+0000","last_undegraded":"2024-09-13T22:45:18.484467+0000","last_fullsized":"2024-09-13T22:45:18.484467+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:37.060657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:44:37.060657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:39.427993+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"172'12","reported_seq":363,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:21.159884+0000","last_change":"2024-09-13T22:44:59.017860+0000","last_active":"2024-09-13T22:45:21.159884+0000","last_peered":"2024-09-13T22:45:21.159884+0000","last_clean":"2024-09-13T22:45:21.159884+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:45:21.159884+0000","last_undegraded":"2024-09-13T22:45:21.159884+0000","last_fullsized":"2024-09-13T22:45:21.159884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'12","last_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_deep_scrub":"172'12","last_deep_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_clean_scrub_stamp":"2024-09-13T22:44:59.017618+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:14.765551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.442259+0000","last_change":"2024-09-13T22:43:56.199092+0000","last_active":"2024-09-13T22:45:18.442259+0000","last_peered":"2024-09-13T22:45:18.442259+0000","last_clean":"2024-09-13T22:45:18.442259+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:45:18.442259+0000","last_undegraded":"2024-09-13T22:45:18.442259+0000","last_fullsized":"2024-09-13T22:45:18.442259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:56.198539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:56.198539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.605890+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":267,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901057+0000","last_change":"2024-09-13T22:44:38.382234+0000","last_active":"2024-09-13T22:45:17.901057+0000","last_peered":"2024-09-13T22:45:17.901057+0000","last_clean":"2024-09-13T22:45:17.901057+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:45:17.901057+0000","last_undegraded":"2024-09-13T22:45:17.901057+0000","last_fullsized":"2024-09-13T22:45:17.901057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:44:38.382116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:38.382116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:56.045341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":256,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.900992+0000","last_change":"2024-09-13T22:44:58.543023+0000","last_active":"2024-09-13T22:45:17.900992+0000","last_peered":"2024-09-13T22:45:17.900992+0000","last_clean":"2024-09-13T22:45:17.900992+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:45:17.900992+0000","last_undegraded":"2024-09-13T22:45:17.900992+0000","last_fullsized":"2024-09-13T22:45:17.900992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:58.542781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.542781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:20.452580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608523+0000","last_change":"2024-09-13T22:42:59.279764+0000","last_active":"2024-09-13T22:45:18.608523+0000","last_peered":"2024-09-13T22:45:18.608523+0000","last_clean":"2024-09-13T22:45:18.608523+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:45:18.608523+0000","last_undegraded":"2024-09-13T22:45:18.608523+0000","last_fullsized":"2024-09-13T22:45:18.608523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:42:59.279506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:42:59.279506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":243,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.905167+0000","last_change":"2024-09-13T22:44:39.003158+0000","last_active":"2024-09-13T22:45:17.905167+0000","last_peered":"2024-09-13T22:45:17.905167+0000","last_clean":"2024-09-13T22:45:17.905167+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:45:17.905167+0000","last_undegraded":"2024-09-13T22:45:17.905167+0000","last_fullsized":"2024-09-13T22:45:17.905167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:39.003066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:39.003066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:40.854289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.610465+0000","last_change":"2024-09-13T22:44:47.023754+0000","last_active":"2024-09-13T22:45:18.610465+0000","last_peered":"2024-09-13T22:45:18.610465+0000","last_clean":"2024-09-13T22:45:18.610465+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:45:18.610465+0000","last_undegraded":"2024-09-13T22:45:18.610465+0000","last_fullsized":"2024-09-13T22:45:18.610465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:47.023644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:47.023644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:15.822893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":271,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.608930+0000","last_change":"2024-09-13T22:44:57.928934+0000","last_active":"2024-09-13T22:45:18.608930+0000","last_peered":"2024-09-13T22:45:18.608930+0000","last_clean":"2024-09-13T22:45:18.608930+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:45:18.608930+0000","last_undegraded":"2024-09-13T22:45:18.608930+0000","last_fullsized":"2024-09-13T22:45:18.608930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:44:57.928755+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:44:57.928755+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:07.937853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:18.444007+0000","last_change":"2024-09-13T22:44:50.489826+0000","last_active":"2024-09-13T22:45:18.444007+0000","last_peered":"2024-09-13T22:45:18.444007+0000","last_clean":"2024-09-13T22:45:18.444007+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:45:18.444007+0000","last_undegraded":"2024-09-13T22:45:18.444007+0000","last_fullsized":"2024-09-13T22:45:18.444007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:50.489640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:44:50.489640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:00.708474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":244,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.902088+0000","last_change":"2024-09-13T22:43:50.727253+0000","last_active":"2024-09-13T22:45:17.902088+0000","last_peered":"2024-09-13T22:45:17.902088+0000","last_clean":"2024-09-13T22:45:17.902088+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:45:17.902088+0000","last_undegraded":"2024-09-13T22:45:17.902088+0000","last_fullsized":"2024-09-13T22:45:17.902088+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:43:50.727116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:43:50.727116+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.512045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":255,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.901846+0000","last_change":"2024-09-13T22:44:49.489483+0000","last_active":"2024-09-13T22:45:17.901846+0000","last_peered":"2024-09-13T22:45:17.901846+0000","last_clean":"2024-09-13T22:45:17.901846+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:45:17.901846+0000","last_undegraded":"2024-09-13T22:45:17.901846+0000","last_fullsized":"2024-09-13T22:45:17.901846+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:49.489414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:49.489414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:56.459622+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"82'9","reported_seq":267,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.905359+0000","last_change":"2024-09-13T22:45:02.968465+0000","last_active":"2024-09-13T22:45:17.905359+0000","last_peered":"2024-09-13T22:45:17.905359+0000","last_clean":"2024-09-13T22:45:17.905359+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:45:17.905359+0000","last_undegraded":"2024-09-13T22:45:17.905359+0000","last_fullsized":"2024-09-13T22:45:17.905359+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:45:02.968155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.968155+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:07.857311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":247,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-13T22:45:17.904867+0000","last_change":"2024-09-13T22:43:38.061761+0000","last_active":"2024-09-13T22:45:17.904867+0000","last_peered":"2024-09-13T22:45:17.904867+0000","last_clean":"2024-09-13T22:45:17.904867+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:45:17.904867+0000","last_undegraded":"2024-09-13T22:45:17.904867+0000","last_fullsized":"2024-09-13T22:45:17.904867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:38.061690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:43:38.061690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:45:21.979294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1309802496,"num_objects":19986,"num_object_clones":0,"num_object_copies":59958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19986,"num_write_kb":1279104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3900047360,"data_stored":3900047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19986,"ondisk_log_size":19986,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808934,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505984,"kb_used_data":398448,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93239168,"statfs":{"total":95995035648,"available":95476908032,"internally_reserved":0,"allocated":408010752,"data_stored":407588399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135985,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472484,"kb_used_data":364948,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93272668,"statfs":{"total":95995035648,"available":95511212032,"internally_reserved":0,"allocated":373706752,"data_stored":373310749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528444,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708452,"kb_used_data":596756,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93036700,"statfs":{"total":95995035648,"available":95269580800,"internally_reserved":0,"allocated":611078144,"data_stored":610681999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888198,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472132,"kb_used_data":360436,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93273020,"statfs":{"total":95995035648,"available":95511572480,"internally_reserved":0,"allocated":369086464,"data_stored":368658081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084432,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641508,"kb_used_data":488148,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93103644,"statfs":{"total":95995035648,"available":95338131456,"internally_reserved":0,"allocated":499863552,"data_stored":499467828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":157022663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378780,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":589604,"kb_used_data":472340,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93155548,"statfs":{"total":95995035648,"available":95391281152,"internally_reserved":0,"allocated":483676160,"data_stored":483280529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":689136,"kb_used_data":541344,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93056016,"statfs":{"total":95995035648,"available":95289360384,"internally_reserved":0,"allocated":554336256,"data_stored":553928732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621752,"kb_used_data":473960,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93123400,"statfs":{"total":95995035648,"available":95358361600,"internally_reserved":0,"allocated":485335040,"data_stored":484920628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":514523136,"data_stored":514523136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":555614208,"data_stored":555614208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":488505344,"data_stored":488505344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":503513088,"data_stored":503513088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":398983168,"data_stored":398983168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":636682240,"data_stored":636682240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":395640832,"data_stored":395640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":406585344,"data_stored":406585344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:45:36.591 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T22:45:36.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-13T22:45:36.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:36.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:36.763 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:36.793 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:36.811 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 17 16 32897 32881 120.872 105.562 0.00812186 0.00826657 2024-09-13T22:45:36.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:36 smithi192 bash[24608]: cluster 2024-09-13T22:45:35.842158+0000 osd.4 (osd.4) 139 : cluster [DBG] 5.e scrub starts 2024-09-13T22:45:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:36 smithi192 bash[24608]: cluster 2024-09-13T22:45:35.853260+0000 osd.4 (osd.4) 140 : cluster [DBG] 5.e scrub ok 2024-09-13T22:45:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:36 smithi192 bash[24608]: cluster 2024-09-13T22:45:36.595307+0000 mgr.y (mgr.14613) 297 : cluster [DBG] pgmap v303: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 681 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T22:45:36.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:36.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:37.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:37.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:37.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:37.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[22840]: cluster 2024-09-13T22:45:35.842158+0000 osd.4 (osd.4) 139 : cluster [DBG] 5.e scrub starts 2024-09-13T22:45:37.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[22840]: cluster 2024-09-13T22:45:35.853260+0000 osd.4 (osd.4) 140 : cluster [DBG] 5.e scrub ok 2024-09-13T22:45:37.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[22840]: cluster 2024-09-13T22:45:36.595307+0000 mgr.y (mgr.14613) 297 : cluster [DBG] pgmap v303: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 681 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T22:45:37.175 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[29889]: cluster 2024-09-13T22:45:35.842158+0000 osd.4 (osd.4) 139 : cluster [DBG] 5.e scrub starts 2024-09-13T22:45:37.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[29889]: cluster 2024-09-13T22:45:35.853260+0000 osd.4 (osd.4) 140 : cluster [DBG] 5.e scrub ok 2024-09-13T22:45:37.176 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:36 smithi178 bash[29889]: cluster 2024-09-13T22:45:36.595307+0000 mgr.y (mgr.14613) 297 : cluster [DBG] pgmap v303: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 681 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T22:45:37.193 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:37.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:37.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:37.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:37.457 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:37.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:37.585 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:37.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:37.714 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:37.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 18 16 34720 34704 120.486 113.938 0.00235768 0.0082939 2024-09-13T22:45:37.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:37.845 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:37.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:37.983 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:37.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:38.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:38.115 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:38.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:37 smithi192 bash[24608]: cluster 2024-09-13T22:45:36.098755+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:45:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:37 smithi192 bash[24608]: cluster 2024-09-13T22:45:36.100521+0000 osd.5 (osd.5) 177 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:45:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:37 smithi192 bash[24608]: cluster 2024-09-13T22:45:36.860155+0000 osd.6 (osd.6) 131 : cluster [DBG] 3.a scrub starts 2024-09-13T22:45:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:37 smithi192 bash[24608]: cluster 2024-09-13T22:45:36.864506+0000 osd.6 (osd.6) 132 : cluster [DBG] 3.a scrub ok 2024-09-13T22:45:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:37 smithi192 bash[24608]: cluster 2024-09-13T22:45:37.775193+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-13T22:45:38.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:37 smithi192 bash[24608]: cluster 2024-09-13T22:45:37.777188+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-13T22:45:38.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[22840]: cluster 2024-09-13T22:45:36.098755+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[22840]: cluster 2024-09-13T22:45:36.100521+0000 osd.5 (osd.5) 177 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[22840]: cluster 2024-09-13T22:45:36.860155+0000 osd.6 (osd.6) 131 : cluster [DBG] 3.a scrub starts 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[22840]: cluster 2024-09-13T22:45:36.864506+0000 osd.6 (osd.6) 132 : cluster [DBG] 3.a scrub ok 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[22840]: cluster 2024-09-13T22:45:37.775193+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[22840]: cluster 2024-09-13T22:45:37.777188+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[29889]: cluster 2024-09-13T22:45:36.098755+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[29889]: cluster 2024-09-13T22:45:36.100521+0000 osd.5 (osd.5) 177 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[29889]: cluster 2024-09-13T22:45:36.860155+0000 osd.6 (osd.6) 131 : cluster [DBG] 3.a scrub starts 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[29889]: cluster 2024-09-13T22:45:36.864506+0000 osd.6 (osd.6) 132 : cluster [DBG] 3.a scrub ok 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[29889]: cluster 2024-09-13T22:45:37.775193+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.a scrub starts 2024-09-13T22:45:38.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:37 smithi178 bash[29889]: cluster 2024-09-13T22:45:37.777188+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.a scrub ok 2024-09-13T22:45:38.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:38.244 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:38.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:38.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:38.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:38.509 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:38.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:38.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:45:38.644 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:38.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:45:38.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:38.784 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:38.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 19 16 36499 36483 119.996 111.188 0.00851707 0.00832869 2024-09-13T22:45:38.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:38.902 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:38 smithi192 bash[24608]: cluster 2024-09-13T22:45:37.837149+0000 osd.4 (osd.4) 141 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:45:38.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:38 smithi192 bash[24608]: cluster 2024-09-13T22:45:37.838729+0000 osd.4 (osd.4) 142 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:45:38.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:38 smithi192 bash[24608]: cluster 2024-09-13T22:45:38.595835+0000 mgr.y (mgr.14613) 298 : cluster [DBG] pgmap v304: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 593 B/s rd, 87 MiB/s wr, 1.38k op/s 2024-09-13T22:45:38.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:38.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:39.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:39.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:39.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:39.145 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:39.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:39.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:38 smithi178 bash[22840]: cluster 2024-09-13T22:45:37.837149+0000 osd.4 (osd.4) 141 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:45:39.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:38 smithi178 bash[22840]: cluster 2024-09-13T22:45:37.838729+0000 osd.4 (osd.4) 142 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:45:39.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:38 smithi178 bash[22840]: cluster 2024-09-13T22:45:38.595835+0000 mgr.y (mgr.14613) 298 : cluster [DBG] pgmap v304: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 593 B/s rd, 87 MiB/s wr, 1.38k op/s 2024-09-13T22:45:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:38 smithi178 bash[29889]: cluster 2024-09-13T22:45:37.837149+0000 osd.4 (osd.4) 141 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:45:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:38 smithi178 bash[29889]: cluster 2024-09-13T22:45:37.838729+0000 osd.4 (osd.4) 142 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:45:39.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:38 smithi178 bash[29889]: cluster 2024-09-13T22:45:38.595835+0000 mgr.y (mgr.14613) 298 : cluster [DBG] pgmap v304: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 593 B/s rd, 87 MiB/s wr, 1.38k op/s 2024-09-13T22:45:39.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:39.300 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:39.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:39.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:39.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:39.567 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:39.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:39.696 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:39.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:39.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:45:39.815898+0000 min lat: 0.00124784 max lat: 0.701883 avg lat: 0.00839025 2024-09-13T22:45:39.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:45:39.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 20 16 38127 38111 119.083 101.75 0.0161793 0.00839025 2024-09-13T22:45:39.824 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:39.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:39.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:39.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:40.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:40.076 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:40.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:39 smithi192 bash[24608]: cluster 2024-09-13T22:45:38.486346+0000 osd.3 (osd.3) 192 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:45:40.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:39 smithi192 bash[24608]: cluster 2024-09-13T22:45:38.494517+0000 osd.3 (osd.3) 193 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:45:40.178 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:45:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:45:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:45:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:39 smithi178 bash[29889]: cluster 2024-09-13T22:45:38.486346+0000 osd.3 (osd.3) 192 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:45:40.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:39 smithi178 bash[29889]: cluster 2024-09-13T22:45:38.494517+0000 osd.3 (osd.3) 193 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:45:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:39 smithi178 bash[22840]: cluster 2024-09-13T22:45:38.486346+0000 osd.3 (osd.3) 192 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:45:40.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:39 smithi178 bash[22840]: cluster 2024-09-13T22:45:38.494517+0000 osd.3 (osd.3) 193 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:45:40.180 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:40.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:40.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:40.336 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:40.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:40.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:40.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:40.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:40.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:40.716 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:40.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 21 15 39753 39738 118.255 101.688 0.0101127 0.00845124 2024-09-13T22:45:40.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:40.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:40.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:40.967 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:40.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:41.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:41.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:40 smithi178 bash[29889]: cluster 2024-09-13T22:45:39.711811+0000 osd.7 (osd.7) 165 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:45:41.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:40 smithi178 bash[29889]: cluster 2024-09-13T22:45:39.713588+0000 osd.7 (osd.7) 166 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:45:41.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:40 smithi178 bash[29889]: cluster 2024-09-13T22:45:40.597534+0000 mgr.y (mgr.14613) 299 : cluster [DBG] pgmap v305: 148 pgs: 148 active+clean; 2.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-13T22:45:41.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:40 smithi178 bash[22840]: cluster 2024-09-13T22:45:39.711811+0000 osd.7 (osd.7) 165 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:45:41.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:40 smithi178 bash[22840]: cluster 2024-09-13T22:45:39.713588+0000 osd.7 (osd.7) 166 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:45:41.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:40 smithi178 bash[22840]: cluster 2024-09-13T22:45:40.597534+0000 mgr.y (mgr.14613) 299 : cluster [DBG] pgmap v305: 148 pgs: 148 active+clean; 2.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-13T22:45:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:40 smithi192 bash[24608]: cluster 2024-09-13T22:45:39.711811+0000 osd.7 (osd.7) 165 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:45:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:40 smithi192 bash[24608]: cluster 2024-09-13T22:45:39.713588+0000 osd.7 (osd.7) 166 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:45:41.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:40 smithi192 bash[24608]: cluster 2024-09-13T22:45:40.597534+0000 mgr.y (mgr.14613) 299 : cluster [DBG] pgmap v305: 148 pgs: 148 active+clean; 2.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-13T22:45:41.180 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:41.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:41.234 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:41.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:41.377 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:41.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:41.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:41.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:41.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:41.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:41.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:41.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 22 16 40713 40697 115.604 59.9375 0.00662331 0.00864547 2024-09-13T22:45:41.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:41.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:41.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:42.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:42.033 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:42.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:42.148 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[29889]: cluster 2024-09-13T22:45:40.482392+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:45:42.148 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[29889]: cluster 2024-09-13T22:45:40.486240+0000 osd.3 (osd.3) 195 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:45:42.148 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[29889]: cluster 2024-09-13T22:45:40.810067+0000 osd.2 (osd.2) 137 : cluster [DBG] 2.e scrub starts 2024-09-13T22:45:42.148 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[29889]: cluster 2024-09-13T22:45:40.816818+0000 osd.2 (osd.2) 138 : cluster [DBG] 2.e scrub ok 2024-09-13T22:45:42.149 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[22840]: cluster 2024-09-13T22:45:40.482392+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:45:42.149 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[22840]: cluster 2024-09-13T22:45:40.486240+0000 osd.3 (osd.3) 195 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:45:42.149 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[22840]: cluster 2024-09-13T22:45:40.810067+0000 osd.2 (osd.2) 137 : cluster [DBG] 2.e scrub starts 2024-09-13T22:45:42.149 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:41 smithi178 bash[22840]: cluster 2024-09-13T22:45:40.816818+0000 osd.2 (osd.2) 138 : cluster [DBG] 2.e scrub ok 2024-09-13T22:45:42.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:42.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:42.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:41 smithi192 bash[24608]: cluster 2024-09-13T22:45:40.482392+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:45:42.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:41 smithi192 bash[24608]: cluster 2024-09-13T22:45:40.486240+0000 osd.3 (osd.3) 195 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:45:42.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:41 smithi192 bash[24608]: cluster 2024-09-13T22:45:40.810067+0000 osd.2 (osd.2) 137 : cluster [DBG] 2.e scrub starts 2024-09-13T22:45:42.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:41 smithi192 bash[24608]: cluster 2024-09-13T22:45:40.816818+0000 osd.2 (osd.2) 138 : cluster [DBG] 2.e scrub ok 2024-09-13T22:45:42.300 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:42.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:42.428 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:42.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:42.557 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:42.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:42.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:42.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:42.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 23 15 42159 42144 114.509 90.4375 0.0222664 0.00872706 2024-09-13T22:45:42.823 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:42.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:42.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:41.733315+0000 osd.7 (osd.7) 167 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:45:42.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:41.734740+0000 osd.7 (osd.7) 168 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:45:42.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:41.789045+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.e scrub starts 2024-09-13T22:45:42.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:42.954 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:42.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:42.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:43.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:43.081 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:41.791337+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.e scrub ok 2024-09-13T22:45:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:41.822122+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:45:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:41.824168+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:45:43.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.030000+0000 osd.5 (osd.5) 178 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.036007+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.598119+0000 mgr.y (mgr.14613) 300 : cluster [DBG] pgmap v306: 148 pgs: 148 active+clean; 2.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:41.733315+0000 osd.7 (osd.7) 167 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:41.734740+0000 osd.7 (osd.7) 168 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:41.789045+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.e scrub starts 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:41.791337+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.e scrub ok 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:41.822122+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:41.824168+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.030000+0000 osd.5 (osd.5) 178 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.036007+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:45:43.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:42 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.598119+0000 mgr.y (mgr.14613) 300 : cluster [DBG] pgmap v306: 148 pgs: 148 active+clean; 2.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-13T22:45:43.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:43.208 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:43.218 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:43.218 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:43.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:43.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:41.733315+0000 osd.7 (osd.7) 167 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:45:43.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:41.734740+0000 osd.7 (osd.7) 168 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:45:43.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:41.789045+0000 osd.4 (osd.4) 143 : cluster [DBG] 6.e scrub starts 2024-09-13T22:45:43.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:41.791337+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.e scrub ok 2024-09-13T22:45:43.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:41.822122+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:45:43.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:41.824168+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:45:43.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.030000+0000 osd.5 (osd.5) 178 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:45:43.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.036007+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:45:43.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:42 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.598119+0000 mgr.y (mgr.14613) 300 : cluster [DBG] pgmap v306: 148 pgs: 148 active+clean; 2.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-13T22:45:43.343 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:43.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:43.475 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:43.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:43.598 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:43.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:43.723 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:43.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 24 16 43147 43131 112.308 61.6875 0.00470659 0.0088982 2024-09-13T22:45:43.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:43.854 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:43.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:43.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:43 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.534592+0000 osd.3 (osd.3) 196 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:45:43.967 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:43 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.535989+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:45:43.968 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:43 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.826830+0000 osd.1 (osd.1) 166 : cluster [DBG] 2.d scrub starts 2024-09-13T22:45:43.968 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:43 smithi192 bash[24608]: cluster 2024-09-13T22:45:42.827977+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.d scrub ok 2024-09-13T22:45:43.987 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:43.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:44.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:44.109 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.534592+0000 osd.3 (osd.3) 196 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.535989+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.826830+0000 osd.1 (osd.1) 166 : cluster [DBG] 2.d scrub starts 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[29889]: cluster 2024-09-13T22:45:42.827977+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.d scrub ok 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.534592+0000 osd.3 (osd.3) 196 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.535989+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.826830+0000 osd.1 (osd.1) 166 : cluster [DBG] 2.d scrub starts 2024-09-13T22:45:44.133 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:43 smithi178 bash[22840]: cluster 2024-09-13T22:45:42.827977+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.d scrub ok 2024-09-13T22:45:44.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:44.237 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:44.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:44.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:44.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:44.488 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:44.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:44.620 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:44.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:44.745 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:44.812 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 25 15 44381 44366 110.903 77.1875 0.139796 0.00896942 2024-09-13T22:45:44.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:44.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:44.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:44.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:45.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:45.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:45.145 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:45.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:44 smithi192 bash[24608]: cluster 2024-09-13T22:45:43.806217+0000 osd.7 (osd.7) 169 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:45:45.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:44 smithi192 bash[24608]: cluster 2024-09-13T22:45:43.808439+0000 osd.7 (osd.7) 170 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:45:45.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:44 smithi192 bash[24608]: cluster 2024-09-13T22:45:43.820316+0000 osd.0 (osd.0) 175 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:45:45.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:44 smithi192 bash[24608]: cluster 2024-09-13T22:45:43.823104+0000 osd.0 (osd.0) 176 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:45:45.146 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:44 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.599392+0000 mgr.y (mgr.14613) 301 : cluster [DBG] pgmap v307: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 920 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:45:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:45.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:45.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[29889]: cluster 2024-09-13T22:45:43.806217+0000 osd.7 (osd.7) 169 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:45:45.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[29889]: cluster 2024-09-13T22:45:43.808439+0000 osd.7 (osd.7) 170 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:45:45.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[29889]: cluster 2024-09-13T22:45:43.820316+0000 osd.0 (osd.0) 175 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:45:45.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[29889]: cluster 2024-09-13T22:45:43.823104+0000 osd.0 (osd.0) 176 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:45:45.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.599392+0000 mgr.y (mgr.14613) 301 : cluster [DBG] pgmap v307: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 920 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:45:45.262 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[22840]: cluster 2024-09-13T22:45:43.806217+0000 osd.7 (osd.7) 169 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:45:45.262 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[22840]: cluster 2024-09-13T22:45:43.808439+0000 osd.7 (osd.7) 170 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:45:45.262 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[22840]: cluster 2024-09-13T22:45:43.820316+0000 osd.0 (osd.0) 175 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:45:45.262 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[22840]: cluster 2024-09-13T22:45:43.823104+0000 osd.0 (osd.0) 176 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:45:45.262 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:44 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.599392+0000 mgr.y (mgr.14613) 301 : cluster [DBG] pgmap v307: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 920 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:45:45.274 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:45.377 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:45.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:45.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:45:45.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:45.539 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:45.645 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:45.672 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:45.778 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:45.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:45.814 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 26 16 45353 45337 108.969 60.6875 0.0155484 0.00917184 2024-09-13T22:45:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:45.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:45.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:45.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:46.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:46.065 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:46.169 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:46.198 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:46.200 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.031437+0000 osd.6 (osd.6) 133 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:45:46.200 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.033120+0000 osd.6 (osd.6) 134 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:45:46.200 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.818794+0000 osd.0 (osd.0) 177 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.827795+0000 osd.0 (osd.0) 178 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.856004+0000 osd.0 (osd.0) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.856084+0000 osd.0 (osd.0) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:44.977665+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: audit 2024-09-13T22:45:45.181750+0000 mgr.y (mgr.14613) 302 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:45.783892+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:45:46.201 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[29889]: cluster 2024-09-13T22:45:45.786256+0000 osd.1 (osd.1) 169 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:45:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:46.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.031437+0000 osd.6 (osd.6) 133 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:45:46.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.033120+0000 osd.6 (osd.6) 134 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:45:46.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.818794+0000 osd.0 (osd.0) 177 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:45:46.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.827795+0000 osd.0 (osd.0) 178 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:45:46.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.856004+0000 osd.0 (osd.0) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:46.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.856084+0000 osd.0 (osd.0) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:46.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:44.977665+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-13T22:45:46.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: audit 2024-09-13T22:45:45.181750+0000 mgr.y (mgr.14613) 302 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:46.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:45.783892+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:45:46.203 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:45 smithi178 bash[22840]: cluster 2024-09-13T22:45:45.786256+0000 osd.1 (osd.1) 169 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:45:46.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:46.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.031437+0000 osd.6 (osd.6) 133 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:45:46.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.033120+0000 osd.6 (osd.6) 134 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:45:46.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.818794+0000 osd.0 (osd.0) 177 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:45:46.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.827795+0000 osd.0 (osd.0) 178 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:45:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.856004+0000 osd.0 (osd.0) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.856084+0000 osd.0 (osd.0) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:44.977665+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-13T22:45:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: audit 2024-09-13T22:45:45.181750+0000 mgr.y (mgr.14613) 302 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:45.783892+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:45:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:45 smithi192 bash[24608]: cluster 2024-09-13T22:45:45.786256+0000 osd.1 (osd.1) 169 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:45:46.333 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:46.439 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:46.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:46.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:46.600 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:46.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:46.728 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:46.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 27 16 46783 46767 108.244 89.375 0.00828015 0.00923299 2024-09-13T22:45:46.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:46.854 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:46.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:46.959 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:45:46.959 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:45:46.965 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:46 smithi192 bash[24608]: cluster 2024-09-13T22:45:45.951265+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:45:46.966 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:46 smithi192 bash[24608]: cluster 2024-09-13T22:45:45.973863+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:45:46.966 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:46 smithi192 bash[24608]: cluster 2024-09-13T22:45:46.599861+0000 mgr.y (mgr.14613) 303 : cluster [DBG] pgmap v309: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-13T22:45:46.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:46 smithi178 bash[29889]: cluster 2024-09-13T22:45:45.951265+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:45:46.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:46 smithi178 bash[29889]: cluster 2024-09-13T22:45:45.973863+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:45:46.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:46 smithi178 bash[29889]: cluster 2024-09-13T22:45:46.599861+0000 mgr.y (mgr.14613) 303 : cluster [DBG] pgmap v309: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-13T22:45:46.976 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:46 smithi178 bash[22840]: cluster 2024-09-13T22:45:45.951265+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:45:46.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:46 smithi178 bash[22840]: cluster 2024-09-13T22:45:45.973863+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:45:46.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:46 smithi178 bash[22840]: cluster 2024-09-13T22:45:46.599861+0000 mgr.y (mgr.14613) 303 : cluster [DBG] pgmap v309: 148 pgs: 148 active+clean; 2.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-13T22:45:46.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:46.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:47.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:47.110 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:47.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:47.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:47.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:47.367 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:47.471 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:47.496 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:47.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:47.625 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:47.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:47.752 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:47.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 28 16 47965 47949 107.016 73.875 0.00840779 0.00933927 2024-09-13T22:45:47.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:47.881 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:47.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:47.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:48.012 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:48.118 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:48.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:48.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:48.278 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:48.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:48.405 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:48.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:48.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:48.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:48.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:48.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:48.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:48.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 29 16 48982 48966 105.517 63.5625 0.00372697 0.00946579 2024-09-13T22:45:48.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:48.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:48.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:49.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:49.050 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:49.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:48 smithi192 bash[24608]: cluster 2024-09-13T22:45:47.009973+0000 osd.5 (osd.5) 180 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:45:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:48 smithi192 bash[24608]: cluster 2024-09-13T22:45:47.036078+0000 osd.5 (osd.5) 181 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:45:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:48 smithi192 bash[24608]: audit 2024-09-13T22:45:48.227730+0000 mon.b (mon.1) 139 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:49.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:48 smithi192 bash[24608]: cluster 2024-09-13T22:45:48.600367+0000 mgr.y (mgr.14613) 304 : cluster [DBG] pgmap v310: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-13T22:45:49.180 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:49.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:49.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:47.009973+0000 osd.5 (osd.5) 180 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:45:49.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:47.036078+0000 osd.5 (osd.5) 181 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:45:49.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: audit 2024-09-13T22:45:48.227730+0000 mon.b (mon.1) 139 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:49.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:48.600367+0000 mgr.y (mgr.14613) 304 : cluster [DBG] pgmap v310: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-13T22:45:49.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:47.009973+0000 osd.5 (osd.5) 180 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:45:49.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:47.036078+0000 osd.5 (osd.5) 181 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:45:49.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: audit 2024-09-13T22:45:48.227730+0000 mon.b (mon.1) 139 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:45:49.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:48.600367+0000 mgr.y (mgr.14613) 304 : cluster [DBG] pgmap v310: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-13T22:45:49.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:49.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:49.445 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:49.549 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:49.575 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:49.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:49.705 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:49.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:49.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 30 16 50237 50221 104.615 78.4375 0.0102353 0.00955356 2024-09-13T22:45:49.819 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:45:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:45:49.836 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:49.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:49.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:49.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:50.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:47.980225+0000 osd.5 (osd.5) 182 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:47.986133+0000 osd.5 (osd.5) 183 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:48.535625+0000 osd.3 (osd.3) 198 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[29889]: cluster 2024-09-13T22:45:48.536948+0000 osd.3 (osd.3) 199 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:47.980225+0000 osd.5 (osd.5) 182 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:47.986133+0000 osd.5 (osd.5) 183 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:48.535625+0000 osd.3 (osd.3) 198 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:45:50.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:49 smithi178 bash[22840]: cluster 2024-09-13T22:45:48.536948+0000 osd.3 (osd.3) 199 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:45:50.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:50.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:49 smithi192 bash[24608]: cluster 2024-09-13T22:45:47.980225+0000 osd.5 (osd.5) 182 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:45:50.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:49 smithi192 bash[24608]: cluster 2024-09-13T22:45:47.986133+0000 osd.5 (osd.5) 183 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:45:50.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:49 smithi192 bash[24608]: cluster 2024-09-13T22:45:48.535625+0000 osd.3 (osd.3) 198 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:45:50.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:49 smithi192 bash[24608]: cluster 2024-09-13T22:45:48.536948+0000 osd.3 (osd.3) 199 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:45:50.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:50.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T22:45:50.288 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:50.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:50.420 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:50.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:50.543 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:50.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:50.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:50.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:50.805 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:50.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 31 16 51460 51444 103.705 76.4375 0.00268814 0.00963574 2024-09-13T22:45:50.835 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:45:50.835 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:45:50.884 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 1 pg_num to 11 2024-09-13T22:45:50.886 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:45:50.887 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:45:50.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:50.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:48.987635+0000 osd.5 (osd.5) 184 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:45:50.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:48.989967+0000 osd.5 (osd.5) 185 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:45:50.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:49.826284+0000 osd.1 (osd.1) 170 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:45:50.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:49.832336+0000 osd.1 (osd.1) 171 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:45:50.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:49.858805+0000 osd.4 (osd.4) 145 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:45:50.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:49.861093+0000 osd.4 (osd.4) 146 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:45:50.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.601336+0000 mgr.y (mgr.14613) 305 : cluster [DBG] pgmap v311: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-13T22:45:50.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: audit 2024-09-13T22:45:50.748643+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.178:0/1874179776' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: audit 2024-09-13T22:45:50.749273+0000 mon.a (mon.0) 1105 : audit [INF] from='client.25030 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[29889]: audit 2024-09-13T22:45:50.803728+0000 mon.b (mon.1) 141 : audit [DBG] from='client.? v1:172.21.15.178:0/2811985494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:48.987635+0000 osd.5 (osd.5) 184 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:48.989967+0000 osd.5 (osd.5) 185 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:49.826284+0000 osd.1 (osd.1) 170 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:49.832336+0000 osd.1 (osd.1) 171 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:45:50.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:49.858805+0000 osd.4 (osd.4) 145 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:45:50.922 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:49.861093+0000 osd.4 (osd.4) 146 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:45:50.922 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.601336+0000 mgr.y (mgr.14613) 305 : cluster [DBG] pgmap v311: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-13T22:45:50.922 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: audit 2024-09-13T22:45:50.748643+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.178:0/1874179776' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-13T22:45:50.922 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: audit 2024-09-13T22:45:50.749273+0000 mon.a (mon.0) 1105 : audit [INF] from='client.25030 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-13T22:45:50.922 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:50 smithi178 bash[22840]: audit 2024-09-13T22:45:50.803728+0000 mon.b (mon.1) 141 : audit [DBG] from='client.? v1:172.21.15.178:0/2811985494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:50.928 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:45:50.928 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:45:50.947 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:50.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:50.982 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:45:50.982 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:45:51.020 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:45:51.021 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:45:51.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.060 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:45:51.060 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:45:51.076 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:51.093 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:45:51.094 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:45:51.128 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:45:51.128 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:45:51.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:48.987635+0000 osd.5 (osd.5) 184 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:45:51.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:48.989967+0000 osd.5 (osd.5) 185 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:45:51.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:49.826284+0000 osd.1 (osd.1) 170 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:49.832336+0000 osd.1 (osd.1) 171 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:49.858805+0000 osd.4 (osd.4) 145 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:49.861093+0000 osd.4 (osd.4) 146 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.601336+0000 mgr.y (mgr.14613) 305 : cluster [DBG] pgmap v311: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: audit 2024-09-13T22:45:50.748643+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.178:0/1874179776' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: audit 2024-09-13T22:45:50.749273+0000 mon.a (mon.0) 1105 : audit [INF] from='client.25030 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-13T22:45:51.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:50 smithi192 bash[24608]: audit 2024-09-13T22:45:50.803728+0000 mon.b (mon.1) 141 : audit [DBG] from='client.? v1:172.21.15.178:0/2811985494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:45:51.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:51.200 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:51.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:51.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:51.458 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:51.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.587 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:51.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.719 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:51.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 32 15 52595 52580 102.683 71 0.00485754 0.00973342 2024-09-13T22:45:51.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.856 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:51.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:51.897 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:45:51.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:51.981 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:51.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:52.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:52.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.014303+0000 osd.5 (osd.5) 186 : cluster [DBG] 3.c scrub starts 2024-09-13T22:45:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.017440+0000 osd.5 (osd.5) 187 : cluster [DBG] 3.c scrub ok 2024-09-13T22:45:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: audit 2024-09-13T22:45:50.880761+0000 mon.a (mon.0) 1106 : audit [INF] from='client.25030 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T22:45:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.894506+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-13T22:45:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.902225+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:45:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.912190+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:45:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: audit 2024-09-13T22:45:51.201642+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.178:0/1652267724' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:51 smithi192 bash[24608]: audit 2024-09-13T22:45:51.202244+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25042 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:52.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.014303+0000 osd.5 (osd.5) 186 : cluster [DBG] 3.c scrub starts 2024-09-13T22:45:52.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.017440+0000 osd.5 (osd.5) 187 : cluster [DBG] 3.c scrub ok 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: audit 2024-09-13T22:45:50.880761+0000 mon.a (mon.0) 1106 : audit [INF] from='client.25030 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.894506+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.902225+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.912190+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: audit 2024-09-13T22:45:51.201642+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.178:0/1652267724' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[29889]: audit 2024-09-13T22:45:51.202244+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25042 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.014303+0000 osd.5 (osd.5) 186 : cluster [DBG] 3.c scrub starts 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.017440+0000 osd.5 (osd.5) 187 : cluster [DBG] 3.c scrub ok 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: audit 2024-09-13T22:45:50.880761+0000 mon.a (mon.0) 1106 : audit [INF] from='client.25030 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T22:45:52.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.894506+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-13T22:45:52.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.902225+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:45:52.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.912190+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:45:52.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: audit 2024-09-13T22:45:51.201642+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.178:0/1652267724' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:52.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:51 smithi178 bash[22840]: audit 2024-09-13T22:45:51.202244+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25042 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:45:52.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:52.251 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:52.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:52.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:52.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:52.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:52.645 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:52.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:52.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:52.813 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 33 16 53907 53891 102.054 81.9375 0.00353142 0.00979413 2024-09-13T22:45:52.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:52.910 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:52.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:53.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:53.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:53.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:50.999084+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.d scrub starts 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.001672+0000 osd.5 (osd.5) 189 : cluster [DBG] 6.d scrub ok 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.537174+0000 osd.3 (osd.3) 200 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.542366+0000 osd.3 (osd.3) 201 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.845046+0000 osd.1 (osd.1) 172 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.851556+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.881004+0000 mon.a (mon.0) 1109 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.881031+0000 mon.a (mon.0) 1110 : cluster [INF] Cluster is now healthy 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: audit 2024-09-13T22:45:51.891430+0000 mon.a (mon.0) 1111 : audit [INF] from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.896348+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.935025+0000 osd.4 (osd.4) 149 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.936001+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:45:53.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.954522+0000 osd.0 (osd.0) 183 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:45:53.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:51.960766+0000 osd.0 (osd.0) 184 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:45:53.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:52.601732+0000 mgr.y (mgr.14613) 306 : cluster [DBG] pgmap v314: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 667 B/s rd, 47 MiB/s wr, 756 op/s 2024-09-13T22:45:53.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: audit 2024-09-13T22:45:52.602821+0000 mon.b (mon.1) 143 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:53.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: audit 2024-09-13T22:45:52.605303+0000 mon.a (mon.0) 1113 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:53.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: audit 2024-09-13T22:45:52.848971+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T22:45:53.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:52 smithi192 bash[24608]: cluster 2024-09-13T22:45:52.852096+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-13T22:45:53.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:53.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:50.999084+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.d scrub starts 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.001672+0000 osd.5 (osd.5) 189 : cluster [DBG] 6.d scrub ok 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.537174+0000 osd.3 (osd.3) 200 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.542366+0000 osd.3 (osd.3) 201 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.845046+0000 osd.1 (osd.1) 172 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.851556+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.881004+0000 mon.a (mon.0) 1109 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.881031+0000 mon.a (mon.0) 1110 : cluster [INF] Cluster is now healthy 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: audit 2024-09-13T22:45:51.891430+0000 mon.a (mon.0) 1111 : audit [INF] from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.896348+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.935025+0000 osd.4 (osd.4) 149 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.936001+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:45:53.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.954522+0000 osd.0 (osd.0) 183 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:51.960766+0000 osd.0 (osd.0) 184 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:52.601732+0000 mgr.y (mgr.14613) 306 : cluster [DBG] pgmap v314: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 667 B/s rd, 47 MiB/s wr, 756 op/s 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: audit 2024-09-13T22:45:52.602821+0000 mon.b (mon.1) 143 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: audit 2024-09-13T22:45:52.605303+0000 mon.a (mon.0) 1113 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: audit 2024-09-13T22:45:52.848971+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[29889]: cluster 2024-09-13T22:45:52.852096+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:50.999084+0000 osd.5 (osd.5) 188 : cluster [DBG] 6.d scrub starts 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.001672+0000 osd.5 (osd.5) 189 : cluster [DBG] 6.d scrub ok 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.537174+0000 osd.3 (osd.3) 200 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.542366+0000 osd.3 (osd.3) 201 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:45:53.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.845046+0000 osd.1 (osd.1) 172 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.851556+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.881004+0000 mon.a (mon.0) 1109 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.881031+0000 mon.a (mon.0) 1110 : cluster [INF] Cluster is now healthy 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: audit 2024-09-13T22:45:51.891430+0000 mon.a (mon.0) 1111 : audit [INF] from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.896348+0000 mon.a (mon.0) 1112 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.935025+0000 osd.4 (osd.4) 149 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.936001+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.954522+0000 osd.0 (osd.0) 183 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:51.960766+0000 osd.0 (osd.0) 184 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:52.601732+0000 mgr.y (mgr.14613) 306 : cluster [DBG] pgmap v314: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 667 B/s rd, 47 MiB/s wr, 756 op/s 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: audit 2024-09-13T22:45:52.602821+0000 mon.b (mon.1) 143 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: audit 2024-09-13T22:45:52.605303+0000 mon.a (mon.0) 1113 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:53.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: audit 2024-09-13T22:45:52.848971+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T22:45:53.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:52 smithi178 bash[22840]: cluster 2024-09-13T22:45:52.852096+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-13T22:45:53.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:53.295 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:53.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:53.424 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:53.526 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:53.549 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:53.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:53.686 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:53.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:53.812 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:53.814 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 34 16 55038 55022 101.13 70.6875 0.0123052 0.00988234 2024-09-13T22:45:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:53.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:53.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:53.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:54.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:54.071 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:54.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:53 smithi192 bash[24608]: cluster 2024-09-13T22:45:52.912524+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:45:54.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:53 smithi192 bash[24608]: cluster 2024-09-13T22:45:52.914939+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:45:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:53 smithi192 bash[24608]: cluster 2024-09-13T22:45:53.857329+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-13T22:45:54.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:54.198 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:54.200 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:53 smithi178 bash[29889]: cluster 2024-09-13T22:45:52.912524+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:45:54.200 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:53 smithi178 bash[29889]: cluster 2024-09-13T22:45:52.914939+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:45:54.200 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:53 smithi178 bash[29889]: cluster 2024-09-13T22:45:53.857329+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-13T22:45:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:54.201 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:53 smithi178 bash[22840]: cluster 2024-09-13T22:45:52.912524+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:45:54.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:53 smithi178 bash[22840]: cluster 2024-09-13T22:45:52.914939+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:45:54.202 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:53 smithi178 bash[22840]: cluster 2024-09-13T22:45:53.857329+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-13T22:45:54.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:54.335 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:54.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:54.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:54.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:54.598 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:54.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:54.724 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:54.814 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 35 16 56425 56409 100.717 86.6875 0.00326181 0.00992232 2024-09-13T22:45:54.830 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:54.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:54.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:54.979 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:54.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:54 smithi192 bash[24608]: cluster 2024-09-13T22:45:53.956373+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T22:45:54.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:54 smithi192 bash[24608]: cluster 2024-09-13T22:45:53.957619+0000 osd.4 (osd.4) 154 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T22:45:54.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:54 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.602400+0000 mgr.y (mgr.14613) 307 : cluster [DBG] pgmap v317: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 60 MiB/s wr, 965 op/s 2024-09-13T22:45:54.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:54 smithi192 bash[24608]: audit 2024-09-13T22:45:54.603470+0000 mon.b (mon.1) 144 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:54.981 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:54 smithi192 bash[24608]: audit 2024-09-13T22:45:54.603925+0000 mon.a (mon.0) 1117 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:54.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:55.083 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:55.108 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:55.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[29889]: cluster 2024-09-13T22:45:53.956373+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T22:45:55.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[29889]: cluster 2024-09-13T22:45:53.957619+0000 osd.4 (osd.4) 154 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T22:45:55.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.602400+0000 mgr.y (mgr.14613) 307 : cluster [DBG] pgmap v317: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 60 MiB/s wr, 965 op/s 2024-09-13T22:45:55.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[29889]: audit 2024-09-13T22:45:54.603470+0000 mon.b (mon.1) 144 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:55.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[29889]: audit 2024-09-13T22:45:54.603925+0000 mon.a (mon.0) 1117 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:55.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[22840]: cluster 2024-09-13T22:45:53.956373+0000 osd.4 (osd.4) 153 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T22:45:55.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[22840]: cluster 2024-09-13T22:45:53.957619+0000 osd.4 (osd.4) 154 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T22:45:55.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.602400+0000 mgr.y (mgr.14613) 307 : cluster [DBG] pgmap v317: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 60 MiB/s wr, 965 op/s 2024-09-13T22:45:55.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[22840]: audit 2024-09-13T22:45:54.603470+0000 mon.b (mon.1) 144 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:55.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:54 smithi178 bash[22840]: audit 2024-09-13T22:45:54.603925+0000 mon.a (mon.0) 1117 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-13T22:45:55.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:55.234 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:55.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:45:55.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:55.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:55.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:55.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:55.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:55.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:55.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:55.748 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:55.814 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 36 16 57724 57708 100.174 81.1875 0.00672708 0.00997639 2024-09-13T22:45:55.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:55.884 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:55.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:55.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:56.013 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:56.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.479640+0000 osd.3 (osd.3) 202 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:45:56.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.489081+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:45:56.015 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.757125+0000 osd.7 (osd.7) 171 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.759991+0000 osd.7 (osd.7) 172 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.811612+0000 osd.4 (osd.4) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.811663+0000 osd.4 (osd.4) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: audit 2024-09-13T22:45:54.900292+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.905483+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.983356+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:54.988983+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: audit 2024-09-13T22:45:55.189511+0000 mgr.y (mgr.14613) 308 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.764632+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:45:56.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:55 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.765450+0000 osd.1 (osd.1) 175 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:45:56.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:56.139 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.479640+0000 osd.3 (osd.3) 202 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.489081+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.757125+0000 osd.7 (osd.7) 171 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.759991+0000 osd.7 (osd.7) 172 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.811612+0000 osd.4 (osd.4) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.811663+0000 osd.4 (osd.4) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: audit 2024-09-13T22:45:54.900292+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.905483+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.983356+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:54.988983+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:45:56.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: audit 2024-09-13T22:45:55.189511+0000 mgr.y (mgr.14613) 308 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.764632+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.765450+0000 osd.1 (osd.1) 175 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.479640+0000 osd.3 (osd.3) 202 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.489081+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.757125+0000 osd.7 (osd.7) 171 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.759991+0000 osd.7 (osd.7) 172 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.811612+0000 osd.4 (osd.4) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.811663+0000 osd.4 (osd.4) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: audit 2024-09-13T22:45:54.900292+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.905483+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.983356+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:45:56.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:54.988983+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:45:56.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: audit 2024-09-13T22:45:55.189511+0000 mgr.y (mgr.14613) 308 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:45:56.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.764632+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:45:56.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:55 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.765450+0000 osd.1 (osd.1) 175 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:45:56.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:56.279 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:56.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:56.416 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:56.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:56.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:56.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:56.682 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:56.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:56.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:56.816 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 37 16 59113 59097 99.8078 86.8125 0.0212034 0.010013 2024-09-13T22:45:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:56.920 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:56.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:56.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:57.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:57.084 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:57.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.786873+0000 osd.7 (osd.7) 173 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:45:57.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.788717+0000 osd.7 (osd.7) 174 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:45:57.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.896892+0000 osd.5 (osd.5) 190 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:45:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.899148+0000 osd.5 (osd.5) 191 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:45:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[29889]: cluster 2024-09-13T22:45:55.932225+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-13T22:45:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[29889]: cluster 2024-09-13T22:45:56.602931+0000 mgr.y (mgr.14613) 309 : cluster [DBG] pgmap v320: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:45:57.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.786873+0000 osd.7 (osd.7) 173 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:45:57.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.788717+0000 osd.7 (osd.7) 174 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:45:57.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.896892+0000 osd.5 (osd.5) 190 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:45:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.899148+0000 osd.5 (osd.5) 191 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:45:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[22840]: cluster 2024-09-13T22:45:55.932225+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-13T22:45:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:56 smithi178 bash[22840]: cluster 2024-09-13T22:45:56.602931+0000 mgr.y (mgr.14613) 309 : cluster [DBG] pgmap v320: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:45:57.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:57.197 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:56 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.786873+0000 osd.7 (osd.7) 173 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:45:57.197 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:56 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.788717+0000 osd.7 (osd.7) 174 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:45:57.197 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:56 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.896892+0000 osd.5 (osd.5) 190 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:45:57.197 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:56 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.899148+0000 osd.5 (osd.5) 191 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:45:57.197 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:56 smithi192 bash[24608]: cluster 2024-09-13T22:45:55.932225+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-13T22:45:57.197 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:56 smithi192 bash[24608]: cluster 2024-09-13T22:45:56.602931+0000 mgr.y (mgr.14613) 309 : cluster [DBG] pgmap v320: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:45:57.214 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:57.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:57.343 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:57.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:45:57.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:45:57.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:57.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:57.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:57.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:57.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:57.727 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:57.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 38 16 60357 60341 99.2272 77.75 0.0244246 0.0100713 2024-09-13T22:45:57.831 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:57.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:57.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:57.991 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:57.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:58.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:58.115 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:45:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[22840]: cluster 2024-09-13T22:45:56.840377+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[22840]: cluster 2024-09-13T22:45:56.842802+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[22840]: cluster 2024-09-13T22:45:56.930790+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[22840]: cluster 2024-09-13T22:45:57.853546+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[29889]: cluster 2024-09-13T22:45:56.840377+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[29889]: cluster 2024-09-13T22:45:56.842802+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[29889]: cluster 2024-09-13T22:45:56.930790+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-13T22:45:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:57 smithi178 bash[29889]: cluster 2024-09-13T22:45:57.853546+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-13T22:45:58.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:58.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:57 smithi192 bash[24608]: cluster 2024-09-13T22:45:56.840377+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:45:58.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:57 smithi192 bash[24608]: cluster 2024-09-13T22:45:56.842802+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:45:58.228 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:57 smithi192 bash[24608]: cluster 2024-09-13T22:45:56.930790+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-13T22:45:58.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:57 smithi192 bash[24608]: cluster 2024-09-13T22:45:57.853546+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-13T22:45:58.250 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:58.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:58.385 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:45:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:45:58.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:58.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:45:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:45:58.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:58.655 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:58.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:58.783 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:58.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 39 16 61683 61667 98.8077 82.875 0.0067826 0.0101158 2024-09-13T22:45:58.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:58.891 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:45:58.907 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:45:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:58.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:45:59.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:59.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:57.926678+0000 osd.5 (osd.5) 192 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:57.927900+0000 osd.5 (osd.5) 193 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:57.974874+0000 osd.2 (osd.2) 141 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:57.976297+0000 osd.2 (osd.2) 142 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:58.014276+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:58.015175+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:58.603471+0000 mgr.y (mgr.14613) 310 : cluster [DBG] pgmap v323: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 573 op/s 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:58.854292+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-13T22:45:59.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:58 smithi192 bash[24608]: cluster 2024-09-13T22:45:58.889582+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:45:59.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:59.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:59.177 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:59.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:59.292 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:57.926678+0000 osd.5 (osd.5) 192 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:45:59.292 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:57.927900+0000 osd.5 (osd.5) 193 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:45:59.292 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:57.974874+0000 osd.2 (osd.2) 141 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:45:59.292 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:57.976297+0000 osd.2 (osd.2) 142 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:58.014276+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:58.015175+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:58.603471+0000 mgr.y (mgr.14613) 310 : cluster [DBG] pgmap v323: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 573 op/s 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:58.854292+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[29889]: cluster 2024-09-13T22:45:58.889582+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:57.926678+0000 osd.5 (osd.5) 192 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:57.927900+0000 osd.5 (osd.5) 193 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:45:59.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:57.974874+0000 osd.2 (osd.2) 141 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:45:59.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:57.976297+0000 osd.2 (osd.2) 142 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:45:59.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:58.014276+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:45:59.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:58.015175+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:45:59.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:58.603471+0000 mgr.y (mgr.14613) 310 : cluster [DBG] pgmap v323: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 573 op/s 2024-09-13T22:45:59.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:58.854292+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-13T22:45:59.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:58 smithi178 bash[22840]: cluster 2024-09-13T22:45:58.889582+0000 osd.5 (osd.5) 194 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:45:59.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:45:59.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:59.451 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:59.556 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:59.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:45:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:45:59.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:59.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:45:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:45:59.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:59.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:45:59.820732+0000 min lat: 0.00124784 max lat: 0.701883 avg lat: 0.010163 2024-09-13T22:45:59.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:45:59.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 40 16 62970 62954 98.3482 80.4375 0.0247178 0.010163 2024-09-13T22:45:59.840 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:45:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:45:59.928 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:45:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:45:59.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:59.976 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:45:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:45:59.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:00.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:00.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:58.899453+0000 osd.5 (osd.5) 195 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:46:00.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.044977+0000 osd.4 (osd.4) 159 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:46:00.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.049664+0000 osd.4 (osd.4) 160 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:46:00.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.108169+0000 osd.6 (osd.6) 135 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.111647+0000 osd.6 (osd.6) 136 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.354058+0000 osd.3 (osd.3) 204 : cluster [DBG] 1.4 deep-scrub starts 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.360623+0000 osd.3 (osd.3) 205 : cluster [DBG] 1.4 deep-scrub ok 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.796745+0000 osd.1 (osd.1) 176 : cluster [DBG] 1.3 deep-scrub starts 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.802222+0000 osd.1 (osd.1) 177 : cluster [DBG] 1.3 deep-scrub ok 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.888024+0000 osd.5 (osd.5) 196 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:46:00.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:45:59 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.889130+0000 osd.5 (osd.5) 197 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:46:00.103 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:00.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:58.899453+0000 osd.5 (osd.5) 195 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:46:00.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.044977+0000 osd.4 (osd.4) 159 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.049664+0000 osd.4 (osd.4) 160 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.108169+0000 osd.6 (osd.6) 135 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.111647+0000 osd.6 (osd.6) 136 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.354058+0000 osd.3 (osd.3) 204 : cluster [DBG] 1.4 deep-scrub starts 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.360623+0000 osd.3 (osd.3) 205 : cluster [DBG] 1.4 deep-scrub ok 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.796745+0000 osd.1 (osd.1) 176 : cluster [DBG] 1.3 deep-scrub starts 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.802222+0000 osd.1 (osd.1) 177 : cluster [DBG] 1.3 deep-scrub ok 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.888024+0000 osd.5 (osd.5) 196 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:46:00.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.889130+0000 osd.5 (osd.5) 197 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:58.899453+0000 osd.5 (osd.5) 195 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.044977+0000 osd.4 (osd.4) 159 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.049664+0000 osd.4 (osd.4) 160 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.108169+0000 osd.6 (osd.6) 135 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.111647+0000 osd.6 (osd.6) 136 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.354058+0000 osd.3 (osd.3) 204 : cluster [DBG] 1.4 deep-scrub starts 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.360623+0000 osd.3 (osd.3) 205 : cluster [DBG] 1.4 deep-scrub ok 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.796745+0000 osd.1 (osd.1) 176 : cluster [DBG] 1.3 deep-scrub starts 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.802222+0000 osd.1 (osd.1) 177 : cluster [DBG] 1.3 deep-scrub ok 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.888024+0000 osd.5 (osd.5) 196 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:46:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:45:59 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.889130+0000 osd.5 (osd.5) 197 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:46:00.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:00.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:00.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:46:00.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:46:00.262 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T22:46:00.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T22:46:00.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:00.362 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:00.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:00.489 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:00.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:00.614 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:00.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:00.742 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:00.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 41 16 63807 63791 97.2253 52.3125 0.00255729 0.0102086 2024-09-13T22:46:00.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:00.870 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:00.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:00.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.003 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:01.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.821860+0000 osd.7 (osd.7) 177 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:46:01.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:45:59.823693+0000 osd.7 (osd.7) 178 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:46:01.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.096208+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:46:01.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.102986+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:46:01.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.604078+0000 mgr.y (mgr.14613) 311 : cluster [DBG] pgmap v325: 158 pgs: 8 peering, 150 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T22:46:01.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.828765+0000 osd.1 (osd.1) 178 : cluster [DBG] 5.12 deep-scrub starts 2024-09-13T22:46:01.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.830734+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.12 deep-scrub ok 2024-09-13T22:46:01.117 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:00 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.891291+0000 osd.5 (osd.5) 198 : cluster [DBG] 1.a scrub starts 2024-09-13T22:46:01.130 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:01.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.821860+0000 osd.7 (osd.7) 177 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:45:59.823693+0000 osd.7 (osd.7) 178 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.096208+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.102986+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.604078+0000 mgr.y (mgr.14613) 311 : cluster [DBG] pgmap v325: 158 pgs: 8 peering, 150 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.828765+0000 osd.1 (osd.1) 178 : cluster [DBG] 5.12 deep-scrub starts 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.830734+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.12 deep-scrub ok 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.891291+0000 osd.5 (osd.5) 198 : cluster [DBG] 1.a scrub starts 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.821860+0000 osd.7 (osd.7) 177 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:45:59.823693+0000 osd.7 (osd.7) 178 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:46:01.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.096208+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:46:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.102986+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:46:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.604078+0000 mgr.y (mgr.14613) 311 : cluster [DBG] pgmap v325: 158 pgs: 8 peering, 150 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T22:46:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.828765+0000 osd.1 (osd.1) 178 : cluster [DBG] 5.12 deep-scrub starts 2024-09-13T22:46:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.830734+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.12 deep-scrub ok 2024-09-13T22:46:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:00 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.891291+0000 osd.5 (osd.5) 198 : cluster [DBG] 1.a scrub starts 2024-09-13T22:46:01.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:01.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:01.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:01.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.526 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:01.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:01.656 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:01.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:01.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:01.792 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:01.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 42 16 65027 65011 96.7257 76.25 0.00785833 0.0103332 2024-09-13T22:46:01.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:01.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:01.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:01.984 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:01.984 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:02.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.836653+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.838795+0000 osd.7 (osd.7) 180 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[29889]: cluster 2024-09-13T22:46:00.895606+0000 osd.5 (osd.5) 199 : cluster [DBG] 1.a scrub ok 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[29889]: cluster 2024-09-13T22:46:01.014928+0000 osd.4 (osd.4) 161 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[29889]: cluster 2024-09-13T22:46:01.019429+0000 osd.4 (osd.4) 162 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.836653+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:46:02.035 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.838795+0000 osd.7 (osd.7) 180 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:46:02.036 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[22840]: cluster 2024-09-13T22:46:00.895606+0000 osd.5 (osd.5) 199 : cluster [DBG] 1.a scrub ok 2024-09-13T22:46:02.036 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[22840]: cluster 2024-09-13T22:46:01.014928+0000 osd.4 (osd.4) 161 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:46:02.036 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:01 smithi178 bash[22840]: cluster 2024-09-13T22:46:01.019429+0000 osd.4 (osd.4) 162 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:46:02.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:02.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:02.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:01 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.836653+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:46:02.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:01 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.838795+0000 osd.7 (osd.7) 180 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:46:02.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:01 smithi192 bash[24608]: cluster 2024-09-13T22:46:00.895606+0000 osd.5 (osd.5) 199 : cluster [DBG] 1.a scrub ok 2024-09-13T22:46:02.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:01 smithi192 bash[24608]: cluster 2024-09-13T22:46:01.014928+0000 osd.4 (osd.4) 161 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:46:02.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:01 smithi192 bash[24608]: cluster 2024-09-13T22:46:01.019429+0000 osd.4 (osd.4) 162 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:46:02.177 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:02.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:02.305 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:02.408 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:02.426 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:02.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:02.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:02.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:02.687 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:02.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:02.814 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:02.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 43 16 66088 66072 96.0182 66.3125 0.00902335 0.0104087 2024-09-13T22:46:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:02.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:02.942 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:02.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:02.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:03.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:03.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:02 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.054030+0000 osd.4 (osd.4) 163 : cluster [DBG] 2.f scrub starts 2024-09-13T22:46:03.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:02 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.057006+0000 osd.4 (osd.4) 164 : cluster [DBG] 2.f scrub ok 2024-09-13T22:46:03.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:02 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.604400+0000 mgr.y (mgr.14613) 312 : cluster [DBG] pgmap v326: 158 pgs: 8 peering, 150 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T22:46:03.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:02 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.797672+0000 osd.5 (osd.5) 200 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:46:03.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:02 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.798634+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:46:03.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:02 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.855817+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-13T22:46:03.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:03.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:03.216 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:03.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.054030+0000 osd.4 (osd.4) 163 : cluster [DBG] 2.f scrub starts 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.057006+0000 osd.4 (osd.4) 164 : cluster [DBG] 2.f scrub ok 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.604400+0000 mgr.y (mgr.14613) 312 : cluster [DBG] pgmap v326: 158 pgs: 8 peering, 150 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.797672+0000 osd.5 (osd.5) 200 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.798634+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.855817+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:02 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.054030+0000 osd.4 (osd.4) 163 : cluster [DBG] 2.f scrub starts 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:02 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.057006+0000 osd.4 (osd.4) 164 : cluster [DBG] 2.f scrub ok 2024-09-13T22:46:03.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:02 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.604400+0000 mgr.y (mgr.14613) 312 : cluster [DBG] pgmap v326: 158 pgs: 8 peering, 150 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T22:46:03.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:02 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.797672+0000 osd.5 (osd.5) 200 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:46:03.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.798634+0000 osd.5 (osd.5) 201 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:46:03.329 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:03 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.855817+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-13T22:46:03.338 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:03.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:03.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:03.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:03.593 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:03.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:03.730 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:03.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 44 16 67274 67258 95.5206 74.125 0.0400917 0.0104594 2024-09-13T22:46:03.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:03.858 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:03.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:03.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:03.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:04.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:04.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:04.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:04.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:04.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:04.388 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:04.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.792928+0000 osd.7 (osd.7) 181 : cluster [DBG] 5.f deep-scrub starts 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.793954+0000 osd.7 (osd.7) 182 : cluster [DBG] 5.f deep-scrub ok 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.903788+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:02.912641+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: audit 2024-09-13T22:46:03.249935+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: audit 2024-09-13T22:46:03.255680+0000 mon.b (mon.1) 145 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.806942+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.807732+0000 osd.5 (osd.5) 203 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.845392+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.15 deep-scrub starts 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.849274+0000 osd.1 (osd.1) 181 : cluster [DBG] 2.15 deep-scrub ok 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.792928+0000 osd.7 (osd.7) 181 : cluster [DBG] 5.f deep-scrub starts 2024-09-13T22:46:04.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.793954+0000 osd.7 (osd.7) 182 : cluster [DBG] 5.f deep-scrub ok 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.903788+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:02.912641+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: audit 2024-09-13T22:46:03.249935+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: audit 2024-09-13T22:46:03.255680+0000 mon.b (mon.1) 145 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.806942+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.807732+0000 osd.5 (osd.5) 203 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.845392+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.15 deep-scrub starts 2024-09-13T22:46:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:04 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.849274+0000 osd.1 (osd.1) 181 : cluster [DBG] 2.15 deep-scrub ok 2024-09-13T22:46:04.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:04.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.792928+0000 osd.7 (osd.7) 181 : cluster [DBG] 5.f deep-scrub starts 2024-09-13T22:46:04.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.793954+0000 osd.7 (osd.7) 182 : cluster [DBG] 5.f deep-scrub ok 2024-09-13T22:46:04.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.903788+0000 osd.0 (osd.0) 187 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:46:04.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:02.912641+0000 osd.0 (osd.0) 188 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:46:04.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: audit 2024-09-13T22:46:03.249935+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:04.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: audit 2024-09-13T22:46:03.255680+0000 mon.b (mon.1) 145 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:04.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.806942+0000 osd.5 (osd.5) 202 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:46:04.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.807732+0000 osd.5 (osd.5) 203 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:46:04.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.845392+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.15 deep-scrub starts 2024-09-13T22:46:04.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:04 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.849274+0000 osd.1 (osd.1) 181 : cluster [DBG] 2.15 deep-scrub ok 2024-09-13T22:46:04.510 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:04.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:04.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:04.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:04.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:04.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:04.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 45 16 68194 68178 94.6757 57.5 0.0249888 0.0105564 2024-09-13T22:46:04.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:04.897 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:04.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:05.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:05.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:05.032 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:05.066 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:46:05.066 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:46:05.066 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:46:05.066 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:46:05.066 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:46:05.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:05.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:05.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.281 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:05.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:05.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:05.399 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.796005+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.796872+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.995981+0000 osd.2 (osd.2) 143 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:03.996950+0000 osd.2 (osd.2) 144 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.033947+0000 osd.4 (osd.4) 165 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.039931+0000 osd.4 (osd.4) 166 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.043772+0000 osd.6 (osd.6) 139 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.047205+0000 osd.6 (osd.6) 140 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.605684+0000 mgr.y (mgr.14613) 313 : cluster [DBG] pgmap v328: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.47k op/s 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.819716+0000 osd.5 (osd.5) 204 : cluster [DBG] 6.a scrub starts 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[22840]: cluster 2024-09-13T22:46:04.822085+0000 osd.5 (osd.5) 205 : cluster [DBG] 6.a scrub ok 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.796005+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:46:05.400 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.796872+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.995981+0000 osd.2 (osd.2) 143 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:03.996950+0000 osd.2 (osd.2) 144 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.033947+0000 osd.4 (osd.4) 165 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.039931+0000 osd.4 (osd.4) 166 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.043772+0000 osd.6 (osd.6) 139 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.047205+0000 osd.6 (osd.6) 140 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.605684+0000 mgr.y (mgr.14613) 313 : cluster [DBG] pgmap v328: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.47k op/s 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.819716+0000 osd.5 (osd.5) 204 : cluster [DBG] 6.a scrub starts 2024-09-13T22:46:05.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:05 smithi178 bash[29889]: cluster 2024-09-13T22:46:04.822085+0000 osd.5 (osd.5) 205 : cluster [DBG] 6.a scrub ok 2024-09-13T22:46:05.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:46:05.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.796005+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.796872+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.995981+0000 osd.2 (osd.2) 143 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:03.996950+0000 osd.2 (osd.2) 144 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.033947+0000 osd.4 (osd.4) 165 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.039931+0000 osd.4 (osd.4) 166 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.043772+0000 osd.6 (osd.6) 139 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.047205+0000 osd.6 (osd.6) 140 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.605684+0000 mgr.y (mgr.14613) 313 : cluster [DBG] pgmap v328: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.47k op/s 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.819716+0000 osd.5 (osd.5) 204 : cluster [DBG] 6.a scrub starts 2024-09-13T22:46:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:05 smithi192 bash[24608]: cluster 2024-09-13T22:46:04.822085+0000 osd.5 (osd.5) 205 : cluster [DBG] 6.a scrub ok 2024-09-13T22:46:05.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:05.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:05.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:05.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:05.667 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:05.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:05.797 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:05.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 46 16 69353 69337 94.1921 72.4375 0.00721935 0.0106114 2024-09-13T22:46:05.902 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:05.932 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:05.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:05.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:06.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:06.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:06.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:06.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:06.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:06.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:06 smithi178 bash[22840]: audit 2024-09-13T22:46:05.197392+0000 mgr.y (mgr.14613) 314 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:06.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:06 smithi178 bash[22840]: cluster 2024-09-13T22:46:05.837076+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:46:06.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:06 smithi178 bash[22840]: cluster 2024-09-13T22:46:05.838782+0000 osd.1 (osd.1) 183 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:46:06.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:06 smithi178 bash[29889]: audit 2024-09-13T22:46:05.197392+0000 mgr.y (mgr.14613) 314 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:06.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:06 smithi178 bash[29889]: cluster 2024-09-13T22:46:05.837076+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:46:06.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:06 smithi178 bash[29889]: cluster 2024-09-13T22:46:05.838782+0000 osd.1 (osd.1) 183 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:46:06.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:06.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:06.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:06 smithi192 bash[24608]: audit 2024-09-13T22:46:05.197392+0000 mgr.y (mgr.14613) 314 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:06.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:06 smithi192 bash[24608]: cluster 2024-09-13T22:46:05.837076+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:46:06.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:06 smithi192 bash[24608]: cluster 2024-09-13T22:46:05.838782+0000 osd.1 (osd.1) 183 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:46:06.455 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:06.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:06.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:06.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:06.709 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:06.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:06.817 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 47 16 70227 70211 93.3502 54.625 0.00866364 0.0107072 2024-09-13T22:46:06.840 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:06.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:06.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:06.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:07.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:07.110 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:07.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:07.338 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:07.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:07 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.606487+0000 mgr.y (mgr.14613) 315 : cluster [DBG] pgmap v329: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 94 MiB/s wr, 1.49k op/s 2024-09-13T22:46:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:07 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.746748+0000 osd.5 (osd.5) 206 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:46:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:07 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.750389+0000 osd.5 (osd.5) 207 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:46:07.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:07 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.606487+0000 mgr.y (mgr.14613) 315 : cluster [DBG] pgmap v329: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 94 MiB/s wr, 1.49k op/s 2024-09-13T22:46:07.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:07 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.746748+0000 osd.5 (osd.5) 206 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:46:07.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:07 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.750389+0000 osd.5 (osd.5) 207 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:46:07.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:07.463 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:07.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:07.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:07 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.606487+0000 mgr.y (mgr.14613) 315 : cluster [DBG] pgmap v329: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 94 MiB/s wr, 1.49k op/s 2024-09-13T22:46:07.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:07 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.746748+0000 osd.5 (osd.5) 206 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:46:07.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:07 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.750389+0000 osd.5 (osd.5) 207 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:46:07.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:07.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:07.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:07.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 48 16 71093 71077 92.533 54.125 0.003386 0.0108005 2024-09-13T22:46:07.839 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:07.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:07.962 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:07.990 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:07.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:08.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:08.121 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:08.224 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:08.251 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:08.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:08.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:08.381 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:08.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.696007+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:46:08.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.698340+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:46:08.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.951435+0000 osd.0 (osd.0) 189 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:46:08.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:06.958938+0000 osd.0 (osd.0) 190 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:46:08.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:07.016424+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.1 deep-scrub starts 2024-09-13T22:46:08.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:07.018482+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.1 deep-scrub ok 2024-09-13T22:46:08.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:07.795263+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:46:08.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:08 smithi192 bash[24608]: cluster 2024-09-13T22:46:07.796840+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:46:08.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:08.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.696007+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.698340+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.951435+0000 osd.0 (osd.0) 189 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:06.958938+0000 osd.0 (osd.0) 190 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:07.016424+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.1 deep-scrub starts 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:07.018482+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.1 deep-scrub ok 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:07.795263+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[29889]: cluster 2024-09-13T22:46:07.796840+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.696007+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.698340+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:46:08.494 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.951435+0000 osd.0 (osd.0) 189 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:46:08.495 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:06.958938+0000 osd.0 (osd.0) 190 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:46:08.495 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:07.016424+0000 osd.4 (osd.4) 167 : cluster [DBG] 4.1 deep-scrub starts 2024-09-13T22:46:08.495 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:07.018482+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.1 deep-scrub ok 2024-09-13T22:46:08.495 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:07.795263+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:46:08.495 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:08 smithi178 bash[22840]: cluster 2024-09-13T22:46:07.796840+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:46:08.512 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:08.622 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:08.648 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:08.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:08.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:08.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 49 16 72113 72097 91.9455 63.75 0.0124406 0.0108714 2024-09-13T22:46:08.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:08.933 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:08.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:09.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:09.058 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:09.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:09.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:09.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:09.308 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:09.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T22:46:09.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:07.736704+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:46:09.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:07.738396+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.016168+0000 osd.4 (osd.4) 169 : cluster [DBG] 3.1a deep-scrub starts 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.020581+0000 osd.2 (osd.2) 145 : cluster [DBG] 2.c scrub starts 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.022531+0000 osd.2 (osd.2) 146 : cluster [DBG] 2.c scrub ok 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.028586+0000 osd.4 (osd.4) 170 : cluster [DBG] 3.1a deep-scrub ok 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.051927+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.055007+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.606929+0000 mgr.y (mgr.14613) 316 : cluster [DBG] pgmap v330: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 61 MiB/s wr, 967 op/s 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.812623+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.819259+0000 osd.1 (osd.1) 185 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.845249+0000 osd.5 (osd.5) 210 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:46:09.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[22840]: cluster 2024-09-13T22:46:08.851573+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:46:09.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:09.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:07.736704+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:46:09.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:07.738396+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:46:09.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.016168+0000 osd.4 (osd.4) 169 : cluster [DBG] 3.1a deep-scrub starts 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.020581+0000 osd.2 (osd.2) 145 : cluster [DBG] 2.c scrub starts 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.022531+0000 osd.2 (osd.2) 146 : cluster [DBG] 2.c scrub ok 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.028586+0000 osd.4 (osd.4) 170 : cluster [DBG] 3.1a deep-scrub ok 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.051927+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.055007+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.606929+0000 mgr.y (mgr.14613) 316 : cluster [DBG] pgmap v330: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 61 MiB/s wr, 967 op/s 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.812623+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.819259+0000 osd.1 (osd.1) 185 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.845249+0000 osd.5 (osd.5) 210 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:46:09.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:09 smithi192 bash[24608]: cluster 2024-09-13T22:46:08.851573+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:46:09.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:09.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:09.564 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:07.736704+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:07.738396+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.016168+0000 osd.4 (osd.4) 169 : cluster [DBG] 3.1a deep-scrub starts 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.020581+0000 osd.2 (osd.2) 145 : cluster [DBG] 2.c scrub starts 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.022531+0000 osd.2 (osd.2) 146 : cluster [DBG] 2.c scrub ok 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.028586+0000 osd.4 (osd.4) 170 : cluster [DBG] 3.1a deep-scrub ok 2024-09-13T22:46:09.656 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.051927+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:46:09.657 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.055007+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:46:09.657 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.606929+0000 mgr.y (mgr.14613) 316 : cluster [DBG] pgmap v330: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 61 MiB/s wr, 967 op/s 2024-09-13T22:46:09.657 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.812623+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:46:09.657 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.819259+0000 osd.1 (osd.1) 185 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:46:09.657 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.845249+0000 osd.5 (osd.5) 210 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:46:09.657 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[29889]: cluster 2024-09-13T22:46:08.851573+0000 osd.5 (osd.5) 211 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:46:09.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:09.691 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:09.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:09.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 50 16 72819 72803 90.9891 44.125 0.011265 0.0109849 2024-09-13T22:46:09.822 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:09.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:09.944 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:09.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:10.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:10.053 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:46:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:46:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:46:10.088 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:10.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:10.220 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:10.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:10.348 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:10.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:10.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:10.606 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:10.634 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:10.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:10.742 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:46:10.742 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:46:10.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:10 smithi178 bash[29889]: cluster 2024-09-13T22:46:09.781697+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.b deep-scrub starts 2024-09-13T22:46:10.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:10 smithi178 bash[29889]: cluster 2024-09-13T22:46:09.789470+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.b deep-scrub ok 2024-09-13T22:46:10.758 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:10 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.607724+0000 mgr.y (mgr.14613) 317 : cluster [DBG] pgmap v331: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-13T22:46:10.758 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:10 smithi178 bash[22840]: cluster 2024-09-13T22:46:09.781697+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.b deep-scrub starts 2024-09-13T22:46:10.758 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:10 smithi178 bash[22840]: cluster 2024-09-13T22:46:09.789470+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.b deep-scrub ok 2024-09-13T22:46:10.758 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:10 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.607724+0000 mgr.y (mgr.14613) 317 : cluster [DBG] pgmap v331: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-13T22:46:10.765 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:10.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:10.787 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:46:10.787 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:46:10.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 51 16 73681 73665 90.2613 53.875 0.010699 0.0110715 2024-09-13T22:46:10.824 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:46:10.824 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:46:10.865 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:46:10.865 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:46:10.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:10.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:10.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:10 smithi192 bash[24608]: cluster 2024-09-13T22:46:09.781697+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.b deep-scrub starts 2024-09-13T22:46:10.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:10 smithi192 bash[24608]: cluster 2024-09-13T22:46:09.789470+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.b deep-scrub ok 2024-09-13T22:46:10.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:10 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.607724+0000 mgr.y (mgr.14613) 317 : cluster [DBG] pgmap v331: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-13T22:46:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:10.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:10.909 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:46:10.909 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:46:10.948 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:46:10.949 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:46:10.981 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:46:10.981 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:46:11.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:11.012 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:46:11.012 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:46:11.029 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:11.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:11.159 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:11.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:11.292 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:11.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:46:11.311 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:46:11.311 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 194, 2024-09-13T22:46:11.311 INFO:teuthology.orchestra.run.smithi178.stdout: "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:46:11.311 INFO:teuthology.orchestra.run.smithi178.stdout: "created": "2024-09-13T22:27:31.500176+0000", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "modified": "2024-09-13T22:46:02.848488+0000", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "last_up_change": "2024-09-13T22:37:07.372146+0000", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "last_in_change": "2024-09-13T22:36:38.469358+0000", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_num": 5799936, 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_set": [ 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "pglog_hardlimit", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "purged_snapdirs", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_deletes", 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "sortbitwise" 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:46:11.312 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_version": 18, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_snapshot": "", 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_max": 8, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "max_osd": 8, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "require_min_compat_client": "luminous", 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "min_compat_client": "luminous", 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "require_osd_release": "squid", 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "allow_crimson": false, 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: "pools": [ 2024-09-13T22:46:11.313 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 1, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".mgr", 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:32:40.780155+0000", 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.314 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 11, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 11, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 11, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 11, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 11, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.315 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "189", 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "187", 2024-09-13T22:46:11.316 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.317 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.318 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:46:11.319 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_max": 32, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_min": 1 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr": {} 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.1800000667572021, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.1800000667572021, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-13T22:46:11.320 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 2, 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".rgw.root", 2024-09-13T22:46:11.321 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:39.227580+0000", 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.322 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.323 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "76", 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 3, 2024-09-13T22:46:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.log", 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.229251+0000", 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.329 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.330 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "78", 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.331 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.332 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.333 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:46:11.334 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 4, 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "datapool", 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.624329+0000", 2024-09-13T22:46:11.335 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 8193, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.336 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 3, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 3, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 3, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 3, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 3, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "85", 2024-09-13T22:46:11.337 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 2, 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 85, 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.338 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.339 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.340 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd": {} 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.6500000953674316, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.6500000953674316, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.341 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 5, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.control", 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:43.199974+0000", 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.342 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:46:11.343 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "80", 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.344 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.345 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.346 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:46:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.25, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.25, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.25, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.25, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.348 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 6, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.meta", 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:45.423695+0000", 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.349 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.350 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "82", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.351 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.352 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.353 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_bias": 4 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.354 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.75, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.75, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.75, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.75, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 8, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "unique_pool_1", 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:45:09.792675+0000", 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:46:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 16, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 16, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 16, 2024-09-13T22:46:11.357 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 16, 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 16, 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "180", 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:46:11.358 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:46:11.359 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.360 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:46:11.361 INFO:teuthology.orchestra.run.smithi178.stdout: "rados": {} 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-13T22:46:11.362 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 1, 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac", 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.363 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 8, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 191, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.364 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6801", 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6802", 2024-09-13T22:46:11.365 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6804", 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.366 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6803", 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6801/1489719354", 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6802/1489719354", 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6804/1489719354", 2024-09-13T22:46:11.367 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6803/1489719354", 2024-09-13T22:46:11.368 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.368 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.368 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.368 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.368 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.369 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.369 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:46:11.369 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa", 2024-09-13T22:46:11.369 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.369 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.369 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 13, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 192, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.370 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.371 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.371 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.371 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6805", 2024-09-13T22:46:11.371 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:46:11.371 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.371 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6806", 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.372 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6808", 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.373 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6807", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6805/4187084287", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6806/4187084287", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6808/4187084287", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6807/4187084287", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.374 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "40a79075-5102-4f6c-b41a-255102064624", 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 19, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 116, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.375 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6809", 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.376 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6810", 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.377 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6812", 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.378 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6811", 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6809/2303802730", 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6810/2303802730", 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6812/2303802730", 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6811/2303802730", 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.379 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad", 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.380 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 28, 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 192, 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6813", 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.381 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6814", 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.382 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6816", 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.383 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6815", 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6813/320675660", 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6814/320675660", 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6816/320675660", 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6815/320675660", 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.384 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b", 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.385 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 34, 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 192, 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.386 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6800", 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6801", 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.387 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6803", 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.388 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6802", 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6800/3593670527", 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6801/3593670527", 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6803/3593670527", 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6802/3593670527", 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.389 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8", 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.390 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.391 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.391 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 45, 2024-09-13T22:46:11.391 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 192, 2024-09-13T22:46:11.391 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.391 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.392 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.392 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.392 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.392 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.392 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6804", 2024-09-13T22:46:11.392 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6805", 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.393 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6807", 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.394 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6806", 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6804/3271425338", 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6805/3271425338", 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6807/3271425338", 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6806/3271425338", 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.395 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98", 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 57, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 192, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.396 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6808", 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.397 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6809", 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6811", 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:46:11.398 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6810", 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6808/684205162", 2024-09-13T22:46:11.399 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6809/684205162", 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6811/684205162", 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6810/684205162", 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52", 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:46:11.400 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 67, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 192, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6812", 2024-09-13T22:46:11.401 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6813", 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.402 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6815", 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.403 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6814", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6812/1303971738", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6813/1303971738", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1303971738", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6814/1303971738", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.404 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_xinfo": [ 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:45:44.856087+0000", 2024-09-13T22:46:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:44:58.997610+0000", 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.406 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:45:16.668315+0000", 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.407 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:45:09.211932+0000", 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.408 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:45:54.811665+0000", 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:44:57.454028+0000", 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.409 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:45:08.979969+0000", 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.410 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:45:33.669872+0000", 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap": [], 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_items": [ 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "2.1f", 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.411 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 7, 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 2 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "5.f", 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 5, 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 7 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.412 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_primaries": [], 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_temp": [], 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_temp": [], 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "blocklist": { 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/268210503": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2039160385": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/669877409": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/129655740": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3632788039": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/388635774": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:46:11.413 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3265105147": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2959920382": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2494442689": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2676020773": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2675812306": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/1834442338": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2835351236": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3836928220": "2024-09-14T22:38:32.396576+0000" 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "range_blocklist": {}, 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profiles": { 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "default": { 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "crush-failure-domain": "osd", 2024-09-13T22:46:11.414 INFO:teuthology.orchestra.run.smithi178.stdout: "k": "2", 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "m": "1", 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "plugin": "jerasure", 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "technique": "reed_sol_van" 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps_queue": [], 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "new_removed_snaps": [], 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "new_purged_snaps": [], 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_node_flags": {}, 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "device_class_flags": {}, 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode": { 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_enabled": false, 2024-09-13T22:46:11.415 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_bucket_count": 0, 2024-09-13T22:46:11.416 INFO:teuthology.orchestra.run.smithi178.stdout: "degraded_stretch_mode": 0, 2024-09-13T22:46:11.416 INFO:teuthology.orchestra.run.smithi178.stdout: "recovering_stretch_mode": 0, 2024-09-13T22:46:11.416 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_bucket": 0 2024-09-13T22:46:11.416 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:11.416 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:46:11.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:11.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:11.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:11.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:11.674 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:46:11.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:11.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:11.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 52 16 74756 74740 89.8175 67.1875 0.0111341 0.011129 2024-09-13T22:46:11.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:11.873 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:11.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:11.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.061250+0000 osd.6 (osd.6) 143 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.063456+0000 osd.6 (osd.6) 144 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: audit 2024-09-13T22:46:10.707232+0000 mon.b (mon.1) 146 : audit [DBG] from='client.? v1:172.21.15.178:0/1152825017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.827969+0000 osd.7 (osd.7) 191 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.833628+0000 osd.7 (osd.7) 192 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: audit 2024-09-13T22:46:10.868889+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.178:0/2255611705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: audit 2024-09-13T22:46:10.873922+0000 mon.a (mon.0) 1126 : audit [INF] from='client.25045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:46:11.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[29889]: audit 2024-09-13T22:46:11.307598+0000 mon.c (mon.2) 40 : audit [DBG] from='client.? v1:172.21.15.178:0/1541219215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.061250+0000 osd.6 (osd.6) 143 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.063456+0000 osd.6 (osd.6) 144 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: audit 2024-09-13T22:46:10.707232+0000 mon.b (mon.1) 146 : audit [DBG] from='client.? v1:172.21.15.178:0/1152825017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.827969+0000 osd.7 (osd.7) 191 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.833628+0000 osd.7 (osd.7) 192 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: audit 2024-09-13T22:46:10.868889+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.178:0/2255611705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: audit 2024-09-13T22:46:10.873922+0000 mon.a (mon.0) 1126 : audit [INF] from='client.25045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:46:11.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:11 smithi178 bash[22840]: audit 2024-09-13T22:46:11.307598+0000 mon.c (mon.2) 40 : audit [DBG] from='client.? v1:172.21.15.178:0/1541219215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:46:11.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:11.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.061250+0000 osd.6 (osd.6) 143 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:46:11.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.063456+0000 osd.6 (osd.6) 144 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:46:11.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: audit 2024-09-13T22:46:10.707232+0000 mon.b (mon.1) 146 : audit [DBG] from='client.? v1:172.21.15.178:0/1152825017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:11.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.827969+0000 osd.7 (osd.7) 191 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:46:11.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.833628+0000 osd.7 (osd.7) 192 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:46:11.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: audit 2024-09-13T22:46:10.868889+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.178:0/2255611705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:46:11.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: audit 2024-09-13T22:46:10.873922+0000 mon.a (mon.0) 1126 : audit [INF] from='client.25045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:46:11.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:11 smithi192 bash[24608]: audit 2024-09-13T22:46:11.307598+0000 mon.c (mon.2) 40 : audit [DBG] from='client.? v1:172.21.15.178:0/1541219215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:46:12.004 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:12.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:12.139 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:12.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:12.266 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:12.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:12.396 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:12.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:12.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:12.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:12.645 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:12.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:12.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.993929+0000 osd.6 (osd.6) 145 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:10.995546+0000 osd.6 (osd.6) 146 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:11.034280+0000 osd.2 (osd.2) 147 : cluster [DBG] 5.d scrub starts 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:11.045375+0000 osd.2 (osd.2) 148 : cluster [DBG] 5.d scrub ok 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:11.664668+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: audit 2024-09-13T22:46:11.672957+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:11.682440+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-13T22:46:12.781 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:12 smithi192 bash[24608]: cluster 2024-09-13T22:46:12.608028+0000 mgr.y (mgr.14613) 318 : cluster [DBG] pgmap v333: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T22:46:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:12.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 53 16 75741 75725 89.2843 61.5625 0.0105675 0.0111953 2024-09-13T22:46:12.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:12.910 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.993929+0000 osd.6 (osd.6) 145 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:46:12.910 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:10.995546+0000 osd.6 (osd.6) 146 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:46:12.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:11.034280+0000 osd.2 (osd.2) 147 : cluster [DBG] 5.d scrub starts 2024-09-13T22:46:12.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:11.045375+0000 osd.2 (osd.2) 148 : cluster [DBG] 5.d scrub ok 2024-09-13T22:46:12.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:11.664668+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:12.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: audit 2024-09-13T22:46:11.672957+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:46:12.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:11.682440+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-13T22:46:12.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[29889]: cluster 2024-09-13T22:46:12.608028+0000 mgr.y (mgr.14613) 318 : cluster [DBG] pgmap v333: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T22:46:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:12.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.993929+0000 osd.6 (osd.6) 145 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:10.995546+0000 osd.6 (osd.6) 146 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:11.034280+0000 osd.2 (osd.2) 147 : cluster [DBG] 5.d scrub starts 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:11.045375+0000 osd.2 (osd.2) 148 : cluster [DBG] 5.d scrub ok 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:11.664668+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: audit 2024-09-13T22:46:11.672957+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:46:12.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:11.682440+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-13T22:46:12.913 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:12 smithi178 bash[22840]: cluster 2024-09-13T22:46:12.608028+0000 mgr.y (mgr.14613) 318 : cluster [DBG] pgmap v333: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T22:46:13.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:13.034 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:13.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:13.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:13.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:13.296 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:13.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:13.416 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:13.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:13.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:13.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:13.673 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:13.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:13.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:13.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 54 16 76653 76637 88.6864 57 0.0111292 0.0112712 2024-09-13T22:46:13.819 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:13 smithi178 bash[22840]: cluster 2024-09-13T22:46:12.254285+0000 osd.5 (osd.5) 212 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:13.819 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:13 smithi178 bash[22840]: cluster 2024-09-13T22:46:12.254403+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:13.819 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:13 smithi178 bash[22840]: cluster 2024-09-13T22:46:12.684409+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-13T22:46:13.819 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:13 smithi178 bash[29889]: cluster 2024-09-13T22:46:12.254285+0000 osd.5 (osd.5) 212 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:13.819 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:13 smithi178 bash[29889]: cluster 2024-09-13T22:46:12.254403+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:13.820 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:13 smithi178 bash[29889]: cluster 2024-09-13T22:46:12.684409+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-13T22:46:13.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:13.931 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:13.932 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:13 smithi192 bash[24608]: cluster 2024-09-13T22:46:12.254285+0000 osd.5 (osd.5) 212 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:13.932 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:13 smithi192 bash[24608]: cluster 2024-09-13T22:46:12.254403+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:13.932 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:13 smithi192 bash[24608]: cluster 2024-09-13T22:46:12.684409+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-13T22:46:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:13.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:14.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:14.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:14.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.190 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:14.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:14.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:14.448 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:14.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:14.574 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:14.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:14.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 55 16 77463 77447 87.9944 50.625 0.00808375 0.0113593 2024-09-13T22:46:14.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:14 smithi192 bash[24608]: cluster 2024-09-13T22:46:14.609367+0000 mgr.y (mgr.14613) 319 : cluster [DBG] pgmap v335: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 956 op/s 2024-09-13T22:46:14.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:14.906 INFO:tasks.thrashosds.thrasher:j is {'epoch': 194, 'fsid': '03ae43e2-721f-11ef-bceb-c7b262605968', 'created': '2024-09-13T22:27:31.500176+0000', 'modified': '2024-09-13T22:46:02.848488+0000', 'last_up_change': '2024-09-13T22:37:07.372146+0000', 'last_in_change': '2024-09-13T22:36:38.469358+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T22:32:40.780155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '187', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T22:37:39.227580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T22:37:41.229251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-09-13T22:37:41.624329+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 85, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T22:37:43.199974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T22:37:45.423695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-13T22:45:09.792675+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '394f2664-bba1-4b15-ace8-78f70f970cac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6801', 'nonce': 1489719354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6802', 'nonce': 1489719354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6804', 'nonce': 1489719354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6803', 'nonce': 1489719354}]}, 'public_addr': '172.21.15.178:6801/1489719354', 'cluster_addr': '172.21.15.178:6802/1489719354', 'heartbeat_back_addr': '172.21.15.178:6804/1489719354', 'heartbeat_front_addr': '172.21.15.178:6803/1489719354', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e65c08b-6431-4849-a6af-2da9cb1efdfa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6805', 'nonce': 4187084287}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6806', 'nonce': 4187084287}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6808', 'nonce': 4187084287}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6807', 'nonce': 4187084287}]}, 'public_addr': '172.21.15.178:6805/4187084287', 'cluster_addr': '172.21.15.178:6806/4187084287', 'heartbeat_back_addr': '172.21.15.178:6808/4187084287', 'heartbeat_front_addr': '172.21.15.178:6807/4187084287', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '40a79075-5102-4f6c-b41a-255102064624', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6809', 'nonce': 2303802730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6810', 'nonce': 2303802730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6812', 'nonce': 2303802730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6811', 'nonce': 2303802730}]}, 'public_addr': '172.21.15.178:6809/2303802730', 'cluster_addr': '172.21.15.178:6810/2303802730', 'heartbeat_back_addr': '172.21.15.178:6812/2303802730', 'heartbeat_front_addr': '172.21.15.178:6811/2303802730', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3962c796-6f97-4cc6-a186-4b6ef58d67ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6813', 'nonce': 320675660}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6814', 'nonce': 320675660}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6816', 'nonce': 320675660}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6815', 'nonce': 320675660}]}, 'public_addr': '172.21.15.178:6813/320675660', 'cluster_addr': '172.21.15.178:6814/320675660', 'heartbeat_back_addr': '172.21.15.178:6816/320675660', 'heartbeat_front_addr': '172.21.15.178:6815/320675660', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b89db2f-9e81-40b1-89bd-8de1dbff133b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 34, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 3593670527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 3593670527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 3593670527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 3593670527}]}, 'public_addr': '172.21.15.192:6800/3593670527', 'cluster_addr': '172.21.15.192:6801/3593670527', 'heartbeat_back_addr': '172.21.15.192:6803/3593670527', 'heartbeat_front_addr': '172.21.15.192:6802/3593670527', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'efa849a2-d789-4fdd-acc7-04f2c1bebfa8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 3271425338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 3271425338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 3271425338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 3271425338}]}, 'public_addr': '172.21.15.192:6804/3271425338', 'cluster_addr': '172.21.15.192:6805/3271425338', 'heartbeat_back_addr': '172.21.15.192:6807/3271425338', 'heartbeat_front_addr': '172.21.15.192:6806/3271425338', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '54a26ae9-bf31-49cf-a924-901eed4a0c98', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 684205162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 684205162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 684205162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 684205162}]}, 'public_addr': '172.21.15.192:6808/684205162', 'cluster_addr': '172.21.15.192:6809/684205162', 'heartbeat_back_addr': '172.21.15.192:6811/684205162', 'heartbeat_front_addr': '172.21.15.192:6810/684205162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '13e146db-61ac-4fe6-8f26-cbf8d1623a52', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 192, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 1303971738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1303971738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1303971738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 1303971738}]}, 'public_addr': '172.21.15.192:6812/1303971738', 'cluster_addr': '172.21.15.192:6813/1303971738', 'heartbeat_back_addr': '172.21.15.192:6815/1303971738', 'heartbeat_front_addr': '172.21.15.192:6814/1303971738', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:45:44.856087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:44:58.997610+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:45:16.668315+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:45:09.211932+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:45:54.811665+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:44:57.454028+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:45:08.979969+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:45:33.669872+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1f', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '5.f', 'mappings': [{'from': 5, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.178:0/268210503': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:6800/2039160385': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/669877409': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/129655740': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/3632788039': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:6800/388635774': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3265105147': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2959920382': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/2494442689': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:6800/2676020773': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2675812306': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/1834442338': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2835351236': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3836928220': '2024-09-14T22:38:32.396576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T22:46:14.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:46:14.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:14.967 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:14.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:15.064 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:14 smithi178 bash[22840]: cluster 2024-09-13T22:46:14.609367+0000 mgr.y (mgr.14613) 319 : cluster [DBG] pgmap v335: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 956 op/s 2024-09-13T22:46:15.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:14 smithi178 bash[29889]: cluster 2024-09-13T22:46:14.609367+0000 mgr.y (mgr.14613) 319 : cluster [DBG] pgmap v335: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 956 op/s 2024-09-13T22:46:15.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:15.111 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:15.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:15.240 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:15.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:15.373 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:15.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:15.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:15.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:15.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:46:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:46:15.638 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:15.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:15.771 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:15.818 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 56 16 78555 78539 87.6417 68.25 0.015952 0.0114051 2024-09-13T22:46:15.875 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:15.901 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:15.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:15 smithi192 bash[24608]: cluster 2024-09-13T22:46:14.143069+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:15 smithi192 bash[24608]: cluster 2024-09-13T22:46:14.143128+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:15 smithi192 bash[24608]: cluster 2024-09-13T22:46:14.702657+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-13T22:46:15.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:15 smithi192 bash[24608]: audit 2024-09-13T22:46:15.206671+0000 mgr.y (mgr.14613) 320 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:15.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:46:15.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[29889]: cluster 2024-09-13T22:46:14.143069+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:15.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[29889]: cluster 2024-09-13T22:46:14.143128+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:15.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[29889]: cluster 2024-09-13T22:46:14.702657+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-13T22:46:15.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[29889]: audit 2024-09-13T22:46:15.206671+0000 mgr.y (mgr.14613) 320 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:15.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[22840]: cluster 2024-09-13T22:46:14.143069+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:15.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[22840]: cluster 2024-09-13T22:46:14.143128+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:15.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[22840]: cluster 2024-09-13T22:46:14.702657+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-13T22:46:15.959 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:15 smithi178 bash[22840]: audit 2024-09-13T22:46:15.206671+0000 mgr.y (mgr.14613) 320 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:16.005 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:16.032 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:16.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.170 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:16.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:16.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:16.300 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:16.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:16.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:16.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:16.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:16.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:16.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:16.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:16.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 57 16 79678 79662 87.3354 70.1875 0.010884 0.011446 2024-09-13T22:46:16.849 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:16.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:16.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:16 smithi192 bash[24608]: cluster 2024-09-13T22:46:16.610258+0000 mgr.y (mgr.14613) 321 : cluster [DBG] pgmap v337: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s 2024-09-13T22:46:16.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:16.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:17.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:46:17.076 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:16 smithi178 bash[29889]: cluster 2024-09-13T22:46:16.610258+0000 mgr.y (mgr.14613) 321 : cluster [DBG] pgmap v337: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s 2024-09-13T22:46:17.076 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:16 smithi178 bash[22840]: cluster 2024-09-13T22:46:16.610258+0000 mgr.y (mgr.14613) 321 : cluster [DBG] pgmap v337: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s 2024-09-13T22:46:17.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:17.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:17.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:17.262 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:17.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:17.395 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:17.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:17.526 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:17.578 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:17.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:17.659 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:17.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:17.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:17.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 58 16 80742 80726 86.9761 66.5 0.0131718 0.0114927 2024-09-13T22:46:17.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:17.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:17.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:18.041 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:18.063 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:18.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:18.210 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:18.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:18.346 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:18.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:18.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:18.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:18.612 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:18.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:18.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:18.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 59 16 81557 81541 86.3652 50.9375 0.00410928 0.0115309 2024-09-13T22:46:18.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:18.865 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:18.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:18.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:18.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:19.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:19.108 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:18 smithi192 bash[24608]: cluster 2024-09-13T22:46:17.863431+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-13T22:46:19.108 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:18 smithi192 bash[24608]: audit 2024-09-13T22:46:18.277534+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:19.108 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:18 smithi192 bash[24608]: audit 2024-09-13T22:46:18.280385+0000 mon.b (mon.1) 148 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:19.109 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:18 smithi192 bash[24608]: cluster 2024-09-13T22:46:18.610646+0000 mgr.y (mgr.14613) 322 : cluster [DBG] pgmap v339: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s 2024-09-13T22:46:19.124 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:19.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:19.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:19.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:19.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[29889]: cluster 2024-09-13T22:46:17.863431+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-13T22:46:19.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[29889]: audit 2024-09-13T22:46:18.277534+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:19.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[29889]: audit 2024-09-13T22:46:18.280385+0000 mon.b (mon.1) 148 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:19.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[29889]: cluster 2024-09-13T22:46:18.610646+0000 mgr.y (mgr.14613) 322 : cluster [DBG] pgmap v339: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s 2024-09-13T22:46:19.370 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[22840]: cluster 2024-09-13T22:46:17.863431+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-13T22:46:19.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[22840]: audit 2024-09-13T22:46:18.277534+0000 mon.a (mon.0) 1133 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:19.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[22840]: audit 2024-09-13T22:46:18.280385+0000 mon.b (mon.1) 148 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:19.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:18 smithi178 bash[22840]: cluster 2024-09-13T22:46:18.610646+0000 mgr.y (mgr.14613) 322 : cluster [DBG] pgmap v339: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s 2024-09-13T22:46:19.392 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:19.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:19.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:19.626 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:19.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:19.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:19.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:19.783 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:19.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:46:19.822629+0000 min lat: 0.00124784 max lat: 0.701883 avg lat: 0.0116428 2024-09-13T22:46:19.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:46:19.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 60 15 82448 82433 85.8548 55.75 0.0157607 0.0116428 2024-09-13T22:46:19.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:19.912 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:19.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:20.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:20.025 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:46:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:46:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:46:20.045 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:20.149 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:20.192 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:20.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:20.329 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:20.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:20.458 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:20.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:20.573 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:20 smithi192 bash[24608]: audit 2024-09-13T22:46:19.981278+0000 mon.b (mon.1) 149 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:46:20.592 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:20.644 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:20.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:20 smithi178 bash[22840]: audit 2024-09-13T22:46:19.981278+0000 mon.b (mon.1) 149 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:46:20.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:20 smithi178 bash[29889]: audit 2024-09-13T22:46:19.981278+0000 mon.b (mon.1) 149 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:46:20.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:20.718 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:20.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 61 16 83347 83331 85.3674 56.125 0.00290787 0.0116894 2024-09-13T22:46:20.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:20.841 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:20.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:20.973 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:20.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:21.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:21.102 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:21.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:21.231 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:21.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:21.365 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[29889]: cluster 2024-09-13T22:46:20.611611+0000 mgr.y (mgr.14613) 323 : cluster [DBG] pgmap v340: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[29889]: audit 2024-09-13T22:46:20.870465+0000 mon.b (mon.1) 150 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[29889]: audit 2024-09-13T22:46:20.871501+0000 mon.b (mon.1) 151 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[29889]: audit 2024-09-13T22:46:20.875552+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[22840]: cluster 2024-09-13T22:46:20.611611+0000 mgr.y (mgr.14613) 323 : cluster [DBG] pgmap v340: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[22840]: audit 2024-09-13T22:46:20.870465+0000 mon.b (mon.1) 150 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[22840]: audit 2024-09-13T22:46:20.871501+0000 mon.b (mon.1) 151 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:46:21.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:21 smithi178 bash[22840]: audit 2024-09-13T22:46:20.875552+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:21.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:21.498 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:21.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:21.625 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:21.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:21 smithi192 bash[24608]: cluster 2024-09-13T22:46:20.611611+0000 mgr.y (mgr.14613) 323 : cluster [DBG] pgmap v340: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:46:21.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:21 smithi192 bash[24608]: audit 2024-09-13T22:46:20.870465+0000 mon.b (mon.1) 150 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:46:21.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:21 smithi192 bash[24608]: audit 2024-09-13T22:46:20.871501+0000 mon.b (mon.1) 151 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:46:21.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:21 smithi192 bash[24608]: audit 2024-09-13T22:46:20.875552+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:46:21.667 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:21.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:21.759 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:21.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 62 16 84268 84252 84.9189 57.5625 0.00625741 0.0117706 2024-09-13T22:46:21.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:21.879 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:21.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:21.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.009 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:22.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:22.110 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:22.140 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:22.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:22.276 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:22.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:22.410 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:22.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:22.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:22.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:22.670 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:22.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:22.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:22.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 63 16 85032 85016 84.3288 47.75 0.00371053 0.0118303 2024-09-13T22:46:22.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:22 smithi192 bash[24608]: cluster 2024-09-13T22:46:22.612089+0000 mgr.y (mgr.14613) 324 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 832 op/s 2024-09-13T22:46:22.923 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:22.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:23.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:22 smithi178 bash[29889]: cluster 2024-09-13T22:46:22.612089+0000 mgr.y (mgr.14613) 324 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 832 op/s 2024-09-13T22:46:23.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:22 smithi178 bash[22840]: cluster 2024-09-13T22:46:22.612089+0000 mgr.y (mgr.14613) 324 : cluster [DBG] pgmap v341: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 832 op/s 2024-09-13T22:46:23.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:23.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:23.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:23.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:23.224 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:46:23.224 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:46:23.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:23.311 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:23.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:23.439 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:23.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:23.577 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:23.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:23.706 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:23.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:23.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 64 16 85945 85929 83.9028 57.0625 0.0114783 0.0119137 2024-09-13T22:46:23.839 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:23.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:23.969 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:23.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:24.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:24.100 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:24.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:24.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:24.236 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:24.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:24.358 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:24.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:24.484 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:24.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:24.618 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:24.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:24.732 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:24 smithi192 bash[24608]: cluster 2024-09-13T22:46:24.612907+0000 mgr.y (mgr.14613) 325 : cluster [DBG] pgmap v342: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 863 op/s 2024-09-13T22:46:24.745 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:24.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 65 16 86830 86814 83.4629 55.3125 0.0204547 0.0119769 2024-09-13T22:46:24.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:24.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:24.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:25.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:25.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:25.118 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:24 smithi178 bash[22840]: cluster 2024-09-13T22:46:24.612907+0000 mgr.y (mgr.14613) 325 : cluster [DBG] pgmap v342: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 863 op/s 2024-09-13T22:46:25.118 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:24 smithi178 bash[29889]: cluster 2024-09-13T22:46:24.612907+0000 mgr.y (mgr.14613) 325 : cluster [DBG] pgmap v342: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 863 op/s 2024-09-13T22:46:25.134 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:25.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:25.266 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:25.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.394 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:25.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:25.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:46:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:46:25.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:25.519 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:25.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:25.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:25.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:25.754 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:25 smithi192 bash[24608]: audit 2024-09-13T22:46:25.219864+0000 mgr.y (mgr.14613) 326 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:25.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:25.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 66 16 87828 87812 83.1433 62.375 0.0187996 0.0120231 2024-09-13T22:46:25.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:25.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:25.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:26.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:26.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:26.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:26.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:25 smithi178 bash[29889]: audit 2024-09-13T22:46:25.219864+0000 mgr.y (mgr.14613) 326 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:26.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:25 smithi178 bash[22840]: audit 2024-09-13T22:46:25.219864+0000 mgr.y (mgr.14613) 326 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:26.182 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:26.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:26.319 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:26.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:26.448 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:26.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:26.593 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:26.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:26.723 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:26.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 67 16 88950 88934 82.9489 70.125 0.04342 0.0120494 2024-09-13T22:46:26.830 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:26.834 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T22:46:26.861 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:26.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:26 smithi192 bash[24608]: cluster 2024-09-13T22:46:26.613658+0000 mgr.y (mgr.14613) 327 : cluster [DBG] pgmap v343: 158 pgs: 158 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 933 op/s 2024-09-13T22:46:26.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:26.987 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:26.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:27.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:27.103 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:26 smithi178 bash[22840]: cluster 2024-09-13T22:46:26.613658+0000 mgr.y (mgr.14613) 327 : cluster [DBG] pgmap v343: 158 pgs: 158 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 933 op/s 2024-09-13T22:46:27.104 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:26 smithi178 bash[29889]: cluster 2024-09-13T22:46:26.613658+0000 mgr.y (mgr.14613) 327 : cluster [DBG] pgmap v343: 158 pgs: 158 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 933 op/s 2024-09-13T22:46:27.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:27.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:27.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:27.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:27.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:27.437 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:46:27.450 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:46:27.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:27.527 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:27.575 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:46:27.575 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:46:27.615 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:46:27.615 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:46:27.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:27.658 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:27.667 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:46:27.667 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:46:27.735 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:46:27.735 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:46:27.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:27.770 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:46:27.771 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:46:27.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:27.814 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:46:27.814 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:46:27.819 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 68 16 89775 89759 82.4873 51.5625 0.00713197 0.0121172 2024-09-13T22:46:27.849 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:46:27.849 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:46:27.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:27.897 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:46:27.898 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:46:27.898 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:27 smithi178 bash[29889]: audit 2024-09-13T22:46:27.419352+0000 mgr.y (mgr.14613) 328 : audit [DBG] from='client.15198 v1:172.21.15.178:0/624125560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:46:27.898 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:27 smithi178 bash[29889]: audit 2024-09-13T22:46:27.530061+0000 mon.a (mon.0) 1135 : audit [DBG] from='client.? v1:172.21.15.178:0/2628246855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:27.898 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:27 smithi178 bash[22840]: audit 2024-09-13T22:46:27.419352+0000 mgr.y (mgr.14613) 328 : audit [DBG] from='client.15198 v1:172.21.15.178:0/624125560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:46:27.898 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:27 smithi178 bash[22840]: audit 2024-09-13T22:46:27.530061+0000 mon.a (mon.0) 1135 : audit [DBG] from='client.? v1:172.21.15.178:0/2628246855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:27.927 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:27.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:28.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:28.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:28.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:27 smithi192 bash[24608]: audit 2024-09-13T22:46:27.419352+0000 mgr.y (mgr.14613) 328 : audit [DBG] from='client.15198 v1:172.21.15.178:0/624125560' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:46:28.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:27 smithi192 bash[24608]: audit 2024-09-13T22:46:27.530061+0000 mon.a (mon.0) 1135 : audit [DBG] from='client.? v1:172.21.15.178:0/2628246855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:28.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:28.213 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:28.317 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:28.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:28.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:28.471 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:28.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:28.602 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:28.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:28.714 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:46:28.739 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:28.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 69 16 90792 90776 82.2129 63.5625 0.00677151 0.0121591 2024-09-13T22:46:28.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:28.853 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:28 smithi192 bash[24608]: audit 2024-09-13T22:46:27.830215+0000 mon.a (mon.0) 1136 : audit [INF] from='client.25066 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:46:28.853 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:28 smithi192 bash[24608]: audit 2024-09-13T22:46:27.830755+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.178:0/711473030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:46:28.853 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:28 smithi192 bash[24608]: cluster 2024-09-13T22:46:28.614450+0000 mgr.y (mgr.14613) 329 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 951 B/s rd, 54 MiB/s wr, 867 op/s 2024-09-13T22:46:28.872 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:28.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:28.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:29.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:29.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:28 smithi178 bash[22840]: audit 2024-09-13T22:46:27.830215+0000 mon.a (mon.0) 1136 : audit [INF] from='client.25066 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:46:29.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:28 smithi178 bash[22840]: audit 2024-09-13T22:46:27.830755+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.178:0/711473030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:46:29.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:28 smithi178 bash[22840]: cluster 2024-09-13T22:46:28.614450+0000 mgr.y (mgr.14613) 329 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 951 B/s rd, 54 MiB/s wr, 867 op/s 2024-09-13T22:46:29.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:28 smithi178 bash[29889]: audit 2024-09-13T22:46:27.830215+0000 mon.a (mon.0) 1136 : audit [INF] from='client.25066 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:46:29.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:28 smithi178 bash[29889]: audit 2024-09-13T22:46:27.830755+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.178:0/711473030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:46:29.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:28 smithi178 bash[29889]: cluster 2024-09-13T22:46:28.614450+0000 mgr.y (mgr.14613) 329 : cluster [DBG] pgmap v344: 158 pgs: 158 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 951 B/s rd, 54 MiB/s wr, 867 op/s 2024-09-13T22:46:29.138 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:29.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:29.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:29.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.400 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:29.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:29.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:29.523 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:29.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:29.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:29.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:29.770 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:29.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 70 16 91523 91507 81.6911 45.6875 0.0131305 0.0122365 2024-09-13T22:46:29.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:29.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:29 smithi192 bash[24608]: cluster 2024-09-13T22:46:28.703130+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:29 smithi192 bash[24608]: audit 2024-09-13T22:46:28.712885+0000 mon.a (mon.0) 1138 : audit [INF] from='client.25066 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:46:29.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:29 smithi192 bash[24608]: cluster 2024-09-13T22:46:28.716767+0000 mon.a (mon.0) 1139 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-13T22:46:29.912 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:29.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:30.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:30.027 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:46:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:46:30.028 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[29889]: cluster 2024-09-13T22:46:28.703130+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:30.028 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[29889]: audit 2024-09-13T22:46:28.712885+0000 mon.a (mon.0) 1138 : audit [INF] from='client.25066 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:46:30.028 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[29889]: cluster 2024-09-13T22:46:28.716767+0000 mon.a (mon.0) 1139 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-13T22:46:30.028 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[22840]: cluster 2024-09-13T22:46:28.703130+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:30.028 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[22840]: audit 2024-09-13T22:46:28.712885+0000 mon.a (mon.0) 1138 : audit [INF] from='client.25066 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:46:30.028 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:29 smithi178 bash[22840]: cluster 2024-09-13T22:46:28.716767+0000 mon.a (mon.0) 1139 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-13T22:46:30.052 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:30.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:30.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:30.280 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:30.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:30.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:30.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:30.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:30.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:30.703 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:30.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:30.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 71 16 92120 92104 81.0661 37.3125 0.028325 0.0123308 2024-09-13T22:46:30.831 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:30.833 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-13T22:46:26.613243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5654583835,"num_objects":86472,"num_object_clones":0,"num_object_copies":259416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86472,"num_whiteouts":0,"num_read":1240,"num_read_kb":992,"num_write":86709,"num_write_kb":5521517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83684,"ondisk_log_size":83684,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17813308,"kb_used_data":16432380,"kb_used_omap":12,"kb_used_meta":1380787,"kb_avail":732147908,"statfs":{"total":767960285184,"available":749719457792,"internally_reserved":0,"allocated":16826757120,"data_stored":16822730817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1413926448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":611057664,"num_objects":9324,"num_object_clones":0,"num_object_copies":27972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9324,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9324,"num_write_kb":596736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003300"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.870159+0000","last_change":"2024-09-13T22:45:02.514835+0000","last_active":"2024-09-13T22:46:17.870159+0000","last_peered":"2024-09-13T22:46:17.870159+0000","last_clean":"2024-09-13T22:46:17.870159+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:46:17.870159+0000","last_undegraded":"2024-09-13T22:46:17.870159+0000","last_fullsized":"2024-09-13T22:46:17.870159+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.514742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862691+0000","last_change":"2024-09-13T22:45:54.989095+0000","last_active":"2024-09-13T22:46:17.862691+0000","last_peered":"2024-09-13T22:46:17.862691+0000","last_clean":"2024-09-13T22:46:17.862691+0000","last_became_active":"2024-09-13T22:45:34.878742+0000","last_became_peered":"2024-09-13T22:45:34.878742+0000","last_unstale":"2024-09-13T22:46:17.862691+0000","last_undegraded":"2024-09-13T22:46:17.862691+0000","last_fullsized":"2024-09-13T22:46:17.862691+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:54.988990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.988990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:15.852558+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"99'15","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866516+0000","last_change":"2024-09-13T22:43:51.778660+0000","last_active":"2024-09-13T22:46:17.866516+0000","last_peered":"2024-09-13T22:46:17.866516+0000","last_clean":"2024-09-13T22:46:17.866516+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:46:17.866516+0000","last_undegraded":"2024-09-13T22:46:17.866516+0000","last_fullsized":"2024-09-13T22:46:17.866516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:43:51.778550+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:43:51.778550+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.191884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857739+0000","last_change":"2024-09-13T22:46:01.019547+0000","last_active":"2024-09-13T22:46:17.857739+0000","last_peered":"2024-09-13T22:46:17.857739+0000","last_clean":"2024-09-13T22:46:17.857739+0000","last_became_active":"2024-09-13T22:37:44.875495+0000","last_became_peered":"2024-09-13T22:37:44.875495+0000","last_unstale":"2024-09-13T22:46:17.857739+0000","last_undegraded":"2024-09-13T22:46:17.857739+0000","last_fullsized":"2024-09-13T22:46:17.857739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:01.019435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:46:01.019435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:23.000289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866373+0000","last_change":"2024-09-13T22:45:04.482417+0000","last_active":"2024-09-13T22:46:17.866373+0000","last_peered":"2024-09-13T22:46:17.866373+0000","last_clean":"2024-09-13T22:46:17.866373+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:46:17.866373+0000","last_undegraded":"2024-09-13T22:46:17.866373+0000","last_fullsized":"2024-09-13T22:46:17.866373+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:04.482214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:04.482214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"102'16","reported_seq":320,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862406+0000","last_change":"2024-09-13T22:46:06.959087+0000","last_active":"2024-09-13T22:46:17.862406+0000","last_peered":"2024-09-13T22:46:17.862406+0000","last_clean":"2024-09-13T22:46:17.862406+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:46:17.862406+0000","last_undegraded":"2024-09-13T22:46:17.862406+0000","last_fullsized":"2024-09-13T22:46:17.862406+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:46:06.958946+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:46:06.958946+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:13.456376+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857862+0000","last_change":"2024-09-13T22:45:33.700682+0000","last_active":"2024-09-13T22:46:17.857862+0000","last_peered":"2024-09-13T22:46:17.857862+0000","last_clean":"2024-09-13T22:46:17.857862+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:46:17.857862+0000","last_undegraded":"2024-09-13T22:46:17.857862+0000","last_fullsized":"2024-09-13T22:46:17.857862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:33.700479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:45:33.700479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:53.533664+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858226+0000","last_change":"2024-09-13T22:45:53.957745+0000","last_active":"2024-09-13T22:46:17.858226+0000","last_peered":"2024-09-13T22:46:17.858226+0000","last_clean":"2024-09-13T22:46:17.858226+0000","last_became_active":"2024-09-13T22:37:46.908632+0000","last_became_peered":"2024-09-13T22:37:46.908632+0000","last_unstale":"2024-09-13T22:46:17.858226+0000","last_undegraded":"2024-09-13T22:46:17.858226+0000","last_fullsized":"2024-09-13T22:46:17.858226+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:45:53.957628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:12.264836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868255+0000","last_change":"2024-09-13T22:45:01.826287+0000","last_active":"2024-09-13T22:46:17.868255+0000","last_peered":"2024-09-13T22:46:17.868255+0000","last_clean":"2024-09-13T22:46:17.868255+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:46:17.868255+0000","last_undegraded":"2024-09-13T22:46:17.868255+0000","last_fullsized":"2024-09-13T22:46:17.868255+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:01.826196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.826196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'22","reported_seq":326,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862055+0000","last_change":"2024-09-13T22:45:55.899327+0000","last_active":"2024-09-13T22:46:17.862055+0000","last_peered":"2024-09-13T22:46:17.862055+0000","last_clean":"2024-09-13T22:46:17.862055+0000","last_became_active":"2024-09-13T22:37:42.757711+0000","last_became_peered":"2024-09-13T22:37:42.757711+0000","last_unstale":"2024-09-13T22:46:17.862055+0000","last_undegraded":"2024-09-13T22:46:17.862055+0000","last_fullsized":"2024-09-13T22:46:17.862055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'22","last_scrub_stamp":"2024-09-13T22:45:55.899175+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:45:55.899175+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:24.268083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":274,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866153+0000","last_change":"2024-09-13T22:44:58.799450+0000","last_active":"2024-09-13T22:46:17.866153+0000","last_peered":"2024-09-13T22:46:17.866153+0000","last_clean":"2024-09-13T22:46:17.866153+0000","last_became_active":"2024-09-13T22:37:44.877601+0000","last_became_peered":"2024-09-13T22:37:44.877601+0000","last_unstale":"2024-09-13T22:46:17.866153+0000","last_undegraded":"2024-09-13T22:46:17.866153+0000","last_fullsized":"2024-09-13T22:46:17.866153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:58.799239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.799239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.891113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861937+0000","last_change":"2024-09-13T22:45:01.888912+0000","last_active":"2024-09-13T22:46:17.861937+0000","last_peered":"2024-09-13T22:46:17.861937+0000","last_clean":"2024-09-13T22:46:17.861937+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:46:17.861937+0000","last_undegraded":"2024-09-13T22:46:17.861937+0000","last_fullsized":"2024-09-13T22:46:17.861937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:30.066021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.869932+0000","last_change":"2024-09-13T22:44:41.802443+0000","last_active":"2024-09-13T22:46:17.869932+0000","last_peered":"2024-09-13T22:46:17.869932+0000","last_clean":"2024-09-13T22:46:17.869932+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:46:17.869932+0000","last_undegraded":"2024-09-13T22:46:17.869932+0000","last_fullsized":"2024-09-13T22:46:17.869932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:41.802331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:41.802331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.891059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"82'12","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862167+0000","last_change":"2024-09-13T22:45:48.990083+0000","last_active":"2024-09-13T22:46:17.862167+0000","last_peered":"2024-09-13T22:46:17.862167+0000","last_clean":"2024-09-13T22:46:17.862167+0000","last_became_active":"2024-09-13T22:37:42.757253+0000","last_became_peered":"2024-09-13T22:37:42.757253+0000","last_unstale":"2024-09-13T22:46:17.862167+0000","last_undegraded":"2024-09-13T22:46:17.862167+0000","last_fullsized":"2024-09-13T22:46:17.862167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:45:48.989973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:48.989973+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:16.829839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":282,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861742+0000","last_change":"2024-09-13T22:44:42.914244+0000","last_active":"2024-09-13T22:46:17.861742+0000","last_peered":"2024-09-13T22:46:17.861742+0000","last_clean":"2024-09-13T22:46:17.861742+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:46:17.861742+0000","last_undegraded":"2024-09-13T22:46:17.861742+0000","last_fullsized":"2024-09-13T22:46:17.861742+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:42.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:42.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.893033+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860709+0000","last_change":"2024-09-13T22:46:02.912752+0000","last_active":"2024-09-13T22:46:17.860709+0000","last_peered":"2024-09-13T22:46:17.860709+0000","last_clean":"2024-09-13T22:46:17.860709+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:46:17.860709+0000","last_undegraded":"2024-09-13T22:46:17.860709+0000","last_fullsized":"2024-09-13T22:46:17.860709+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:02.912649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.912649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:20.458403+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860884+0000","last_change":"2024-09-13T22:45:59.889203+0000","last_active":"2024-09-13T22:46:17.860884+0000","last_peered":"2024-09-13T22:46:17.860884+0000","last_clean":"2024-09-13T22:46:17.860884+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:46:17.860884+0000","last_undegraded":"2024-09-13T22:46:17.860884+0000","last_fullsized":"2024-09-13T22:46:17.860884+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.889134+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.889134+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:14.252207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"198'5466","reported_seq":5521,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.104449+0000","last_change":"2024-09-13T22:45:10.877405+0000","last_active":"2024-09-13T22:46:24.104449+0000","last_peered":"2024-09-13T22:46:24.104449+0000","last_clean":"2024-09-13T22:46:24.104449+0000","last_became_active":"2024-09-13T22:45:10.877045+0000","last_became_peered":"2024-09-13T22:45:10.877045+0000","last_unstale":"2024-09-13T22:46:24.104449+0000","last_undegraded":"2024-09-13T22:46:24.104449+0000","last_fullsized":"2024-09-13T22:46:24.104449+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:27.160558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358219776,"num_objects":5466,"num_object_clones":0,"num_object_copies":16398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5466,"num_write_kb":349824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"114'28","reported_seq":343,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864248+0000","last_change":"2024-09-13T22:45:36.864593+0000","last_active":"2024-09-13T22:46:17.864248+0000","last_peered":"2024-09-13T22:46:17.864248+0000","last_clean":"2024-09-13T22:46:17.864248+0000","last_became_active":"2024-09-13T22:37:42.762329+0000","last_became_peered":"2024-09-13T22:37:42.762329+0000","last_unstale":"2024-09-13T22:46:17.864248+0000","last_undegraded":"2024-09-13T22:46:17.864248+0000","last_fullsized":"2024-09-13T22:46:17.864248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'28","last_scrub_stamp":"2024-09-13T22:45:36.864510+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:45:36.864510+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:36.895840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868529+0000","last_change":"2024-09-13T22:46:09.789583+0000","last_active":"2024-09-13T22:46:17.868529+0000","last_peered":"2024-09-13T22:46:17.868529+0000","last_clean":"2024-09-13T22:46:17.868529+0000","last_became_active":"2024-09-13T22:37:40.756755+0000","last_became_peered":"2024-09-13T22:37:40.756755+0000","last_unstale":"2024-09-13T22:46:17.868529+0000","last_undegraded":"2024-09-13T22:46:17.868529+0000","last_fullsized":"2024-09-13T22:46:17.868529+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:46:09.789477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:22.371823+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857457+0000","last_change":"2024-09-13T22:44:43.533920+0000","last_active":"2024-09-13T22:46:17.857457+0000","last_peered":"2024-09-13T22:46:17.857457+0000","last_clean":"2024-09-13T22:46:17.857457+0000","last_became_active":"2024-09-13T22:37:44.875771+0000","last_became_peered":"2024-09-13T22:37:44.875771+0000","last_unstale":"2024-09-13T22:46:17.857457+0000","last_undegraded":"2024-09-13T22:46:17.857457+0000","last_fullsized":"2024-09-13T22:46:17.857457+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:43.533803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:43.533803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.754350+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.874097+0000","last_change":"2024-09-13T22:45:02.748405+0000","last_active":"2024-09-13T22:46:17.874097+0000","last_peered":"2024-09-13T22:46:17.874097+0000","last_clean":"2024-09-13T22:46:17.874097+0000","last_became_active":"2024-09-13T22:37:46.920494+0000","last_became_peered":"2024-09-13T22:37:46.920494+0000","last_unstale":"2024-09-13T22:46:17.874097+0000","last_undegraded":"2024-09-13T22:46:17.874097+0000","last_fullsized":"2024-09-13T22:46:17.874097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.748330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.748330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:29.239492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":569,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865536+0000","last_change":"2024-09-13T22:45:59.823771+0000","last_active":"2024-09-13T22:46:17.865536+0000","last_peered":"2024-09-13T22:46:17.865536+0000","last_clean":"2024-09-13T22:46:17.865536+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:46:17.865536+0000","last_undegraded":"2024-09-13T22:46:17.865536+0000","last_fullsized":"2024-09-13T22:46:17.865536+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.823697+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.823697+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:29.604397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"198'5462","reported_seq":5518,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.782911+0000","last_change":"2024-09-13T22:45:10.877879+0000","last_active":"2024-09-13T22:46:24.782911+0000","last_peered":"2024-09-13T22:46:24.782911+0000","last_clean":"2024-09-13T22:46:24.782911+0000","last_became_active":"2024-09-13T22:45:10.877303+0000","last_became_peered":"2024-09-13T22:45:10.877303+0000","last_unstale":"2024-09-13T22:46:24.782911+0000","last_undegraded":"2024-09-13T22:46:24.782911+0000","last_fullsized":"2024-09-13T22:46:24.782911+0000","mapping_epoch":175,"log_start":"180'500","ondisk_log_start":"180'500","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357957632,"num_objects":5462,"num_object_clones":0,"num_object_copies":16386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5462,"num_write_kb":349568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"110'13","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.874962+0000","last_change":"2024-09-13T22:43:57.825534+0000","last_active":"2024-09-13T22:46:17.874962+0000","last_peered":"2024-09-13T22:46:17.874962+0000","last_clean":"2024-09-13T22:46:17.874962+0000","last_became_active":"2024-09-13T22:37:42.757493+0000","last_became_peered":"2024-09-13T22:37:42.757493+0000","last_unstale":"2024-09-13T22:46:17.874962+0000","last_undegraded":"2024-09-13T22:46:17.874962+0000","last_fullsized":"2024-09-13T22:46:17.874962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-09-13T22:43:57.825403+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.825403+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.191937+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860500+0000","last_change":"2024-09-13T22:44:56.623755+0000","last_active":"2024-09-13T22:46:17.860500+0000","last_peered":"2024-09-13T22:46:17.860500+0000","last_clean":"2024-09-13T22:46:17.860500+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:46:17.860500+0000","last_undegraded":"2024-09-13T22:46:17.860500+0000","last_fullsized":"2024-09-13T22:46:17.860500+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:56.623456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:56.623456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:24.628189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":406,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.872116+0000","last_change":"2024-09-13T22:46:11.045638+0000","last_active":"2024-09-13T22:46:17.872116+0000","last_peered":"2024-09-13T22:46:17.872116+0000","last_clean":"2024-09-13T22:46:17.872116+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:46:17.872116+0000","last_undegraded":"2024-09-13T22:46:17.872116+0000","last_fullsized":"2024-09-13T22:46:17.872116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:46:11.045383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:11.045383+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:24.677682+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858401+0000","last_change":"2024-09-13T22:45:41.791449+0000","last_active":"2024-09-13T22:46:17.858401+0000","last_peered":"2024-09-13T22:46:17.858401+0000","last_clean":"2024-09-13T22:46:17.858401+0000","last_became_active":"2024-09-13T22:37:46.910428+0000","last_became_peered":"2024-09-13T22:37:46.910428+0000","last_unstale":"2024-09-13T22:46:17.858401+0000","last_undegraded":"2024-09-13T22:46:17.858401+0000","last_fullsized":"2024-09-13T22:46:17.858401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:41.791344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:41.791344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:50.802267+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861587+0000","last_change":"2024-09-13T22:46:00.895765+0000","last_active":"2024-09-13T22:46:17.861587+0000","last_peered":"2024-09-13T22:46:17.861587+0000","last_clean":"2024-09-13T22:46:17.861587+0000","last_became_active":"2024-09-13T22:45:58.880701+0000","last_became_peered":"2024-09-13T22:45:58.880701+0000","last_unstale":"2024-09-13T22:46:17.861587+0000","last_undegraded":"2024-09-13T22:46:17.861587+0000","last_fullsized":"2024-09-13T22:46:17.861587+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:46:00.895612+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:46:00.895612+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:10.072551+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"198'5335","reported_seq":5391,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.045266+0000","last_change":"2024-09-13T22:45:10.877854+0000","last_active":"2024-09-13T22:46:24.045266+0000","last_peered":"2024-09-13T22:46:24.045266+0000","last_clean":"2024-09-13T22:46:24.045266+0000","last_became_active":"2024-09-13T22:45:10.877601+0000","last_became_peered":"2024-09-13T22:45:10.877601+0000","last_unstale":"2024-09-13T22:46:24.045266+0000","last_undegraded":"2024-09-13T22:46:24.045266+0000","last_fullsized":"2024-09-13T22:46:24.045266+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.590501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349634560,"num_objects":5335,"num_object_clones":0,"num_object_copies":16005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5335,"num_write_kb":341440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"82'15","reported_seq":313,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865728+0000","last_change":"2024-09-13T22:44:37.351243+0000","last_active":"2024-09-13T22:46:17.865728+0000","last_peered":"2024-09-13T22:46:17.865728+0000","last_clean":"2024-09-13T22:46:17.865728+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:46:17.865728+0000","last_undegraded":"2024-09-13T22:46:17.865728+0000","last_fullsized":"2024-09-13T22:46:17.865728+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-09-13T22:44:37.351117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:37.351117+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.322993+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861034+0000","last_change":"2024-09-13T22:45:45.786377+0000","last_active":"2024-09-13T22:46:17.861034+0000","last_peered":"2024-09-13T22:46:17.861034+0000","last_clean":"2024-09-13T22:46:17.861034+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:46:17.861034+0000","last_undegraded":"2024-09-13T22:46:17.861034+0000","last_fullsized":"2024-09-13T22:46:17.861034+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:45.786264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:45.786264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.759013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"102'12","reported_seq":399,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859996+0000","last_change":"2024-09-13T22:45:35.853389+0000","last_active":"2024-09-13T22:46:17.859996+0000","last_peered":"2024-09-13T22:46:17.859996+0000","last_clean":"2024-09-13T22:46:17.859996+0000","last_became_active":"2024-09-13T22:37:44.874615+0000","last_became_peered":"2024-09-13T22:37:44.874615+0000","last_unstale":"2024-09-13T22:46:17.859996+0000","last_undegraded":"2024-09-13T22:46:17.859996+0000","last_fullsized":"2024-09-13T22:46:17.859996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'12","last_scrub_stamp":"2024-09-13T22:45:35.853268+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:45:35.853268+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:49.605657+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861921+0000","last_change":"2024-09-13T22:45:51.001746+0000","last_active":"2024-09-13T22:46:17.861921+0000","last_peered":"2024-09-13T22:46:17.861921+0000","last_clean":"2024-09-13T22:46:17.861921+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:46:17.861921+0000","last_undegraded":"2024-09-13T22:46:17.861921+0000","last_fullsized":"2024-09-13T22:46:17.861921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:51.001676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.001676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:17.018739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"198'5324","reported_seq":5380,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.546734+0000","last_change":"2024-09-13T22:45:10.874698+0000","last_active":"2024-09-13T22:46:23.546734+0000","last_peered":"2024-09-13T22:46:23.546734+0000","last_clean":"2024-09-13T22:46:23.546734+0000","last_became_active":"2024-09-13T22:45:10.874312+0000","last_became_peered":"2024-09-13T22:45:10.874312+0000","last_unstale":"2024-09-13T22:46:23.546734+0000","last_undegraded":"2024-09-13T22:46:23.546734+0000","last_fullsized":"2024-09-13T22:46:23.546734+0000","mapping_epoch":175,"log_start":"180'600","ondisk_log_start":"180'600","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:32.487521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348913664,"num_objects":5324,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5324,"num_write_kb":340736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"82'12","reported_seq":319,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858460+0000","last_change":"2024-09-13T22:46:04.040076+0000","last_active":"2024-09-13T22:46:17.858460+0000","last_peered":"2024-09-13T22:46:17.858460+0000","last_clean":"2024-09-13T22:46:17.858460+0000","last_became_active":"2024-09-13T22:37:43.301465+0000","last_became_peered":"2024-09-13T22:37:43.301465+0000","last_unstale":"2024-09-13T22:46:17.858460+0000","last_undegraded":"2024-09-13T22:46:17.858460+0000","last_fullsized":"2024-09-13T22:46:17.858460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:46:04.039938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:04.039938+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:23.442060+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865788+0000","last_change":"2024-09-13T22:44:51.782458+0000","last_active":"2024-09-13T22:46:17.865788+0000","last_peered":"2024-09-13T22:46:17.865788+0000","last_clean":"2024-09-13T22:46:17.865788+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:46:17.865788+0000","last_undegraded":"2024-09-13T22:46:17.865788+0000","last_fullsized":"2024-09-13T22:46:17.865788+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:51.782326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:51.782326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:23.899854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":87,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867677+0000","last_change":"2024-09-13T22:46:02.794026+0000","last_active":"2024-09-13T22:46:17.867677+0000","last_peered":"2024-09-13T22:46:17.867677+0000","last_clean":"2024-09-13T22:46:17.867677+0000","last_became_active":"2024-09-13T22:44:17.652323+0000","last_became_peered":"2024-09-13T22:44:17.652323+0000","last_unstale":"2024-09-13T22:46:17.867677+0000","last_undegraded":"2024-09-13T22:46:17.867677+0000","last_fullsized":"2024-09-13T22:46:17.867677+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.793958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:22.857754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866658+0000","last_change":"2024-09-13T22:45:03.478116+0000","last_active":"2024-09-13T22:46:17.866658+0000","last_peered":"2024-09-13T22:46:17.866658+0000","last_clean":"2024-09-13T22:46:17.866658+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:46:17.866658+0000","last_undegraded":"2024-09-13T22:46:17.866658+0000","last_fullsized":"2024-09-13T22:46:17.866658+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.477932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.477932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866608+0000","last_change":"2024-09-13T22:45:59.360761+0000","last_active":"2024-09-13T22:46:17.866608+0000","last_peered":"2024-09-13T22:46:17.866608+0000","last_clean":"2024-09-13T22:46:17.866608+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:46:17.866608+0000","last_undegraded":"2024-09-13T22:46:17.866608+0000","last_fullsized":"2024-09-13T22:46:17.866608+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.360631+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:19.619807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"198'5381","reported_seq":5437,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.746387+0000","last_change":"2024-09-13T22:45:11.075836+0000","last_active":"2024-09-13T22:46:23.746387+0000","last_peered":"2024-09-13T22:46:23.746387+0000","last_clean":"2024-09-13T22:46:23.746387+0000","last_became_active":"2024-09-13T22:45:11.075425+0000","last_became_peered":"2024-09-13T22:45:11.075425+0000","last_unstale":"2024-09-13T22:46:23.746387+0000","last_undegraded":"2024-09-13T22:46:23.746387+0000","last_fullsized":"2024-09-13T22:46:23.746387+0000","mapping_epoch":175,"log_start":"180'600","ondisk_log_start":"180'600","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352649216,"num_objects":5381,"num_object_clones":0,"num_object_copies":16143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":344384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"82'12","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868144+0000","last_change":"2024-09-13T22:44:37.772426+0000","last_active":"2024-09-13T22:46:17.868144+0000","last_peered":"2024-09-13T22:46:17.868144+0000","last_clean":"2024-09-13T22:46:17.868144+0000","last_became_active":"2024-09-13T22:37:42.759869+0000","last_became_peered":"2024-09-13T22:37:42.759869+0000","last_unstale":"2024-09-13T22:46:17.868144+0000","last_undegraded":"2024-09-13T22:46:17.868144+0000","last_fullsized":"2024-09-13T22:46:17.868144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:44:37.772333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:37.772333+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864921+0000","last_change":"2024-09-13T22:46:08.055128+0000","last_active":"2024-09-13T22:46:17.864921+0000","last_peered":"2024-09-13T22:46:17.864921+0000","last_clean":"2024-09-13T22:46:17.864921+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:46:17.864921+0000","last_undegraded":"2024-09-13T22:46:17.864921+0000","last_fullsized":"2024-09-13T22:46:17.864921+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:08.055012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.055012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:22.514209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858238+0000","last_change":"2024-09-13T22:46:07.018597+0000","last_active":"2024-09-13T22:46:17.858238+0000","last_peered":"2024-09-13T22:46:17.858238+0000","last_clean":"2024-09-13T22:46:17.858238+0000","last_became_active":"2024-09-13T22:37:42.764575+0000","last_became_peered":"2024-09-13T22:37:42.764575+0000","last_unstale":"2024-09-13T22:46:17.858238+0000","last_undegraded":"2024-09-13T22:46:17.858238+0000","last_fullsized":"2024-09-13T22:46:17.858238+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:46:07.018490+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:12.090624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"82'11","reported_seq":399,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865475+0000","last_change":"2024-09-13T22:45:38.494596+0000","last_active":"2024-09-13T22:46:17.865475+0000","last_peered":"2024-09-13T22:46:17.865475+0000","last_clean":"2024-09-13T22:46:17.865475+0000","last_became_active":"2024-09-13T22:37:44.862853+0000","last_became_peered":"2024-09-13T22:37:44.862853+0000","last_unstale":"2024-09-13T22:46:17.865475+0000","last_undegraded":"2024-09-13T22:46:17.865475+0000","last_fullsized":"2024-09-13T22:46:17.865475+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:38.494521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:38.494521+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:54.521018+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":271,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866543+0000","last_change":"2024-09-13T22:44:03.606280+0000","last_active":"2024-09-13T22:46:17.866543+0000","last_peered":"2024-09-13T22:46:17.866543+0000","last_clean":"2024-09-13T22:46:17.866543+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:46:17.866543+0000","last_undegraded":"2024-09-13T22:46:17.866543+0000","last_fullsized":"2024-09-13T22:46:17.866543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:03.606176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:03.606176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.890936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859760+0000","last_change":"2024-09-13T22:45:59.049789+0000","last_active":"2024-09-13T22:46:17.859760+0000","last_peered":"2024-09-13T22:46:17.859760+0000","last_clean":"2024-09-13T22:46:17.859760+0000","last_became_active":"2024-09-13T22:45:58.861436+0000","last_became_peered":"2024-09-13T22:45:58.861436+0000","last_unstale":"2024-09-13T22:46:17.859760+0000","last_undegraded":"2024-09-13T22:46:17.859760+0000","last_fullsized":"2024-09-13T22:46:17.859760+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.049671+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.049671+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:05.863022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"198'5354","reported_seq":5410,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.653972+0000","last_change":"2024-09-13T22:45:11.071616+0000","last_active":"2024-09-13T22:46:24.653972+0000","last_peered":"2024-09-13T22:46:24.653972+0000","last_clean":"2024-09-13T22:46:24.653972+0000","last_became_active":"2024-09-13T22:45:11.071318+0000","last_became_peered":"2024-09-13T22:45:11.071318+0000","last_unstale":"2024-09-13T22:46:24.653972+0000","last_undegraded":"2024-09-13T22:46:24.653972+0000","last_fullsized":"2024-09-13T22:46:24.653972+0000","mapping_epoch":175,"log_start":"180'200","ondisk_log_start":"180'200","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:34.473688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350879744,"num_objects":5354,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":342656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"103'18","reported_seq":320,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867753+0000","last_change":"2024-09-13T22:45:01.550550+0000","last_active":"2024-09-13T22:46:17.867753+0000","last_peered":"2024-09-13T22:46:17.867753+0000","last_clean":"2024-09-13T22:46:17.867753+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:46:17.867753+0000","last_undegraded":"2024-09-13T22:46:17.867753+0000","last_fullsized":"2024-09-13T22:46:17.867753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:45:01.550400+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.550400+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.323145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"75'1","reported_seq":293,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860802+0000","last_change":"2024-09-13T22:44:49.488944+0000","last_active":"2024-09-13T22:46:17.860802+0000","last_peered":"2024-09-13T22:46:17.860802+0000","last_clean":"2024-09-13T22:46:17.860802+0000","last_became_active":"2024-09-13T22:37:40.746250+0000","last_became_peered":"2024-09-13T22:37:40.746250+0000","last_unstale":"2024-09-13T22:46:17.860802+0000","last_undegraded":"2024-09-13T22:46:17.860802+0000","last_fullsized":"2024-09-13T22:46:17.860802+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:49.488854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:49.488854+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:24.628163+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":800,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.135326+0000","last_change":"2024-09-13T22:45:54.489191+0000","last_active":"2024-09-13T22:46:23.135326+0000","last_peered":"2024-09-13T22:46:23.135326+0000","last_clean":"2024-09-13T22:46:23.135326+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:46:23.135326+0000","last_undegraded":"2024-09-13T22:46:23.135326+0000","last_fullsized":"2024-09-13T22:46:23.135326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:45:54.489086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.489086+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:15.841934+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":502,"num_read_kb":497,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857951+0000","last_change":"2024-09-13T22:45:51.936085+0000","last_active":"2024-09-13T22:46:17.857951+0000","last_peered":"2024-09-13T22:46:17.857951+0000","last_clean":"2024-09-13T22:46:17.857951+0000","last_became_active":"2024-09-13T22:37:44.864942+0000","last_became_peered":"2024-09-13T22:37:44.864942+0000","last_unstale":"2024-09-13T22:46:17.857951+0000","last_undegraded":"2024-09-13T22:46:17.857951+0000","last_fullsized":"2024-09-13T22:46:17.857951+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:51.936005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.936005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:16.788125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858074+0000","last_change":"2024-09-13T22:45:37.838815+0000","last_active":"2024-09-13T22:46:17.858074+0000","last_peered":"2024-09-13T22:46:17.858074+0000","last_clean":"2024-09-13T22:46:17.858074+0000","last_became_active":"2024-09-13T22:37:46.923721+0000","last_became_peered":"2024-09-13T22:37:46.923721+0000","last_unstale":"2024-09-13T22:46:17.858074+0000","last_undegraded":"2024-09-13T22:46:17.858074+0000","last_fullsized":"2024-09-13T22:46:17.858074+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:37.838733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:45:37.838733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:46.977696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":350,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864614+0000","last_change":"2024-09-13T22:46:00.103065+0000","last_active":"2024-09-13T22:46:17.864614+0000","last_peered":"2024-09-13T22:46:17.864614+0000","last_clean":"2024-09-13T22:46:17.864614+0000","last_became_active":"2024-09-13T22:45:58.866066+0000","last_became_peered":"2024-09-13T22:45:58.866066+0000","last_unstale":"2024-09-13T22:46:17.864614+0000","last_undegraded":"2024-09-13T22:46:17.864614+0000","last_fullsized":"2024-09-13T22:46:17.864614+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:46:00.102990+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:46:00.102990+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:20.933188+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"198'5454","reported_seq":5510,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.787192+0000","last_change":"2024-09-13T22:45:10.879441+0000","last_active":"2024-09-13T22:46:24.787192+0000","last_peered":"2024-09-13T22:46:24.787192+0000","last_clean":"2024-09-13T22:46:24.787192+0000","last_became_active":"2024-09-13T22:45:10.879109+0000","last_became_peered":"2024-09-13T22:45:10.879109+0000","last_unstale":"2024-09-13T22:46:24.787192+0000","last_undegraded":"2024-09-13T22:46:24.787192+0000","last_fullsized":"2024-09-13T22:46:24.787192+0000","mapping_epoch":175,"log_start":"180'500","ondisk_log_start":"180'500","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.324752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357433344,"num_objects":5454,"num_object_clones":0,"num_object_copies":16362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5454,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"82'30","reported_seq":443,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860089+0000","last_change":"2024-09-13T22:45:00.620616+0000","last_active":"2024-09-13T22:46:17.860089+0000","last_peered":"2024-09-13T22:46:17.860089+0000","last_clean":"2024-09-13T22:46:17.860089+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:46:17.860089+0000","last_undegraded":"2024-09-13T22:46:17.860089+0000","last_fullsized":"2024-09-13T22:46:17.860089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'30","last_scrub_stamp":"2024-09-13T22:45:00.620468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:00.620468+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.590538+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868284+0000","last_change":"2024-09-13T22:46:03.796946+0000","last_active":"2024-09-13T22:46:17.868284+0000","last_peered":"2024-09-13T22:46:17.868284+0000","last_clean":"2024-09-13T22:46:17.868284+0000","last_became_active":"2024-09-13T22:37:40.753535+0000","last_became_peered":"2024-09-13T22:37:40.753535+0000","last_unstale":"2024-09-13T22:46:17.868284+0000","last_undegraded":"2024-09-13T22:46:17.868284+0000","last_fullsized":"2024-09-13T22:46:17.868284+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:03.796876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:46:03.796876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:23.589339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864136+0000","last_change":"2024-09-13T22:46:04.047287+0000","last_active":"2024-09-13T22:46:17.864136+0000","last_peered":"2024-09-13T22:46:17.864136+0000","last_clean":"2024-09-13T22:46:17.864136+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:46:17.864136+0000","last_undegraded":"2024-09-13T22:46:17.864136+0000","last_fullsized":"2024-09-13T22:46:17.864136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:04.047209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:04.047209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:10.419272+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857789+0000","last_change":"2024-09-13T22:44:54.588476+0000","last_active":"2024-09-13T22:46:17.857789+0000","last_peered":"2024-09-13T22:46:17.857789+0000","last_clean":"2024-09-13T22:46:17.857789+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:46:17.857789+0000","last_undegraded":"2024-09-13T22:46:17.857789+0000","last_fullsized":"2024-09-13T22:46:17.857789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:54.588262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:54.588262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:20.709337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":350,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864492+0000","last_change":"2024-09-13T22:45:59.111852+0000","last_active":"2024-09-13T22:46:17.864492+0000","last_peered":"2024-09-13T22:46:17.864492+0000","last_clean":"2024-09-13T22:46:17.864492+0000","last_became_active":"2024-09-13T22:45:58.862759+0000","last_became_peered":"2024-09-13T22:45:58.862759+0000","last_unstale":"2024-09-13T22:46:17.864492+0000","last_undegraded":"2024-09-13T22:46:17.864492+0000","last_fullsized":"2024-09-13T22:46:17.864492+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.111662+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.111662+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:25.737936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"198'5436","reported_seq":5492,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.630676+0000","last_change":"2024-09-13T22:45:10.878616+0000","last_active":"2024-09-13T22:46:24.630676+0000","last_peered":"2024-09-13T22:46:24.630676+0000","last_clean":"2024-09-13T22:46:24.630676+0000","last_became_active":"2024-09-13T22:45:10.877979+0000","last_became_peered":"2024-09-13T22:45:10.877979+0000","last_unstale":"2024-09-13T22:46:24.630676+0000","last_undegraded":"2024-09-13T22:46:24.630676+0000","last_fullsized":"2024-09-13T22:46:24.630676+0000","mapping_epoch":175,"log_start":"180'300","ondisk_log_start":"180'300","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:38.861678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356253696,"num_objects":5436,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":347904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"103'24","reported_seq":422,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862251+0000","last_change":"2024-09-13T22:44:58.932861+0000","last_active":"2024-09-13T22:46:17.862251+0000","last_peered":"2024-09-13T22:46:17.862251+0000","last_clean":"2024-09-13T22:46:17.862251+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:46:17.862251+0000","last_undegraded":"2024-09-13T22:46:17.862251+0000","last_fullsized":"2024-09-13T22:46:17.862251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:44:58.932607+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.932607+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:28.776680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857649+0000","last_change":"2024-09-13T22:45:51.851690+0000","last_active":"2024-09-13T22:46:17.857649+0000","last_peered":"2024-09-13T22:46:17.857649+0000","last_clean":"2024-09-13T22:46:17.857649+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:46:17.857649+0000","last_undegraded":"2024-09-13T22:46:17.857649+0000","last_fullsized":"2024-09-13T22:46:17.857649+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:45:51.851565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.851565+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.005276+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860408+0000","last_change":"2024-09-13T22:46:05.839012+0000","last_active":"2024-09-13T22:46:17.860408+0000","last_peered":"2024-09-13T22:46:17.860408+0000","last_clean":"2024-09-13T22:46:17.860408+0000","last_became_active":"2024-09-13T22:37:42.755709+0000","last_became_peered":"2024-09-13T22:37:42.755709+0000","last_unstale":"2024-09-13T22:46:17.860408+0000","last_undegraded":"2024-09-13T22:46:17.860408+0000","last_fullsized":"2024-09-13T22:46:17.860408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:46:05.838788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:05.838788+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:34.243906+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":408,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861712+0000","last_change":"2024-09-13T22:45:32.781118+0000","last_active":"2024-09-13T22:46:17.861712+0000","last_peered":"2024-09-13T22:46:17.861712+0000","last_clean":"2024-09-13T22:46:17.861712+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:46:17.861712+0000","last_undegraded":"2024-09-13T22:46:17.861712+0000","last_fullsized":"2024-09-13T22:46:17.861712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:45:32.781043+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:45:32.781043+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:38.849221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860830+0000","last_change":"2024-09-13T22:44:55.859959+0000","last_active":"2024-09-13T22:46:17.860830+0000","last_peered":"2024-09-13T22:46:17.860830+0000","last_clean":"2024-09-13T22:46:17.860830+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:46:17.860830+0000","last_undegraded":"2024-09-13T22:46:17.860830+0000","last_fullsized":"2024-09-13T22:46:17.860830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:55.859847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.859847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861556+0000","last_change":"2024-09-13T22:45:58.899570+0000","last_active":"2024-09-13T22:46:17.861556+0000","last_peered":"2024-09-13T22:46:17.861556+0000","last_clean":"2024-09-13T22:46:17.861556+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:46:17.861556+0000","last_undegraded":"2024-09-13T22:46:17.861556+0000","last_fullsized":"2024-09-13T22:46:17.861556+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:58.899459+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:58.899459+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:10.690347+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"198'5278","reported_seq":5334,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.717846+0000","last_change":"2024-09-13T22:45:11.072418+0000","last_active":"2024-09-13T22:46:23.717846+0000","last_peered":"2024-09-13T22:46:23.717846+0000","last_clean":"2024-09-13T22:46:23.717846+0000","last_became_active":"2024-09-13T22:45:11.071983+0000","last_became_peered":"2024-09-13T22:45:11.071983+0000","last_unstale":"2024-09-13T22:46:23.717846+0000","last_undegraded":"2024-09-13T22:46:23.717846+0000","last_fullsized":"2024-09-13T22:46:23.717846+0000","mapping_epoch":175,"log_start":"180'500","ondisk_log_start":"180'500","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:29.204118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345899008,"num_objects":5278,"num_object_clones":0,"num_object_copies":15834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5278,"num_write_kb":337792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"82'19","reported_seq":332,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858453+0000","last_change":"2024-09-13T22:45:52.915083+0000","last_active":"2024-09-13T22:46:17.858453+0000","last_peered":"2024-09-13T22:46:17.858453+0000","last_clean":"2024-09-13T22:46:17.858453+0000","last_became_active":"2024-09-13T22:37:42.764340+0000","last_became_peered":"2024-09-13T22:37:42.764340+0000","last_unstale":"2024-09-13T22:46:17.858453+0000","last_undegraded":"2024-09-13T22:46:17.858453+0000","last_fullsized":"2024-09-13T22:46:17.858453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:45:52.914946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:52.914946+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:08.812183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865069+0000","last_change":"2024-09-13T22:45:42.036125+0000","last_active":"2024-09-13T22:46:17.865069+0000","last_peered":"2024-09-13T22:46:17.865069+0000","last_clean":"2024-09-13T22:46:17.865069+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:46:17.865069+0000","last_undegraded":"2024-09-13T22:46:17.865069+0000","last_fullsized":"2024-09-13T22:46:17.865069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.036015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.036015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:03.182339+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862480+0000","last_change":"2024-09-13T22:45:44.827947+0000","last_active":"2024-09-13T22:46:17.862480+0000","last_peered":"2024-09-13T22:46:17.862480+0000","last_clean":"2024-09-13T22:46:17.862480+0000","last_became_active":"2024-09-13T22:37:44.875518+0000","last_became_peered":"2024-09-13T22:37:44.875518+0000","last_unstale":"2024-09-13T22:46:17.862480+0000","last_undegraded":"2024-09-13T22:46:17.862480+0000","last_fullsized":"2024-09-13T22:46:17.862480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:44.827803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:45:44.827803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.246638+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866236+0000","last_change":"2024-09-13T22:45:42.536122+0000","last_active":"2024-09-13T22:46:17.866236+0000","last_peered":"2024-09-13T22:46:17.866236+0000","last_clean":"2024-09-13T22:46:17.866236+0000","last_became_active":"2024-09-13T22:37:46.918627+0000","last_became_peered":"2024-09-13T22:37:46.918627+0000","last_unstale":"2024-09-13T22:46:17.866236+0000","last_undegraded":"2024-09-13T22:46:17.866236+0000","last_fullsized":"2024-09-13T22:46:17.866236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.535993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.535993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:54.671556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":569,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866277+0000","last_change":"2024-09-13T22:46:00.838997+0000","last_active":"2024-09-13T22:46:17.866277+0000","last_peered":"2024-09-13T22:46:17.866277+0000","last_clean":"2024-09-13T22:46:17.866277+0000","last_became_active":"2024-09-13T22:45:58.867668+0000","last_became_peered":"2024-09-13T22:45:58.867668+0000","last_unstale":"2024-09-13T22:46:17.866277+0000","last_undegraded":"2024-09-13T22:46:17.866277+0000","last_fullsized":"2024-09-13T22:46:17.866277+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:46:00.838808+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:46:00.838808+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:26.089611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"198'5366","reported_seq":5421,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.808169+0000","last_change":"2024-09-13T22:45:10.881296+0000","last_active":"2024-09-13T22:46:24.808169+0000","last_peered":"2024-09-13T22:46:24.808169+0000","last_clean":"2024-09-13T22:46:24.808169+0000","last_became_active":"2024-09-13T22:45:10.880563+0000","last_became_peered":"2024-09-13T22:45:10.880563+0000","last_unstale":"2024-09-13T22:46:24.808169+0000","last_undegraded":"2024-09-13T22:46:24.808169+0000","last_fullsized":"2024-09-13T22:46:24.808169+0000","mapping_epoch":175,"log_start":"180'400","ondisk_log_start":"180'400","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:25.324786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351666176,"num_objects":5366,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":343424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":328,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857335+0000","last_change":"2024-09-13T22:44:55.600907+0000","last_active":"2024-09-13T22:46:17.857335+0000","last_peered":"2024-09-13T22:46:17.857335+0000","last_clean":"2024-09-13T22:46:17.857335+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:46:17.857335+0000","last_undegraded":"2024-09-13T22:46:17.857335+0000","last_fullsized":"2024-09-13T22:46:17.857335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-09-13T22:44:55.600799+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.600799+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:22.514411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.871714+0000","last_change":"2024-09-13T22:45:41.824246+0000","last_active":"2024-09-13T22:46:17.871714+0000","last_peered":"2024-09-13T22:46:17.871714+0000","last_clean":"2024-09-13T22:46:17.871714+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:46:17.871714+0000","last_undegraded":"2024-09-13T22:46:17.871714+0000","last_fullsized":"2024-09-13T22:46:17.871714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:45:41.824173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:41.824173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:59.277858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.874299+0000","last_change":"2024-09-13T22:45:57.976559+0000","last_active":"2024-09-13T22:46:17.874299+0000","last_peered":"2024-09-13T22:46:17.874299+0000","last_clean":"2024-09-13T22:46:17.874299+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:46:17.874299+0000","last_undegraded":"2024-09-13T22:46:17.874299+0000","last_fullsized":"2024-09-13T22:46:17.874299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:57.976304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:45:57.976304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:13.972793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868043+0000","last_change":"2024-09-13T22:45:43.808511+0000","last_active":"2024-09-13T22:46:17.868043+0000","last_peered":"2024-09-13T22:46:17.868043+0000","last_clean":"2024-09-13T22:46:17.868043+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:46:17.868043+0000","last_undegraded":"2024-09-13T22:46:17.868043+0000","last_fullsized":"2024-09-13T22:46:17.868043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:43.808443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:43.808443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:45.844028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857209+0000","last_change":"2024-09-13T22:45:59.802335+0000","last_active":"2024-09-13T22:46:17.857209+0000","last_peered":"2024-09-13T22:46:17.857209+0000","last_clean":"2024-09-13T22:46:17.857209+0000","last_became_active":"2024-09-13T22:45:58.871527+0000","last_became_peered":"2024-09-13T22:45:58.871527+0000","last_unstale":"2024-09-13T22:46:17.857209+0000","last_undegraded":"2024-09-13T22:46:17.857209+0000","last_fullsized":"2024-09-13T22:46:17.857209+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.802229+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.802229+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.802229+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:18.223900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"198'5399","reported_seq":5455,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.099059+0000","last_change":"2024-09-13T22:45:10.878791+0000","last_active":"2024-09-13T22:46:24.099059+0000","last_peered":"2024-09-13T22:46:24.099059+0000","last_clean":"2024-09-13T22:46:24.099059+0000","last_became_active":"2024-09-13T22:45:10.878487+0000","last_became_peered":"2024-09-13T22:45:10.878487+0000","last_unstale":"2024-09-13T22:46:24.099059+0000","last_undegraded":"2024-09-13T22:46:24.099059+0000","last_fullsized":"2024-09-13T22:46:24.099059+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5399,"log_dups_size":0,"ondisk_log_size":5399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353828864,"num_objects":5399,"num_object_clones":0,"num_object_copies":16197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":345536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"82'14","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860939+0000","last_change":"2024-09-13T22:45:51.960851+0000","last_active":"2024-09-13T22:46:17.860939+0000","last_peered":"2024-09-13T22:46:17.860939+0000","last_clean":"2024-09-13T22:46:17.860939+0000","last_became_active":"2024-09-13T22:37:42.760533+0000","last_became_peered":"2024-09-13T22:37:42.760533+0000","last_unstale":"2024-09-13T22:46:17.860939+0000","last_undegraded":"2024-09-13T22:46:17.860939+0000","last_fullsized":"2024-09-13T22:46:17.860939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'14","last_scrub_stamp":"2024-09-13T22:45:51.960770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.960770+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.139158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865282+0000","last_change":"2024-09-13T22:44:52.527561+0000","last_active":"2024-09-13T22:46:17.865282+0000","last_peered":"2024-09-13T22:46:17.865282+0000","last_clean":"2024-09-13T22:46:17.865282+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:46:17.865282+0000","last_undegraded":"2024-09-13T22:46:17.865282+0000","last_fullsized":"2024-09-13T22:46:17.865282+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:44:01.750360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.890869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862621+0000","last_change":"2024-09-13T22:46:07.796968+0000","last_active":"2024-09-13T22:46:17.862621+0000","last_peered":"2024-09-13T22:46:17.862621+0000","last_clean":"2024-09-13T22:46:17.862621+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:46:17.862621+0000","last_undegraded":"2024-09-13T22:46:17.862621+0000","last_fullsized":"2024-09-13T22:46:17.862621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:07.796849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:46:07.796849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:19.001254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857359+0000","last_change":"2024-09-13T22:44:42.511563+0000","last_active":"2024-09-13T22:46:17.857359+0000","last_peered":"2024-09-13T22:46:17.857359+0000","last_clean":"2024-09-13T22:46:17.857359+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:46:17.857359+0000","last_undegraded":"2024-09-13T22:46:17.857359+0000","last_fullsized":"2024-09-13T22:46:17.857359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:42.511177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:44:42.511177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.754258+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"198'5417","reported_seq":5473,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.726068+0000","last_change":"2024-09-13T22:45:10.876077+0000","last_active":"2024-09-13T22:46:23.726068+0000","last_peered":"2024-09-13T22:46:23.726068+0000","last_clean":"2024-09-13T22:46:23.726068+0000","last_became_active":"2024-09-13T22:45:10.875657+0000","last_became_peered":"2024-09-13T22:45:10.875657+0000","last_unstale":"2024-09-13T22:46:23.726068+0000","last_undegraded":"2024-09-13T22:46:23.726068+0000","last_fullsized":"2024-09-13T22:46:23.726068+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:24.945120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355008512,"num_objects":5417,"num_object_clones":0,"num_object_copies":16251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":346688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'10","reported_seq":303,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865952+0000","last_change":"2024-09-13T22:44:55.563811+0000","last_active":"2024-09-13T22:46:17.865952+0000","last_peered":"2024-09-13T22:46:17.865952+0000","last_clean":"2024-09-13T22:46:17.865952+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:46:17.865952+0000","last_undegraded":"2024-09-13T22:46:17.865952+0000","last_fullsized":"2024-09-13T22:46:17.865952+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:44:55.563593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.563593+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861906+0000","last_change":"2024-09-13T22:46:03.807801+0000","last_active":"2024-09-13T22:46:17.861906+0000","last_peered":"2024-09-13T22:46:17.861906+0000","last_clean":"2024-09-13T22:46:17.861906+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:46:17.861906+0000","last_undegraded":"2024-09-13T22:46:17.861906+0000","last_fullsized":"2024-09-13T22:46:17.861906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:03.807736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:46:03.807736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:23.153119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":561,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865943+0000","last_change":"2024-09-13T22:45:54.760070+0000","last_active":"2024-09-13T22:46:17.865943+0000","last_peered":"2024-09-13T22:46:17.865943+0000","last_clean":"2024-09-13T22:46:17.865943+0000","last_became_active":"2024-09-13T22:45:53.877322+0000","last_became_peered":"2024-09-13T22:45:53.877322+0000","last_unstale":"2024-09-13T22:46:17.865943+0000","last_undegraded":"2024-09-13T22:46:17.865943+0000","last_fullsized":"2024-09-13T22:46:17.865943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:54.759995+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.759995+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:15.848331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867740+0000","last_change":"2024-09-13T22:44:59.821248+0000","last_active":"2024-09-13T22:46:17.867740+0000","last_peered":"2024-09-13T22:46:17.867740+0000","last_clean":"2024-09-13T22:46:17.867740+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:46:17.867740+0000","last_undegraded":"2024-09-13T22:46:17.867740+0000","last_fullsized":"2024-09-13T22:46:17.867740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:59.820971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:44:59.820971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.857639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862505+0000","last_change":"2024-09-13T22:45:47.036211+0000","last_active":"2024-09-13T22:46:17.862505+0000","last_peered":"2024-09-13T22:46:17.862505+0000","last_clean":"2024-09-13T22:46:17.862505+0000","last_became_active":"2024-09-13T22:37:46.912399+0000","last_became_peered":"2024-09-13T22:37:46.912399+0000","last_unstale":"2024-09-13T22:46:17.862505+0000","last_undegraded":"2024-09-13T22:46:17.862505+0000","last_fullsized":"2024-09-13T22:46:17.862505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:47.036097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:47.036097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:08.330189+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"198'5433","reported_seq":5489,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.646090+0000","last_change":"2024-09-13T22:45:11.071425+0000","last_active":"2024-09-13T22:46:24.646090+0000","last_peered":"2024-09-13T22:46:24.646090+0000","last_clean":"2024-09-13T22:46:24.646090+0000","last_became_active":"2024-09-13T22:45:11.071153+0000","last_became_peered":"2024-09-13T22:45:11.071153+0000","last_unstale":"2024-09-13T22:46:24.646090+0000","last_undegraded":"2024-09-13T22:46:24.646090+0000","last_fullsized":"2024-09-13T22:46:24.646090+0000","mapping_epoch":175,"log_start":"180'300","ondisk_log_start":"180'300","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:27.100031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356057088,"num_objects":5433,"num_object_clones":0,"num_object_copies":16299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":347712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"82'17","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868162+0000","last_change":"2024-09-13T22:44:48.862741+0000","last_active":"2024-09-13T22:46:17.868162+0000","last_peered":"2024-09-13T22:46:17.868162+0000","last_clean":"2024-09-13T22:46:17.868162+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:46:17.868162+0000","last_undegraded":"2024-09-13T22:46:17.868162+0000","last_fullsized":"2024-09-13T22:46:17.868162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'17","last_scrub_stamp":"2024-09-13T22:44:48.862647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:48.862647+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.872333+0000","last_change":"2024-09-13T22:46:08.022638+0000","last_active":"2024-09-13T22:46:17.872333+0000","last_peered":"2024-09-13T22:46:17.872333+0000","last_clean":"2024-09-13T22:46:17.872333+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:46:17.872333+0000","last_undegraded":"2024-09-13T22:46:17.872333+0000","last_fullsized":"2024-09-13T22:46:17.872333+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:08.022538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.022538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:25.754828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.871801+0000","last_change":"2024-09-13T22:44:54.771739+0000","last_active":"2024-09-13T22:46:17.871801+0000","last_peered":"2024-09-13T22:46:17.871801+0000","last_clean":"2024-09-13T22:46:17.871801+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:46:17.871801+0000","last_undegraded":"2024-09-13T22:46:17.871801+0000","last_fullsized":"2024-09-13T22:46:17.871801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:54.771552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:54.771552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:19.451322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865848+0000","last_change":"2024-09-13T22:46:10.833710+0000","last_active":"2024-09-13T22:46:17.865848+0000","last_peered":"2024-09-13T22:46:17.865848+0000","last_clean":"2024-09-13T22:46:17.865848+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:46:17.865848+0000","last_undegraded":"2024-09-13T22:46:17.865848+0000","last_fullsized":"2024-09-13T22:46:17.865848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:10.833635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:46:10.833635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:31.305211+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"198'5375","reported_seq":5431,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:24.107941+0000","last_change":"2024-09-13T22:45:10.879726+0000","last_active":"2024-09-13T22:46:24.107941+0000","last_peered":"2024-09-13T22:46:24.107941+0000","last_clean":"2024-09-13T22:46:24.107941+0000","last_became_active":"2024-09-13T22:45:10.879140+0000","last_became_peered":"2024-09-13T22:45:10.879140+0000","last_unstale":"2024-09-13T22:46:24.107941+0000","last_undegraded":"2024-09-13T22:46:24.107941+0000","last_fullsized":"2024-09-13T22:46:24.107941+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:37.379391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256000,"num_objects":5375,"num_object_clones":0,"num_object_copies":16125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5375,"num_write_kb":344000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"82'10","reported_seq":309,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862141+0000","last_change":"2024-09-13T22:45:50.017524+0000","last_active":"2024-09-13T22:46:17.862141+0000","last_peered":"2024-09-13T22:46:17.862141+0000","last_clean":"2024-09-13T22:46:17.862141+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:46:17.862141+0000","last_undegraded":"2024-09-13T22:46:17.862141+0000","last_fullsized":"2024-09-13T22:46:17.862141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:45:50.017444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:50.017444+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:17.060795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860270+0000","last_change":"2024-09-13T22:45:42.828054+0000","last_active":"2024-09-13T22:46:17.860270+0000","last_peered":"2024-09-13T22:46:17.860270+0000","last_clean":"2024-09-13T22:46:17.860270+0000","last_became_active":"2024-09-13T22:37:40.733119+0000","last_became_peered":"2024-09-13T22:37:40.733119+0000","last_unstale":"2024-09-13T22:46:17.860270+0000","last_undegraded":"2024-09-13T22:46:17.860270+0000","last_fullsized":"2024-09-13T22:46:17.860270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.827981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.827981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:44.176382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.872237+0000","last_change":"2024-09-13T22:45:37.777432+0000","last_active":"2024-09-13T22:46:17.872237+0000","last_peered":"2024-09-13T22:46:17.872237+0000","last_clean":"2024-09-13T22:46:17.872237+0000","last_became_active":"2024-09-13T22:37:44.862328+0000","last_became_peered":"2024-09-13T22:37:44.862328+0000","last_unstale":"2024-09-13T22:46:17.872237+0000","last_undegraded":"2024-09-13T22:46:17.872237+0000","last_fullsized":"2024-09-13T22:46:17.872237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:37.777194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:37.777194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:53.772400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860857+0000","last_change":"2024-09-13T22:44:44.797246+0000","last_active":"2024-09-13T22:46:17.860857+0000","last_peered":"2024-09-13T22:46:17.860857+0000","last_clean":"2024-09-13T22:46:17.860857+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:46:17.860857+0000","last_undegraded":"2024-09-13T22:46:17.860857+0000","last_fullsized":"2024-09-13T22:46:17.860857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:44.796884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:44:44.796884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.855866+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"198'5404","reported_seq":5459,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.744986+0000","last_change":"2024-09-13T22:45:10.873543+0000","last_active":"2024-09-13T22:46:23.744986+0000","last_peered":"2024-09-13T22:46:23.744986+0000","last_clean":"2024-09-13T22:46:23.744986+0000","last_became_active":"2024-09-13T22:45:10.873170+0000","last_became_peered":"2024-09-13T22:45:10.873170+0000","last_unstale":"2024-09-13T22:46:23.744986+0000","last_undegraded":"2024-09-13T22:46:23.744986+0000","last_fullsized":"2024-09-13T22:46:23.744986+0000","mapping_epoch":175,"log_start":"180'700","ondisk_log_start":"180'700","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:27.152804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354156544,"num_objects":5404,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5404,"num_write_kb":345856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"107'22","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867779+0000","last_change":"2024-09-13T22:44:47.859269+0000","last_active":"2024-09-13T22:46:17.867779+0000","last_peered":"2024-09-13T22:46:17.867779+0000","last_clean":"2024-09-13T22:46:17.867779+0000","last_became_active":"2024-09-13T22:37:42.764180+0000","last_became_peered":"2024-09-13T22:37:42.764180+0000","last_unstale":"2024-09-13T22:46:17.867779+0000","last_undegraded":"2024-09-13T22:46:17.867779+0000","last_fullsized":"2024-09-13T22:46:17.867779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'22","last_scrub_stamp":"2024-09-13T22:44:47.859159+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:44:47.859159+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.891031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.874500+0000","last_change":"2024-09-13T22:45:40.816983+0000","last_active":"2024-09-13T22:46:17.874500+0000","last_peered":"2024-09-13T22:46:17.874500+0000","last_clean":"2024-09-13T22:46:17.874500+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:46:17.874500+0000","last_undegraded":"2024-09-13T22:46:17.874500+0000","last_fullsized":"2024-09-13T22:46:17.874500+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:40.816827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:45:40.816827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:09.031317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"82'11","reported_seq":397,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868217+0000","last_change":"2024-09-13T22:44:02.641797+0000","last_active":"2024-09-13T22:46:17.868217+0000","last_peered":"2024-09-13T22:46:17.868217+0000","last_clean":"2024-09-13T22:46:17.868217+0000","last_became_active":"2024-09-13T22:37:44.877440+0000","last_became_peered":"2024-09-13T22:37:44.877440+0000","last_unstale":"2024-09-13T22:46:17.868217+0000","last_undegraded":"2024-09-13T22:46:17.868217+0000","last_fullsized":"2024-09-13T22:46:17.868217+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:44:02.641728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:44:02.641728+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:23.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861802+0000","last_change":"2024-09-13T22:46:04.822159+0000","last_active":"2024-09-13T22:46:17.861802+0000","last_peered":"2024-09-13T22:46:17.861802+0000","last_clean":"2024-09-13T22:46:17.861802+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:46:17.861802+0000","last_undegraded":"2024-09-13T22:46:17.861802+0000","last_fullsized":"2024-09-13T22:46:17.861802+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:04.822089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:04.822089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:33.282382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"198'5389","reported_seq":5445,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:23.720287+0000","last_change":"2024-09-13T22:45:11.076685+0000","last_active":"2024-09-13T22:46:23.720287+0000","last_peered":"2024-09-13T22:46:23.720287+0000","last_clean":"2024-09-13T22:46:23.720287+0000","last_became_active":"2024-09-13T22:45:11.075580+0000","last_became_peered":"2024-09-13T22:45:11.075580+0000","last_unstale":"2024-09-13T22:46:23.720287+0000","last_undegraded":"2024-09-13T22:46:23.720287+0000","last_fullsized":"2024-09-13T22:46:23.720287+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:09.843138+0000","last_clean_scrub_stamp":"2024-09-13T22:45:09.843138+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:30.512256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353173504,"num_objects":5389,"num_object_clones":0,"num_object_copies":16167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":344896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"82'11","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.869016+0000","last_change":"2024-09-13T22:44:55.856200+0000","last_active":"2024-09-13T22:46:17.869016+0000","last_peered":"2024-09-13T22:46:17.869016+0000","last_clean":"2024-09-13T22:46:17.869016+0000","last_became_active":"2024-09-13T22:37:42.758169+0000","last_became_peered":"2024-09-13T22:37:42.758169+0000","last_unstale":"2024-09-13T22:46:17.869016+0000","last_undegraded":"2024-09-13T22:46:17.869016+0000","last_fullsized":"2024-09-13T22:46:17.869016+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:44:55.856068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.856068+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.890968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859932+0000","last_change":"2024-09-13T22:46:02.057133+0000","last_active":"2024-09-13T22:46:17.859932+0000","last_peered":"2024-09-13T22:46:17.859932+0000","last_clean":"2024-09-13T22:46:17.859932+0000","last_became_active":"2024-09-13T22:37:40.745315+0000","last_became_peered":"2024-09-13T22:37:40.745315+0000","last_unstale":"2024-09-13T22:46:17.859932+0000","last_undegraded":"2024-09-13T22:46:17.859932+0000","last_fullsized":"2024-09-13T22:46:17.859932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:46:02.057013+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.057013+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:14.062261+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.871387+0000","last_change":"2024-09-13T22:45:01.719814+0000","last_active":"2024-09-13T22:46:17.871387+0000","last_peered":"2024-09-13T22:46:17.871387+0000","last_clean":"2024-09-13T22:46:17.871387+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:46:17.871387+0000","last_undegraded":"2024-09-13T22:46:17.871387+0000","last_fullsized":"2024-09-13T22:46:17.871387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.870391+0000","last_change":"2024-09-13T22:44:05.968876+0000","last_active":"2024-09-13T22:46:17.870391+0000","last_peered":"2024-09-13T22:46:17.870391+0000","last_clean":"2024-09-13T22:46:17.870391+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:46:17.870391+0000","last_undegraded":"2024-09-13T22:46:17.870391+0000","last_fullsized":"2024-09-13T22:46:17.870391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:44:05.968767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.191961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"112'16","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868326+0000","last_change":"2024-09-13T22:45:39.713716+0000","last_active":"2024-09-13T22:46:17.868326+0000","last_peered":"2024-09-13T22:46:17.868326+0000","last_clean":"2024-09-13T22:46:17.868326+0000","last_became_active":"2024-09-13T22:37:42.760887+0000","last_became_peered":"2024-09-13T22:37:42.760887+0000","last_unstale":"2024-09-13T22:46:17.868326+0000","last_undegraded":"2024-09-13T22:46:17.868326+0000","last_fullsized":"2024-09-13T22:46:17.868326+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'16","last_scrub_stamp":"2024-09-13T22:45:39.713591+0000","last_deep_scrub":"112'16","last_deep_scrub_stamp":"2024-09-13T22:40:26.503359+0000","last_clean_scrub_stamp":"2024-09-13T22:45:39.713591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:04.554354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.871522+0000","last_change":"2024-09-13T22:45:03.702368+0000","last_active":"2024-09-13T22:46:17.871522+0000","last_peered":"2024-09-13T22:46:17.871522+0000","last_clean":"2024-09-13T22:46:17.871522+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:46:17.871522+0000","last_undegraded":"2024-09-13T22:46:17.871522+0000","last_fullsized":"2024-09-13T22:46:17.871522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.702179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.702179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:28.781648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866667+0000","last_change":"2024-09-13T22:43:58.778941+0000","last_active":"2024-09-13T22:46:17.866667+0000","last_peered":"2024-09-13T22:46:17.866667+0000","last_clean":"2024-09-13T22:46:17.866667+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:46:17.866667+0000","last_undegraded":"2024-09-13T22:46:17.866667+0000","last_fullsized":"2024-09-13T22:46:17.866667+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:43:58.778835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:43:58.778835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.191910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.871872+0000","last_change":"2024-09-13T22:46:03.997039+0000","last_active":"2024-09-13T22:46:17.871872+0000","last_peered":"2024-09-13T22:46:17.871872+0000","last_clean":"2024-09-13T22:46:17.871872+0000","last_became_active":"2024-09-13T22:37:46.910447+0000","last_became_peered":"2024-09-13T22:37:46.910447+0000","last_unstale":"2024-09-13T22:46:17.871872+0000","last_undegraded":"2024-09-13T22:46:17.871872+0000","last_fullsized":"2024-09-13T22:46:17.871872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:03.996954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:03.996954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:04.036347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"82'4","reported_seq":309,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864007+0000","last_change":"2024-09-13T22:46:10.063585+0000","last_active":"2024-09-13T22:46:17.864007+0000","last_peered":"2024-09-13T22:46:17.864007+0000","last_clean":"2024-09-13T22:46:17.864007+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:46:17.864007+0000","last_undegraded":"2024-09-13T22:46:17.864007+0000","last_fullsized":"2024-09-13T22:46:17.864007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'4","last_scrub_stamp":"2024-09-13T22:46:10.063463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:10.063463+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:33.458944+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864878+0000","last_change":"2024-09-13T22:46:10.995650+0000","last_active":"2024-09-13T22:46:17.864878+0000","last_peered":"2024-09-13T22:46:17.864878+0000","last_clean":"2024-09-13T22:46:17.864878+0000","last_became_active":"2024-09-13T22:37:40.750369+0000","last_became_peered":"2024-09-13T22:37:40.750369+0000","last_unstale":"2024-09-13T22:46:17.864878+0000","last_undegraded":"2024-09-13T22:46:17.864878+0000","last_fullsized":"2024-09-13T22:46:17.864878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:10.995551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_clean_scrub_stamp":"2024-09-13T22:46:10.995551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:18.797482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862110+0000","last_change":"2024-09-13T22:46:08.851682+0000","last_active":"2024-09-13T22:46:17.862110+0000","last_peered":"2024-09-13T22:46:17.862110+0000","last_clean":"2024-09-13T22:46:17.862110+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:46:17.862110+0000","last_undegraded":"2024-09-13T22:46:17.862110+0000","last_fullsized":"2024-09-13T22:46:17.862110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:08.851579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.851579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:29.557927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867969+0000","last_change":"2024-09-13T22:46:06.698541+0000","last_active":"2024-09-13T22:46:17.867969+0000","last_peered":"2024-09-13T22:46:17.867969+0000","last_clean":"2024-09-13T22:46:17.867969+0000","last_became_active":"2024-09-13T22:37:47.689827+0000","last_became_peered":"2024-09-13T22:37:47.689827+0000","last_unstale":"2024-09-13T22:46:17.867969+0000","last_undegraded":"2024-09-13T22:46:17.867969+0000","last_fullsized":"2024-09-13T22:46:17.867969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:06.698355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:06.698355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.234471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"82'11","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868466+0000","last_change":"2024-09-13T22:45:41.734825+0000","last_active":"2024-09-13T22:46:17.868466+0000","last_peered":"2024-09-13T22:46:17.868466+0000","last_clean":"2024-09-13T22:46:17.868466+0000","last_became_active":"2024-09-13T22:37:43.301346+0000","last_became_peered":"2024-09-13T22:37:43.301346+0000","last_unstale":"2024-09-13T22:46:17.868466+0000","last_undegraded":"2024-09-13T22:46:17.868466+0000","last_fullsized":"2024-09-13T22:46:17.868466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:41.734745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:41.734745+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:54.391214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864994+0000","last_change":"2024-09-13T22:45:47.986200+0000","last_active":"2024-09-13T22:46:17.864994+0000","last_peered":"2024-09-13T22:46:17.864994+0000","last_clean":"2024-09-13T22:46:17.864994+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:46:17.864994+0000","last_undegraded":"2024-09-13T22:46:17.864994+0000","last_fullsized":"2024-09-13T22:46:17.864994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:47.986137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:45:47.986137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:03.333949+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":408,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862073+0000","last_change":"2024-09-13T22:45:36.100609+0000","last_active":"2024-09-13T22:46:17.862073+0000","last_peered":"2024-09-13T22:46:17.862073+0000","last_clean":"2024-09-13T22:46:17.862073+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:46:17.862073+0000","last_undegraded":"2024-09-13T22:46:17.862073+0000","last_fullsized":"2024-09-13T22:46:17.862073+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:36.100527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:36.100527+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:49.448091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.869067+0000","last_change":"2024-09-13T22:44:41.721931+0000","last_active":"2024-09-13T22:46:17.869067+0000","last_peered":"2024-09-13T22:46:17.869067+0000","last_clean":"2024-09-13T22:46:17.869067+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:46:17.869067+0000","last_undegraded":"2024-09-13T22:46:17.869067+0000","last_fullsized":"2024-09-13T22:46:17.869067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:41.721714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:41.721714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":303,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.872581+0000","last_change":"2024-09-13T22:43:57.436556+0000","last_active":"2024-09-13T22:46:17.872581+0000","last_peered":"2024-09-13T22:46:17.872581+0000","last_clean":"2024-09-13T22:46:17.872581+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:46:17.872581+0000","last_undegraded":"2024-09-13T22:46:17.872581+0000","last_fullsized":"2024-09-13T22:46:17.872581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:43:57.436436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:43:57.436436+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862046+0000","last_change":"2024-09-13T22:45:03.920511+0000","last_active":"2024-09-13T22:46:17.862046+0000","last_peered":"2024-09-13T22:46:17.862046+0000","last_clean":"2024-09-13T22:46:17.862046+0000","last_became_active":"2024-09-13T22:37:40.738659+0000","last_became_peered":"2024-09-13T22:37:40.738659+0000","last_unstale":"2024-09-13T22:46:17.862046+0000","last_undegraded":"2024-09-13T22:46:17.862046+0000","last_fullsized":"2024-09-13T22:46:17.862046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.920319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.920319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.344069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":406,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866945+0000","last_change":"2024-09-13T22:45:48.537059+0000","last_active":"2024-09-13T22:46:17.866945+0000","last_peered":"2024-09-13T22:46:17.866945+0000","last_clean":"2024-09-13T22:46:17.866945+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:46:17.866945+0000","last_undegraded":"2024-09-13T22:46:17.866945+0000","last_fullsized":"2024-09-13T22:46:17.866945+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:48.536952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:48.536952+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:09.191051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859725+0000","last_change":"2024-09-13T22:44:43.044133+0000","last_active":"2024-09-13T22:46:17.859725+0000","last_peered":"2024-09-13T22:46:17.859725+0000","last_clean":"2024-09-13T22:46:17.859725+0000","last_became_active":"2024-09-13T22:37:46.913054+0000","last_became_peered":"2024-09-13T22:37:46.913054+0000","last_unstale":"2024-09-13T22:46:17.859725+0000","last_undegraded":"2024-09-13T22:46:17.859725+0000","last_fullsized":"2024-09-13T22:46:17.859725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:43.044031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:43.044031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:19.031007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"82'9","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.868138+0000","last_change":"2024-09-13T22:45:55.788849+0000","last_active":"2024-09-13T22:46:17.868138+0000","last_peered":"2024-09-13T22:46:17.868138+0000","last_clean":"2024-09-13T22:46:17.868138+0000","last_became_active":"2024-09-13T22:37:42.762132+0000","last_became_peered":"2024-09-13T22:37:42.762132+0000","last_unstale":"2024-09-13T22:46:17.868138+0000","last_undegraded":"2024-09-13T22:46:17.868138+0000","last_fullsized":"2024-09-13T22:46:17.868138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:45:55.788723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:55.788723+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:08.913728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864301+0000","last_change":"2024-09-13T22:44:52.084394+0000","last_active":"2024-09-13T22:46:17.864301+0000","last_peered":"2024-09-13T22:46:17.864301+0000","last_clean":"2024-09-13T22:46:17.864301+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:46:17.864301+0000","last_undegraded":"2024-09-13T22:46:17.864301+0000","last_fullsized":"2024-09-13T22:46:17.864301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:52.084196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:52.084196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:17.993655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.874809+0000","last_change":"2024-09-13T22:45:00.534688+0000","last_active":"2024-09-13T22:46:17.874809+0000","last_peered":"2024-09-13T22:46:17.874809+0000","last_clean":"2024-09-13T22:46:17.874809+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:46:17.874809+0000","last_undegraded":"2024-09-13T22:46:17.874809+0000","last_fullsized":"2024-09-13T22:46:17.874809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:00.534596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:45:00.534596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860614+0000","last_change":"2024-09-13T22:45:43.823216+0000","last_active":"2024-09-13T22:46:17.860614+0000","last_peered":"2024-09-13T22:46:17.860614+0000","last_clean":"2024-09-13T22:46:17.860614+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:46:17.860614+0000","last_undegraded":"2024-09-13T22:46:17.860614+0000","last_fullsized":"2024-09-13T22:46:17.860614+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:43.823110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:43.823110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:51.567038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"82'10","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860101+0000","last_change":"2024-09-13T22:45:49.861224+0000","last_active":"2024-09-13T22:46:17.860101+0000","last_peered":"2024-09-13T22:46:17.860101+0000","last_clean":"2024-09-13T22:46:17.860101+0000","last_became_active":"2024-09-13T22:37:42.764515+0000","last_became_peered":"2024-09-13T22:37:42.764515+0000","last_unstale":"2024-09-13T22:46:17.860101+0000","last_undegraded":"2024-09-13T22:46:17.860101+0000","last_fullsized":"2024-09-13T22:46:17.860101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:45:49.861100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:49.861100+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.261360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860176+0000","last_change":"2024-09-13T22:46:03.849387+0000","last_active":"2024-09-13T22:46:17.860176+0000","last_peered":"2024-09-13T22:46:17.860176+0000","last_clean":"2024-09-13T22:46:17.860176+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:46:17.860176+0000","last_undegraded":"2024-09-13T22:46:17.860176+0000","last_fullsized":"2024-09-13T22:46:17.860176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:03.849281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:03.849281+0000","last_clean_scrub_stamp":"2024-09-13T22:46:03.849281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:16.943280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860290+0000","last_change":"2024-09-13T22:46:00.830826+0000","last_active":"2024-09-13T22:46:17.860290+0000","last_peered":"2024-09-13T22:46:17.860290+0000","last_clean":"2024-09-13T22:46:17.860290+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:46:17.860290+0000","last_undegraded":"2024-09-13T22:46:17.860290+0000","last_fullsized":"2024-09-13T22:46:17.860290+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:46:00.830738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:09.043295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865469+0000","last_change":"2024-09-13T22:44:45.461864+0000","last_active":"2024-09-13T22:46:17.865469+0000","last_peered":"2024-09-13T22:46:17.865469+0000","last_clean":"2024-09-13T22:46:17.865469+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:46:17.865469+0000","last_undegraded":"2024-09-13T22:46:17.865469+0000","last_fullsized":"2024-09-13T22:46:17.865469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:45.461720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:44:45.461720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192123+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"82'6","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862521+0000","last_change":"2024-09-13T22:45:45.974044+0000","last_active":"2024-09-13T22:46:17.862521+0000","last_peered":"2024-09-13T22:46:17.862521+0000","last_clean":"2024-09-13T22:46:17.862521+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:46:17.862521+0000","last_undegraded":"2024-09-13T22:46:17.862521+0000","last_fullsized":"2024-09-13T22:46:17.862521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'6","last_scrub_stamp":"2024-09-13T22:45:45.973873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:45.973873+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:48.010489+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862330+0000","last_change":"2024-09-13T22:46:02.798750+0000","last_active":"2024-09-13T22:46:17.862330+0000","last_peered":"2024-09-13T22:46:17.862330+0000","last_clean":"2024-09-13T22:46:17.862330+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:46:17.862330+0000","last_undegraded":"2024-09-13T22:46:17.862330+0000","last_fullsized":"2024-09-13T22:46:17.862330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:02.798641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.798641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:27.119341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864051+0000","last_change":"2024-09-13T22:45:02.990024+0000","last_active":"2024-09-13T22:46:17.864051+0000","last_peered":"2024-09-13T22:46:17.864051+0000","last_clean":"2024-09-13T22:46:17.864051+0000","last_became_active":"2024-09-13T22:37:44.879209+0000","last_became_peered":"2024-09-13T22:37:44.879209+0000","last_unstale":"2024-09-13T22:46:17.864051+0000","last_undegraded":"2024-09-13T22:46:17.864051+0000","last_fullsized":"2024-09-13T22:46:17.864051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.989795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.989795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:24.890892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":270,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866069+0000","last_change":"2024-09-13T22:44:54.870749+0000","last_active":"2024-09-13T22:46:17.866069+0000","last_peered":"2024-09-13T22:46:17.866069+0000","last_clean":"2024-09-13T22:46:17.866069+0000","last_became_active":"2024-09-13T22:37:46.914846+0000","last_became_peered":"2024-09-13T22:37:46.914846+0000","last_unstale":"2024-09-13T22:46:17.866069+0000","last_undegraded":"2024-09-13T22:46:17.866069+0000","last_fullsized":"2024-09-13T22:46:17.866069+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:54.870619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:44:54.870619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.891086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"82'9","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.862202+0000","last_change":"2024-09-13T22:46:06.750745+0000","last_active":"2024-09-13T22:46:17.862202+0000","last_peered":"2024-09-13T22:46:17.862202+0000","last_clean":"2024-09-13T22:46:17.862202+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:46:17.862202+0000","last_undegraded":"2024-09-13T22:46:17.862202+0000","last_fullsized":"2024-09-13T22:46:17.862202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:46:06.750407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:06.750407+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:26.660719+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864798+0000","last_change":"2024-09-13T22:45:44.033193+0000","last_active":"2024-09-13T22:46:17.864798+0000","last_peered":"2024-09-13T22:46:17.864798+0000","last_clean":"2024-09-13T22:46:17.864798+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:46:17.864798+0000","last_undegraded":"2024-09-13T22:46:17.864798+0000","last_fullsized":"2024-09-13T22:46:17.864798+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:44.033125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:45:44.033125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:57.574169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":282,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865372+0000","last_change":"2024-09-13T22:46:07.738472+0000","last_active":"2024-09-13T22:46:17.865372+0000","last_peered":"2024-09-13T22:46:17.865372+0000","last_clean":"2024-09-13T22:46:17.865372+0000","last_became_active":"2024-09-13T22:37:44.877902+0000","last_became_peered":"2024-09-13T22:37:44.877902+0000","last_unstale":"2024-09-13T22:46:17.865372+0000","last_undegraded":"2024-09-13T22:46:17.865372+0000","last_fullsized":"2024-09-13T22:46:17.865372+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:07.738400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:07.738400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:33.115278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.874873+0000","last_change":"2024-09-13T22:45:40.486348+0000","last_active":"2024-09-13T22:46:17.874873+0000","last_peered":"2024-09-13T22:46:17.874873+0000","last_clean":"2024-09-13T22:46:17.874873+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:46:17.874873+0000","last_undegraded":"2024-09-13T22:46:17.874873+0000","last_fullsized":"2024-09-13T22:46:17.874873+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:40.486247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:45:40.486247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:58.576067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867680+0000","last_change":"2024-09-13T22:45:56.843016+0000","last_active":"2024-09-13T22:46:17.867680+0000","last_peered":"2024-09-13T22:46:17.867680+0000","last_clean":"2024-09-13T22:46:17.867680+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:46:17.867680+0000","last_undegraded":"2024-09-13T22:46:17.867680+0000","last_fullsized":"2024-09-13T22:46:17.867680+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:45:56.842809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:56.842809+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:07.997019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"111'22","reported_seq":327,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857932+0000","last_change":"2024-09-13T22:45:49.832578+0000","last_active":"2024-09-13T22:46:17.857932+0000","last_peered":"2024-09-13T22:46:17.857932+0000","last_clean":"2024-09-13T22:46:17.857932+0000","last_became_active":"2024-09-13T22:37:42.759986+0000","last_became_peered":"2024-09-13T22:37:42.759986+0000","last_unstale":"2024-09-13T22:46:17.857932+0000","last_undegraded":"2024-09-13T22:46:17.857932+0000","last_fullsized":"2024-09-13T22:46:17.857932+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-13T22:45:49.832343+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:45:49.832343+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:15.942350+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.860991+0000","last_change":"2024-09-13T22:45:57.927989+0000","last_active":"2024-09-13T22:46:17.860991+0000","last_peered":"2024-09-13T22:46:17.860991+0000","last_clean":"2024-09-13T22:46:17.860991+0000","last_became_active":"2024-09-13T22:37:40.761204+0000","last_became_peered":"2024-09-13T22:37:40.761204+0000","last_unstale":"2024-09-13T22:46:17.860991+0000","last_undegraded":"2024-09-13T22:46:17.860991+0000","last_fullsized":"2024-09-13T22:46:17.860991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:57.927907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:45:57.927907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:01.964614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"172'12","reported_seq":411,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865004+0000","last_change":"2024-09-13T22:44:59.017860+0000","last_active":"2024-09-13T22:46:17.865004+0000","last_peered":"2024-09-13T22:46:17.865004+0000","last_clean":"2024-09-13T22:46:17.865004+0000","last_became_active":"2024-09-13T22:37:44.880383+0000","last_became_peered":"2024-09-13T22:37:44.880383+0000","last_unstale":"2024-09-13T22:46:17.865004+0000","last_undegraded":"2024-09-13T22:46:17.865004+0000","last_fullsized":"2024-09-13T22:46:17.865004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'12","last_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_deep_scrub":"172'12","last_deep_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_clean_scrub_stamp":"2024-09-13T22:44:59.017618+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.979557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.857760+0000","last_change":"2024-09-13T22:45:55.765519+0000","last_active":"2024-09-13T22:46:17.857760+0000","last_peered":"2024-09-13T22:46:17.857760+0000","last_clean":"2024-09-13T22:46:17.857760+0000","last_became_active":"2024-09-13T22:37:46.902752+0000","last_became_peered":"2024-09-13T22:37:46.902752+0000","last_unstale":"2024-09-13T22:46:17.857760+0000","last_undegraded":"2024-09-13T22:46:17.857760+0000","last_fullsized":"2024-09-13T22:46:17.857760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:55.765454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:55.765454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:05.480739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"82'9","reported_seq":303,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865646+0000","last_change":"2024-09-13T22:44:38.382234+0000","last_active":"2024-09-13T22:46:17.865646+0000","last_peered":"2024-09-13T22:46:17.865646+0000","last_clean":"2024-09-13T22:46:17.865646+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:46:17.865646+0000","last_undegraded":"2024-09-13T22:46:17.865646+0000","last_fullsized":"2024-09-13T22:46:17.865646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:44:38.382116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:38.382116+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.865739+0000","last_change":"2024-09-13T22:44:58.543023+0000","last_active":"2024-09-13T22:46:17.865739+0000","last_peered":"2024-09-13T22:46:17.865739+0000","last_clean":"2024-09-13T22:46:17.865739+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:46:17.865739+0000","last_undegraded":"2024-09-13T22:46:17.865739+0000","last_fullsized":"2024-09-13T22:46:17.865739+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:58.542781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.542781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:20.452580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.867953+0000","last_change":"2024-09-13T22:45:31.769164+0000","last_active":"2024-09-13T22:46:17.867953+0000","last_peered":"2024-09-13T22:46:17.867953+0000","last_clean":"2024-09-13T22:46:17.867953+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:46:17.867953+0000","last_undegraded":"2024-09-13T22:46:17.867953+0000","last_fullsized":"2024-09-13T22:46:17.867953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:31.769051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:03.677120+0000","last_clean_scrub_stamp":"2024-09-13T22:45:31.769051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:47.243315+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859840+0000","last_change":"2024-09-13T22:45:58.015245+0000","last_active":"2024-09-13T22:46:17.859840+0000","last_peered":"2024-09-13T22:46:17.859840+0000","last_clean":"2024-09-13T22:46:17.859840+0000","last_became_active":"2024-09-13T22:37:47.684249+0000","last_became_peered":"2024-09-13T22:37:47.684249+0000","last_unstale":"2024-09-13T22:46:17.859840+0000","last_undegraded":"2024-09-13T22:46:17.859840+0000","last_fullsized":"2024-09-13T22:46:17.859840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:58.015179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:58.015179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:27.666717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.864465+0000","last_change":"2024-09-13T22:44:47.023754+0000","last_active":"2024-09-13T22:46:17.864465+0000","last_peered":"2024-09-13T22:46:17.864465+0000","last_clean":"2024-09-13T22:46:17.864465+0000","last_became_active":"2024-09-13T22:37:40.757871+0000","last_became_peered":"2024-09-13T22:37:40.757871+0000","last_unstale":"2024-09-13T22:46:17.864465+0000","last_undegraded":"2024-09-13T22:46:17.864465+0000","last_fullsized":"2024-09-13T22:46:17.864465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:47.023644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:47.023644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:21.979583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"100'7","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.861912+0000","last_change":"2024-09-13T22:44:57.928934+0000","last_active":"2024-09-13T22:46:17.861912+0000","last_peered":"2024-09-13T22:46:17.861912+0000","last_clean":"2024-09-13T22:46:17.861912+0000","last_became_active":"2024-09-13T22:37:42.763268+0000","last_became_peered":"2024-09-13T22:37:42.763268+0000","last_unstale":"2024-09-13T22:46:17.861912+0000","last_undegraded":"2024-09-13T22:46:17.861912+0000","last_fullsized":"2024-09-13T22:46:17.861912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:44:57.928755+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:44:57.928755+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.858036+0000","last_change":"2024-09-13T22:46:08.819373+0000","last_active":"2024-09-13T22:46:17.858036+0000","last_peered":"2024-09-13T22:46:17.858036+0000","last_clean":"2024-09-13T22:46:17.858036+0000","last_became_active":"2024-09-13T22:37:44.875623+0000","last_became_peered":"2024-09-13T22:37:44.875623+0000","last_unstale":"2024-09-13T22:46:17.858036+0000","last_undegraded":"2024-09-13T22:46:17.858036+0000","last_fullsized":"2024-09-13T22:46:17.858036+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:08.819266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.819266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:37.200209+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":289,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.866582+0000","last_change":"2024-09-13T22:45:51.542656+0000","last_active":"2024-09-13T22:46:17.866582+0000","last_peered":"2024-09-13T22:46:17.866582+0000","last_clean":"2024-09-13T22:46:17.866582+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:46:17.866582+0000","last_undegraded":"2024-09-13T22:46:17.866582+0000","last_fullsized":"2024-09-13T22:46:17.866582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:45:51.542373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.542373+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:10.130127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.875048+0000","last_change":"2024-09-13T22:44:49.489483+0000","last_active":"2024-09-13T22:46:17.875048+0000","last_peered":"2024-09-13T22:46:17.875048+0000","last_clean":"2024-09-13T22:46:17.875048+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:46:17.875048+0000","last_undegraded":"2024-09-13T22:46:17.875048+0000","last_fullsized":"2024-09-13T22:46:17.875048+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:49.489414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:49.489414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:46:18.192050+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"194'13","reported_seq":315,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859712+0000","last_change":"2024-09-13T22:46:08.028792+0000","last_active":"2024-09-13T22:46:17.859712+0000","last_peered":"2024-09-13T22:46:17.859712+0000","last_clean":"2024-09-13T22:46:17.859712+0000","last_became_active":"2024-09-13T22:37:43.301616+0000","last_became_peered":"2024-09-13T22:37:43.301616+0000","last_unstale":"2024-09-13T22:46:17.859712+0000","last_undegraded":"2024-09-13T22:46:17.859712+0000","last_fullsized":"2024-09-13T22:46:17.859712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.028593+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:14.356195+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-13T22:46:17.859881+0000","last_change":"2024-09-13T22:45:50.912275+0000","last_active":"2024-09-13T22:46:17.859881+0000","last_peered":"2024-09-13T22:46:17.859881+0000","last_clean":"2024-09-13T22:46:17.859881+0000","last_became_active":"2024-09-13T22:37:44.860731+0000","last_became_peered":"2024-09-13T22:37:44.860731+0000","last_unstale":"2024-09-13T22:46:17.859881+0000","last_undegraded":"2024-09-13T22:46:17.859881+0000","last_fullsized":"2024-09-13T22:46:17.859881+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:50.912196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:45:50.912196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:15.133102+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5653987328,"num_objects":86273,"num_object_clones":0,"num_object_copies":258819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86273,"num_write_kb":5521472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16938172416,"data_stored":16938172416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81673,"ondisk_log_size":81673,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":502,"num_read_kb":497,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808945,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1835476,"kb_used_data":1696004,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91909676,"statfs":{"total":95995035648,"available":94115508224,"internally_reserved":0,"allocated":1736708096,"data_stored":1736193029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135996,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1845432,"kb_used_data":1705960,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91899720,"statfs":{"total":95995035648,"available":94105313280,"internally_reserved":0,"allocated":1746903040,"data_stored":1746414323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142801350},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528455,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2878208,"kb_used_data":2724848,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":90866944,"statfs":{"total":95995035648,"available":93047750656,"internally_reserved":0,"allocated":2790244352,"data_stored":2789748389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888209,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1848796,"kb_used_data":1705164,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91896356,"statfs":{"total":95995035648,"available":94101868544,"internally_reserved":0,"allocated":1746087936,"data_stored":1745563287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084443,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2311808,"kb_used_data":2064048,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91433344,"statfs":{"total":95995035648,"available":93627744256,"internally_reserved":0,"allocated":2113585152,"data_stored":2113089610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":253688263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378791,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201536,"kb_used_data":2057904,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91543616,"statfs":{"total":95995035648,"available":93740662784,"internally_reserved":0,"allocated":2107293696,"data_stored":2106798247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2586440,"kb_used_data":2402552,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91158712,"statfs":{"total":95995035648,"available":93346521088,"internally_reserved":0,"allocated":2460213248,"data_stored":2459709458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305612,"kb_used_data":2075900,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91439540,"statfs":{"total":95995035648,"available":93634088960,"internally_reserved":0,"allocated":2125721600,"data_stored":2125214474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2134376448,"data_stored":2134376448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2474508288,"data_stored":2474508288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2127429632,"data_stored":2127429632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2122579968,"data_stored":2122579968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1755906048,"data_stored":1755906048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2812346368,"data_stored":2812346368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1759248384,"data_stored":1759248384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1751777280,"data_stored":1751777280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:46:30.840 INFO:tasks.thrashosds.thrasher:Setting 8.5 to [5, 0, 4, 1, 6, 3] 2024-09-13T22:46:30.840 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.5', '5', '0', '4', '1', '6', '3'] 2024-09-13T22:46:30.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 8.5 5 0 4 1 6 3 2024-09-13T22:46:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:30.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[29889]: cluster 2024-09-13T22:46:28.972738+0000 osd.1 (osd.1) 186 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[29889]: cluster 2024-09-13T22:46:28.972787+0000 osd.1 (osd.1) 187 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[29889]: cluster 2024-09-13T22:46:29.902918+0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[29889]: cluster 2024-09-13T22:46:30.615191+0000 mgr.y (mgr.14613) 330 : cluster [DBG] pgmap v347: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[22840]: cluster 2024-09-13T22:46:28.972738+0000 osd.1 (osd.1) 186 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[22840]: cluster 2024-09-13T22:46:28.972787+0000 osd.1 (osd.1) 187 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:30.954 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[22840]: cluster 2024-09-13T22:46:29.902918+0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-13T22:46:30.955 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:30 smithi178 bash[22840]: cluster 2024-09-13T22:46:30.615191+0000 mgr.y (mgr.14613) 330 : cluster [DBG] pgmap v347: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T22:46:30.979 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:30.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:31.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:31.118 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:31.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:30 smithi192 bash[24608]: cluster 2024-09-13T22:46:28.972738+0000 osd.1 (osd.1) 186 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:30 smithi192 bash[24608]: cluster 2024-09-13T22:46:28.972787+0000 osd.1 (osd.1) 187 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:30 smithi192 bash[24608]: cluster 2024-09-13T22:46:29.902918+0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-13T22:46:31.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:30 smithi192 bash[24608]: cluster 2024-09-13T22:46:30.615191+0000 mgr.y (mgr.14613) 330 : cluster [DBG] pgmap v347: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T22:46:31.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:31.249 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:31.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:31.386 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:31.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:31.513 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:31.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:31.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:31.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:31.789 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:31.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 72 16 93149 93133 80.8333 64.3125 0.0113139 0.0123671 2024-09-13T22:46:31.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:31.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:31.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:32.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:32.043 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:31 smithi178 bash[29889]: cluster 2024-09-13T22:46:31.825250+0000 osd.2 (osd.2) 149 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:32.043 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:31 smithi178 bash[29889]: cluster 2024-09-13T22:46:31.825313+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:32.043 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:31 smithi178 bash[29889]: cluster 2024-09-13T22:46:31.927186+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-13T22:46:32.043 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:31 smithi178 bash[22840]: cluster 2024-09-13T22:46:31.825250+0000 osd.2 (osd.2) 149 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:32.043 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:31 smithi178 bash[22840]: cluster 2024-09-13T22:46:31.825313+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:32.044 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:31 smithi178 bash[22840]: cluster 2024-09-13T22:46:31.927186+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-13T22:46:32.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:32.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:31 smithi192 bash[24608]: cluster 2024-09-13T22:46:31.825250+0000 osd.2 (osd.2) 149 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:32.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:31 smithi192 bash[24608]: cluster 2024-09-13T22:46:31.825313+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:32.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:31 smithi192 bash[24608]: cluster 2024-09-13T22:46:31.927186+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-13T22:46:32.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:32.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:32.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.328 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:32.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:32.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:32.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:32.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:32.584 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:32.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:32.709 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:32.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:32.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 73 15 94261 94246 80.6788 69.5625 0.0106299 0.0123902 2024-09-13T22:46:32.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:32.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:32.972 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:32.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:33.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:33.101 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:33.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:32 smithi192 bash[24608]: cluster 2024-09-13T22:46:32.615640+0000 mgr.y (mgr.14613) 331 : cluster [DBG] pgmap v349: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 801 op/s 2024-09-13T22:46:33.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:32 smithi192 bash[24608]: cluster 2024-09-13T22:46:32.868543+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-13T22:46:33.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:32 smithi178 bash[29889]: cluster 2024-09-13T22:46:32.615640+0000 mgr.y (mgr.14613) 331 : cluster [DBG] pgmap v349: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 801 op/s 2024-09-13T22:46:33.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:32 smithi178 bash[29889]: cluster 2024-09-13T22:46:32.868543+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-13T22:46:33.215 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:32 smithi178 bash[22840]: cluster 2024-09-13T22:46:32.615640+0000 mgr.y (mgr.14613) 331 : cluster [DBG] pgmap v349: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 801 op/s 2024-09-13T22:46:33.215 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:32 smithi178 bash[22840]: cluster 2024-09-13T22:46:32.868543+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-13T22:46:33.230 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:33.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.355 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:33.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:33.492 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:33.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:46:33.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:46:33.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.688 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:33.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 74 16 95271 95255 80.4404 63.0625 0.0226225 0.0124258 2024-09-13T22:46:33.836 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:33.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:33.966 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:33.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:34.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:34.105 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:33 smithi178 bash[29889]: cluster 2024-09-13T22:46:32.048667+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:34.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:33 smithi178 bash[29889]: cluster 2024-09-13T22:46:32.048732+0000 osd.3 (osd.3) 207 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:34.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:33 smithi178 bash[29889]: audit 2024-09-13T22:46:33.297473+0000 mon.b (mon.1) 153 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:33 smithi178 bash[22840]: cluster 2024-09-13T22:46:32.048667+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:33 smithi178 bash[22840]: cluster 2024-09-13T22:46:32.048732+0000 osd.3 (osd.3) 207 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:34.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:33 smithi178 bash[22840]: audit 2024-09-13T22:46:33.297473+0000 mon.b (mon.1) 153 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:34.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:34.235 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:34.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:33 smithi192 bash[24608]: cluster 2024-09-13T22:46:32.048667+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:34.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:33 smithi192 bash[24608]: cluster 2024-09-13T22:46:32.048732+0000 osd.3 (osd.3) 207 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:34.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:33 smithi192 bash[24608]: audit 2024-09-13T22:46:33.297473+0000 mon.b (mon.1) 153 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:34.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:34.365 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:34.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:34.500 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:34.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:34.635 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:34.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:34.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:34.820 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 75 16 96352 96336 80.2686 67.5625 0.00954889 0.0124536 2024-09-13T22:46:34.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:34.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:35.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:35.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:35.054 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:35.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:35.161 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:34 smithi178 bash[22840]: cluster 2024-09-13T22:46:34.616483+0000 mgr.y (mgr.14613) 332 : cluster [DBG] pgmap v351: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 867 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-13T22:46:35.161 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:34 smithi178 bash[29889]: cluster 2024-09-13T22:46:34.616483+0000 mgr.y (mgr.14613) 332 : cluster [DBG] pgmap v351: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 867 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-13T22:46:35.167 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:35.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:35.302 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:35.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:46:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:46:35.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:34 smithi192 bash[24608]: cluster 2024-09-13T22:46:34.616483+0000 mgr.y (mgr.14613) 332 : cluster [DBG] pgmap v351: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 867 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-13T22:46:35.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:35.436 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:35.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:35.574 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:35.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:35.706 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:35.809 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:35.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 76 15 97388 97373 80.0651 64.8125 0.0101694 0.0124851 2024-09-13T22:46:35.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:35.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:35.968 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:35.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:36.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:36.083 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:36.101 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:36.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:36.228 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:36.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:36.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:35 smithi192 bash[24608]: audit 2024-09-13T22:46:35.229145+0000 mgr.y (mgr.14613) 333 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:36.362 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:36.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:36 smithi178 bash[29889]: audit 2024-09-13T22:46:35.229145+0000 mgr.y (mgr.14613) 333 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:36.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:36 smithi178 bash[22840]: audit 2024-09-13T22:46:35.229145+0000 mgr.y (mgr.14613) 333 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:36.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:36.496 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:36.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:36.616 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:36.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:36.743 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:36.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 77 16 98242 98226 79.7176 53.3125 0.00615604 0.012527 2024-09-13T22:46:36.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:36.868 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:36.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:36.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:37.003 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:37.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:37.130 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:37.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:37.269 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:37.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:37.385 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:37 smithi178 bash[29889]: cluster 2024-09-13T22:46:36.617060+0000 mgr.y (mgr.14613) 334 : cluster [DBG] pgmap v352: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 762 B/s rd, 43 MiB/s wr, 683 op/s 2024-09-13T22:46:37.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:37 smithi178 bash[22840]: cluster 2024-09-13T22:46:36.617060+0000 mgr.y (mgr.14613) 334 : cluster [DBG] pgmap v352: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 762 B/s rd, 43 MiB/s wr, 683 op/s 2024-09-13T22:46:37.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:37 smithi192 bash[24608]: cluster 2024-09-13T22:46:36.617060+0000 mgr.y (mgr.14613) 334 : cluster [DBG] pgmap v352: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 762 B/s rd, 43 MiB/s wr, 683 op/s 2024-09-13T22:46:37.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:37.522 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:37.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:37.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:37.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:37.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:37.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 78 15 98984 98969 79.2909 46.4375 0.0176967 0.0126071 2024-09-13T22:46:37.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:37.923 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:37.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:38.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:38.050 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:38.125 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:38.125 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:38.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:38.183 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:38.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:38.311 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:38.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:38.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:38.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:38.606 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:38.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:38.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:38.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 79 16 99908 99892 79.0174 57.6875 0.008437 0.0126508 2024-09-13T22:46:38.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:38.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:38.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:38.877 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:38 smithi178 bash[29889]: cluster 2024-09-13T22:46:38.617440+0000 mgr.y (mgr.14613) 335 : cluster [DBG] pgmap v353: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 573 op/s 2024-09-13T22:46:38.877 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:38 smithi178 bash[22840]: cluster 2024-09-13T22:46:38.617440+0000 mgr.y (mgr.14613) 335 : cluster [DBG] pgmap v353: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 573 op/s 2024-09-13T22:46:38.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:38 smithi192 bash[24608]: cluster 2024-09-13T22:46:38.617440+0000 mgr.y (mgr.14613) 335 : cluster [DBG] pgmap v353: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 573 op/s 2024-09-13T22:46:38.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:39.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:39.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:39.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:39.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:39.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:39.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:39.407 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:39.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:39.540 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:39.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:39.676 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:39.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:39.807 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:39.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:46:39.824772+0000 min lat: 0.00124784 max lat: 0.701883 avg lat: 0.0126874 2024-09-13T22:46:39.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:46:39.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 80 16 100880 100864 78.789 60.75 0.0145893 0.0126874 2024-09-13T22:46:39.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:39.933 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:39.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:40.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:40.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:40.084 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:46:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:46:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:46:40.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:40.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:40.301 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:40.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:40.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:40.458 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:40.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:40.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:40.588 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:40.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:40.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:40.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:40.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 81 16 101732 101716 78.4737 53.25 0.00712651 0.0127382 2024-09-13T22:46:40.838 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:40.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:40.950 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:40 smithi192 bash[24608]: cluster 2024-09-13T22:46:40.618602+0000 mgr.y (mgr.14613) 336 : cluster [DBG] pgmap v354: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:46:40.964 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:40.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:41.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:41.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:40 smithi178 bash[22840]: cluster 2024-09-13T22:46:40.618602+0000 mgr.y (mgr.14613) 336 : cluster [DBG] pgmap v354: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:46:41.077 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:40 smithi178 bash[29889]: cluster 2024-09-13T22:46:40.618602+0000 mgr.y (mgr.14613) 336 : cluster [DBG] pgmap v354: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:46:41.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:102009 8.3 8:ceb77869:::benchmark_data_smithi178_80607_object102008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.103697+0000", 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0064188939999999996, 2024-09-13T22:46:41.108 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0064510569999999996, 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 102009 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103697+0000", 2024-09-13T22:46:41.109 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103697+0000", 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103699+0000", 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.34e-06 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103723+0000", 2024-09-13T22:46:41.110 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3726e-05 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103734+0000", 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.135e-05 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103743+0000", 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.7409999999999998e-06 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:46:41.111 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106606+0000", 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0028629229999999999 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.109539+0000", 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0029326669999999999 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.109691+0000", 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00015248400000000001 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.112 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:102010 8.3 8.d8014393 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.104399+0000", 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0057173399999999996, 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0058213960000000004, 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "queued for pg", 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 102010 2024-09-13T22:46:41.113 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.104399+0000", 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.104399+0000", 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.104400+0000", 2024-09-13T22:46:41.114 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.9800000000000002e-07 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.104416+0000", 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6011e-05 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.104423+0000", 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.182e-06 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.104428+0000", 2024-09-13T22:46:41.115 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0499999999999999e-06 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop_reply(client.25037.0:102005 8.3 e202/175)", 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.106448+0000", 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0036676920000000002, 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0038181199999999999, 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "queued for pg", 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.116 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106448+0000", 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106448+0000", 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.117 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106450+0000", 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.237e-06 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106453+0000", 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9009999999999996e-06 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106463+0000", 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4849999999999998e-06 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.118 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106468+0000", 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.814e-06 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop_reply(client.25037.0:102005 8.3 e202/175)", 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.106968+0000", 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0031479149999999998, 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.003337329, 2024-09-13T22:46:41.119 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "queued for pg", 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106968+0000", 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106968+0000", 2024-09-13T22:46:41.120 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106969+0000", 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4330000000000001e-06 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106977+0000", 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6420000000000007e-06 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.121 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106991+0000", 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3949e-05 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.106998+0000", 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.153e-06 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:101987 8.7 8:ea9c3a0d:::benchmark_data_smithi178_80607_object101986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.079084+0000", 2024-09-13T22:46:41.122 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.031032224000000001, 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.031269816999999998, 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 101987 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.079084+0000", 2024-09-13T22:46:41.123 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.079084+0000", 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.079085+0000", 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.55e-06 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.124 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.079107+0000", 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1820000000000001e-05 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.079120+0000", 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2687000000000001e-05 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.079137+0000", 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7413000000000001e-05 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.125 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.110222+0000", 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.031085181999999999 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:101984 8.f e202/175 8:f3b50998:::benchmark_data_smithi178_80607_object101983:head v 202'6392, mlcod=202'6389)", 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.080948+0000", 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.029168394, 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.029460837, 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.126 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:46:41.127 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.127 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.127 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.127 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.080948+0000", 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.080948+0000", 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.080949+0000", 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2020000000000001e-06 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.128 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.080972+0000", 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3093000000000001e-05 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.080992+0000", 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0077000000000001e-05 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.081000+0000", 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5680000000000002e-06 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:46:41.129 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.098421+0000", 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.017421478000000001 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.098452+0000", 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.1146000000000003e-05 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:101990 8.f e202/175 8:f2305efe:::benchmark_data_smithi178_80607_object101989:head v 202'6393, mlcod=202'6389)", 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.095029+0000", 2024-09-13T22:46:41.130 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.015087287, 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.015432718999999999, 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.095029+0000", 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.095029+0000", 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.131 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.095030+0000", 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.9999999999999995e-07 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.095054+0000", 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4091999999999999e-05 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.095060+0000", 2024-09-13T22:46:41.132 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2559999999999999e-06 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.095064+0000", 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6909999999999999e-06 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.101102+0000", 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0060382500000000002 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.101123+0000", 2024-09-13T22:46:41.133 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1056e-05 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:102004 8.7 8.9be50aa7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.099024+0000", 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.011092288, 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011488208999999999, 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "queued for pg", 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:46:41.134 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 102004 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099024+0000", 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099024+0000", 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.135 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099025+0000", 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0300000000000001e-06 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099068+0000", 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.3318000000000001e-05 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099077+0000", 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3729999999999999e-06 2024-09-13T22:46:41.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099084+0000", 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1310000000000003e-06 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:102005 8.3 8:c435edf8:::benchmark_data_smithi178_80607_object102004:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.099092+0000", 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.01102445, 2024-09-13T22:46:41.137 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011475282, 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 102005 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099092+0000", 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.138 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099092+0000", 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099092+0000", 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9800000000000003e-07 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099113+0000", 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0394000000000002e-05 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.139 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099120+0000", 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3880000000000002e-06 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099125+0000", 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0860000000000001e-06 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103588+0000", 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.004463252 2024-09-13T22:46:41.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103679+0000", 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1248000000000005e-05 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103774+0000", 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4384999999999998e-05 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.141 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:101999 8.f e202/175 8:f5a182c8:::benchmark_data_smithi178_80607_object101998:head v 202'6394, mlcod=202'6389)", 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:41.099186+0000", 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.01093006, 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011443607999999999, 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:46:41.142 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:46:41.143 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099186+0000", 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099186+0000", 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099186+0000", 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.5100000000000001e-07 2024-09-13T22:46:41.144 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099208+0000", 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2078000000000001e-05 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099215+0000", 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.691e-06 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.099220+0000", 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1189999999999996e-06 2024-09-13T22:46:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103765+0000", 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0045443710000000002 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:41.103791+0000", 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6046000000000001e-05 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout: "num_ops": 10 2024-09-13T22:46:41.146 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:46:41.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.234 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:41.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:41.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:41.360 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:41.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:41.489 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:41.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:41.624 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:41.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:41.748 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:41.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 82 16 102754 102738 78.2956 63.875 0.00309902 0.0127577 2024-09-13T22:46:41.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:41.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:41.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:42.008 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:42.112 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:42.135 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:42.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:42.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:42.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:42.388 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:42.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:42.491 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:42.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:42.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:42.646 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:42.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:42.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:42.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 83 16 103345 103329 77.7973 36.9375 0.0160188 0.0128491 2024-09-13T22:46:42.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:42.907 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:42.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:42.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:42 smithi178 bash[29889]: cluster 2024-09-13T22:46:42.619114+0000 mgr.y (mgr.14613) 337 : cluster [DBG] pgmap v355: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 920 op/s 2024-09-13T22:46:42.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:42 smithi178 bash[22840]: cluster 2024-09-13T22:46:42.619114+0000 mgr.y (mgr.14613) 337 : cluster [DBG] pgmap v355: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 920 op/s 2024-09-13T22:46:43.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:43.038 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:43.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:43.148 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:42 smithi192 bash[24608]: cluster 2024-09-13T22:46:42.619114+0000 mgr.y (mgr.14613) 337 : cluster [DBG] pgmap v355: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 920 op/s 2024-09-13T22:46:43.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:43.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:43.298 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:43.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:43.431 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:43.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:43.556 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:43.659 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:43.679 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:43.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:43.807 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:43.821 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 84 16 104190 104174 77.4998 52.8125 0.0138924 0.0128985 2024-09-13T22:46:43.871 INFO:teuthology.orchestra.run.smithi178.stderr:set 8.5 pg_upmap_items mapping to [5->0,4->1,6->3] 2024-09-13T22:46:43.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:43.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:43.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:43.947 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:43 smithi178 bash[29889]: audit 2024-09-13T22:46:43.647088+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? v1:172.21.15.178:0/93651896' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.5", "id": [5, 0, 4, 1, 6, 3]}]: dispatch 2024-09-13T22:46:43.948 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:43 smithi178 bash[22840]: audit 2024-09-13T22:46:43.647088+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? v1:172.21.15.178:0/93651896' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.5", "id": [5, 0, 4, 1, 6, 3]}]: dispatch 2024-09-13T22:46:44.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:44.055 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:44.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:43 smithi192 bash[24608]: audit 2024-09-13T22:46:43.647088+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? v1:172.21.15.178:0/93651896' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.5", "id": [5, 0, 4, 1, 6, 3]}]: dispatch 2024-09-13T22:46:44.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:44.178 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:44.280 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:44.305 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:44.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T22:46:44.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:44.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:44.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:44.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:44.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:44.679 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:44.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:44.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 85 16 104438 104422 76.7704 15.5 0.00289649 0.0128937 2024-09-13T22:46:44.885 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:46:44.885 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:46:44.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:44.906 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:46:44.906 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:46:44.923 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:46:44.923 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:46:44.933 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:44.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:44.938 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:46:44.939 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:46:44.954 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:46:44.955 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:46:44.973 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:46:44.973 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:46:44.990 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:46:44.990 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:46:45.006 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:46:45.007 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:46:45.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:45.061 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:45.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:44 smithi178 bash[29889]: audit 2024-09-13T22:46:43.871106+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? v1:172.21.15.178:0/93651896' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.5", "id": [5, 0, 4, 1, 6, 3]}]': finished 2024-09-13T22:46:45.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:44 smithi178 bash[29889]: cluster 2024-09-13T22:46:43.874237+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-13T22:46:45.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:44 smithi178 bash[29889]: cluster 2024-09-13T22:46:44.619578+0000 mgr.y (mgr.14613) 338 : cluster [DBG] pgmap v357: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T22:46:45.096 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:44 smithi178 bash[22840]: audit 2024-09-13T22:46:43.871106+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? v1:172.21.15.178:0/93651896' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.5", "id": [5, 0, 4, 1, 6, 3]}]': finished 2024-09-13T22:46:45.096 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:44 smithi178 bash[22840]: cluster 2024-09-13T22:46:43.874237+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-13T22:46:45.096 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:44 smithi178 bash[22840]: cluster 2024-09-13T22:46:44.619578+0000 mgr.y (mgr.14613) 338 : cluster [DBG] pgmap v357: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T22:46:45.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:44 smithi192 bash[24608]: audit 2024-09-13T22:46:43.871106+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? v1:172.21.15.178:0/93651896' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.5", "id": [5, 0, 4, 1, 6, 3]}]': finished 2024-09-13T22:46:45.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:44 smithi192 bash[24608]: cluster 2024-09-13T22:46:43.874237+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-13T22:46:45.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:44 smithi192 bash[24608]: cluster 2024-09-13T22:46:44.619578+0000 mgr.y (mgr.14613) 338 : cluster [DBG] pgmap v357: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T22:46:45.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:45.185 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:45.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:45.310 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:45.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:45.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:45.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:45.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:46:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:46:45.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:45.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:45.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:45.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:45.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:45.821 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:45.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 86 16 104438 104422 75.8777 0 - 0.0128937 2024-09-13T22:46:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:45.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:45.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:45 smithi192 bash[24608]: audit 2024-09-13T22:46:44.871561+0000 mon.a (mon.0) 1146 : audit [DBG] from='client.? v1:172.21.15.178:0/3088211744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:45.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:45 smithi192 bash[24608]: cluster 2024-09-13T22:46:44.906027+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-13T22:46:45.936 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:45 smithi192 bash[24608]: audit 2024-09-13T22:46:45.237234+0000 mgr.y (mgr.14613) 339 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:45.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:45.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:46.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:46.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:46.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:45 smithi178 bash[29889]: audit 2024-09-13T22:46:44.871561+0000 mon.a (mon.0) 1146 : audit [DBG] from='client.? v1:172.21.15.178:0/3088211744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:45 smithi178 bash[29889]: cluster 2024-09-13T22:46:44.906027+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-13T22:46:46.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:45 smithi178 bash[29889]: audit 2024-09-13T22:46:45.237234+0000 mgr.y (mgr.14613) 339 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:45 smithi178 bash[22840]: audit 2024-09-13T22:46:44.871561+0000 mon.a (mon.0) 1146 : audit [DBG] from='client.? v1:172.21.15.178:0/3088211744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:46:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:45 smithi178 bash[22840]: cluster 2024-09-13T22:46:44.906027+0000 mon.a (mon.0) 1147 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-13T22:46:46.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:45 smithi178 bash[22840]: audit 2024-09-13T22:46:45.237234+0000 mgr.y (mgr.14613) 339 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:46.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:46.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:46.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:46.350 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:46.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:46.486 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:46.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:46.618 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:46.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:46.757 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:46.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 87 16 105050 105034 75.4452 19.125 0.0161497 0.0132501 2024-09-13T22:46:46.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:46.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:46.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:46.914 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:46:46.989 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:47.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:47.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:46 smithi192 bash[24608]: cluster 2024-09-13T22:46:45.914102+0000 mon.a (mon.0) 1148 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-13T22:46:47.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:46 smithi192 bash[24608]: cluster 2024-09-13T22:46:45.915302+0000 osd.4 (osd.4) 171 : cluster [DBG] 8.5 starting backfill to osd.1 from (0'0,0'0] MAX to 202'6541 2024-09-13T22:46:47.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:46 smithi192 bash[24608]: cluster 2024-09-13T22:46:45.935153+0000 osd.4 (osd.4) 172 : cluster [DBG] 8.5 starting backfill to osd.3 from (0'0,0'0] MAX to 202'6541 2024-09-13T22:46:47.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:46 smithi192 bash[24608]: audit 2024-09-13T22:46:46.376609+0000 mon.c (mon.2) 41 : audit [INF] from='client.? v1:172.21.15.178:0/982485766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:46:47.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:46 smithi192 bash[24608]: audit 2024-09-13T22:46:46.377197+0000 mon.a (mon.0) 1149 : audit [INF] from='client.25121 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:46:47.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:46 smithi192 bash[24608]: cluster 2024-09-13T22:46:46.619990+0000 mgr.y (mgr.14613) 340 : cluster [DBG] pgmap v360: 158 pgs: 1 unknown, 157 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:46:47.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[22840]: cluster 2024-09-13T22:46:45.914102+0000 mon.a (mon.0) 1148 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-13T22:46:47.178 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[22840]: cluster 2024-09-13T22:46:45.915302+0000 osd.4 (osd.4) 171 : cluster [DBG] 8.5 starting backfill to osd.1 from (0'0,0'0] MAX to 202'6541 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[22840]: cluster 2024-09-13T22:46:45.935153+0000 osd.4 (osd.4) 172 : cluster [DBG] 8.5 starting backfill to osd.3 from (0'0,0'0] MAX to 202'6541 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[22840]: audit 2024-09-13T22:46:46.376609+0000 mon.c (mon.2) 41 : audit [INF] from='client.? v1:172.21.15.178:0/982485766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[22840]: audit 2024-09-13T22:46:46.377197+0000 mon.a (mon.0) 1149 : audit [INF] from='client.25121 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[22840]: cluster 2024-09-13T22:46:46.619990+0000 mgr.y (mgr.14613) 340 : cluster [DBG] pgmap v360: 158 pgs: 1 unknown, 157 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[29889]: cluster 2024-09-13T22:46:45.914102+0000 mon.a (mon.0) 1148 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[29889]: cluster 2024-09-13T22:46:45.915302+0000 osd.4 (osd.4) 171 : cluster [DBG] 8.5 starting backfill to osd.1 from (0'0,0'0] MAX to 202'6541 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[29889]: cluster 2024-09-13T22:46:45.935153+0000 osd.4 (osd.4) 172 : cluster [DBG] 8.5 starting backfill to osd.3 from (0'0,0'0] MAX to 202'6541 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[29889]: audit 2024-09-13T22:46:46.376609+0000 mon.c (mon.2) 41 : audit [INF] from='client.? v1:172.21.15.178:0/982485766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[29889]: audit 2024-09-13T22:46:46.377197+0000 mon.a (mon.0) 1149 : audit [INF] from='client.25121 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:46:47.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:46 smithi178 bash[29889]: cluster 2024-09-13T22:46:46.619990+0000 mgr.y (mgr.14613) 340 : cluster [DBG] pgmap v360: 158 pgs: 1 unknown, 157 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:46:47.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:47.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:47.387 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:47.411 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:47.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:47.542 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:47.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:47.691 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:47.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:47.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:47.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 88 16 106005 105989 75.2661 59.6875 0.0114741 0.0132814 2024-09-13T22:46:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:47.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:47.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:48.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:48.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:47 smithi192 bash[24608]: cluster 2024-09-13T22:46:46.906101+0000 mon.a (mon.0) 1150 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:48.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:47 smithi192 bash[24608]: audit 2024-09-13T22:46:46.908060+0000 mon.a (mon.0) 1151 : audit [INF] from='client.25121 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:46:48.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:47 smithi192 bash[24608]: cluster 2024-09-13T22:46:46.912712+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-13T22:46:48.077 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:48.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:47 smithi178 bash[29889]: cluster 2024-09-13T22:46:46.906101+0000 mon.a (mon.0) 1150 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:47 smithi178 bash[29889]: audit 2024-09-13T22:46:46.908060+0000 mon.a (mon.0) 1151 : audit [INF] from='client.25121 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:46:48.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:47 smithi178 bash[29889]: cluster 2024-09-13T22:46:46.912712+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-13T22:46:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:47 smithi178 bash[22840]: cluster 2024-09-13T22:46:46.906101+0000 mon.a (mon.0) 1150 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:46:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:47 smithi178 bash[22840]: audit 2024-09-13T22:46:46.908060+0000 mon.a (mon.0) 1151 : audit [INF] from='client.25121 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:46:48.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:47 smithi178 bash[22840]: cluster 2024-09-13T22:46:46.912712+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-13T22:46:48.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:48.206 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:48.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:48.347 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:48.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:48.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:48.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:48.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:48.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:48.733 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:48.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 89 16 107049 107033 75.1535 65.25 0.0101615 0.0133013 2024-09-13T22:46:48.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:48.873 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:48.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:48.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:47.387164+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:47.409612+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:47.970790+0000 osd.5 (osd.5) 214 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:47.974830+0000 osd.5 (osd.5) 215 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:47.977407+0000 osd.2 (osd.2) 151 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:47.978879+0000 osd.2 (osd.2) 152 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: audit 2024-09-13T22:46:48.316622+0000 mon.b (mon.1) 154 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:48.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:48 smithi192 bash[24608]: cluster 2024-09-13T22:46:48.620425+0000 mgr.y (mgr.14613) 341 : cluster [DBG] pgmap v362: 158 pgs: 1 unknown, 157 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:46:49.000 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:49.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:46:49.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.129 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:49.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:47.387164+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:47.409612+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:47.970790+0000 osd.5 (osd.5) 214 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:47.974830+0000 osd.5 (osd.5) 215 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:47.977407+0000 osd.2 (osd.2) 151 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:47.978879+0000 osd.2 (osd.2) 152 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: audit 2024-09-13T22:46:48.316622+0000 mon.b (mon.1) 154 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[29889]: cluster 2024-09-13T22:46:48.620425+0000 mgr.y (mgr.14613) 341 : cluster [DBG] pgmap v362: 158 pgs: 1 unknown, 157 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:47.387164+0000 osd.0 (osd.0) 191 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:47.409612+0000 osd.0 (osd.0) 192 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:47.970790+0000 osd.5 (osd.5) 214 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:47.974830+0000 osd.5 (osd.5) 215 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:47.977407+0000 osd.2 (osd.2) 151 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:47.978879+0000 osd.2 (osd.2) 152 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: audit 2024-09-13T22:46:48.316622+0000 mon.b (mon.1) 154 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:46:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:48 smithi178 bash[22840]: cluster 2024-09-13T22:46:48.620425+0000 mgr.y (mgr.14613) 341 : cluster [DBG] pgmap v362: 158 pgs: 1 unknown, 157 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:46:49.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:49.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:49.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:49.381 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:49.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:49.522 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:49.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:49.656 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:49.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:49.789 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:49.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total time run: 90.0083 2024-09-13T22:46:49.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total writes made: 108042 2024-09-13T22:46:49.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Write size: 65536 2024-09-13T22:46:49.822 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object size: 65536 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Bandwidth (MB/sec): 75.0222 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Bandwidth: 33.4555 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max bandwidth (MB/sec): 194.062 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min bandwidth (MB/sec): 0 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average IOPS: 1200 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev IOPS: 535.288 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max IOPS: 3105 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min IOPS: 0 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average Latency(s): 0.0133255 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Latency(s): 0.0343653 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max latency(s): 2.2767 2024-09-13T22:46:49.823 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min latency(s): 0.00124784 2024-09-13T22:46:49.826 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T22:46:49.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:49.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:49.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:49.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:46:50.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:50.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:50.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:50.169 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:46:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:46:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:46:50.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:50.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:50.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:50.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:50.445 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:50.479 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:50.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:50.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:50.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:50.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:50.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:50.869 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:50.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:50.982 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:50 smithi192 bash[24608]: cluster 2024-09-13T22:46:49.149583+0000 osd.0 (osd.0) 193 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:50.982 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:50 smithi192 bash[24608]: cluster 2024-09-13T22:46:49.149661+0000 osd.0 (osd.0) 194 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:50.982 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:50 smithi192 bash[24608]: cluster 2024-09-13T22:46:49.946495+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-13T22:46:50.982 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:50 smithi192 bash[24608]: cluster 2024-09-13T22:46:50.620841+0000 mgr.y (mgr.14613) 342 : cluster [DBG] pgmap v364: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 896 B/s rd, 106 MiB/s wr, 1.69k op/s; 12880/322914 objects misplaced (3.989%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T22:46:50.996 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:50.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:51.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:51.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[29889]: cluster 2024-09-13T22:46:49.149583+0000 osd.0 (osd.0) 193 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:51.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[29889]: cluster 2024-09-13T22:46:49.149661+0000 osd.0 (osd.0) 194 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:51.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[29889]: cluster 2024-09-13T22:46:49.946495+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-13T22:46:51.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[29889]: cluster 2024-09-13T22:46:50.620841+0000 mgr.y (mgr.14613) 342 : cluster [DBG] pgmap v364: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 896 B/s rd, 106 MiB/s wr, 1.69k op/s; 12880/322914 objects misplaced (3.989%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T22:46:51.109 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[22840]: cluster 2024-09-13T22:46:49.149583+0000 osd.0 (osd.0) 193 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:51.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[22840]: cluster 2024-09-13T22:46:49.149661+0000 osd.0 (osd.0) 194 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:51.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[22840]: cluster 2024-09-13T22:46:49.946495+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-13T22:46:51.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[22840]: cluster 2024-09-13T22:46:50.620841+0000 mgr.y (mgr.14613) 342 : cluster [DBG] pgmap v364: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 896 B/s rd, 106 MiB/s wr, 1.69k op/s; 12880/322914 objects misplaced (3.989%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T22:46:51.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:51.209 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:46:51.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:46:51.209 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:46:51.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T22:46:51.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:51.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:51.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:51.377 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:51.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:51.517 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:51.620 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:51.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:51.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:51.786 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:51.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:51.922 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:51.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:52.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:52.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:51 smithi192 bash[24608]: cluster 2024-09-13T22:46:49.999847+0000 osd.2 (osd.2) 153 : cluster [DBG] 5.b scrub starts 2024-09-13T22:46:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:51 smithi192 bash[24608]: cluster 2024-09-13T22:46:50.007286+0000 osd.2 (osd.2) 154 : cluster [DBG] 5.b scrub ok 2024-09-13T22:46:52.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:52.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[29889]: cluster 2024-09-13T22:46:49.999847+0000 osd.2 (osd.2) 153 : cluster [DBG] 5.b scrub starts 2024-09-13T22:46:52.178 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[29889]: cluster 2024-09-13T22:46:50.007286+0000 osd.2 (osd.2) 154 : cluster [DBG] 5.b scrub ok 2024-09-13T22:46:52.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[22840]: cluster 2024-09-13T22:46:49.999847+0000 osd.2 (osd.2) 153 : cluster [DBG] 5.b scrub starts 2024-09-13T22:46:52.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:51 smithi178 bash[22840]: cluster 2024-09-13T22:46:50.007286+0000 osd.2 (osd.2) 154 : cluster [DBG] 5.b scrub ok 2024-09-13T22:46:52.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:52.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:52.332 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:52.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:52.469 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:52.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:52.594 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:52.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:52.728 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:52.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:52.861 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:52.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:52.982 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:52.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:53.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:53.115 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:53.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:53.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:53.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:53.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:52 smithi192 bash[24608]: cluster 2024-09-13T22:46:51.969584+0000 osd.2 (osd.2) 155 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:46:53.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:52 smithi192 bash[24608]: cluster 2024-09-13T22:46:51.976338+0000 osd.2 (osd.2) 156 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:46:53.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:52 smithi192 bash[24608]: cluster 2024-09-13T22:46:52.136270+0000 osd.5 (osd.5) 216 : cluster [DBG] 8.d scrub starts 2024-09-13T22:46:53.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:52 smithi192 bash[24608]: cluster 2024-09-13T22:46:52.621255+0000 mgr.y (mgr.14613) 343 : cluster [DBG] pgmap v365: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 762 B/s rd, 90 MiB/s wr, 1.44k op/s; 12880/322914 objects misplaced (3.989%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T22:46:53.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:52 smithi192 bash[24608]: cluster 2024-09-13T22:46:52.869074+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-13T22:46:53.358 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:53.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:53.487 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:53.502 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:51.969584+0000 osd.2 (osd.2) 155 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:51.976338+0000 osd.2 (osd.2) 156 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:52.136270+0000 osd.5 (osd.5) 216 : cluster [DBG] 8.d scrub starts 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:52.621255+0000 mgr.y (mgr.14613) 343 : cluster [DBG] pgmap v365: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 762 B/s rd, 90 MiB/s wr, 1.44k op/s; 12880/322914 objects misplaced (3.989%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:52.869074+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:51.969584+0000 osd.2 (osd.2) 155 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:51.976338+0000 osd.2 (osd.2) 156 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:52.136270+0000 osd.5 (osd.5) 216 : cluster [DBG] 8.d scrub starts 2024-09-13T22:46:53.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:52.621255+0000 mgr.y (mgr.14613) 343 : cluster [DBG] pgmap v365: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 762 B/s rd, 90 MiB/s wr, 1.44k op/s; 12880/322914 objects misplaced (3.989%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T22:46:53.524 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:52.869074+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-13T22:46:53.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:53.615 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:53.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:53.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:53.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:53.878 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:53.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:53.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:54.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:54.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:54.115 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.091064+0000 osd.5 (osd.5) 217 : cluster [DBG] 8.4 scrub starts 2024-09-13T22:46:54.115 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.499507+0000 osd.5 (osd.5) 218 : cluster [DBG] 8.d scrub ok 2024-09-13T22:46:54.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.091064+0000 osd.5 (osd.5) 217 : cluster [DBG] 8.4 scrub starts 2024-09-13T22:46:54.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:53 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.499507+0000 osd.5 (osd.5) 218 : cluster [DBG] 8.d scrub ok 2024-09-13T22:46:54.137 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:54.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:54.249 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:53 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.091064+0000 osd.5 (osd.5) 217 : cluster [DBG] 8.4 scrub starts 2024-09-13T22:46:54.250 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:53 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.499507+0000 osd.5 (osd.5) 218 : cluster [DBG] 8.d scrub ok 2024-09-13T22:46:54.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:54.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:54.397 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:54.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:54.516 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:54.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:54.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:54.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:54.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:54.759 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:54.785 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:54.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:54.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:54.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:55.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:55.046 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:55.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:55.170 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:55.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:52.998220+0000 osd.2 (osd.2) 157 : cluster [DBG] 6.f scrub starts 2024-09-13T22:46:55.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.000157+0000 osd.2 (osd.2) 158 : cluster [DBG] 6.f scrub ok 2024-09-13T22:46:55.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.527948+0000 osd.0 (osd.0) 195 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:46:55.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.530235+0000 osd.0 (osd.0) 196 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:46:55.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.985727+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.a scrub starts 2024-09-13T22:46:55.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:53.986565+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.a scrub ok 2024-09-13T22:46:55.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:54.244545+0000 osd.5 (osd.5) 219 : cluster [DBG] 8.4 scrub ok 2024-09-13T22:46:55.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:54 smithi192 bash[24608]: cluster 2024-09-13T22:46:54.622154+0000 mgr.y (mgr.14613) 344 : cluster [DBG] pgmap v367: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 796 B/s rd, 83 MiB/s wr, 1.64k op/s; 11500/317304 objects misplaced (3.624%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:52.998220+0000 osd.2 (osd.2) 157 : cluster [DBG] 6.f scrub starts 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.000157+0000 osd.2 (osd.2) 158 : cluster [DBG] 6.f scrub ok 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.527948+0000 osd.0 (osd.0) 195 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.530235+0000 osd.0 (osd.0) 196 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.985727+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.a scrub starts 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:53.986565+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.a scrub ok 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:54.244545+0000 osd.5 (osd.5) 219 : cluster [DBG] 8.4 scrub ok 2024-09-13T22:46:55.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[22840]: cluster 2024-09-13T22:46:54.622154+0000 mgr.y (mgr.14613) 344 : cluster [DBG] pgmap v367: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 796 B/s rd, 83 MiB/s wr, 1.64k op/s; 11500/317304 objects misplaced (3.624%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:52.998220+0000 osd.2 (osd.2) 157 : cluster [DBG] 6.f scrub starts 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.000157+0000 osd.2 (osd.2) 158 : cluster [DBG] 6.f scrub ok 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.527948+0000 osd.0 (osd.0) 195 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.530235+0000 osd.0 (osd.0) 196 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.985727+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.a scrub starts 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:53.986565+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.a scrub ok 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:54.244545+0000 osd.5 (osd.5) 219 : cluster [DBG] 8.4 scrub ok 2024-09-13T22:46:55.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:54 smithi178 bash[29889]: cluster 2024-09-13T22:46:54.622154+0000 mgr.y (mgr.14613) 344 : cluster [DBG] pgmap v367: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 796 B/s rd, 83 MiB/s wr, 1.64k op/s; 11500/317304 objects misplaced (3.624%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T22:46:55.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:55.308 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:55.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:46:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:46:55.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:55.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:55.533 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:46:55.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:55.562 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:55.664 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:55.694 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:55.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:55.831 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:55.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:55.957 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:55.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:56.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:56.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:55 smithi192 bash[24608]: cluster 2024-09-13T22:46:54.743670+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:46:56.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:55 smithi192 bash[24608]: cluster 2024-09-13T22:46:54.746088+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:46:56.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:55 smithi192 bash[24608]: audit 2024-09-13T22:46:55.248435+0000 mgr.y (mgr.14613) 345 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:56.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:56.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.217 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:56.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:56.218 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:55 smithi178 bash[22840]: cluster 2024-09-13T22:46:54.743670+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:46:56.218 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:55 smithi178 bash[22840]: cluster 2024-09-13T22:46:54.746088+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:46:56.218 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:55 smithi178 bash[22840]: audit 2024-09-13T22:46:55.248435+0000 mgr.y (mgr.14613) 345 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:56.218 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:55 smithi178 bash[29889]: cluster 2024-09-13T22:46:54.743670+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:46:56.218 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:55 smithi178 bash[29889]: cluster 2024-09-13T22:46:54.746088+0000 osd.7 (osd.7) 194 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:46:56.218 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:55 smithi178 bash[29889]: audit 2024-09-13T22:46:55.248435+0000 mgr.y (mgr.14613) 345 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:46:56.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:56.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:56.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:56.482 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:56.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:56.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:56.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:56.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:56.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:56.878 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:56.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:56.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:57.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:57.094 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:46:57.094 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:46:57.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:55.556368+0000 osd.0 (osd.0) 197 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:55.557575+0000 osd.0 (osd.0) 198 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:55.711377+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:55.715366+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.047502+0000 osd.5 (osd.5) 220 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.049679+0000 osd.5 (osd.5) 221 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:46:57.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:56 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.622763+0000 mgr.y (mgr.14613) 346 : cluster [DBG] pgmap v368: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.79k op/s; 11500/312408 objects misplaced (3.681%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T22:46:57.131 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:55.556368+0000 osd.0 (osd.0) 197 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:55.557575+0000 osd.0 (osd.0) 198 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:55.711377+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:55.715366+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.047502+0000 osd.5 (osd.5) 220 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.049679+0000 osd.5 (osd.5) 221 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:46:57.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.622763+0000 mgr.y (mgr.14613) 346 : cluster [DBG] pgmap v368: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.79k op/s; 11500/312408 objects misplaced (3.681%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:55.556368+0000 osd.0 (osd.0) 197 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:55.557575+0000 osd.0 (osd.0) 198 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:55.711377+0000 osd.7 (osd.7) 195 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:55.715366+0000 osd.7 (osd.7) 196 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.047502+0000 osd.5 (osd.5) 220 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.049679+0000 osd.5 (osd.5) 221 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:46:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:56 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.622763+0000 mgr.y (mgr.14613) 346 : cluster [DBG] pgmap v368: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.79k op/s; 11500/312408 objects misplaced (3.681%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T22:46:57.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:57.266 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:46:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:46:57.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:57.401 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:57.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:57.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:57.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:57.660 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:57.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:57.794 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:57.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:57.921 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:46:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:57.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:46:58.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:58.033 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:57 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.517338+0000 osd.0 (osd.0) 199 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:46:58.033 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:57 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.519325+0000 osd.0 (osd.0) 200 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:46:58.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:57 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.703140+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:46:58.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:57 smithi192 bash[24608]: cluster 2024-09-13T22:46:56.706431+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:46:58.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:58.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:58.169 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:58.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:58.294 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:58.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.517338+0000 osd.0 (osd.0) 199 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:46:58.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.519325+0000 osd.0 (osd.0) 200 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:46:58.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.703140+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:46:58.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[29889]: cluster 2024-09-13T22:46:56.706431+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:46:58.374 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.517338+0000 osd.0 (osd.0) 199 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:46:58.375 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.519325+0000 osd.0 (osd.0) 200 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:46:58.375 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.703140+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:46:58.375 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:57 smithi178 bash[22840]: cluster 2024-09-13T22:46:56.706431+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:46:58.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:58.429 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:46:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:46:58.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:58.567 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:58.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:58.704 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:58.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:58.830 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:58.932 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:58.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:59.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:59.094 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:46:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:46:59.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:59.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:59.336 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:59.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:57.698813+0000 osd.7 (osd.7) 199 : cluster [DBG] 3.e scrub starts 2024-09-13T22:46:59.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:57.700452+0000 osd.7 (osd.7) 200 : cluster [DBG] 3.e scrub ok 2024-09-13T22:46:59.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.032222+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:46:59.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.038408+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.419997+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.420089+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.623187+0000 mgr.y (mgr.14613) 347 : cluster [DBG] pgmap v369: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 707 B/s rd, 4.3 MiB/s wr, 543 op/s; 11500/312408 objects misplaced (3.681%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:57.698813+0000 osd.7 (osd.7) 199 : cluster [DBG] 3.e scrub starts 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:57.700452+0000 osd.7 (osd.7) 200 : cluster [DBG] 3.e scrub ok 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.032222+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.038408+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.419997+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:59.346 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.420089+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:59.347 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.623187+0000 mgr.y (mgr.14613) 347 : cluster [DBG] pgmap v369: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 707 B/s rd, 4.3 MiB/s wr, 543 op/s; 11500/312408 objects misplaced (3.681%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T22:46:59.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:46:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:46:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:57.698813+0000 osd.7 (osd.7) 199 : cluster [DBG] 3.e scrub starts 2024-09-13T22:46:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:57.700452+0000 osd.7 (osd.7) 200 : cluster [DBG] 3.e scrub ok 2024-09-13T22:46:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.032222+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:46:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.038408+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:46:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.419997+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:46:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.420089+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:46:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:58 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.623187+0000 mgr.y (mgr.14613) 347 : cluster [DBG] pgmap v369: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 707 B/s rd, 4.3 MiB/s wr, 543 op/s; 11500/312408 objects misplaced (3.681%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T22:46:59.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:59.497 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:59.600 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:59.634 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:46:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:46:59.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:59.761 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:46:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:46:59.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:59.888 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:46:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:46:59.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:46:59.891 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:46:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:46:59.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:00.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:00.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:00.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:00.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:59 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.544818+0000 osd.0 (osd.0) 201 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:47:00.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:59 smithi192 bash[24608]: cluster 2024-09-13T22:46:58.546261+0000 osd.0 (osd.0) 202 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:47:00.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:59 smithi192 bash[24608]: cluster 2024-09-13T22:46:59.001218+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:47:00.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:59 smithi192 bash[24608]: cluster 2024-09-13T22:46:59.003662+0000 osd.6 (osd.6) 152 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:47:00.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:46:59 smithi192 bash[24608]: cluster 2024-09-13T22:46:59.033724+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-13T22:47:00.187 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.544818+0000 osd.0 (osd.0) 201 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:47:00.187 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:58.546261+0000 osd.0 (osd.0) 202 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:47:00.187 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:59.001218+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:47:00.187 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:59.003662+0000 osd.6 (osd.6) 152 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:47:00.187 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:46:59 smithi178 bash[22840]: cluster 2024-09-13T22:46:59.033724+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-13T22:47:00.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:00.289 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:00.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:00.418 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:00.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:00.544 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:00.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:00.663 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.544818+0000 osd.0 (osd.0) 201 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:47:00.663 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:58.546261+0000 osd.0 (osd.0) 202 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:47:00.663 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:59.001218+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:47:00.663 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:59.003662+0000 osd.6 (osd.6) 152 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:47:00.663 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:46:59.033724+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-13T22:47:00.677 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:00.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:00.811 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:00.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:00.947 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:00.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:01.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:01.090 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:01.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:01.236 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:01.259 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:47:00.624239+0000 mgr.y (mgr.14613) 348 : cluster [DBG] pgmap v371: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.7 MiB/s wr, 1.16k op/s; 10178/298626 objects misplaced (3.408%); 7.6 MiB/s, 122 objects/s recovering 2024-09-13T22:47:01.259 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:47:00.956869+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:47:01.259 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[29889]: cluster 2024-09-13T22:47:00.957910+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:47:01.259 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[22840]: cluster 2024-09-13T22:47:00.624239+0000 mgr.y (mgr.14613) 348 : cluster [DBG] pgmap v371: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.7 MiB/s wr, 1.16k op/s; 10178/298626 objects misplaced (3.408%); 7.6 MiB/s, 122 objects/s recovering 2024-09-13T22:47:01.259 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[22840]: cluster 2024-09-13T22:47:00.956869+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:47:01.259 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:00 smithi178 bash[22840]: cluster 2024-09-13T22:47:00.957910+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:47:01.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:01.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:00 smithi192 bash[24608]: cluster 2024-09-13T22:47:00.624239+0000 mgr.y (mgr.14613) 348 : cluster [DBG] pgmap v371: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.7 MiB/s wr, 1.16k op/s; 10178/298626 objects misplaced (3.408%); 7.6 MiB/s, 122 objects/s recovering 2024-09-13T22:47:01.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:00 smithi192 bash[24608]: cluster 2024-09-13T22:47:00.956869+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:47:01.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:00 smithi192 bash[24608]: cluster 2024-09-13T22:47:00.957910+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:47:01.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T22:47:01.368 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:01.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:01.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:01.502 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:01.551 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:47:01.551 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:47:01.603 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:47:01.603 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:47:01.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:01.644 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:01.655 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:47:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:47:01.691 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:47:01.691 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:47:01.719 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:47:01.719 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:47:01.748 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:47:01.748 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:47:01.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:01.773 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:01.783 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:47:01.783 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:47:01.807 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:47:01.808 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:47:01.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:01.909 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:01.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:02.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:02.022 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:01 smithi192 bash[24608]: cluster 2024-09-13T22:47:00.037743+0000 osd.2 (osd.2) 161 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:47:02.022 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:01 smithi192 bash[24608]: cluster 2024-09-13T22:47:00.045051+0000 osd.2 (osd.2) 162 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:47:02.022 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:01 smithi192 bash[24608]: cluster 2024-09-13T22:47:01.059841+0000 osd.5 (osd.5) 222 : cluster [DBG] 8.8 scrub starts 2024-09-13T22:47:02.022 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:01 smithi192 bash[24608]: audit 2024-09-13T22:47:01.524138+0000 mon.a (mon.0) 1156 : audit [DBG] from='client.? v1:172.21.15.178:0/1199871284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:02.041 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[22840]: cluster 2024-09-13T22:47:00.037743+0000 osd.2 (osd.2) 161 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[22840]: cluster 2024-09-13T22:47:00.045051+0000 osd.2 (osd.2) 162 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[22840]: cluster 2024-09-13T22:47:01.059841+0000 osd.5 (osd.5) 222 : cluster [DBG] 8.8 scrub starts 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[22840]: audit 2024-09-13T22:47:01.524138+0000 mon.a (mon.0) 1156 : audit [DBG] from='client.? v1:172.21.15.178:0/1199871284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[29889]: cluster 2024-09-13T22:47:00.037743+0000 osd.2 (osd.2) 161 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[29889]: cluster 2024-09-13T22:47:00.045051+0000 osd.2 (osd.2) 162 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:47:02.114 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[29889]: cluster 2024-09-13T22:47:01.059841+0000 osd.5 (osd.5) 222 : cluster [DBG] 8.8 scrub starts 2024-09-13T22:47:02.115 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:01 smithi178 bash[29889]: audit 2024-09-13T22:47:01.524138+0000 mon.a (mon.0) 1156 : audit [DBG] from='client.? v1:172.21.15.178:0/1199871284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:02.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:02.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:02.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:02.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:02.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:02.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:02 smithi178 bash[22840]: debug 2024-09-13T22:47:02.354+0000 7f8b8e05d640 -1 mon.a@0(leader).osd e209 definitely_dead 0 2024-09-13T22:47:02.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:02.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:02.589 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:02.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:02.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:02.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:02.838 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:02.877 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:47:02.881 INFO:teuthology.orchestra.run.smithi178.stderr:marked out osd.4. 2024-09-13T22:47:02.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:02.957 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:02.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:03.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:03.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: audit 2024-09-13T22:47:02.353876+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.178:0/3895294853' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: audit 2024-09-13T22:47:02.354996+0000 mon.a (mon.0) 1157 : audit [INF] from='client.25133 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.355266+0000 mon.a (mon.0) 1158 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: audit 2024-09-13T22:47:02.395937+0000 mon.a (mon.0) 1159 : audit [INF] from='client.25090 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: audit 2024-09-13T22:47:02.396398+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.178:0/2835939688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.624728+0000 mgr.y (mgr.14613) 349 : cluster [DBG] pgmap v372: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.9 MiB/s wr, 954 op/s; 10178/298626 objects misplaced (3.408%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.658899+0000 osd.5 (osd.5) 223 : cluster [DBG] 8.8 scrub ok 2024-09-13T22:47:03.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.865927+0000 mon.a (mon.0) 1160 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:47:03.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.865980+0000 mon.a (mon.0) 1161 : cluster [INF] Cluster is now healthy 2024-09-13T22:47:03.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: audit 2024-09-13T22:47:02.867664+0000 mon.a (mon.0) 1162 : audit [INF] from='client.25133 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T22:47:03.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: audit 2024-09-13T22:47:02.867816+0000 mon.a (mon.0) 1163 : audit [INF] from='client.25090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:47:03.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:03 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.874298+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-09-13T22:47:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: audit 2024-09-13T22:47:02.353876+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.178:0/3895294853' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:47:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: audit 2024-09-13T22:47:02.354996+0000 mon.a (mon.0) 1157 : audit [INF] from='client.25133 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:47:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.355266+0000 mon.a (mon.0) 1158 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T22:47:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: audit 2024-09-13T22:47:02.395937+0000 mon.a (mon.0) 1159 : audit [INF] from='client.25090 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:03.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: audit 2024-09-13T22:47:02.396398+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.178:0/2835939688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.624728+0000 mgr.y (mgr.14613) 349 : cluster [DBG] pgmap v372: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.9 MiB/s wr, 954 op/s; 10178/298626 objects misplaced (3.408%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.658899+0000 osd.5 (osd.5) 223 : cluster [DBG] 8.8 scrub ok 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.865927+0000 mon.a (mon.0) 1160 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.865980+0000 mon.a (mon.0) 1161 : cluster [INF] Cluster is now healthy 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: audit 2024-09-13T22:47:02.867664+0000 mon.a (mon.0) 1162 : audit [INF] from='client.25133 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: audit 2024-09-13T22:47:02.867816+0000 mon.a (mon.0) 1163 : audit [INF] from='client.25090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.874298+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: audit 2024-09-13T22:47:02.353876+0000 mon.c (mon.2) 42 : audit [INF] from='client.? v1:172.21.15.178:0/3895294853' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: audit 2024-09-13T22:47:02.354996+0000 mon.a (mon.0) 1157 : audit [INF] from='client.25133 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:47:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.355266+0000 mon.a (mon.0) 1158 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: audit 2024-09-13T22:47:02.395937+0000 mon.a (mon.0) 1159 : audit [INF] from='client.25090 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: audit 2024-09-13T22:47:02.396398+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.178:0/2835939688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.624728+0000 mgr.y (mgr.14613) 349 : cluster [DBG] pgmap v372: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.9 MiB/s wr, 954 op/s; 10178/298626 objects misplaced (3.408%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.658899+0000 osd.5 (osd.5) 223 : cluster [DBG] 8.8 scrub ok 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.865927+0000 mon.a (mon.0) 1160 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.865980+0000 mon.a (mon.0) 1161 : cluster [INF] Cluster is now healthy 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: audit 2024-09-13T22:47:02.867664+0000 mon.a (mon.0) 1162 : audit [INF] from='client.25133 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: audit 2024-09-13T22:47:02.867816+0000 mon.a (mon.0) 1163 : audit [INF] from='client.25090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:47:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:03 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.874298+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-09-13T22:47:03.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:03.209 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:03.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:03.339 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:03.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:03.469 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:03.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:03.598 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:03.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:03.728 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:03.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:03.854 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:03.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:03.976 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:03.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:04.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:04.101 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:04.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:04.230 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:04.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:04.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.989554+0000 osd.7 (osd.7) 203 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:47:04.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:02.999671+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:47:04.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: audit 2024-09-13T22:47:03.339619+0000 mon.a (mon.0) 1165 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:04.347 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: audit 2024-09-13T22:47:03.343051+0000 mon.b (mon.1) 156 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:04.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.368598+0000 osd.3 (osd.3) 208 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:47:04.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.372352+0000 osd.3 (osd.3) 209 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:47:04.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.873951+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-09-13T22:47:04.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.876164+0000 osd.4 (osd.4) 173 : cluster [DBG] 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 209'7501 2024-09-13T22:47:04.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:04 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.944532+0000 osd.4 (osd.4) 174 : cluster [INF] 8.5 continuing backfill to osd.3 from (184'2754,209'7501] 8:a3f0bef2:::benchmark_data_smithi178_80607_object20510:head to 209'7501 2024-09-13T22:47:04.360 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:04.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:04.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.989554+0000 osd.7 (osd.7) 203 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:02.999671+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: audit 2024-09-13T22:47:03.339619+0000 mon.a (mon.0) 1165 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: audit 2024-09-13T22:47:03.343051+0000 mon.b (mon.1) 156 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.368598+0000 osd.3 (osd.3) 208 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.372352+0000 osd.3 (osd.3) 209 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.873951+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.876164+0000 osd.4 (osd.4) 173 : cluster [DBG] 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 209'7501 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.944532+0000 osd.4 (osd.4) 174 : cluster [INF] 8.5 continuing backfill to osd.3 from (184'2754,209'7501] 8:a3f0bef2:::benchmark_data_smithi178_80607_object20510:head to 209'7501 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.989554+0000 osd.7 (osd.7) 203 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:47:04.492 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:02.999671+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: audit 2024-09-13T22:47:03.339619+0000 mon.a (mon.0) 1165 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: audit 2024-09-13T22:47:03.343051+0000 mon.b (mon.1) 156 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.368598+0000 osd.3 (osd.3) 208 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.372352+0000 osd.3 (osd.3) 209 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.873951+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.876164+0000 osd.4 (osd.4) 173 : cluster [DBG] 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 209'7501 2024-09-13T22:47:04.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:04 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.944532+0000 osd.4 (osd.4) 174 : cluster [INF] 8.5 continuing backfill to osd.3 from (184'2754,209'7501] 8:a3f0bef2:::benchmark_data_smithi178_80607_object20510:head to 209'7501 2024-09-13T22:47:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:04.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:04.622 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:04.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:04.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:04.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:04.889 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:04.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:04.994 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:05.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:05.152 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:05.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:05.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:05.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:05.392 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.639116+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:03.640573+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:04.625929+0000 mgr.y (mgr.14613) 350 : cluster [DBG] pgmap v375: 158 pgs: 3 remapped+peering, 30 peering, 1 active+recovering, 124 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 5.0 KiB/s rd, 37 KiB/s wr, 1.07k op/s; 1/293211 objects misplaced (0.000%); 8.2 MiB/s, 133 objects/s recovering 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:04.876345+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:04.878258+0000 osd.0 (osd.0) 203 : cluster [DBG] 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 209'7484 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:04.879891+0000 osd.0 (osd.0) 204 : cluster [DBG] 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 209'7556 2024-09-13T22:47:05.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:05 smithi192 bash[24608]: cluster 2024-09-13T22:47:04.885030+0000 osd.5 (osd.5) 224 : cluster [DBG] 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 209'7503 2024-09-13T22:47:05.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:05.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.639116+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:47:05.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:03.640573+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:47:05.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:04.625929+0000 mgr.y (mgr.14613) 350 : cluster [DBG] pgmap v375: 158 pgs: 3 remapped+peering, 30 peering, 1 active+recovering, 124 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 5.0 KiB/s rd, 37 KiB/s wr, 1.07k op/s; 1/293211 objects misplaced (0.000%); 8.2 MiB/s, 133 objects/s recovering 2024-09-13T22:47:05.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:04.876345+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-09-13T22:47:05.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:04.878258+0000 osd.0 (osd.0) 203 : cluster [DBG] 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 209'7484 2024-09-13T22:47:05.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:04.879891+0000 osd.0 (osd.0) 204 : cluster [DBG] 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 209'7556 2024-09-13T22:47:05.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[29889]: cluster 2024-09-13T22:47:04.885030+0000 osd.5 (osd.5) 224 : cluster [DBG] 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 209'7503 2024-09-13T22:47:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.546 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:05.546 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.639116+0000 osd.7 (osd.7) 205 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:47:05.546 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:03.640573+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:47:05.546 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:04.625929+0000 mgr.y (mgr.14613) 350 : cluster [DBG] pgmap v375: 158 pgs: 3 remapped+peering, 30 peering, 1 active+recovering, 124 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 5.0 KiB/s rd, 37 KiB/s wr, 1.07k op/s; 1/293211 objects misplaced (0.000%); 8.2 MiB/s, 133 objects/s recovering 2024-09-13T22:47:05.546 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:04.876345+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-09-13T22:47:05.547 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:04.878258+0000 osd.0 (osd.0) 203 : cluster [DBG] 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 209'7484 2024-09-13T22:47:05.547 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:04.879891+0000 osd.0 (osd.0) 204 : cluster [DBG] 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 209'7556 2024-09-13T22:47:05.547 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:05 smithi178 bash[22840]: cluster 2024-09-13T22:47:04.885030+0000 osd.5 (osd.5) 224 : cluster [DBG] 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 209'7503 2024-09-13T22:47:05.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.676 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:05.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:05.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:47:05.807 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:05.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:05.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:06.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:06.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:06.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:06.216 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:06.319 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:06.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:06.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:06.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.485 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:06.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:06.586 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[29889]: cluster 2024-09-13T22:47:04.888658+0000 osd.3 (osd.3) 210 : cluster [DBG] 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 209'7397 2024-09-13T22:47:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[29889]: audit 2024-09-13T22:47:05.264657+0000 mgr.y (mgr.14613) 351 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[29889]: cluster 2024-09-13T22:47:05.337020+0000 mon.a (mon.0) 1168 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-13T22:47:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[29889]: cluster 2024-09-13T22:47:05.877968+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-09-13T22:47:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[29889]: cluster 2024-09-13T22:47:05.962664+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[29889]: cluster 2024-09-13T22:47:05.962720+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:06.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[22840]: cluster 2024-09-13T22:47:04.888658+0000 osd.3 (osd.3) 210 : cluster [DBG] 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 209'7397 2024-09-13T22:47:06.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[22840]: audit 2024-09-13T22:47:05.264657+0000 mgr.y (mgr.14613) 351 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:06.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[22840]: cluster 2024-09-13T22:47:05.337020+0000 mon.a (mon.0) 1168 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-13T22:47:06.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[22840]: cluster 2024-09-13T22:47:05.877968+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-09-13T22:47:06.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[22840]: cluster 2024-09-13T22:47:05.962664+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:06.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:06 smithi178 bash[22840]: cluster 2024-09-13T22:47:05.962720+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:06.613 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:06.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:06 smithi192 bash[24608]: cluster 2024-09-13T22:47:04.888658+0000 osd.3 (osd.3) 210 : cluster [DBG] 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 209'7397 2024-09-13T22:47:06.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:06 smithi192 bash[24608]: audit 2024-09-13T22:47:05.264657+0000 mgr.y (mgr.14613) 351 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:06.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:06 smithi192 bash[24608]: cluster 2024-09-13T22:47:05.337020+0000 mon.a (mon.0) 1168 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-13T22:47:06.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:06 smithi192 bash[24608]: cluster 2024-09-13T22:47:05.877968+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-09-13T22:47:06.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:06 smithi192 bash[24608]: cluster 2024-09-13T22:47:05.962664+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:06.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:06 smithi192 bash[24608]: cluster 2024-09-13T22:47:05.962720+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:06.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:06.746 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:06.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:06.879 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:06.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:06.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:07.018 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:07.121 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:07.145 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:07.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:07.283 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:07.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:47:07.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:07.413 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[29889]: cluster 2024-09-13T22:47:06.627882+0000 mgr.y (mgr.14613) 352 : cluster [DBG] pgmap v378: 158 pgs: 1 activating+degraded+remapped, 5 remapped+peering, 30 peering, 1 active+recovering, 121 active+clean; 5.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 7.2 KiB/s rd, 55 KiB/s wr, 770 op/s; 2/289479 objects degraded (0.001%); 5966/289479 objects misplaced (2.061%); 5.7 MiB/s, 102 objects/s recovering 2024-09-13T22:47:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[29889]: cluster 2024-09-13T22:47:06.880241+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-09-13T22:47:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[29889]: cluster 2024-09-13T22:47:07.014122+0000 osd.1 (osd.1) 188 : cluster [DBG] 2.6 deep-scrub starts 2024-09-13T22:47:07.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[29889]: cluster 2024-09-13T22:47:07.022788+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 deep-scrub ok 2024-09-13T22:47:07.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[22840]: cluster 2024-09-13T22:47:06.627882+0000 mgr.y (mgr.14613) 352 : cluster [DBG] pgmap v378: 158 pgs: 1 activating+degraded+remapped, 5 remapped+peering, 30 peering, 1 active+recovering, 121 active+clean; 5.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 7.2 KiB/s rd, 55 KiB/s wr, 770 op/s; 2/289479 objects degraded (0.001%); 5966/289479 objects misplaced (2.061%); 5.7 MiB/s, 102 objects/s recovering 2024-09-13T22:47:07.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[22840]: cluster 2024-09-13T22:47:06.880241+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-09-13T22:47:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[22840]: cluster 2024-09-13T22:47:07.014122+0000 osd.1 (osd.1) 188 : cluster [DBG] 2.6 deep-scrub starts 2024-09-13T22:47:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:07 smithi178 bash[22840]: cluster 2024-09-13T22:47:07.022788+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 deep-scrub ok 2024-09-13T22:47:07.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:07.559 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:07.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:07 smithi192 bash[24608]: cluster 2024-09-13T22:47:06.627882+0000 mgr.y (mgr.14613) 352 : cluster [DBG] pgmap v378: 158 pgs: 1 activating+degraded+remapped, 5 remapped+peering, 30 peering, 1 active+recovering, 121 active+clean; 5.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 7.2 KiB/s rd, 55 KiB/s wr, 770 op/s; 2/289479 objects degraded (0.001%); 5966/289479 objects misplaced (2.061%); 5.7 MiB/s, 102 objects/s recovering 2024-09-13T22:47:07.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:07 smithi192 bash[24608]: cluster 2024-09-13T22:47:06.880241+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-09-13T22:47:07.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:07 smithi192 bash[24608]: cluster 2024-09-13T22:47:07.014122+0000 osd.1 (osd.1) 188 : cluster [DBG] 2.6 deep-scrub starts 2024-09-13T22:47:07.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:07 smithi192 bash[24608]: cluster 2024-09-13T22:47:07.022788+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 deep-scrub ok 2024-09-13T22:47:07.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:07.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:07.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:07.823 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:07.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:07.957 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:07.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:08.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:08.087 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:08.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:08.216 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:08.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:08.345 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:08.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:08.482 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:08.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:08.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:08.626 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:08 smithi178 bash[29889]: cluster 2024-09-13T22:47:07.355024+0000 mon.a (mon.0) 1171 : cluster [WRN] Health check failed: Degraded data redundancy: 2/289479 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T22:47:08.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:08 smithi178 bash[22840]: cluster 2024-09-13T22:47:07.355024+0000 mon.a (mon.0) 1171 : cluster [WRN] Health check failed: Degraded data redundancy: 2/289479 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T22:47:08.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:08 smithi192 bash[24608]: cluster 2024-09-13T22:47:07.355024+0000 mon.a (mon.0) 1171 : cluster [WRN] Health check failed: Degraded data redundancy: 2/289479 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T22:47:08.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:08.751 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:08.814 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:08.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:08.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:08.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:08.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.007 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:09.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:09.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:09.132 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:09.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:09.259 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:09.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:09.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:09.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:09 smithi192 bash[24608]: cluster 2024-09-13T22:47:08.628475+0000 mgr.y (mgr.14613) 353 : cluster [DBG] pgmap v380: 158 pgs: 1 activating+degraded+remapped, 4 remapped+peering, 30 peering, 1 active+remapped+backfill_toofull, 1 active+recovering, 121 active+clean; 5.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 6.3 KiB/s rd, 46 KiB/s wr, 478 op/s; 2/288795 objects degraded (0.001%); 16162/288795 objects misplaced (5.596%); 33 KiB/s, 10 objects/s recovering 2024-09-13T22:47:09.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:09 smithi192 bash[24608]: cluster 2024-09-13T22:47:09.030384+0000 osd.1 (osd.1) 190 : cluster [DBG] 5.c scrub starts 2024-09-13T22:47:09.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:09 smithi192 bash[24608]: cluster 2024-09-13T22:47:09.036137+0000 osd.1 (osd.1) 191 : cluster [DBG] 5.c scrub ok 2024-09-13T22:47:09.538 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:09.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:09.666 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:09.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[29889]: cluster 2024-09-13T22:47:08.628475+0000 mgr.y (mgr.14613) 353 : cluster [DBG] pgmap v380: 158 pgs: 1 activating+degraded+remapped, 4 remapped+peering, 30 peering, 1 active+remapped+backfill_toofull, 1 active+recovering, 121 active+clean; 5.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 6.3 KiB/s rd, 46 KiB/s wr, 478 op/s; 2/288795 objects degraded (0.001%); 16162/288795 objects misplaced (5.596%); 33 KiB/s, 10 objects/s recovering 2024-09-13T22:47:09.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[29889]: cluster 2024-09-13T22:47:09.030384+0000 osd.1 (osd.1) 190 : cluster [DBG] 5.c scrub starts 2024-09-13T22:47:09.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[29889]: cluster 2024-09-13T22:47:09.036137+0000 osd.1 (osd.1) 191 : cluster [DBG] 5.c scrub ok 2024-09-13T22:47:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[22840]: cluster 2024-09-13T22:47:08.628475+0000 mgr.y (mgr.14613) 353 : cluster [DBG] pgmap v380: 158 pgs: 1 activating+degraded+remapped, 4 remapped+peering, 30 peering, 1 active+remapped+backfill_toofull, 1 active+recovering, 121 active+clean; 5.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 6.3 KiB/s rd, 46 KiB/s wr, 478 op/s; 2/288795 objects degraded (0.001%); 16162/288795 objects misplaced (5.596%); 33 KiB/s, 10 objects/s recovering 2024-09-13T22:47:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[22840]: cluster 2024-09-13T22:47:09.030384+0000 osd.1 (osd.1) 190 : cluster [DBG] 5.c scrub starts 2024-09-13T22:47:09.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[22840]: cluster 2024-09-13T22:47:09.036137+0000 osd.1 (osd.1) 191 : cluster [DBG] 5.c scrub ok 2024-09-13T22:47:09.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:09.792 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:09.841 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:09.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:09.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:09.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:10.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:10.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:10.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:10.175 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:10.197 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:47:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:47:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:47:10.280 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:47:10.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T22:47:10.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:10.322 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:10.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:10.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:10.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:10.573 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:10 smithi178 bash[29889]: cluster 2024-09-13T22:47:09.364337+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:47:10.573 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:10 smithi178 bash[22840]: cluster 2024-09-13T22:47:09.364337+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:47:10.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:10.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:10 smithi192 bash[24608]: cluster 2024-09-13T22:47:09.364337+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:47:10.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:10.732 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:10.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:10.861 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:10.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:10.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:11.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:11.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:11.131 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:11.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:11.270 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:11.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:11.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:11.507 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:11 smithi178 bash[29889]: cluster 2024-09-13T22:47:10.629575+0000 mgr.y (mgr.14613) 354 : cluster [DBG] pgmap v381: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 932 op/s; 30499/276444 objects misplaced (11.033%); 27 MiB/s, 453 objects/s recovering 2024-09-13T22:47:11.507 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:11 smithi178 bash[22840]: cluster 2024-09-13T22:47:10.629575+0000 mgr.y (mgr.14613) 354 : cluster [DBG] pgmap v381: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 932 op/s; 30499/276444 objects misplaced (11.033%); 27 MiB/s, 453 objects/s recovering 2024-09-13T22:47:11.526 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:11.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:11 smithi192 bash[24608]: cluster 2024-09-13T22:47:10.629575+0000 mgr.y (mgr.14613) 354 : cluster [DBG] pgmap v381: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 932 op/s; 30499/276444 objects misplaced (11.033%); 27 MiB/s, 453 objects/s recovering 2024-09-13T22:47:11.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:11.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:11.790 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:11.879 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:11.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:11.915 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:11.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:12.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:12.043 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:12.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:12.175 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:12.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:12.306 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:12.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:12.445 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:12.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:12.560 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:12 smithi192 bash[24608]: cluster 2024-09-13T22:47:11.376530+0000 mon.a (mon.0) 1173 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-09-13T22:47:12.560 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:12 smithi192 bash[24608]: cluster 2024-09-13T22:47:11.376574+0000 mon.a (mon.0) 1174 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/288795 objects degraded (0.001%), 1 pg degraded) 2024-09-13T22:47:12.587 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:12.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:12 smithi178 bash[29889]: cluster 2024-09-13T22:47:11.376530+0000 mon.a (mon.0) 1173 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-09-13T22:47:12.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:12 smithi178 bash[29889]: cluster 2024-09-13T22:47:11.376574+0000 mon.a (mon.0) 1174 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/288795 objects degraded (0.001%), 1 pg degraded) 2024-09-13T22:47:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:12 smithi178 bash[22840]: cluster 2024-09-13T22:47:11.376530+0000 mon.a (mon.0) 1173 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-09-13T22:47:12.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:12 smithi178 bash[22840]: cluster 2024-09-13T22:47:11.376574+0000 mon.a (mon.0) 1174 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/288795 objects degraded (0.001%), 1 pg degraded) 2024-09-13T22:47:12.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:12.723 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:12.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:12.862 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:12.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:12.987 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:12.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:13.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:13.109 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:13.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:13.233 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:13.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:13.366 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:13.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:13.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:13.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:13 smithi192 bash[24608]: cluster 2024-09-13T22:47:12.629972+0000 mgr.y (mgr.14613) 355 : cluster [DBG] pgmap v382: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 622 op/s; 30499/276444 objects misplaced (11.033%); 21 MiB/s, 348 objects/s recovering 2024-09-13T22:47:13.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:13.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:13.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:13.753 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:13.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:13.888 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:13.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:13.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:14.017 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:14.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:14.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:13 smithi178 bash[29889]: cluster 2024-09-13T22:47:12.629972+0000 mgr.y (mgr.14613) 355 : cluster [DBG] pgmap v382: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 622 op/s; 30499/276444 objects misplaced (11.033%); 21 MiB/s, 348 objects/s recovering 2024-09-13T22:47:14.133 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:13 smithi178 bash[22840]: cluster 2024-09-13T22:47:12.629972+0000 mgr.y (mgr.14613) 355 : cluster [DBG] pgmap v382: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 622 op/s; 30499/276444 objects misplaced (11.033%); 21 MiB/s, 348 objects/s recovering 2024-09-13T22:47:14.155 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:14.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:14.283 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:14.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:14.402 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:14 smithi178 bash[22840]: cluster 2024-09-13T22:47:13.082069+0000 osd.1 (osd.1) 192 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:47:14.402 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:14 smithi178 bash[22840]: cluster 2024-09-13T22:47:13.085153+0000 osd.1 (osd.1) 193 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:47:14.407 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:14.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:14.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:14.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:14.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:14 smithi192 bash[24608]: cluster 2024-09-13T22:47:13.082069+0000 osd.1 (osd.1) 192 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:47:14.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:14 smithi192 bash[24608]: cluster 2024-09-13T22:47:13.085153+0000 osd.1 (osd.1) 193 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:47:14.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:14.764 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:14 smithi178 bash[29889]: cluster 2024-09-13T22:47:13.082069+0000 osd.1 (osd.1) 192 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:47:14.764 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:14 smithi178 bash[29889]: cluster 2024-09-13T22:47:13.085153+0000 osd.1 (osd.1) 193 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:47:14.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:14.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:14.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:14.933 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:14.937 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:14.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:15.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.071 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:15.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.205 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:15.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:15.345 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:15.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:15.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:15.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.589 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:47:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:47:15.589 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:15 smithi192 bash[24608]: cluster 2024-09-13T22:47:14.137263+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.a scrub starts 2024-09-13T22:47:15.589 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:15 smithi192 bash[24608]: cluster 2024-09-13T22:47:14.139213+0000 osd.1 (osd.1) 195 : cluster [DBG] 2.a scrub ok 2024-09-13T22:47:15.589 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:15 smithi192 bash[24608]: cluster 2024-09-13T22:47:14.630787+0000 mgr.y (mgr.14613) 356 : cluster [DBG] pgmap v383: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 793 op/s; 28410/268647 objects misplaced (10.575%); 29 MiB/s, 471 objects/s recovering 2024-09-13T22:47:15.607 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:15.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:15.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:15.862 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:15.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:15.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:15.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:16.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:16.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:15 smithi178 bash[29889]: cluster 2024-09-13T22:47:14.137263+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.a scrub starts 2024-09-13T22:47:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:15 smithi178 bash[29889]: cluster 2024-09-13T22:47:14.139213+0000 osd.1 (osd.1) 195 : cluster [DBG] 2.a scrub ok 2024-09-13T22:47:16.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:15 smithi178 bash[29889]: cluster 2024-09-13T22:47:14.630787+0000 mgr.y (mgr.14613) 356 : cluster [DBG] pgmap v383: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 793 op/s; 28410/268647 objects misplaced (10.575%); 29 MiB/s, 471 objects/s recovering 2024-09-13T22:47:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:15 smithi178 bash[22840]: cluster 2024-09-13T22:47:14.137263+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.a scrub starts 2024-09-13T22:47:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:15 smithi178 bash[22840]: cluster 2024-09-13T22:47:14.139213+0000 osd.1 (osd.1) 195 : cluster [DBG] 2.a scrub ok 2024-09-13T22:47:16.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:15 smithi178 bash[22840]: cluster 2024-09-13T22:47:14.630787+0000 mgr.y (mgr.14613) 356 : cluster [DBG] pgmap v383: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 793 op/s; 28410/268647 objects misplaced (10.575%); 29 MiB/s, 471 objects/s recovering 2024-09-13T22:47:16.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:16.267 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:16.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:16.394 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:16.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:16.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:16.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:16.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:16 smithi192 bash[24608]: audit 2024-09-13T22:47:15.272784+0000 mgr.y (mgr.14613) 357 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:16.670 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:16.673 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:16 smithi178 bash[22840]: audit 2024-09-13T22:47:15.272784+0000 mgr.y (mgr.14613) 357 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:16.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:16 smithi178 bash[29889]: audit 2024-09-13T22:47:15.272784+0000 mgr.y (mgr.14613) 357 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:16.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:16.801 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:16.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:16.932 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:16.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:17.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:17.058 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:17.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.196 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:17.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:17.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:17.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:17.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:17.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:17 smithi192 bash[24608]: cluster 2024-09-13T22:47:16.631802+0000 mgr.y (mgr.14613) 358 : cluster [DBG] pgmap v384: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 900 op/s; 26121/262494 objects misplaced (9.951%); 35 MiB/s, 567 objects/s recovering 2024-09-13T22:47:17.447 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:17.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:17.569 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:17.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:17.689 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:17.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:17.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:17 smithi178 bash[29889]: cluster 2024-09-13T22:47:16.631802+0000 mgr.y (mgr.14613) 358 : cluster [DBG] pgmap v384: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 900 op/s; 26121/262494 objects misplaced (9.951%); 35 MiB/s, 567 objects/s recovering 2024-09-13T22:47:17.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:17 smithi178 bash[22840]: cluster 2024-09-13T22:47:16.631802+0000 mgr.y (mgr.14613) 358 : cluster [DBG] pgmap v384: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 900 op/s; 26121/262494 objects misplaced (9.951%); 35 MiB/s, 567 objects/s recovering 2024-09-13T22:47:17.823 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:17.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:17.960 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:17.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:18.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:18.095 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:18.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:18.225 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:18.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:18.349 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:18.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:18.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:18.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:18.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:18.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:18.727 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:18.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:18.852 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:18.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:18.979 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:18.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:19.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:19.108 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:19.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:19.232 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:19.336 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:19.373 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:19.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:19.485 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:19 smithi192 bash[24608]: audit 2024-09-13T22:47:18.389476+0000 mon.a (mon.0) 1175 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:19.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:19 smithi192 bash[24608]: audit 2024-09-13T22:47:18.393059+0000 mon.b (mon.1) 157 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:19.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:19 smithi192 bash[24608]: cluster 2024-09-13T22:47:18.632258+0000 mgr.y (mgr.14613) 359 : cluster [DBG] pgmap v385: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 871 B/s rd, 0 B/s wr, 790 op/s; 25619/261645 objects misplaced (9.792%); 30 MiB/s, 482 objects/s recovering 2024-09-13T22:47:19.500 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:19.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:19.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[29889]: audit 2024-09-13T22:47:18.389476+0000 mon.a (mon.0) 1175 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:19.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[29889]: audit 2024-09-13T22:47:18.393059+0000 mon.b (mon.1) 157 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:19.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[29889]: cluster 2024-09-13T22:47:18.632258+0000 mgr.y (mgr.14613) 359 : cluster [DBG] pgmap v385: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 871 B/s rd, 0 B/s wr, 790 op/s; 25619/261645 objects misplaced (9.792%); 30 MiB/s, 482 objects/s recovering 2024-09-13T22:47:19.613 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[22840]: audit 2024-09-13T22:47:18.389476+0000 mon.a (mon.0) 1175 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:19.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[22840]: audit 2024-09-13T22:47:18.393059+0000 mon.b (mon.1) 157 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:19.614 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[22840]: cluster 2024-09-13T22:47:18.632258+0000 mgr.y (mgr.14613) 359 : cluster [DBG] pgmap v385: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 871 B/s rd, 0 B/s wr, 790 op/s; 25619/261645 objects misplaced (9.792%); 30 MiB/s, 482 objects/s recovering 2024-09-13T22:47:19.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:19.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:19.762 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:19.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:19.892 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:19.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:19.928 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:47:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:47:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:47:19.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:20.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:20.176 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:20.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:20.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:20.333 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:47:20.333 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:47:20.384 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:47:20.384 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:47:20.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:20.431 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:47:20.431 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:47:20.441 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:20.484 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:47:20.484 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:47:20.519 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:47:20.519 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:47:20.547 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:20.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:20 smithi178 bash[29889]: audit 2024-09-13T22:47:20.278789+0000 mon.a (mon.0) 1176 : audit [DBG] from='client.? v1:172.21.15.178:0/1085174223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:20.556 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:20 smithi178 bash[22840]: audit 2024-09-13T22:47:20.278789+0000 mon.a (mon.0) 1176 : audit [DBG] from='client.? v1:172.21.15.178:0/1085174223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:20.557 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:47:20.558 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:47:20.584 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:20.614 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:47:20.614 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:47:20.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:20 smithi192 bash[24608]: audit 2024-09-13T22:47:20.278789+0000 mon.a (mon.0) 1176 : audit [DBG] from='client.? v1:172.21.15.178:0/1085174223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:20.659 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:47:20.660 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:47:20.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:20.719 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:47:20.719 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:47:20.750 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:20.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:20.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:20.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:21.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:21.046 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:21.149 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:21.185 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:21.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:21.341 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:21.398 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:47:21.404 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T22:47:21.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:21.467 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:21.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:21.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[22840]: cluster 2024-09-13T22:47:20.633261+0000 mgr.y (mgr.14613) 360 : cluster [DBG] pgmap v386: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 21448/249045 objects misplaced (8.612%); 47 MiB/s, 755 objects/s recovering 2024-09-13T22:47:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[22840]: audit 2024-09-13T22:47:20.968213+0000 mon.b (mon.1) 158 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:47:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[22840]: audit 2024-09-13T22:47:20.997707+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.178:0/3488454802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:47:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[22840]: audit 2024-09-13T22:47:20.998791+0000 mon.a (mon.0) 1177 : audit [INF] from='client.25148 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:47:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[22840]: cluster 2024-09-13T22:47:21.069246+0000 osd.1 (osd.1) 196 : cluster [DBG] 8.3 scrub starts 2024-09-13T22:47:21.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[22840]: audit 2024-09-13T22:47:21.091199+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? v1:172.21.15.178:0/1369462250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:47:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[29889]: cluster 2024-09-13T22:47:20.633261+0000 mgr.y (mgr.14613) 360 : cluster [DBG] pgmap v386: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 21448/249045 objects misplaced (8.612%); 47 MiB/s, 755 objects/s recovering 2024-09-13T22:47:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[29889]: audit 2024-09-13T22:47:20.968213+0000 mon.b (mon.1) 158 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:47:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[29889]: audit 2024-09-13T22:47:20.997707+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.178:0/3488454802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:47:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[29889]: audit 2024-09-13T22:47:20.998791+0000 mon.a (mon.0) 1177 : audit [INF] from='client.25148 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:47:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[29889]: cluster 2024-09-13T22:47:21.069246+0000 osd.1 (osd.1) 196 : cluster [DBG] 8.3 scrub starts 2024-09-13T22:47:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:21 smithi178 bash[29889]: audit 2024-09-13T22:47:21.091199+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? v1:172.21.15.178:0/1369462250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:47:21.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:21.723 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:21 smithi192 bash[24608]: cluster 2024-09-13T22:47:20.633261+0000 mgr.y (mgr.14613) 360 : cluster [DBG] pgmap v386: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 21448/249045 objects misplaced (8.612%); 47 MiB/s, 755 objects/s recovering 2024-09-13T22:47:21.723 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:21 smithi192 bash[24608]: audit 2024-09-13T22:47:20.968213+0000 mon.b (mon.1) 158 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:47:21.723 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:21 smithi192 bash[24608]: audit 2024-09-13T22:47:20.997707+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.178:0/3488454802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:47:21.723 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:21 smithi192 bash[24608]: audit 2024-09-13T22:47:20.998791+0000 mon.a (mon.0) 1177 : audit [INF] from='client.25148 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:47:21.723 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:21 smithi192 bash[24608]: cluster 2024-09-13T22:47:21.069246+0000 osd.1 (osd.1) 196 : cluster [DBG] 8.3 scrub starts 2024-09-13T22:47:21.723 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:21 smithi192 bash[24608]: audit 2024-09-13T22:47:21.091199+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? v1:172.21.15.178:0/1369462250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:47:21.740 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:21.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:21.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:21.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:21.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:22.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:22.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:22.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:22.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:22.259 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:22.362 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:22.391 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:22.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:22.519 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:22.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:22.659 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: cluster 2024-09-13T22:47:21.398994+0000 mon.a (mon.0) 1179 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: audit 2024-09-13T22:47:21.400346+0000 mon.a (mon.0) 1180 : audit [INF] from='client.25148 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: audit 2024-09-13T22:47:21.400424+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? v1:172.21.15.178:0/1369462250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: cluster 2024-09-13T22:47:21.406123+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: audit 2024-09-13T22:47:21.961899+0000 mon.b (mon.1) 159 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: audit 2024-09-13T22:47:21.964994+0000 mon.b (mon.1) 160 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[29889]: audit 2024-09-13T22:47:21.971761+0000 mon.a (mon.0) 1183 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: cluster 2024-09-13T22:47:21.398994+0000 mon.a (mon.0) 1179 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:47:22.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: audit 2024-09-13T22:47:21.400346+0000 mon.a (mon.0) 1180 : audit [INF] from='client.25148 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:47:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: audit 2024-09-13T22:47:21.400424+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? v1:172.21.15.178:0/1369462250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:47:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: cluster 2024-09-13T22:47:21.406123+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-09-13T22:47:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: audit 2024-09-13T22:47:21.961899+0000 mon.b (mon.1) 159 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:47:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: audit 2024-09-13T22:47:21.964994+0000 mon.b (mon.1) 160 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:47:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:22 smithi178 bash[22840]: audit 2024-09-13T22:47:21.971761+0000 mon.a (mon.0) 1183 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:22.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:22.790 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:22.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: cluster 2024-09-13T22:47:21.398994+0000 mon.a (mon.0) 1179 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: audit 2024-09-13T22:47:21.400346+0000 mon.a (mon.0) 1180 : audit [INF] from='client.25148 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: audit 2024-09-13T22:47:21.400424+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? v1:172.21.15.178:0/1369462250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: cluster 2024-09-13T22:47:21.406123+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: audit 2024-09-13T22:47:21.961899+0000 mon.b (mon.1) 159 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: audit 2024-09-13T22:47:21.964994+0000 mon.b (mon.1) 160 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:47:22.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:22 smithi192 bash[24608]: audit 2024-09-13T22:47:21.971761+0000 mon.a (mon.0) 1183 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:22.921 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:22.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:23.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:23.053 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:23.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:23.182 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:23.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:23.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:23.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:23.445 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:23.548 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:23.582 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:23.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:23 smithi178 bash[29889]: cluster 2024-09-13T22:47:22.633723+0000 mgr.y (mgr.14613) 361 : cluster [DBG] pgmap v388: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 913 op/s; 21448/249045 objects misplaced (8.612%); 40 MiB/s, 636 objects/s recovering 2024-09-13T22:47:23.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:23 smithi178 bash[29889]: cluster 2024-09-13T22:47:22.876679+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-09-13T22:47:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:23 smithi178 bash[22840]: cluster 2024-09-13T22:47:22.633723+0000 mgr.y (mgr.14613) 361 : cluster [DBG] pgmap v388: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 913 op/s; 21448/249045 objects misplaced (8.612%); 40 MiB/s, 636 objects/s recovering 2024-09-13T22:47:23.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:23 smithi178 bash[22840]: cluster 2024-09-13T22:47:22.876679+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-09-13T22:47:23.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:23.720 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:23.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:23.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:23.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:23 smithi192 bash[24608]: cluster 2024-09-13T22:47:22.633723+0000 mgr.y (mgr.14613) 361 : cluster [DBG] pgmap v388: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 913 op/s; 21448/249045 objects misplaced (8.612%); 40 MiB/s, 636 objects/s recovering 2024-09-13T22:47:23.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:23 smithi192 bash[24608]: cluster 2024-09-13T22:47:22.876679+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-09-13T22:47:23.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:23.993 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:23.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:24.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:24.117 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:24.222 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:24.270 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:24.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:24.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:24.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:24.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:24.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:24.662 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:24.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:24.786 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:24.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:24.913 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:24.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:25.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:25.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:25.137 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:25.162 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:25.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:25.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:25.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:25.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:25.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:25.540 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:25.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:25 smithi178 bash[22840]: cluster 2024-09-13T22:47:24.497031+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-09-13T22:47:25.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:25 smithi178 bash[22840]: cluster 2024-09-13T22:47:24.634277+0000 mgr.y (mgr.14613) 362 : cluster [DBG] pgmap v391: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 19557/241614 objects misplaced (8.094%); 50 MiB/s, 795 objects/s recovering 2024-09-13T22:47:25.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:25 smithi178 bash[29889]: cluster 2024-09-13T22:47:24.497031+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-09-13T22:47:25.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:25 smithi178 bash[29889]: cluster 2024-09-13T22:47:24.634277+0000 mgr.y (mgr.14613) 362 : cluster [DBG] pgmap v391: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 19557/241614 objects misplaced (8.094%); 50 MiB/s, 795 objects/s recovering 2024-09-13T22:47:25.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:25.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:47:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:47:25.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:25 smithi192 bash[24608]: cluster 2024-09-13T22:47:24.497031+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-09-13T22:47:25.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:25 smithi192 bash[24608]: cluster 2024-09-13T22:47:24.634277+0000 mgr.y (mgr.14613) 362 : cluster [DBG] pgmap v391: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 19557/241614 objects misplaced (8.094%); 50 MiB/s, 795 objects/s recovering 2024-09-13T22:47:25.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:25.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:25.808 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:25.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:25.942 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:25.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:26.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:26.077 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:26.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:26.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:26.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:26.329 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:26.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:26.449 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:26.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:26.575 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:26.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:26.699 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:26.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:26.810 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:26 smithi192 bash[24608]: audit 2024-09-13T22:47:25.280780+0000 mgr.y (mgr.14613) 363 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:26.810 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:26 smithi192 bash[24608]: cluster 2024-09-13T22:47:25.505164+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-09-13T22:47:26.825 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:26.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:26.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:26 smithi178 bash[29889]: audit 2024-09-13T22:47:25.280780+0000 mgr.y (mgr.14613) 363 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:26.942 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:26 smithi178 bash[29889]: cluster 2024-09-13T22:47:25.505164+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-09-13T22:47:26.942 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:26 smithi178 bash[22840]: audit 2024-09-13T22:47:25.280780+0000 mgr.y (mgr.14613) 363 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:26.942 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:26 smithi178 bash[22840]: cluster 2024-09-13T22:47:25.505164+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-09-13T22:47:26.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:26.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:27.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:27.082 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:27.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:27.209 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:27.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:27.335 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:27.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:27.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:27.596 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:27.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:27.720 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:27.813 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[22840]: cluster 2024-09-13T22:47:26.407914+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:27.813 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[22840]: cluster 2024-09-13T22:47:26.407978+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:27.813 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[22840]: cluster 2024-09-13T22:47:26.510194+0000 mon.a (mon.0) 1187 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-09-13T22:47:27.813 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[22840]: cluster 2024-09-13T22:47:26.634761+0000 mgr.y (mgr.14613) 364 : cluster [DBG] pgmap v394: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 17257/236112 objects misplaced (7.309%); 50 MiB/s, 803 objects/s recovering 2024-09-13T22:47:27.814 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[29889]: cluster 2024-09-13T22:47:26.407914+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:27.814 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[29889]: cluster 2024-09-13T22:47:26.407978+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:27.814 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[29889]: cluster 2024-09-13T22:47:26.510194+0000 mon.a (mon.0) 1187 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-09-13T22:47:27.814 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:27 smithi178 bash[29889]: cluster 2024-09-13T22:47:26.634761+0000 mgr.y (mgr.14613) 364 : cluster [DBG] pgmap v394: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 17257/236112 objects misplaced (7.309%); 50 MiB/s, 803 objects/s recovering 2024-09-13T22:47:27.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:27.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:27 smithi192 bash[24608]: cluster 2024-09-13T22:47:26.407914+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:27.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:27 smithi192 bash[24608]: cluster 2024-09-13T22:47:26.407978+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:27.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:27 smithi192 bash[24608]: cluster 2024-09-13T22:47:26.510194+0000 mon.a (mon.0) 1187 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-09-13T22:47:27.837 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:27 smithi192 bash[24608]: cluster 2024-09-13T22:47:26.634761+0000 mgr.y (mgr.14613) 364 : cluster [DBG] pgmap v394: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 17257/236112 objects misplaced (7.309%); 50 MiB/s, 803 objects/s recovering 2024-09-13T22:47:27.852 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:27.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:27.981 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:27.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:28.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:28.110 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:28.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T22:47:28.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:28.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T22:47:28.252 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:28.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.406 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:28.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:28.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:28.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:28.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:28.663 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:28.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:28.811 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:28.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:28.926 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:28 smithi178 bash[29889]: cluster 2024-09-13T22:47:27.879179+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-09-13T22:47:28.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:28 smithi178 bash[29889]: cluster 2024-09-13T22:47:28.635272+0000 mgr.y (mgr.14613) 365 : cluster [DBG] pgmap v396: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 496 op/s; 16883/235470 objects misplaced (7.170%); 28 MiB/s, 443 objects/s recovering 2024-09-13T22:47:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:28 smithi178 bash[22840]: cluster 2024-09-13T22:47:27.879179+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-09-13T22:47:28.927 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:28 smithi178 bash[22840]: cluster 2024-09-13T22:47:28.635272+0000 mgr.y (mgr.14613) 365 : cluster [DBG] pgmap v396: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 496 op/s; 16883/235470 objects misplaced (7.170%); 28 MiB/s, 443 objects/s recovering 2024-09-13T22:47:28.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:28.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:29.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:29.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:28 smithi192 bash[24608]: cluster 2024-09-13T22:47:27.879179+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-09-13T22:47:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:28 smithi192 bash[24608]: cluster 2024-09-13T22:47:28.635272+0000 mgr.y (mgr.14613) 365 : cluster [DBG] pgmap v396: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 496 op/s; 16883/235470 objects misplaced (7.170%); 28 MiB/s, 443 objects/s recovering 2024-09-13T22:47:29.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:29.226 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:29.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:29.352 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:29.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:29.480 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:29.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:29.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:29.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:29.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:29.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:29.857 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:47:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:47:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:47:29.875 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:29.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:29.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:30.015 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:30.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:30.125 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:47:30.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:30.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:47:30.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:30.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:30.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.400 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:30.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:30.501 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:30.544 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:30.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:30.670 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:30.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:30.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:30.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:30.910 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:30 smithi178 bash[29889]: cluster 2024-09-13T22:47:30.636249+0000 mgr.y (mgr.14613) 366 : cluster [DBG] pgmap v397: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 925 op/s; 13296/224979 objects misplaced (5.910%); 53 MiB/s, 854 objects/s recovering 2024-09-13T22:47:30.910 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:30 smithi178 bash[22840]: cluster 2024-09-13T22:47:30.636249+0000 mgr.y (mgr.14613) 366 : cluster [DBG] pgmap v397: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 925 op/s; 13296/224979 objects misplaced (5.910%); 53 MiB/s, 854 objects/s recovering 2024-09-13T22:47:30.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:30.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:31.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:31.048 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:30 smithi192 bash[24608]: cluster 2024-09-13T22:47:30.636249+0000 mgr.y (mgr.14613) 366 : cluster [DBG] pgmap v397: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 925 op/s; 13296/224979 objects misplaced (5.910%); 53 MiB/s, 854 objects/s recovering 2024-09-13T22:47:31.066 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:31.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:31.190 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:31.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:31.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:31.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.460 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:31.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:31.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:31.588 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:31.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:31.712 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:31.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:31.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:31.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:31.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:31.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:32.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:32.104 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:32.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:32.240 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:32.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:32.381 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:32.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:32.511 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:32.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:32.657 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:32.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:32.770 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:32 smithi192 bash[24608]: cluster 2024-09-13T22:47:32.636891+0000 mgr.y (mgr.14613) 367 : cluster [DBG] pgmap v398: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 717 B/s rd, 0 B/s wr, 520 op/s; 13296/224979 objects misplaced (5.910%); 29 MiB/s, 461 objects/s recovering 2024-09-13T22:47:32.785 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:32.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:32.910 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:32.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:32.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:32 smithi178 bash[29889]: cluster 2024-09-13T22:47:32.636891+0000 mgr.y (mgr.14613) 367 : cluster [DBG] pgmap v398: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 717 B/s rd, 0 B/s wr, 520 op/s; 13296/224979 objects misplaced (5.910%); 29 MiB/s, 461 objects/s recovering 2024-09-13T22:47:32.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:32 smithi178 bash[22840]: cluster 2024-09-13T22:47:32.636891+0000 mgr.y (mgr.14613) 367 : cluster [DBG] pgmap v398: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 717 B/s rd, 0 B/s wr, 520 op/s; 13296/224979 objects misplaced (5.910%); 29 MiB/s, 461 objects/s recovering 2024-09-13T22:47:33.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:33.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:33.105 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:33.105 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:33.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.166 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:33.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.289 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:33.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:33.419 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:33.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:33.547 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:33.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:33.679 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:33.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:33.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.939 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:33.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:34.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:34.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:34.128 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:34.128 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:34.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:34.213 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:34.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:34.340 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:34.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:34.470 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:34.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:34.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:34.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:34 smithi192 bash[24608]: audit 2024-09-13T22:47:33.407702+0000 mon.a (mon.0) 1189 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:34.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:34 smithi192 bash[24608]: audit 2024-09-13T22:47:33.413087+0000 mon.b (mon.1) 161 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:34.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:34 smithi192 bash[24608]: cluster 2024-09-13T22:47:33.995163+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:34.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:34 smithi192 bash[24608]: cluster 2024-09-13T22:47:33.995226+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[22840]: audit 2024-09-13T22:47:33.407702+0000 mon.a (mon.0) 1189 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[22840]: audit 2024-09-13T22:47:33.413087+0000 mon.b (mon.1) 161 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[22840]: cluster 2024-09-13T22:47:33.995163+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:34.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[22840]: cluster 2024-09-13T22:47:33.995226+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:34.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[29889]: audit 2024-09-13T22:47:33.407702+0000 mon.a (mon.0) 1189 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:47:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[29889]: audit 2024-09-13T22:47:33.413087+0000 mon.b (mon.1) 161 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[29889]: cluster 2024-09-13T22:47:33.995163+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:34 smithi178 bash[29889]: cluster 2024-09-13T22:47:33.995226+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:34.704 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:34.740 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:34.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:34.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:34.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:34.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:35.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:35.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:35.124 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:35.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:35.248 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:35.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:35.390 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:35.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:35.501 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:47:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:47:35.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:35 smithi192 bash[24608]: cluster 2024-09-13T22:47:34.419676+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-09-13T22:47:35.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:35 smithi192 bash[24608]: cluster 2024-09-13T22:47:34.637404+0000 mgr.y (mgr.14613) 368 : cluster [DBG] pgmap v400: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 825 op/s; 12710/216315 objects misplaced (5.876%); 26 MiB/s, 411 objects/s recovering 2024-09-13T22:47:35.511 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:35.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:35.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:35.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:35 smithi178 bash[29889]: cluster 2024-09-13T22:47:34.419676+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-09-13T22:47:35.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:35 smithi178 bash[29889]: cluster 2024-09-13T22:47:34.637404+0000 mgr.y (mgr.14613) 368 : cluster [DBG] pgmap v400: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 825 op/s; 12710/216315 objects misplaced (5.876%); 26 MiB/s, 411 objects/s recovering 2024-09-13T22:47:35.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:35 smithi178 bash[22840]: cluster 2024-09-13T22:47:34.419676+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-09-13T22:47:35.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:35 smithi178 bash[22840]: cluster 2024-09-13T22:47:34.637404+0000 mgr.y (mgr.14613) 368 : cluster [DBG] pgmap v400: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 825 op/s; 12710/216315 objects misplaced (5.876%); 26 MiB/s, 411 objects/s recovering 2024-09-13T22:47:35.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:35.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:35.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:35.910 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:35.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:36.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:36.055 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:36.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:36.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:36.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:36.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:36.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:36.463 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:36.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:36.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:36.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:36 smithi178 bash[29889]: audit 2024-09-13T22:47:35.290413+0000 mgr.y (mgr.14613) 369 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:36.679 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:36 smithi178 bash[22840]: audit 2024-09-13T22:47:35.290413+0000 mgr.y (mgr.14613) 369 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:36.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:36.707 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:36 smithi192 bash[24608]: audit 2024-09-13T22:47:35.290413+0000 mgr.y (mgr.14613) 369 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:36.718 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:36.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:36.844 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:36.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:36.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:36.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:37.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:37.099 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:37.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:37.234 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:37.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:37.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:37.385 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:37.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:37.497 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:37 smithi178 bash[22840]: cluster 2024-09-13T22:47:36.638233+0000 mgr.y (mgr.14613) 370 : cluster [DBG] pgmap v401: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 993 op/s; 10732/210042 objects misplaced (5.109%); 36 MiB/s, 574 objects/s recovering 2024-09-13T22:47:37.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:37 smithi178 bash[29889]: cluster 2024-09-13T22:47:36.638233+0000 mgr.y (mgr.14613) 370 : cluster [DBG] pgmap v401: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 993 op/s; 10732/210042 objects misplaced (5.109%); 36 MiB/s, 574 objects/s recovering 2024-09-13T22:47:37.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:37.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:37.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:37 smithi192 bash[24608]: cluster 2024-09-13T22:47:36.638233+0000 mgr.y (mgr.14613) 370 : cluster [DBG] pgmap v401: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 993 op/s; 10732/210042 objects misplaced (5.109%); 36 MiB/s, 574 objects/s recovering 2024-09-13T22:47:37.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:37.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:37.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:37.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:37.908 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:37.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:38.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:38.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:38.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:38.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:38.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:38.319 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:38.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:38.452 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:38.555 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:38.581 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:38.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:38.714 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:38.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:38.834 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:38.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:38.962 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:38.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:39.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:39.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[22840]: cluster 2024-09-13T22:47:37.597862+0000 osd.3 (osd.3) 211 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:39.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[22840]: cluster 2024-09-13T22:47:37.597919+0000 osd.3 (osd.3) 212 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:39.077 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[22840]: cluster 2024-09-13T22:47:37.613073+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[22840]: cluster 2024-09-13T22:47:37.613134+0000 osd.6 (osd.6) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[22840]: cluster 2024-09-13T22:47:37.882961+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[22840]: cluster 2024-09-13T22:47:38.638713+0000 mgr.y (mgr.14613) 371 : cluster [DBG] pgmap v403: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 654 op/s; 10284/209280 objects misplaced (4.914%); 14 MiB/s, 216 objects/s recovering 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[29889]: cluster 2024-09-13T22:47:37.597862+0000 osd.3 (osd.3) 211 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[29889]: cluster 2024-09-13T22:47:37.597919+0000 osd.3 (osd.3) 212 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[29889]: cluster 2024-09-13T22:47:37.613073+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[29889]: cluster 2024-09-13T22:47:37.613134+0000 osd.6 (osd.6) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[29889]: cluster 2024-09-13T22:47:37.882961+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-09-13T22:47:39.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:38 smithi178 bash[29889]: cluster 2024-09-13T22:47:38.638713+0000 mgr.y (mgr.14613) 371 : cluster [DBG] pgmap v403: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 654 op/s; 10284/209280 objects misplaced (4.914%); 14 MiB/s, 216 objects/s recovering 2024-09-13T22:47:39.091 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:38 smithi192 bash[24608]: cluster 2024-09-13T22:47:37.597862+0000 osd.3 (osd.3) 211 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:38 smithi192 bash[24608]: cluster 2024-09-13T22:47:37.597919+0000 osd.3 (osd.3) 212 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:38 smithi192 bash[24608]: cluster 2024-09-13T22:47:37.613073+0000 osd.6 (osd.6) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:38 smithi192 bash[24608]: cluster 2024-09-13T22:47:37.613134+0000 osd.6 (osd.6) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:38 smithi192 bash[24608]: cluster 2024-09-13T22:47:37.882961+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-09-13T22:47:39.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:38 smithi192 bash[24608]: cluster 2024-09-13T22:47:38.638713+0000 mgr.y (mgr.14613) 371 : cluster [DBG] pgmap v403: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 654 op/s; 10284/209280 objects misplaced (4.914%); 14 MiB/s, 216 objects/s recovering 2024-09-13T22:47:39.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:39.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:39.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.356 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:39.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:39.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:39.488 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:39.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:39.623 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:39.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:39.756 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:39.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:39.888 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:39.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:39.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:40.011 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:40.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:40.145 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:47:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:47:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:47:40.166 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:40.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:40.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:40.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:40.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:40.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:40.537 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:40.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:40.667 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:40.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:40.795 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:40.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:40.921 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:40.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:41.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:41.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:41.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:41 smithi178 bash[29889]: cluster 2024-09-13T22:47:40.091389+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-09-13T22:47:41.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:41 smithi178 bash[29889]: cluster 2024-09-13T22:47:40.639502+0000 mgr.y (mgr.14613) 372 : cluster [DBG] pgmap v405: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.07k op/s; 8419/196338 objects misplaced (4.288%); 35 MiB/s, 556 objects/s recovering 2024-09-13T22:47:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:41 smithi178 bash[22840]: cluster 2024-09-13T22:47:40.091389+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-09-13T22:47:41.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:41 smithi178 bash[22840]: cluster 2024-09-13T22:47:40.639502+0000 mgr.y (mgr.14613) 372 : cluster [DBG] pgmap v405: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.07k op/s; 8419/196338 objects misplaced (4.288%); 35 MiB/s, 556 objects/s recovering 2024-09-13T22:47:41.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:41.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:41.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:41.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:41 smithi192 bash[24608]: cluster 2024-09-13T22:47:40.091389+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-09-13T22:47:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:41 smithi192 bash[24608]: cluster 2024-09-13T22:47:40.639502+0000 mgr.y (mgr.14613) 372 : cluster [DBG] pgmap v405: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.07k op/s; 8419/196338 objects misplaced (4.288%); 35 MiB/s, 556 objects/s recovering 2024-09-13T22:47:41.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:41.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:41.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:41.552 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:41.655 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:41.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:41.778 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:41.804 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:41.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:41.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:41.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:42.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:42.063 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:42.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:42 smithi178 bash[22840]: cluster 2024-09-13T22:47:41.096937+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-09-13T22:47:42.166 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:42.195 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:42.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:42.324 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:42.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:42 smithi192 bash[24608]: cluster 2024-09-13T22:47:41.096937+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-09-13T22:47:42.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:42 smithi178 bash[29889]: cluster 2024-09-13T22:47:41.096937+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-09-13T22:47:42.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:42.450 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:42.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:42.577 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:42.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:42.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:42.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:42.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:42.971 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:42.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:43.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:43.116 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:43.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:43.246 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:43.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:43.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:43 smithi178 bash[29889]: cluster 2024-09-13T22:47:42.102298+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-09-13T22:47:43.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:43 smithi178 bash[29889]: cluster 2024-09-13T22:47:42.640066+0000 mgr.y (mgr.14613) 373 : cluster [DBG] pgmap v408: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 960 op/s; 8419/196338 objects misplaced (4.288%); 23 MiB/s, 363 objects/s recovering 2024-09-13T22:47:43.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:43 smithi178 bash[29889]: cluster 2024-09-13T22:47:42.888557+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-09-13T22:47:43.364 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:43 smithi178 bash[22840]: cluster 2024-09-13T22:47:42.102298+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-09-13T22:47:43.364 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:43 smithi178 bash[22840]: cluster 2024-09-13T22:47:42.640066+0000 mgr.y (mgr.14613) 373 : cluster [DBG] pgmap v408: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 960 op/s; 8419/196338 objects misplaced (4.288%); 23 MiB/s, 363 objects/s recovering 2024-09-13T22:47:43.364 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:43 smithi178 bash[22840]: cluster 2024-09-13T22:47:42.888557+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-09-13T22:47:43.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:43.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:43 smithi192 bash[24608]: cluster 2024-09-13T22:47:42.102298+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-09-13T22:47:43.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:43 smithi192 bash[24608]: cluster 2024-09-13T22:47:42.640066+0000 mgr.y (mgr.14613) 373 : cluster [DBG] pgmap v408: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 960 op/s; 8419/196338 objects misplaced (4.288%); 23 MiB/s, 363 objects/s recovering 2024-09-13T22:47:43.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:43 smithi192 bash[24608]: cluster 2024-09-13T22:47:42.888557+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-09-13T22:47:43.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:43.516 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:43.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:43.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:43.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:43.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:43.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:43.908 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:43.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:44.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:44.041 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:44.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:44.168 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:44.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:44.305 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:44.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:44.429 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:44.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:44.550 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:44.576 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:47:44.576 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:47:44.626 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:47:44.626 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:47:44.651 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:47:44.651 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:47:44.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:44.673 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:44.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:44 smithi192 bash[24608]: audit 2024-09-13T22:47:44.535694+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.178:0/3263319513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:44.679 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:47:44.679 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:47:44.725 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:47:44.726 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:47:44.771 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:47:44.771 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:47:44.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:44.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:44.814 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:47:44.814 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:47:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:44.842 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:47:44.843 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:47:44.883 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:47:44.884 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:47:44.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:44 smithi178 bash[29889]: audit 2024-09-13T22:47:44.535694+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.178:0/3263319513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:44.902 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:44 smithi178 bash[22840]: audit 2024-09-13T22:47:44.535694+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? v1:172.21.15.178:0/3263319513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:47:44.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:44.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:44.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:45.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:45.095 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:45.198 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:45.220 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:45.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:45.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:47:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:47:45.351 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:45.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:45.479 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:45.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:45.603 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:47:45.606 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T22:47:45.626 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[29889]: cluster 2024-09-13T22:47:44.640756+0000 mgr.y (mgr.14613) 374 : cluster [DBG] pgmap v410: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 534 op/s; 8101/191676 objects misplaced (4.226%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T22:47:45.626 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[29889]: audit 2024-09-13T22:47:44.862857+0000 mon.a (mon.0) 1197 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:45.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[29889]: audit 2024-09-13T22:47:44.864004+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.178:0/628272564' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:45.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[29889]: audit 2024-09-13T22:47:45.015100+0000 mon.a (mon.0) 1198 : audit [INF] from='client.25111 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:47:45.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[29889]: audit 2024-09-13T22:47:45.015816+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.178:0/1529476533' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:47:45.630 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[22840]: cluster 2024-09-13T22:47:44.640756+0000 mgr.y (mgr.14613) 374 : cluster [DBG] pgmap v410: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 534 op/s; 8101/191676 objects misplaced (4.226%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T22:47:45.630 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[22840]: audit 2024-09-13T22:47:44.862857+0000 mon.a (mon.0) 1197 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:45.630 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[22840]: audit 2024-09-13T22:47:44.864004+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.178:0/628272564' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:45.630 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[22840]: audit 2024-09-13T22:47:45.015100+0000 mon.a (mon.0) 1198 : audit [INF] from='client.25111 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:47:45.630 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:45 smithi178 bash[22840]: audit 2024-09-13T22:47:45.015816+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.178:0/1529476533' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:47:45.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:45.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:45.771 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:45.875 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:45 smithi192 bash[24608]: cluster 2024-09-13T22:47:44.640756+0000 mgr.y (mgr.14613) 374 : cluster [DBG] pgmap v410: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 534 op/s; 8101/191676 objects misplaced (4.226%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T22:47:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:45 smithi192 bash[24608]: audit 2024-09-13T22:47:44.862857+0000 mon.a (mon.0) 1197 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:45 smithi192 bash[24608]: audit 2024-09-13T22:47:44.864004+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.178:0/628272564' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:47:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:45 smithi192 bash[24608]: audit 2024-09-13T22:47:45.015100+0000 mon.a (mon.0) 1198 : audit [INF] from='client.25111 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:47:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:45 smithi192 bash[24608]: audit 2024-09-13T22:47:45.015816+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.178:0/1529476533' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:47:45.910 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:45.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:46.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:46.045 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:46.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:46.184 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:46.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:46.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:46.418 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.453 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:46.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:46.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:46.583 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[29889]: audit 2024-09-13T22:47:45.300500+0000 mgr.y (mgr.14613) 375 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[29889]: cluster 2024-09-13T22:47:45.587988+0000 mon.a (mon.0) 1199 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:47:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[29889]: audit 2024-09-13T22:47:45.598874+0000 mon.a (mon.0) 1200 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:47:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[29889]: audit 2024-09-13T22:47:45.599028+0000 mon.a (mon.0) 1201 : audit [INF] from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:47:46.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[29889]: cluster 2024-09-13T22:47:45.620040+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-09-13T22:47:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[22840]: audit 2024-09-13T22:47:45.300500+0000 mgr.y (mgr.14613) 375 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[22840]: cluster 2024-09-13T22:47:45.587988+0000 mon.a (mon.0) 1199 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:47:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[22840]: audit 2024-09-13T22:47:45.598874+0000 mon.a (mon.0) 1200 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:47:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[22840]: audit 2024-09-13T22:47:45.599028+0000 mon.a (mon.0) 1201 : audit [INF] from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:47:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:46 smithi178 bash[22840]: cluster 2024-09-13T22:47:45.620040+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-09-13T22:47:46.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:46.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:46.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:46.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:46 smithi192 bash[24608]: audit 2024-09-13T22:47:45.300500+0000 mgr.y (mgr.14613) 375 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:46.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:46 smithi192 bash[24608]: cluster 2024-09-13T22:47:45.587988+0000 mon.a (mon.0) 1199 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:47:46.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:46 smithi192 bash[24608]: audit 2024-09-13T22:47:45.598874+0000 mon.a (mon.0) 1200 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:47:46.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:46 smithi192 bash[24608]: audit 2024-09-13T22:47:45.599028+0000 mon.a (mon.0) 1201 : audit [INF] from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:47:46.826 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:46 smithi192 bash[24608]: cluster 2024-09-13T22:47:45.620040+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-09-13T22:47:46.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:46.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:46.982 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:46.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:47.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:47.109 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:47.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:47.241 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:47.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:47.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:47.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:47.501 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:47.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:47.627 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:47.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:47.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:47.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:47.906 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:47.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:47 smithi192 bash[24608]: cluster 2024-09-13T22:47:46.641184+0000 mgr.y (mgr.14613) 376 : cluster [DBG] pgmap v412: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 496 op/s; 7043/188085 objects misplaced (3.745%); 12 MiB/s, 189 objects/s recovering 2024-09-13T22:47:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:47.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:47.964 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:47 smithi178 bash[22840]: cluster 2024-09-13T22:47:46.641184+0000 mgr.y (mgr.14613) 376 : cluster [DBG] pgmap v412: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 496 op/s; 7043/188085 objects misplaced (3.745%); 12 MiB/s, 189 objects/s recovering 2024-09-13T22:47:47.964 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:47 smithi178 bash[29889]: cluster 2024-09-13T22:47:46.641184+0000 mgr.y (mgr.14613) 376 : cluster [DBG] pgmap v412: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 496 op/s; 7043/188085 objects misplaced (3.745%); 12 MiB/s, 189 objects/s recovering 2024-09-13T22:47:48.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:48.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:48.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:48.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:48.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:48.308 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:48.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:48.435 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:48.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:48.576 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:48.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:48.699 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:48.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:48.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:48.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:48.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:48.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:48.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:49.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T22:47:49.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:49.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:49.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:48 smithi192 bash[24608]: cluster 2024-09-13T22:47:47.892724+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-09-13T22:47:49.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:48 smithi192 bash[24608]: audit 2024-09-13T22:47:48.429316+0000 mon.b (mon.1) 164 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:49.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:48 smithi192 bash[24608]: cluster 2024-09-13T22:47:48.641625+0000 mgr.y (mgr.14613) 377 : cluster [DBG] pgmap v414: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 509 op/s; 6515/187185 objects misplaced (3.481%); 11 MiB/s, 175 objects/s recovering 2024-09-13T22:47:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:48 smithi178 bash[29889]: cluster 2024-09-13T22:47:47.892724+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-09-13T22:47:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:48 smithi178 bash[29889]: audit 2024-09-13T22:47:48.429316+0000 mon.b (mon.1) 164 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:49.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:48 smithi178 bash[29889]: cluster 2024-09-13T22:47:48.641625+0000 mgr.y (mgr.14613) 377 : cluster [DBG] pgmap v414: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 509 op/s; 6515/187185 objects misplaced (3.481%); 11 MiB/s, 175 objects/s recovering 2024-09-13T22:47:49.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:48 smithi178 bash[22840]: cluster 2024-09-13T22:47:47.892724+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-09-13T22:47:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:48 smithi178 bash[22840]: audit 2024-09-13T22:47:48.429316+0000 mon.b (mon.1) 164 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:47:49.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:48 smithi178 bash[22840]: cluster 2024-09-13T22:47:48.641625+0000 mgr.y (mgr.14613) 377 : cluster [DBG] pgmap v414: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 509 op/s; 6515/187185 objects misplaced (3.481%); 11 MiB/s, 175 objects/s recovering 2024-09-13T22:47:49.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:49.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:49.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:49.373 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:49.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:49.510 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:49.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:49.644 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:49.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:49.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:49.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:49.899 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:49.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:50.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:50.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.157 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:50.172 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:47:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:47:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:47:50.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T22:47:50.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:50.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:50.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:50.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:50.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.554 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:50.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.681 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:50.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:50.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:50.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:50 smithi192 bash[24608]: cluster 2024-09-13T22:47:50.642670+0000 mgr.y (mgr.14613) 378 : cluster [DBG] pgmap v415: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.03k op/s; 6515/173337 objects misplaced (3.759%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T22:47:50.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:50.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:51.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:51.069 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:50 smithi178 bash[22840]: cluster 2024-09-13T22:47:50.642670+0000 mgr.y (mgr.14613) 378 : cluster [DBG] pgmap v415: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.03k op/s; 6515/173337 objects misplaced (3.759%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T22:47:51.069 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:50 smithi178 bash[29889]: cluster 2024-09-13T22:47:50.642670+0000 mgr.y (mgr.14613) 378 : cluster [DBG] pgmap v415: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.03k op/s; 6515/173337 objects misplaced (3.759%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T22:47:51.081 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:51.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:51.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:51.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:51.338 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:51.443 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:51.469 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:51.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:47:51.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:51.592 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:51.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:51.718 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:51.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:51.854 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:51.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:51.988 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:51.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:52.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:52.119 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:52.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:47:52.222 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:52.259 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:52.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:52.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:52.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:52.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:52.620 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:52.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:52.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:52.778 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:52.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:52.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[22840]: cluster 2024-09-13T22:47:51.843941+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:52.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[22840]: cluster 2024-09-13T22:47:51.844010+0000 osd.1 (osd.1) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:52.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[22840]: cluster 2024-09-13T22:47:51.901183+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-09-13T22:47:52.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[22840]: cluster 2024-09-13T22:47:52.643425+0000 mgr.y (mgr.14613) 379 : cluster [DBG] pgmap v417: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 25 KiB/s rd, 0 B/s wr, 740 op/s; 6515/173337 objects misplaced (3.759%) 2024-09-13T22:47:52.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[22840]: cluster 2024-09-13T22:47:52.894054+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-09-13T22:47:52.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:52.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:53.018 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:53.036 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:53.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:52 smithi192 bash[24608]: cluster 2024-09-13T22:47:51.843941+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:52 smithi192 bash[24608]: cluster 2024-09-13T22:47:51.844010+0000 osd.1 (osd.1) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:52 smithi192 bash[24608]: cluster 2024-09-13T22:47:51.901183+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-09-13T22:47:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:52 smithi192 bash[24608]: cluster 2024-09-13T22:47:52.643425+0000 mgr.y (mgr.14613) 379 : cluster [DBG] pgmap v417: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 25 KiB/s rd, 0 B/s wr, 740 op/s; 6515/173337 objects misplaced (3.759%) 2024-09-13T22:47:53.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:52 smithi192 bash[24608]: cluster 2024-09-13T22:47:52.894054+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-09-13T22:47:53.166 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:53.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[29889]: cluster 2024-09-13T22:47:51.843941+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:47:53.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[29889]: cluster 2024-09-13T22:47:51.844010+0000 osd.1 (osd.1) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:47:53.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[29889]: cluster 2024-09-13T22:47:51.901183+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-09-13T22:47:53.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[29889]: cluster 2024-09-13T22:47:52.643425+0000 mgr.y (mgr.14613) 379 : cluster [DBG] pgmap v417: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 25 KiB/s rd, 0 B/s wr, 740 op/s; 6515/173337 objects misplaced (3.759%) 2024-09-13T22:47:53.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:52 smithi178 bash[29889]: cluster 2024-09-13T22:47:52.894054+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-09-13T22:47:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:53.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:53.291 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:53.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:53.418 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:53.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:53.556 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:53.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:53.687 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:53.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:53.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:53.887 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:53.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:53.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:47:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:53.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:47:54.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:54.084 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:54.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:54.219 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:54.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:54.349 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:54.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:54.479 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:54.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:54.617 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:54.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:54.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:54.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:54.901 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:54.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:54.909 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:54 smithi178 bash[29889]: cluster 2024-09-13T22:47:54.644288+0000 mgr.y (mgr.14613) 380 : cluster [DBG] pgmap v419: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.25k op/s; 5969/164502 objects misplaced (3.629%) 2024-09-13T22:47:54.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:54 smithi178 bash[22840]: cluster 2024-09-13T22:47:54.644288+0000 mgr.y (mgr.14613) 380 : cluster [DBG] pgmap v419: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.25k op/s; 5969/164502 objects misplaced (3.629%) 2024-09-13T22:47:55.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:55.025 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:55.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:55.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:54 smithi192 bash[24608]: cluster 2024-09-13T22:47:54.644288+0000 mgr.y (mgr.14613) 380 : cluster [DBG] pgmap v419: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.25k op/s; 5969/164502 objects misplaced (3.629%) 2024-09-13T22:47:55.155 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:55.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:55.287 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:55.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:55.413 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:55.416 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:47:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:47:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:55.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:55.540 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:55.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:55.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:55.787 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:55.805 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:55.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:55.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:55 smithi178 bash[29889]: audit 2024-09-13T22:47:55.308504+0000 mgr.y (mgr.14613) 381 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:55.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:55 smithi178 bash[22840]: audit 2024-09-13T22:47:55.308504+0000 mgr.y (mgr.14613) 381 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:55.932 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:55.933 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:55.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:56.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:56.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:56.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:55 smithi192 bash[24608]: audit 2024-09-13T22:47:55.308504+0000 mgr.y (mgr.14613) 381 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:47:56.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:56.196 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:56.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:56.324 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:56.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:56.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:56.548 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:56.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:56.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:56.700 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:47:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:47:56.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:56.829 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:56.861 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:56 smithi178 bash[22840]: cluster 2024-09-13T22:47:56.645419+0000 mgr.y (mgr.14613) 382 : cluster [DBG] pgmap v420: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.32k op/s; 5969/157665 objects misplaced (3.786%) 2024-09-13T22:47:56.861 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:56 smithi178 bash[29889]: cluster 2024-09-13T22:47:56.645419+0000 mgr.y (mgr.14613) 382 : cluster [DBG] pgmap v420: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.32k op/s; 5969/157665 objects misplaced (3.786%) 2024-09-13T22:47:56.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:56.942 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:47:56.952 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:56.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:57.055 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:57.080 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:57.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:56 smithi192 bash[24608]: cluster 2024-09-13T22:47:56.645419+0000 mgr.y (mgr.14613) 382 : cluster [DBG] pgmap v420: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.32k op/s; 5969/157665 objects misplaced (3.786%) 2024-09-13T22:47:57.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:57.209 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:57.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:57.352 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:57.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:57.483 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:57.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:57.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:57.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:57.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:57.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:57.865 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:57.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:57.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:58.000 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:58.104 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:58.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:58.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:58.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:47:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:47:58.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:58.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:58.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:58.540 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:58.643 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:58.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:58.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:58.810 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:58.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:58.939 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:47:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:58.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:47:59.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:59.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:47:58 smithi192 bash[24608]: cluster 2024-09-13T22:47:58.645950+0000 mgr.y (mgr.14613) 383 : cluster [DBG] pgmap v421: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 32 KiB/s rd, 0 B/s wr, 739 op/s; 5504/156873 objects misplaced (3.509%) 2024-09-13T22:47:59.075 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:59.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:59.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:47:58 smithi178 bash[29889]: cluster 2024-09-13T22:47:58.645950+0000 mgr.y (mgr.14613) 383 : cluster [DBG] pgmap v421: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 32 KiB/s rd, 0 B/s wr, 739 op/s; 5504/156873 objects misplaced (3.509%) 2024-09-13T22:47:59.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:47:58 smithi178 bash[22840]: cluster 2024-09-13T22:47:58.645950+0000 mgr.y (mgr.14613) 383 : cluster [DBG] pgmap v421: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 32 KiB/s rd, 0 B/s wr, 739 op/s; 5504/156873 objects misplaced (3.509%) 2024-09-13T22:47:59.203 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:59.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:59.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:47:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:47:59.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:59.464 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:47:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:47:59.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:59.607 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:59.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:59.736 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:47:59.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:59.862 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:47:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:47:59.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:47:59.932 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:47:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:47:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:47:59.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:47:59.996 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:00.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:00.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:00.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:00.259 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:00.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:00.381 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:00.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:00.503 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:00.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:00.628 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:00.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:00.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:00.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:00.905 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:00.907 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:00 smithi178 bash[22840]: cluster 2024-09-13T22:48:00.646960+0000 mgr.y (mgr.14613) 384 : cluster [DBG] pgmap v422: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.13k op/s; 5504/144990 objects misplaced (3.796%) 2024-09-13T22:48:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:00.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:00.908 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:00 smithi178 bash[29889]: cluster 2024-09-13T22:48:00.646960+0000 mgr.y (mgr.14613) 384 : cluster [DBG] pgmap v422: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.13k op/s; 5504/144990 objects misplaced (3.796%) 2024-09-13T22:48:01.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:01.018 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:00 smithi192 bash[24608]: cluster 2024-09-13T22:48:00.646960+0000 mgr.y (mgr.14613) 384 : cluster [DBG] pgmap v422: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.13k op/s; 5504/144990 objects misplaced (3.796%) 2024-09-13T22:48:01.034 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:01.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.163 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:01.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:01.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:01.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:01.414 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:01.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:01.542 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:01.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.670 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:01.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:01.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:01.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:02.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:02.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:02.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:02.191 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:02.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:02.326 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:02.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:02.462 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:02.549 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:48:02.549 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:48:02.549 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:48:02.549 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:48:02.549 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:48:02.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:02.596 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:02.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:02.726 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:02.830 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:02.840 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:02 smithi178 bash[29889]: cluster 2024-09-13T22:48:02.648563+0000 mgr.y (mgr.14613) 385 : cluster [DBG] pgmap v423: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 26 KiB/s rd, 0 B/s wr, 987 op/s; 5504/144990 objects misplaced (3.796%) 2024-09-13T22:48:02.840 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:02 smithi178 bash[22840]: cluster 2024-09-13T22:48:02.648563+0000 mgr.y (mgr.14613) 385 : cluster [DBG] pgmap v423: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 26 KiB/s rd, 0 B/s wr, 987 op/s; 5504/144990 objects misplaced (3.796%) 2024-09-13T22:48:02.857 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:02.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:02.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:02 smithi192 bash[24608]: cluster 2024-09-13T22:48:02.648563+0000 mgr.y (mgr.14613) 385 : cluster [DBG] pgmap v423: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 26 KiB/s rd, 0 B/s wr, 987 op/s; 5504/144990 objects misplaced (3.796%) 2024-09-13T22:48:02.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:02.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:03.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:03.110 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:03.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:03.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:03.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:03.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.506 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:03.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:03.729 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T22:48:03.729 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:48:03.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:03.771 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:03.875 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.893 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:03.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:03.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:04.006 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:03 smithi192 bash[24608]: audit 2024-09-13T22:48:03.448303+0000 mon.b (mon.1) 165 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:04.006 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:03 smithi192 bash[24608]: audit 2024-09-13T22:48:03.514028+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.178:0/990524104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:48:04.006 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:03 smithi192 bash[24608]: audit 2024-09-13T22:48:03.517382+0000 mon.a (mon.0) 1206 : audit [INF] from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:48:04.007 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:03 smithi192 bash[24608]: audit 2024-09-13T22:48:03.685562+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.178:0/451556340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:48:04.007 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:03 smithi192 bash[24608]: audit 2024-09-13T22:48:03.686652+0000 mon.a (mon.0) 1207 : audit [INF] from='client.25178 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:48:04.025 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:04.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:04.156 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:04.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[29889]: audit 2024-09-13T22:48:03.448303+0000 mon.b (mon.1) 165 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:04.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[29889]: audit 2024-09-13T22:48:03.514028+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.178:0/990524104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:48:04.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[29889]: audit 2024-09-13T22:48:03.517382+0000 mon.a (mon.0) 1206 : audit [INF] from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:48:04.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[29889]: audit 2024-09-13T22:48:03.685562+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.178:0/451556340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:48:04.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[29889]: audit 2024-09-13T22:48:03.686652+0000 mon.a (mon.0) 1207 : audit [INF] from='client.25178 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:48:04.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[22840]: audit 2024-09-13T22:48:03.448303+0000 mon.b (mon.1) 165 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[22840]: audit 2024-09-13T22:48:03.514028+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.178:0/990524104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:48:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[22840]: audit 2024-09-13T22:48:03.517382+0000 mon.a (mon.0) 1206 : audit [INF] from='client.25114 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:48:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[22840]: audit 2024-09-13T22:48:03.685562+0000 mon.c (mon.2) 44 : audit [INF] from='client.? v1:172.21.15.178:0/451556340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:48:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:03 smithi178 bash[22840]: audit 2024-09-13T22:48:03.686652+0000 mon.a (mon.0) 1207 : audit [INF] from='client.25178 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:48:04.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:04.276 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:04.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:04.397 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:04.431 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:48:04.431 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:48:04.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:04.521 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:04.597 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:48:04.597 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:48:04.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:04.634 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:48:04.634 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:48:04.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:04.674 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:48:04.674 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:48:04.709 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:48:04.709 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:48:04.747 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:48:04.747 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:48:04.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:04.781 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:48:04.781 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:48:04.817 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:48:04.818 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: cluster 2024-09-13T22:48:03.718662+0000 mon.a (mon.0) 1208 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: audit 2024-09-13T22:48:03.725858+0000 mon.a (mon.0) 1209 : audit [INF] from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: audit 2024-09-13T22:48:03.725982+0000 mon.a (mon.0) 1210 : audit [INF] from='client.25178 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: cluster 2024-09-13T22:48:03.730300+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: audit 2024-09-13T22:48:04.401712+0000 mon.a (mon.0) 1212 : audit [DBG] from='client.? v1:172.21.15.178:0/2714931042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.609296+0000 osd.0 (osd.0) 205 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.615321+0000 osd.0 (osd.0) 206 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.649420+0000 mgr.y (mgr.14613) 386 : cluster [DBG] pgmap v425: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 921 op/s; 4928/137217 objects misplaced (3.591%) 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: cluster 2024-09-13T22:48:03.718662+0000 mon.a (mon.0) 1208 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: audit 2024-09-13T22:48:03.725858+0000 mon.a (mon.0) 1209 : audit [INF] from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:48:04.842 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: audit 2024-09-13T22:48:03.725982+0000 mon.a (mon.0) 1210 : audit [INF] from='client.25178 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:48:04.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: cluster 2024-09-13T22:48:03.730300+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-09-13T22:48:04.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: audit 2024-09-13T22:48:04.401712+0000 mon.a (mon.0) 1212 : audit [DBG] from='client.? v1:172.21.15.178:0/2714931042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:04.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.609296+0000 osd.0 (osd.0) 205 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:48:04.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.615321+0000 osd.0 (osd.0) 206 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:48:04.843 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:04 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.649420+0000 mgr.y (mgr.14613) 386 : cluster [DBG] pgmap v425: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 921 op/s; 4928/137217 objects misplaced (3.591%) 2024-09-13T22:48:04.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: cluster 2024-09-13T22:48:03.718662+0000 mon.a (mon.0) 1208 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: audit 2024-09-13T22:48:03.725858+0000 mon.a (mon.0) 1209 : audit [INF] from='client.25114 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: audit 2024-09-13T22:48:03.725982+0000 mon.a (mon.0) 1210 : audit [INF] from='client.25178 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: cluster 2024-09-13T22:48:03.730300+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: audit 2024-09-13T22:48:04.401712+0000 mon.a (mon.0) 1212 : audit [DBG] from='client.? v1:172.21.15.178:0/2714931042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.609296+0000 osd.0 (osd.0) 205 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:48:04.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.615321+0000 osd.0 (osd.0) 206 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:48:04.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:04 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.649420+0000 mgr.y (mgr.14613) 386 : cluster [DBG] pgmap v425: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 921 op/s; 4928/137217 objects misplaced (3.591%) 2024-09-13T22:48:04.913 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:04.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:05.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:05.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:05.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:05.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:05.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:05.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:05.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:05.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:05.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:05.570 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:05.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:48:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:05.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:05.695 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:05.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:05.826 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:05.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:05.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:05.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:06.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T22:48:06.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:03.948868+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:03.950981+0000 osd.5 (osd.5) 228 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.630188+0000 osd.0 (osd.0) 207 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.630248+0000 osd.0 (osd.0) 208 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.743581+0000 mon.a (mon.0) 1213 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.913479+0000 osd.5 (osd.5) 229 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:48:06.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: audit 2024-09-13T22:48:05.316277+0000 mgr.y (mgr.14613) 387 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:06.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:05.436666+0000 osd.7 (osd.7) 207 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:48:06.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:05 smithi192 bash[24608]: cluster 2024-09-13T22:48:05.437515+0000 osd.7 (osd.7) 208 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:48:06.085 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:06.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:03.948868+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:48:06.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:03.950981+0000 osd.5 (osd.5) 228 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:48:06.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.630188+0000 osd.0 (osd.0) 207 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:06.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.630248+0000 osd.0 (osd.0) 208 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:06.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.743581+0000 mon.a (mon.0) 1213 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-09-13T22:48:06.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.913479+0000 osd.5 (osd.5) 229 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: audit 2024-09-13T22:48:05.316277+0000 mgr.y (mgr.14613) 387 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:05.436666+0000 osd.7 (osd.7) 207 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[29889]: cluster 2024-09-13T22:48:05.437515+0000 osd.7 (osd.7) 208 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:03.948868+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:03.950981+0000 osd.5 (osd.5) 228 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.630188+0000 osd.0 (osd.0) 207 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.630248+0000 osd.0 (osd.0) 208 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.743581+0000 mon.a (mon.0) 1213 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.913479+0000 osd.5 (osd.5) 229 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:48:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: audit 2024-09-13T22:48:05.316277+0000 mgr.y (mgr.14613) 387 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:05.436666+0000 osd.7 (osd.7) 207 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:48:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:05 smithi178 bash[22840]: cluster 2024-09-13T22:48:05.437515+0000 osd.7 (osd.7) 208 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:48:06.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:06.223 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:06.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:06.350 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:06.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:06.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:06.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:06.598 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:06.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:06.725 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:06.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:06.855 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:06.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:04.929555+0000 osd.5 (osd.5) 230 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:48:06.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:05.138376+0000 osd.1 (osd.1) 199 : cluster [DBG] 8.3 scrub starts 2024-09-13T22:48:06.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:05.914623+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:48:06.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:05.917122+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:48:06.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:06.527661+0000 osd.7 (osd.7) 209 : cluster [DBG] 3.f scrub starts 2024-09-13T22:48:06.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:06.530182+0000 osd.7 (osd.7) 210 : cluster [DBG] 3.f scrub ok 2024-09-13T22:48:06.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:06 smithi192 bash[24608]: cluster 2024-09-13T22:48:06.650501+0000 mgr.y (mgr.14613) 388 : cluster [DBG] pgmap v427: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 4928/131541 objects misplaced (3.746%) 2024-09-13T22:48:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:06.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:06.980 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:06.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:07.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:07.112 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:07.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:04.929555+0000 osd.5 (osd.5) 230 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:48:07.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:05.138376+0000 osd.1 (osd.1) 199 : cluster [DBG] 8.3 scrub starts 2024-09-13T22:48:07.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:05.914623+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:48:07.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:05.917122+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:48:07.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:06.527661+0000 osd.7 (osd.7) 209 : cluster [DBG] 3.f scrub starts 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:06.530182+0000 osd.7 (osd.7) 210 : cluster [DBG] 3.f scrub ok 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[29889]: cluster 2024-09-13T22:48:06.650501+0000 mgr.y (mgr.14613) 388 : cluster [DBG] pgmap v427: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 4928/131541 objects misplaced (3.746%) 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:04.929555+0000 osd.5 (osd.5) 230 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:05.138376+0000 osd.1 (osd.1) 199 : cluster [DBG] 8.3 scrub starts 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:05.914623+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:05.917122+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:06.527661+0000 osd.7 (osd.7) 209 : cluster [DBG] 3.f scrub starts 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:06.530182+0000 osd.7 (osd.7) 210 : cluster [DBG] 3.f scrub ok 2024-09-13T22:48:07.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:06 smithi178 bash[22840]: cluster 2024-09-13T22:48:06.650501+0000 mgr.y (mgr.14613) 388 : cluster [DBG] pgmap v427: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 4928/131541 objects misplaced (3.746%) 2024-09-13T22:48:07.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:07.243 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:07.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:07.372 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:07.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:07.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:07.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:07.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:07.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:07.753 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:07.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:07.887 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:07.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:07 smithi178 bash[22840]: cluster 2024-09-13T22:48:06.353941+0000 osd.1 (osd.1) 200 : cluster [DBG] 8.3 scrub ok 2024-09-13T22:48:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:07 smithi178 bash[22840]: cluster 2024-09-13T22:48:07.414501+0000 osd.7 (osd.7) 211 : cluster [DBG] 3.d scrub starts 2024-09-13T22:48:07.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:07 smithi178 bash[22840]: cluster 2024-09-13T22:48:07.421957+0000 osd.7 (osd.7) 212 : cluster [DBG] 3.d scrub ok 2024-09-13T22:48:07.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:07 smithi178 bash[29889]: cluster 2024-09-13T22:48:06.353941+0000 osd.1 (osd.1) 200 : cluster [DBG] 8.3 scrub ok 2024-09-13T22:48:07.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:07 smithi178 bash[29889]: cluster 2024-09-13T22:48:07.414501+0000 osd.7 (osd.7) 211 : cluster [DBG] 3.d scrub starts 2024-09-13T22:48:07.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:07 smithi178 bash[29889]: cluster 2024-09-13T22:48:07.421957+0000 osd.7 (osd.7) 212 : cluster [DBG] 3.d scrub ok 2024-09-13T22:48:07.989 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:08.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:08.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:08.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:07 smithi192 bash[24608]: cluster 2024-09-13T22:48:06.353941+0000 osd.1 (osd.1) 200 : cluster [DBG] 8.3 scrub ok 2024-09-13T22:48:08.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:07 smithi192 bash[24608]: cluster 2024-09-13T22:48:07.414501+0000 osd.7 (osd.7) 211 : cluster [DBG] 3.d scrub starts 2024-09-13T22:48:08.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:07 smithi192 bash[24608]: cluster 2024-09-13T22:48:07.421957+0000 osd.7 (osd.7) 212 : cluster [DBG] 3.d scrub ok 2024-09-13T22:48:08.157 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:08.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:08.292 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:08.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:08.421 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:08.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:08.547 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:08.657 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:08.685 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:08.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:08.820 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:08.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:08.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:08 smithi178 bash[29889]: cluster 2024-09-13T22:48:07.139931+0000 osd.2 (osd.2) 165 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:48:08.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:08 smithi178 bash[29889]: cluster 2024-09-13T22:48:07.145342+0000 osd.2 (osd.2) 166 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:48:08.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:08 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.651376+0000 mgr.y (mgr.14613) 389 : cluster [DBG] pgmap v428: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 591 op/s; 4490/130791 objects misplaced (3.433%) 2024-09-13T22:48:08.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:08 smithi178 bash[22840]: cluster 2024-09-13T22:48:07.139931+0000 osd.2 (osd.2) 165 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:48:08.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:08 smithi178 bash[22840]: cluster 2024-09-13T22:48:07.145342+0000 osd.2 (osd.2) 166 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:48:08.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:08 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.651376+0000 mgr.y (mgr.14613) 389 : cluster [DBG] pgmap v428: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 591 op/s; 4490/130791 objects misplaced (3.433%) 2024-09-13T22:48:08.951 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:08.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:08.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T22:48:09.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:09.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:08 smithi192 bash[24608]: cluster 2024-09-13T22:48:07.139931+0000 osd.2 (osd.2) 165 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:48:09.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:08 smithi192 bash[24608]: cluster 2024-09-13T22:48:07.145342+0000 osd.2 (osd.2) 166 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:48:09.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:08 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.651376+0000 mgr.y (mgr.14613) 389 : cluster [DBG] pgmap v428: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 591 op/s; 4490/130791 objects misplaced (3.433%) 2024-09-13T22:48:09.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:09.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:09.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:09.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:09.519 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:09.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.661 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:09.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.800 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:09.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.941 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:09.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:10.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:10.060 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:48:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:48:10.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.142219+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:48:10.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.144893+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:48:10.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.213398+0000 osd.6 (osd.6) 157 : cluster [DBG] 8.9 scrub starts 2024-09-13T22:48:10.061 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.879754+0000 osd.6 (osd.6) 158 : cluster [DBG] 8.9 scrub ok 2024-09-13T22:48:10.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.142219+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:48:10.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.144893+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:48:10.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.213398+0000 osd.6 (osd.6) 157 : cluster [DBG] 8.9 scrub starts 2024-09-13T22:48:10.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:09 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.879754+0000 osd.6 (osd.6) 158 : cluster [DBG] 8.9 scrub ok 2024-09-13T22:48:10.088 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:10.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:10.202 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:09 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.142219+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:48:10.202 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:09 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.144893+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:48:10.202 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:09 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.213398+0000 osd.6 (osd.6) 157 : cluster [DBG] 8.9 scrub starts 2024-09-13T22:48:10.202 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:09 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.879754+0000 osd.6 (osd.6) 158 : cluster [DBG] 8.9 scrub ok 2024-09-13T22:48:10.217 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:10.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:10.347 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:10.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:10.479 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:10.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:10.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:10.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:10.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:10.740 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:10.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:48:10.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:10.891 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:10.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:10.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:11.001 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:10 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.984359+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.e scrub starts 2024-09-13T22:48:11.001 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:10 smithi192 bash[24608]: cluster 2024-09-13T22:48:08.989689+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.e scrub ok 2024-09-13T22:48:11.001 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:10 smithi192 bash[24608]: cluster 2024-09-13T22:48:10.363941+0000 osd.7 (osd.7) 213 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:48:11.001 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:10 smithi192 bash[24608]: cluster 2024-09-13T22:48:10.365475+0000 osd.7 (osd.7) 214 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:48:11.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:10 smithi192 bash[24608]: cluster 2024-09-13T22:48:10.652234+0000 mgr.y (mgr.14613) 390 : cluster [DBG] pgmap v429: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4490/118722 objects misplaced (3.782%) 2024-09-13T22:48:11.013 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:11.041 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.984359+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.e scrub starts 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[29889]: cluster 2024-09-13T22:48:08.989689+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.e scrub ok 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[29889]: cluster 2024-09-13T22:48:10.363941+0000 osd.7 (osd.7) 213 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[29889]: cluster 2024-09-13T22:48:10.365475+0000 osd.7 (osd.7) 214 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[29889]: cluster 2024-09-13T22:48:10.652234+0000 mgr.y (mgr.14613) 390 : cluster [DBG] pgmap v429: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4490/118722 objects misplaced (3.782%) 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.984359+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.e scrub starts 2024-09-13T22:48:11.054 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[22840]: cluster 2024-09-13T22:48:08.989689+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.e scrub ok 2024-09-13T22:48:11.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[22840]: cluster 2024-09-13T22:48:10.363941+0000 osd.7 (osd.7) 213 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:48:11.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[22840]: cluster 2024-09-13T22:48:10.365475+0000 osd.7 (osd.7) 214 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:48:11.055 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:10 smithi178 bash[22840]: cluster 2024-09-13T22:48:10.652234+0000 mgr.y (mgr.14613) 390 : cluster [DBG] pgmap v429: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4490/118722 objects misplaced (3.782%) 2024-09-13T22:48:11.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:11.162 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:11.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:11.292 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:11.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:11.420 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:11.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:11.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:11.655 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:11.682 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:11.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:11.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:11.910 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:11.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:11.946 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:11.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:12.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:12.076 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:12.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[22840]: cluster 2024-09-13T22:48:10.234951+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[22840]: cluster 2024-09-13T22:48:10.240213+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.349693+0000 osd.7 (osd.7) 215 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.350538+0000 osd.7 (osd.7) 216 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.560526+0000 osd.0 (osd.0) 209 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.570496+0000 osd.0 (osd.0) 210 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[29889]: cluster 2024-09-13T22:48:10.234951+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[29889]: cluster 2024-09-13T22:48:10.240213+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.349693+0000 osd.7 (osd.7) 215 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.350538+0000 osd.7 (osd.7) 216 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.560526+0000 osd.0 (osd.0) 209 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:48:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:11 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.570496+0000 osd.0 (osd.0) 210 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:48:12.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:12.208 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:12.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:12.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:11 smithi192 bash[24608]: cluster 2024-09-13T22:48:10.234951+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:48:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:11 smithi192 bash[24608]: cluster 2024-09-13T22:48:10.240213+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:48:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:11 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.349693+0000 osd.7 (osd.7) 215 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:48:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:11 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.350538+0000 osd.7 (osd.7) 216 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:48:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:11 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.560526+0000 osd.0 (osd.0) 209 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:48:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:11 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.570496+0000 osd.0 (osd.0) 210 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:48:12.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:12.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:12.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:12.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:12.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:12.733 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:12.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:12.861 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:12.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:12.931 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:12.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.212185+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.214827+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.237205+0000 osd.6 (osd.6) 161 : cluster [DBG] 3.a scrub starts 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[29889]: cluster 2024-09-13T22:48:11.242199+0000 osd.6 (osd.6) 162 : cluster [DBG] 3.a scrub ok 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[29889]: cluster 2024-09-13T22:48:12.652988+0000 mgr.y (mgr.14613) 391 : cluster [DBG] pgmap v430: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 4490/118722 objects misplaced (3.782%) 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.212185+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.214827+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.237205+0000 osd.6 (osd.6) 161 : cluster [DBG] 3.a scrub starts 2024-09-13T22:48:12.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[22840]: cluster 2024-09-13T22:48:11.242199+0000 osd.6 (osd.6) 162 : cluster [DBG] 3.a scrub ok 2024-09-13T22:48:12.976 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:12 smithi178 bash[22840]: cluster 2024-09-13T22:48:12.652988+0000 mgr.y (mgr.14613) 391 : cluster [DBG] pgmap v430: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 4490/118722 objects misplaced (3.782%) 2024-09-13T22:48:13.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:13.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:13.113 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:12 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.212185+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:48:13.113 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:12 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.214827+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:48:13.113 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:12 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.237205+0000 osd.6 (osd.6) 161 : cluster [DBG] 3.a scrub starts 2024-09-13T22:48:13.113 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:12 smithi192 bash[24608]: cluster 2024-09-13T22:48:11.242199+0000 osd.6 (osd.6) 162 : cluster [DBG] 3.a scrub ok 2024-09-13T22:48:13.114 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:12 smithi192 bash[24608]: cluster 2024-09-13T22:48:12.652988+0000 mgr.y (mgr.14613) 391 : cluster [DBG] pgmap v430: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 4490/118722 objects misplaced (3.782%) 2024-09-13T22:48:13.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:13.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:13.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:13.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:13.378 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:13.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:13.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:13.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:13.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:13.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:13.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:13.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:13.909 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:13.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:14.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:14.021 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[29889]: cluster 2024-09-13T22:48:12.845959+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:48:14.021 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[29889]: cluster 2024-09-13T22:48:12.849203+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:48:14.021 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.476813+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:14.021 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.476879+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:14.021 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.713975+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.b scrub starts 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.725627+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.b scrub ok 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[22840]: cluster 2024-09-13T22:48:12.845959+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[22840]: cluster 2024-09-13T22:48:12.849203+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.476813+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.476879+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.713975+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.b scrub starts 2024-09-13T22:48:14.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:13 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.725627+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.b scrub ok 2024-09-13T22:48:14.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:14.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:14.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:14.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:14.276 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:13 smithi192 bash[24608]: cluster 2024-09-13T22:48:12.845959+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:48:14.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:13 smithi192 bash[24608]: cluster 2024-09-13T22:48:12.849203+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:48:14.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:13 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.476813+0000 osd.7 (osd.7) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:14.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:13 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.476879+0000 osd.7 (osd.7) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:14.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:13 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.713975+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.b scrub starts 2024-09-13T22:48:14.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:13 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.725627+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.b scrub ok 2024-09-13T22:48:14.286 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:14.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:14.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:14.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:14.543 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:14.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:14.682 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:14.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:14.815 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:14.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:14.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:14.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:14.975 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:14.977 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:15.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:15.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.612171+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.617664+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.646957+0000 osd.1 (osd.1) 203 : cluster [DBG] 8.a scrub starts 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:13.946384+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.602870+0000 osd.0 (osd.0) 211 : cluster [DBG] 8.e scrub starts 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.653500+0000 mgr.y (mgr.14613) 392 : cluster [DBG] pgmap v432: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 3929/110718 objects misplaced (3.549%) 2024-09-13T22:48:15.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.832280+0000 osd.5 (osd.5) 237 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:48:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:14 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.833195+0000 osd.5 (osd.5) 238 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:48:15.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:15.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:15.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:15.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:15.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.612171+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:48:15.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.617664+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:48:15.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.646957+0000 osd.1 (osd.1) 203 : cluster [DBG] 8.a scrub starts 2024-09-13T22:48:15.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:13.946384+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.602870+0000 osd.0 (osd.0) 211 : cluster [DBG] 8.e scrub starts 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.653500+0000 mgr.y (mgr.14613) 392 : cluster [DBG] pgmap v432: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 3929/110718 objects misplaced (3.549%) 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.832280+0000 osd.5 (osd.5) 237 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.833195+0000 osd.5 (osd.5) 238 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.612171+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.617664+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.646957+0000 osd.1 (osd.1) 203 : cluster [DBG] 8.a scrub starts 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:13.946384+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.602870+0000 osd.0 (osd.0) 211 : cluster [DBG] 8.e scrub starts 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.653500+0000 mgr.y (mgr.14613) 392 : cluster [DBG] pgmap v432: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 3929/110718 objects misplaced (3.549%) 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.832280+0000 osd.5 (osd.5) 237 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:48:15.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:14 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.833195+0000 osd.5 (osd.5) 238 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:48:15.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:15.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:48:15.444 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:15.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:15.569 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:15.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:15.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:15.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:15.821 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:15.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:15.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:15.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:16.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.207433+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.210083+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: cluster 2024-09-13T22:48:14.428441+0000 osd.1 (osd.1) 204 : cluster [DBG] 8.a scrub ok 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: audit 2024-09-13T22:48:15.325633+0000 mgr.y (mgr.14613) 393 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.546333+0000 osd.0 (osd.0) 212 : cluster [DBG] 8.2 scrub starts 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.795186+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.803895+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:48:16.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.207433+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:48:16.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.210083+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:48:16.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: cluster 2024-09-13T22:48:14.428441+0000 osd.1 (osd.1) 204 : cluster [DBG] 8.a scrub ok 2024-09-13T22:48:16.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: audit 2024-09-13T22:48:15.325633+0000 mgr.y (mgr.14613) 393 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:16.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.546333+0000 osd.0 (osd.0) 212 : cluster [DBG] 8.2 scrub starts 2024-09-13T22:48:16.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.795186+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:48:16.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:15 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.803895+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:48:16.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:16.175 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:16.197 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:16.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:16.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.207433+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:48:16.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.210083+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:48:16.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: cluster 2024-09-13T22:48:14.428441+0000 osd.1 (osd.1) 204 : cluster [DBG] 8.a scrub ok 2024-09-13T22:48:16.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: audit 2024-09-13T22:48:15.325633+0000 mgr.y (mgr.14613) 393 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:16.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.546333+0000 osd.0 (osd.0) 212 : cluster [DBG] 8.2 scrub starts 2024-09-13T22:48:16.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.795186+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:48:16.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:15 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.803895+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:48:16.326 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:16.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:16.453 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:16.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:16.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:16.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:16.715 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:16.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:16.840 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:16.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:16.970 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:16.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:17.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:17.092 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:17.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.184455+0000 osd.1 (osd.1) 205 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.188131+0000 osd.1 (osd.1) 206 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.195750+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.197320+0000 osd.6 (osd.6) 168 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.734482+0000 osd.0 (osd.0) 213 : cluster [DBG] 8.e scrub ok 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.773713+0000 osd.3 (osd.3) 215 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:15.778853+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:48:17.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:16.359787+0000 osd.7 (osd.7) 219 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:48:17.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:16.361313+0000 osd.7 (osd.7) 220 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:48:17.205 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:16 smithi192 bash[24608]: cluster 2024-09-13T22:48:16.654176+0000 mgr.y (mgr.14613) 394 : cluster [DBG] pgmap v433: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 898 op/s; 3929/104529 objects misplaced (3.759%) 2024-09-13T22:48:17.222 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:17.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:17.350 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:17.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.184455+0000 osd.1 (osd.1) 205 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:48:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.188131+0000 osd.1 (osd.1) 206 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:48:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.195750+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:48:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.197320+0000 osd.6 (osd.6) 168 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:48:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.734482+0000 osd.0 (osd.0) 213 : cluster [DBG] 8.e scrub ok 2024-09-13T22:48:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.773713+0000 osd.3 (osd.3) 215 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:48:17.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:15.778853+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:48:17.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:16.359787+0000 osd.7 (osd.7) 219 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:48:17.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:16.361313+0000 osd.7 (osd.7) 220 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:48:17.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[29889]: cluster 2024-09-13T22:48:16.654176+0000 mgr.y (mgr.14613) 394 : cluster [DBG] pgmap v433: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 898 op/s; 3929/104529 objects misplaced (3.759%) 2024-09-13T22:48:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.184455+0000 osd.1 (osd.1) 205 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.188131+0000 osd.1 (osd.1) 206 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.195750+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.197320+0000 osd.6 (osd.6) 168 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.734482+0000 osd.0 (osd.0) 213 : cluster [DBG] 8.e scrub ok 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.773713+0000 osd.3 (osd.3) 215 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:15.778853+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:48:17.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:16.359787+0000 osd.7 (osd.7) 219 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:48:17.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:16.361313+0000 osd.7 (osd.7) 220 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:48:17.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:16 smithi178 bash[22840]: cluster 2024-09-13T22:48:16.654176+0000 mgr.y (mgr.14613) 394 : cluster [DBG] pgmap v433: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 898 op/s; 3929/104529 objects misplaced (3.759%) 2024-09-13T22:48:17.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:17.488 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:17.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:17.611 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:17.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:17.743 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:17.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:17.871 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:17.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:17.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:17.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:18.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:18.117 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:18.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[22840]: cluster 2024-09-13T22:48:16.351015+0000 osd.6 (osd.6) 169 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:48:18.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[22840]: cluster 2024-09-13T22:48:16.353779+0000 osd.6 (osd.6) 170 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[22840]: cluster 2024-09-13T22:48:16.684804+0000 osd.0 (osd.0) 214 : cluster [DBG] 8.2 scrub ok 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.367628+0000 osd.7 (osd.7) 221 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.369545+0000 osd.7 (osd.7) 222 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[29889]: cluster 2024-09-13T22:48:16.351015+0000 osd.6 (osd.6) 169 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[29889]: cluster 2024-09-13T22:48:16.353779+0000 osd.6 (osd.6) 170 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[29889]: cluster 2024-09-13T22:48:16.684804+0000 osd.0 (osd.0) 214 : cluster [DBG] 8.2 scrub ok 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.367628+0000 osd.7 (osd.7) 221 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:48:18.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:18 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.369545+0000 osd.7 (osd.7) 222 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:48:18.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:18.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:17 smithi192 bash[24608]: cluster 2024-09-13T22:48:16.351015+0000 osd.6 (osd.6) 169 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:48:18.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:17 smithi192 bash[24608]: cluster 2024-09-13T22:48:16.353779+0000 osd.6 (osd.6) 170 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:48:18.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:17 smithi192 bash[24608]: cluster 2024-09-13T22:48:16.684804+0000 osd.0 (osd.0) 214 : cluster [DBG] 8.2 scrub ok 2024-09-13T22:48:18.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:17 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.367628+0000 osd.7 (osd.7) 221 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:48:18.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:17 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.369545+0000 osd.7 (osd.7) 222 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:48:18.244 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:18.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:18.382 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:18.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:18.526 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:18.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:18.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:18.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:18.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:18.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:18.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:18.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:19.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:19.058 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:19.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:19.184 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:19.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:19.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.159451+0000 osd.1 (osd.1) 207 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.165319+0000 osd.1 (osd.1) 208 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.190023+0000 osd.2 (osd.2) 169 : cluster [DBG] 2.e scrub starts 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.191234+0000 osd.2 (osd.2) 170 : cluster [DBG] 2.e scrub ok 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.835310+0000 osd.3 (osd.3) 217 : cluster [DBG] 6.b scrub starts 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:17.839510+0000 osd.3 (osd.3) 218 : cluster [DBG] 6.b scrub ok 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:18.319581+0000 osd.7 (osd.7) 223 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:18.321066+0000 osd.7 (osd.7) 224 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: audit 2024-09-13T22:48:18.467749+0000 mon.b (mon.1) 167 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:18.650315+0000 osd.0 (osd.0) 215 : cluster [DBG] 8.6 scrub starts 2024-09-13T22:48:19.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[22840]: cluster 2024-09-13T22:48:18.654834+0000 mgr.y (mgr.14613) 395 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 903 op/s; 3396/103620 objects misplaced (3.277%) 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.159451+0000 osd.1 (osd.1) 207 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.165319+0000 osd.1 (osd.1) 208 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.190023+0000 osd.2 (osd.2) 169 : cluster [DBG] 2.e scrub starts 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.191234+0000 osd.2 (osd.2) 170 : cluster [DBG] 2.e scrub ok 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.835310+0000 osd.3 (osd.3) 217 : cluster [DBG] 6.b scrub starts 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:17.839510+0000 osd.3 (osd.3) 218 : cluster [DBG] 6.b scrub ok 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:18.319581+0000 osd.7 (osd.7) 223 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:18.321066+0000 osd.7 (osd.7) 224 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: audit 2024-09-13T22:48:18.467749+0000 mon.b (mon.1) 167 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:18.650315+0000 osd.0 (osd.0) 215 : cluster [DBG] 8.6 scrub starts 2024-09-13T22:48:19.297 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[29889]: cluster 2024-09-13T22:48:18.654834+0000 mgr.y (mgr.14613) 395 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 903 op/s; 3396/103620 objects misplaced (3.277%) 2024-09-13T22:48:19.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.159451+0000 osd.1 (osd.1) 207 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.165319+0000 osd.1 (osd.1) 208 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.190023+0000 osd.2 (osd.2) 169 : cluster [DBG] 2.e scrub starts 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.191234+0000 osd.2 (osd.2) 170 : cluster [DBG] 2.e scrub ok 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.835310+0000 osd.3 (osd.3) 217 : cluster [DBG] 6.b scrub starts 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:17.839510+0000 osd.3 (osd.3) 218 : cluster [DBG] 6.b scrub ok 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:18.319581+0000 osd.7 (osd.7) 223 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:18.321066+0000 osd.7 (osd.7) 224 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: audit 2024-09-13T22:48:18.467749+0000 mon.b (mon.1) 167 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:18.650315+0000 osd.0 (osd.0) 215 : cluster [DBG] 8.6 scrub starts 2024-09-13T22:48:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:18 smithi192 bash[24608]: cluster 2024-09-13T22:48:18.654834+0000 mgr.y (mgr.14613) 395 : cluster [DBG] pgmap v434: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 903 op/s; 3396/103620 objects misplaced (3.277%) 2024-09-13T22:48:19.409 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:19.442 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:19.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:19.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:19.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:19.710 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:19.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:19.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:19.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:19.963 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:19.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:20.054 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:48:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:48:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:48:20.058 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:18.112397+0000 osd.6 (osd.6) 171 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:48:20.058 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:18.126314+0000 osd.6 (osd.6) 172 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:48:20.059 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:19.228552+0000 osd.0 (osd.0) 216 : cluster [DBG] 8.6 scrub ok 2024-09-13T22:48:20.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:20.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:19 smithi192 bash[24608]: cluster 2024-09-13T22:48:18.112397+0000 osd.6 (osd.6) 171 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:48:20.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:19 smithi192 bash[24608]: cluster 2024-09-13T22:48:18.126314+0000 osd.6 (osd.6) 172 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:48:20.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:19 smithi192 bash[24608]: cluster 2024-09-13T22:48:19.228552+0000 osd.0 (osd.0) 216 : cluster [DBG] 8.6 scrub ok 2024-09-13T22:48:20.100 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:20.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:20.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:20.310 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:18.112397+0000 osd.6 (osd.6) 171 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:48:20.310 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:18.126314+0000 osd.6 (osd.6) 172 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:48:20.310 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:19.228552+0000 osd.0 (osd.0) 216 : cluster [DBG] 8.6 scrub ok 2024-09-13T22:48:20.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:20.370 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:20.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:20.497 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:20.553 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:48:20.553 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:48:20.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:20.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:20.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:20.751 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:20.854 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:20.880 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:20.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:20.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:21.011 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:21.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:21.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:19.109119+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:48:21.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:19.112029+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:48:21.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:19.135567+0000 osd.1 (osd.1) 209 : cluster [DBG] 8.1 scrub starts 2024-09-13T22:48:21.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:20.273097+0000 osd.7 (osd.7) 225 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:48:21.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:20.275123+0000 osd.7 (osd.7) 226 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[29889]: cluster 2024-09-13T22:48:20.657996+0000 mgr.y (mgr.14613) 396 : cluster [DBG] pgmap v435: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s; 3396/91869 objects misplaced (3.697%) 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:19.109119+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:19.112029+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:19.135567+0000 osd.1 (osd.1) 209 : cluster [DBG] 8.1 scrub starts 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:20.273097+0000 osd.7 (osd.7) 225 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:20.275123+0000 osd.7 (osd.7) 226 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:48:21.126 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:20 smithi178 bash[22840]: cluster 2024-09-13T22:48:20.657996+0000 mgr.y (mgr.14613) 396 : cluster [DBG] pgmap v435: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s; 3396/91869 objects misplaced (3.697%) 2024-09-13T22:48:21.147 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:21.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:21.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:21.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:21.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:20 smithi192 bash[24608]: cluster 2024-09-13T22:48:19.109119+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:48:21.398 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:20 smithi192 bash[24608]: cluster 2024-09-13T22:48:19.112029+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:48:21.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:20 smithi192 bash[24608]: cluster 2024-09-13T22:48:19.135567+0000 osd.1 (osd.1) 209 : cluster [DBG] 8.1 scrub starts 2024-09-13T22:48:21.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:20 smithi192 bash[24608]: cluster 2024-09-13T22:48:20.273097+0000 osd.7 (osd.7) 225 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:48:21.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:20 smithi192 bash[24608]: cluster 2024-09-13T22:48:20.275123+0000 osd.7 (osd.7) 226 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:48:21.399 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:20 smithi192 bash[24608]: cluster 2024-09-13T22:48:20.657996+0000 mgr.y (mgr.14613) 396 : cluster [DBG] pgmap v435: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s; 3396/91869 objects misplaced (3.697%) 2024-09-13T22:48:21.414 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:21.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:21.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:21.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:21.687 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:21.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:21.822 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:21.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:21.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:21.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:21.976 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T22:48:22.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:22.074 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:22.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:22.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:22.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:22.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: cluster 2024-09-13T22:48:20.166754+0000 osd.2 (osd.2) 171 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:48:22.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: cluster 2024-09-13T22:48:20.191520+0000 osd.2 (osd.2) 172 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:48:22.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: cluster 2024-09-13T22:48:20.675369+0000 osd.1 (osd.1) 210 : cluster [DBG] 8.1 scrub ok 2024-09-13T22:48:22.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: cluster 2024-09-13T22:48:21.233073+0000 osd.7 (osd.7) 227 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: cluster 2024-09-13T22:48:21.238458+0000 osd.7 (osd.7) 228 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: cluster 2024-09-13T22:48:21.629475+0000 osd.0 (osd.0) 217 : cluster [DBG] 8.c scrub starts 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[22840]: audit 2024-09-13T22:48:21.962465+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.178:0/748752207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: cluster 2024-09-13T22:48:20.166754+0000 osd.2 (osd.2) 171 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: cluster 2024-09-13T22:48:20.191520+0000 osd.2 (osd.2) 172 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: cluster 2024-09-13T22:48:20.675369+0000 osd.1 (osd.1) 210 : cluster [DBG] 8.1 scrub ok 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: cluster 2024-09-13T22:48:21.233073+0000 osd.7 (osd.7) 227 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: cluster 2024-09-13T22:48:21.238458+0000 osd.7 (osd.7) 228 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: cluster 2024-09-13T22:48:21.629475+0000 osd.0 (osd.0) 217 : cluster [DBG] 8.c scrub starts 2024-09-13T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:21 smithi178 bash[29889]: audit 2024-09-13T22:48:21.962465+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.178:0/748752207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:48:22.346 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: cluster 2024-09-13T22:48:20.166754+0000 osd.2 (osd.2) 171 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: cluster 2024-09-13T22:48:20.191520+0000 osd.2 (osd.2) 172 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: cluster 2024-09-13T22:48:20.675369+0000 osd.1 (osd.1) 210 : cluster [DBG] 8.1 scrub ok 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: cluster 2024-09-13T22:48:21.233073+0000 osd.7 (osd.7) 227 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: cluster 2024-09-13T22:48:21.238458+0000 osd.7 (osd.7) 228 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: cluster 2024-09-13T22:48:21.629475+0000 osd.0 (osd.0) 217 : cluster [DBG] 8.c scrub starts 2024-09-13T22:48:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:21 smithi192 bash[24608]: audit 2024-09-13T22:48:21.962465+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.178:0/748752207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:48:22.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:22.586 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:22.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:22.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:22.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:22.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:22.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:22.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:23.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:23.029 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:23.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:23.155 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:23.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: audit 2024-09-13T22:48:21.976222+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? v1:172.21.15.178:0/748752207' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:21.979140+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.042982+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.045187+0000 osd.6 (osd.6) 176 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.084216+0000 osd.0 (osd.0) 218 : cluster [DBG] 8.c scrub ok 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: audit 2024-09-13T22:48:22.091786+0000 mon.b (mon.1) 168 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.095358+0000 osd.1 (osd.1) 211 : cluster [DBG] 8.7 scrub starts 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.658513+0000 mgr.y (mgr.14613) 397 : cluster [DBG] pgmap v437: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s; 3396/91869 objects misplaced (3.697%) 2024-09-13T22:48:23.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.897291+0000 osd.5 (osd.5) 241 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.907488+0000 osd.5 (osd.5) 242 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: audit 2024-09-13T22:48:22.957444+0000 mon.b (mon.1) 169 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: audit 2024-09-13T22:48:22.958549+0000 mon.b (mon.1) 170 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[22840]: audit 2024-09-13T22:48:22.967114+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: audit 2024-09-13T22:48:21.976222+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? v1:172.21.15.178:0/748752207' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:21.979140+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.042982+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.045187+0000 osd.6 (osd.6) 176 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.084216+0000 osd.0 (osd.0) 218 : cluster [DBG] 8.c scrub ok 2024-09-13T22:48:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: audit 2024-09-13T22:48:22.091786+0000 mon.b (mon.1) 168 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.095358+0000 osd.1 (osd.1) 211 : cluster [DBG] 8.7 scrub starts 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.658513+0000 mgr.y (mgr.14613) 397 : cluster [DBG] pgmap v437: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s; 3396/91869 objects misplaced (3.697%) 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.897291+0000 osd.5 (osd.5) 241 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.907488+0000 osd.5 (osd.5) 242 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: audit 2024-09-13T22:48:22.957444+0000 mon.b (mon.1) 169 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: audit 2024-09-13T22:48:22.958549+0000 mon.b (mon.1) 170 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:48:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:22 smithi178 bash[29889]: audit 2024-09-13T22:48:22.967114+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:23.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:23.268 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: audit 2024-09-13T22:48:21.976222+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? v1:172.21.15.178:0/748752207' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:21.979140+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.042982+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.045187+0000 osd.6 (osd.6) 176 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.084216+0000 osd.0 (osd.0) 218 : cluster [DBG] 8.c scrub ok 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: audit 2024-09-13T22:48:22.091786+0000 mon.b (mon.1) 168 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.095358+0000 osd.1 (osd.1) 211 : cluster [DBG] 8.7 scrub starts 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.658513+0000 mgr.y (mgr.14613) 397 : cluster [DBG] pgmap v437: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s; 3396/91869 objects misplaced (3.697%) 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.897291+0000 osd.5 (osd.5) 241 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.907488+0000 osd.5 (osd.5) 242 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: audit 2024-09-13T22:48:22.957444+0000 mon.b (mon.1) 169 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: audit 2024-09-13T22:48:22.958549+0000 mon.b (mon.1) 170 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:48:23.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:22 smithi192 bash[24608]: audit 2024-09-13T22:48:22.967114+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:23.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:23.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:23.402 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:23.506 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:23.524 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:23.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:23.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:23.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:23.791 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:23.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:23.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:23.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:23.994 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:48:24.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:24.039 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:24.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:24.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:24.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:24 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.139501+0000 osd.2 (osd.2) 173 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:48:24.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:24 smithi192 bash[24608]: cluster 2024-09-13T22:48:22.142322+0000 osd.2 (osd.2) 174 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:48:24.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:24 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.104017+0000 osd.1 (osd.1) 212 : cluster [DBG] 2.d scrub starts 2024-09-13T22:48:24.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:24 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.112110+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.d scrub ok 2024-09-13T22:48:24.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:24 smithi192 bash[24608]: audit 2024-09-13T22:48:23.632164+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? v1:172.21.15.178:0/3159115073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:24.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.139501+0000 osd.2 (osd.2) 173 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:48:24.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[29889]: cluster 2024-09-13T22:48:22.142322+0000 osd.2 (osd.2) 174 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:48:24.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.104017+0000 osd.1 (osd.1) 212 : cluster [DBG] 2.d scrub starts 2024-09-13T22:48:24.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.112110+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.d scrub ok 2024-09-13T22:48:24.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[29889]: audit 2024-09-13T22:48:23.632164+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? v1:172.21.15.178:0/3159115073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:24.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.139501+0000 osd.2 (osd.2) 173 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:48:24.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[22840]: cluster 2024-09-13T22:48:22.142322+0000 osd.2 (osd.2) 174 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:48:24.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.104017+0000 osd.1 (osd.1) 212 : cluster [DBG] 2.d scrub starts 2024-09-13T22:48:24.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.112110+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.d scrub ok 2024-09-13T22:48:24.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:24 smithi178 bash[22840]: audit 2024-09-13T22:48:23.632164+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? v1:172.21.15.178:0/3159115073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:24.275 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:48:24.275 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:48:24.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:24.299 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:24.317 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:48:24.317 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:48:24.354 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:48:24.355 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:48:24.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:24.404 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:48:24.405 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:48:24.423 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:24.437 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:48:24.437 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:48:24.464 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:48:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:48:24.481 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:48:24.481 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:48:24.499 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:48:24.499 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:48:24.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:24.548 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:24.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:24.684 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:24.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:24.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:24.920 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:24.948 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:24.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:25.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:25.079 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:25.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.351740+0000 osd.1 (osd.1) 214 : cluster [DBG] 8.7 scrub ok 2024-09-13T22:48:25.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.926172+0000 osd.3 (osd.3) 219 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.929336+0000 osd.3 (osd.3) 220 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.995700+0000 mon.a (mon.0) 1220 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: audit 2024-09-13T22:48:23.996628+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? v1:172.21.15.178:0/3159115073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:23.999208+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.164249+0000 osd.7 (osd.7) 229 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.168389+0000 osd.7 (osd.7) 230 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: audit 2024-09-13T22:48:24.239571+0000 mon.a (mon.0) 1223 : audit [DBG] from='client.? v1:172.21.15.178:0/2593043206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.659067+0000 mgr.y (mgr.14613) 398 : cluster [DBG] pgmap v439: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 675 op/s; 3008/88314 objects misplaced (3.406%) 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.857156+0000 osd.5 (osd.5) 243 : cluster [DBG] 1.a scrub starts 2024-09-13T22:48:25.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.860769+0000 osd.5 (osd.5) 244 : cluster [DBG] 1.a scrub ok 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.351740+0000 osd.1 (osd.1) 214 : cluster [DBG] 8.7 scrub ok 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.926172+0000 osd.3 (osd.3) 219 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.929336+0000 osd.3 (osd.3) 220 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.995700+0000 mon.a (mon.0) 1220 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: audit 2024-09-13T22:48:23.996628+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? v1:172.21.15.178:0/3159115073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:23.999208+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.164249+0000 osd.7 (osd.7) 229 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.168389+0000 osd.7 (osd.7) 230 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: audit 2024-09-13T22:48:24.239571+0000 mon.a (mon.0) 1223 : audit [DBG] from='client.? v1:172.21.15.178:0/2593043206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.659067+0000 mgr.y (mgr.14613) 398 : cluster [DBG] pgmap v439: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 675 op/s; 3008/88314 objects misplaced (3.406%) 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.857156+0000 osd.5 (osd.5) 243 : cluster [DBG] 1.a scrub starts 2024-09-13T22:48:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:25 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.860769+0000 osd.5 (osd.5) 244 : cluster [DBG] 1.a scrub ok 2024-09-13T22:48:25.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:25.209 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:25.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.351740+0000 osd.1 (osd.1) 214 : cluster [DBG] 8.7 scrub ok 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.926172+0000 osd.3 (osd.3) 219 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.929336+0000 osd.3 (osd.3) 220 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.995700+0000 mon.a (mon.0) 1220 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: audit 2024-09-13T22:48:23.996628+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? v1:172.21.15.178:0/3159115073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:23.999208+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.164249+0000 osd.7 (osd.7) 229 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.168389+0000 osd.7 (osd.7) 230 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: audit 2024-09-13T22:48:24.239571+0000 mon.a (mon.0) 1223 : audit [DBG] from='client.? v1:172.21.15.178:0/2593043206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.659067+0000 mgr.y (mgr.14613) 398 : cluster [DBG] pgmap v439: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 675 op/s; 3008/88314 objects misplaced (3.406%) 2024-09-13T22:48:25.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.857156+0000 osd.5 (osd.5) 243 : cluster [DBG] 1.a scrub starts 2024-09-13T22:48:25.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.860769+0000 osd.5 (osd.5) 244 : cluster [DBG] 1.a scrub ok 2024-09-13T22:48:25.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:25.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:25.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:25.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:25.594 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:25.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:48:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:48:25.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:25.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:25.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:25.852 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:25.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:25.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:26.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:26 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.142623+0000 osd.2 (osd.2) 175 : cluster [DBG] 6.14 deep-scrub starts 2024-09-13T22:48:26.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:26 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.158633+0000 osd.2 (osd.2) 176 : cluster [DBG] 6.14 deep-scrub ok 2024-09-13T22:48:26.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:26 smithi178 bash[22840]: audit 2024-09-13T22:48:25.336048+0000 mgr.y (mgr.14613) 399 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:26.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:26.106 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:26.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:26.226 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:26 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.142623+0000 osd.2 (osd.2) 175 : cluster [DBG] 6.14 deep-scrub starts 2024-09-13T22:48:26.226 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:26 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.158633+0000 osd.2 (osd.2) 176 : cluster [DBG] 6.14 deep-scrub ok 2024-09-13T22:48:26.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:26 smithi192 bash[24608]: audit 2024-09-13T22:48:25.336048+0000 mgr.y (mgr.14613) 399 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:26.248 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:26.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:26.374 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:26.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T22:48:26.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:26 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.142623+0000 osd.2 (osd.2) 175 : cluster [DBG] 6.14 deep-scrub starts 2024-09-13T22:48:26.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:26 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.158633+0000 osd.2 (osd.2) 176 : cluster [DBG] 6.14 deep-scrub ok 2024-09-13T22:48:26.450 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:26 smithi178 bash[29889]: audit 2024-09-13T22:48:25.336048+0000 mgr.y (mgr.14613) 399 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:26.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:26.498 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:26.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:26.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:26.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:26.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:26.754 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:26.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:26.885 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:26.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:26.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:27.012 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:27.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.906531+0000 osd.3 (osd.3) 221 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:48:27.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:24.913927+0000 osd.3 (osd.3) 222 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:48:27.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.070038+0000 osd.5 (osd.5) 245 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:48:27.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.075938+0000 osd.5 (osd.5) 246 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:48:27.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.086999+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.e scrub starts 2024-09-13T22:48:27.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.090966+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.e scrub ok 2024-09-13T22:48:27.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.659801+0000 mgr.y (mgr.14613) 400 : cluster [DBG] pgmap v440: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s; 3008/79716 objects misplaced (3.773%) 2024-09-13T22:48:27.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.826770+0000 osd.5 (osd.5) 247 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T22:48:27.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[22840]: cluster 2024-09-13T22:48:26.832687+0000 osd.5 (osd.5) 248 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T22:48:27.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:27.133 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:27.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:27.278 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:27.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.906531+0000 osd.3 (osd.3) 221 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:24.913927+0000 osd.3 (osd.3) 222 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.070038+0000 osd.5 (osd.5) 245 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.075938+0000 osd.5 (osd.5) 246 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.086999+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.e scrub starts 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.090966+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.e scrub ok 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.659801+0000 mgr.y (mgr.14613) 400 : cluster [DBG] pgmap v440: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s; 3008/79716 objects misplaced (3.773%) 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.826770+0000 osd.5 (osd.5) 247 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T22:48:27.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:27 smithi178 bash[29889]: cluster 2024-09-13T22:48:26.832687+0000 osd.5 (osd.5) 248 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.906531+0000 osd.3 (osd.3) 221 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:24.913927+0000 osd.3 (osd.3) 222 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.070038+0000 osd.5 (osd.5) 245 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.075938+0000 osd.5 (osd.5) 246 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.086999+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.e scrub starts 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.090966+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.e scrub ok 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.659801+0000 mgr.y (mgr.14613) 400 : cluster [DBG] pgmap v440: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s; 3008/79716 objects misplaced (3.773%) 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.826770+0000 osd.5 (osd.5) 247 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T22:48:27.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:27 smithi192 bash[24608]: cluster 2024-09-13T22:48:26.832687+0000 osd.5 (osd.5) 248 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T22:48:27.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:27.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:27.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:27.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:27.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:27.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:27.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:27.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:27.953 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:27.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:28.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:28.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:28.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[29889]: cluster 2024-09-13T22:48:27.192439+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.f scrub starts 2024-09-13T22:48:28.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[29889]: cluster 2024-09-13T22:48:27.194209+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.f scrub ok 2024-09-13T22:48:28.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[29889]: cluster 2024-09-13T22:48:27.799311+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:48:28.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[29889]: cluster 2024-09-13T22:48:27.803918+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:48:28.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[22840]: cluster 2024-09-13T22:48:27.192439+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.f scrub starts 2024-09-13T22:48:28.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[22840]: cluster 2024-09-13T22:48:27.194209+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.f scrub ok 2024-09-13T22:48:28.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[22840]: cluster 2024-09-13T22:48:27.799311+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:48:28.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:28 smithi178 bash[22840]: cluster 2024-09-13T22:48:27.803918+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:48:28.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:28.224 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:28.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:28.363 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:28 smithi192 bash[24608]: cluster 2024-09-13T22:48:27.192439+0000 osd.7 (osd.7) 231 : cluster [DBG] 5.f scrub starts 2024-09-13T22:48:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:28 smithi192 bash[24608]: cluster 2024-09-13T22:48:27.194209+0000 osd.7 (osd.7) 232 : cluster [DBG] 5.f scrub ok 2024-09-13T22:48:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:28 smithi192 bash[24608]: cluster 2024-09-13T22:48:27.799311+0000 osd.5 (osd.5) 249 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:48:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:28 smithi192 bash[24608]: cluster 2024-09-13T22:48:27.803918+0000 osd.5 (osd.5) 250 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:48:28.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:28.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:28.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:28.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:28.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:28.754 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:28.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:28.884 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:28.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:28.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:29.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:29.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:29.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.136132+0000 osd.1 (osd.1) 217 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:48:29.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.140645+0000 osd.1 (osd.1) 218 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:48:29.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.660279+0000 mgr.y (mgr.14613) 401 : cluster [DBG] pgmap v441: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s; 2549/78936 objects misplaced (3.229%) 2024-09-13T22:48:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.136132+0000 osd.1 (osd.1) 217 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:48:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.140645+0000 osd.1 (osd.1) 218 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:48:29.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.660279+0000 mgr.y (mgr.14613) 401 : cluster [DBG] pgmap v441: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s; 2549/78936 objects misplaced (3.229%) 2024-09-13T22:48:29.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:29.269 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:29.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:29 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.136132+0000 osd.1 (osd.1) 217 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:48:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:29 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.140645+0000 osd.1 (osd.1) 218 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:48:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:29 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.660279+0000 mgr.y (mgr.14613) 401 : cluster [DBG] pgmap v441: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s; 2549/78936 objects misplaced (3.229%) 2024-09-13T22:48:29.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:29.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:29.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:29.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:29.672 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:29.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:29.801 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:29.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:48:29.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:29.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:48:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:48:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:48:29.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:48:29.967 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:29.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:30.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:30.084 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.088332+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.089545+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.128548+0000 osd.0 (osd.0) 219 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.132772+0000 osd.0 (osd.0) 220 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.188180+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.d scrub starts 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:28.189500+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.d scrub ok 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:29.165524+0000 osd.7 (osd.7) 233 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[29889]: cluster 2024-09-13T22:48:29.166584+0000 osd.7 (osd.7) 234 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.088332+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:48:30.085 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.089545+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:48:30.086 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.128548+0000 osd.0 (osd.0) 219 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:48:30.086 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.132772+0000 osd.0 (osd.0) 220 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:48:30.086 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.188180+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.d scrub starts 2024-09-13T22:48:30.086 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:28.189500+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.d scrub ok 2024-09-13T22:48:30.086 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:29.165524+0000 osd.7 (osd.7) 233 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:48:30.086 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:30 smithi178 bash[22840]: cluster 2024-09-13T22:48:29.166584+0000 osd.7 (osd.7) 234 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:48:30.120 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:30.210 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:30.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:30.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:30.350 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:30.376 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.088332+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.089545+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.128548+0000 osd.0 (osd.0) 219 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.132772+0000 osd.0 (osd.0) 220 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.188180+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.d scrub starts 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:28.189500+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.d scrub ok 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:29.165524+0000 osd.7 (osd.7) 233 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:48:30.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:30 smithi192 bash[24608]: cluster 2024-09-13T22:48:29.166584+0000 osd.7 (osd.7) 234 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:48:30.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:30.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:30.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:30.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:30.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:30.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:30.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:30.924 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:30.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:31.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:31.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:31.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:31.198 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:31.229 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:31.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:31.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.079110+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:48:31.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.082432+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:48:31.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.132490+0000 osd.1 (osd.1) 219 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:48:31.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.135252+0000 osd.1 (osd.1) 220 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:48:31.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.661260+0000 mgr.y (mgr.14613) 402 : cluster [DBG] pgmap v442: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 2549/67287 objects misplaced (3.788%) 2024-09-13T22:48:31.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.079110+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:48:31.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.082432+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:48:31.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.132490+0000 osd.1 (osd.1) 219 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:48:31.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.135252+0000 osd.1 (osd.1) 220 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:48:31.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:31 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.661260+0000 mgr.y (mgr.14613) 402 : cluster [DBG] pgmap v442: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 2549/67287 objects misplaced (3.788%) 2024-09-13T22:48:31.335 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:31.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:31 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.079110+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:48:31.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:31 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.082432+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:48:31.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:31 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.132490+0000 osd.1 (osd.1) 219 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:48:31.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:31 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.135252+0000 osd.1 (osd.1) 220 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:48:31.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:31 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.661260+0000 mgr.y (mgr.14613) 402 : cluster [DBG] pgmap v442: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 2549/67287 objects misplaced (3.788%) 2024-09-13T22:48:31.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:31.463 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:31.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:31.594 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:31.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:31.716 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:31.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.853 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:31.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:31.953 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:31.983 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:31.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:32.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:32.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:32 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.080637+0000 osd.0 (osd.0) 221 : cluster [DBG] 5.1e deep-scrub starts 2024-09-13T22:48:32.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:32 smithi178 bash[29889]: cluster 2024-09-13T22:48:30.083562+0000 osd.0 (osd.0) 222 : cluster [DBG] 5.1e deep-scrub ok 2024-09-13T22:48:32.095 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:32 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.080637+0000 osd.0 (osd.0) 221 : cluster [DBG] 5.1e deep-scrub starts 2024-09-13T22:48:32.095 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:32 smithi178 bash[22840]: cluster 2024-09-13T22:48:30.083562+0000 osd.0 (osd.0) 222 : cluster [DBG] 5.1e deep-scrub ok 2024-09-13T22:48:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:32.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:32.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:32.257 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:32.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:32.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:32.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:32 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.080637+0000 osd.0 (osd.0) 221 : cluster [DBG] 5.1e deep-scrub starts 2024-09-13T22:48:32.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:32 smithi192 bash[24608]: cluster 2024-09-13T22:48:30.083562+0000 osd.0 (osd.0) 222 : cluster [DBG] 5.1e deep-scrub ok 2024-09-13T22:48:32.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:32.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:32.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:32.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:32.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:32.759 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:32.864 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:32.905 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:32.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:33.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:33.028 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:33.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:33.155 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:33.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:31.169286+0000 osd.1 (osd.1) 221 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:31.173741+0000 osd.1 (osd.1) 222 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:31.986138+0000 osd.6 (osd.6) 179 : cluster [DBG] 3.10 deep-scrub starts 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:31.992115+0000 osd.6 (osd.6) 180 : cluster [DBG] 3.10 deep-scrub ok 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:32.047326+0000 osd.0 (osd.0) 223 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:32.050874+0000 osd.0 (osd.0) 224 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:32.195794+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:32.199483+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:48:33.274 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[22840]: cluster 2024-09-13T22:48:32.663011+0000 mgr.y (mgr.14613) 403 : cluster [DBG] pgmap v443: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 820 op/s; 2549/67287 objects misplaced (3.788%) 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:31.169286+0000 osd.1 (osd.1) 221 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:31.173741+0000 osd.1 (osd.1) 222 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:31.986138+0000 osd.6 (osd.6) 179 : cluster [DBG] 3.10 deep-scrub starts 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:31.992115+0000 osd.6 (osd.6) 180 : cluster [DBG] 3.10 deep-scrub ok 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:32.047326+0000 osd.0 (osd.0) 223 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:32.050874+0000 osd.0 (osd.0) 224 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:32.195794+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:32.199483+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:48:33.275 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:33 smithi178 bash[29889]: cluster 2024-09-13T22:48:32.663011+0000 mgr.y (mgr.14613) 403 : cluster [DBG] pgmap v443: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 820 op/s; 2549/67287 objects misplaced (3.788%) 2024-09-13T22:48:33.284 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:33.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:31.169286+0000 osd.1 (osd.1) 221 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:31.173741+0000 osd.1 (osd.1) 222 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:31.986138+0000 osd.6 (osd.6) 179 : cluster [DBG] 3.10 deep-scrub starts 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:31.992115+0000 osd.6 (osd.6) 180 : cluster [DBG] 3.10 deep-scrub ok 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:32.047326+0000 osd.0 (osd.0) 223 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:32.050874+0000 osd.0 (osd.0) 224 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:32.195794+0000 osd.7 (osd.7) 235 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:32.199483+0000 osd.7 (osd.7) 236 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:48:33.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:33 smithi192 bash[24608]: cluster 2024-09-13T22:48:32.663011+0000 mgr.y (mgr.14613) 403 : cluster [DBG] pgmap v443: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 820 op/s; 2549/67287 objects misplaced (3.788%) 2024-09-13T22:48:33.413 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:33.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:33.550 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:33.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:33.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:33.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:33.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:33.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:33.935 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:33.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:34.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:34.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:34.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:34.205 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:34.283 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:34.283 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[22840]: cluster 2024-09-13T22:48:33.053251+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[22840]: cluster 2024-09-13T22:48:33.071745+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[22840]: audit 2024-09-13T22:48:33.483926+0000 mon.a (mon.0) 1224 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[22840]: audit 2024-09-13T22:48:33.487589+0000 mon.b (mon.1) 171 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[22840]: audit 2024-09-13T22:48:33.906914+0000 mon.a (mon.0) 1225 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]: dispatch 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[22840]: audit 2024-09-13T22:48:33.908373+0000 mon.b (mon.1) 172 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]: dispatch 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[29889]: cluster 2024-09-13T22:48:33.053251+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[29889]: cluster 2024-09-13T22:48:33.071745+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:48:34.307 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[29889]: audit 2024-09-13T22:48:33.483926+0000 mon.a (mon.0) 1224 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:34.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[29889]: audit 2024-09-13T22:48:33.487589+0000 mon.b (mon.1) 171 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:34.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[29889]: audit 2024-09-13T22:48:33.906914+0000 mon.a (mon.0) 1225 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]: dispatch 2024-09-13T22:48:34.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:34 smithi178 bash[29889]: audit 2024-09-13T22:48:33.908373+0000 mon.b (mon.1) 172 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]: dispatch 2024-09-13T22:48:34.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:34.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:34 smithi192 bash[24608]: cluster 2024-09-13T22:48:33.053251+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:48:34.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:34 smithi192 bash[24608]: cluster 2024-09-13T22:48:33.071745+0000 osd.0 (osd.0) 226 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:48:34.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:34 smithi192 bash[24608]: audit 2024-09-13T22:48:33.483926+0000 mon.a (mon.0) 1224 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:34.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:34 smithi192 bash[24608]: audit 2024-09-13T22:48:33.487589+0000 mon.b (mon.1) 171 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:34.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:34 smithi192 bash[24608]: audit 2024-09-13T22:48:33.906914+0000 mon.a (mon.0) 1225 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]: dispatch 2024-09-13T22:48:34.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:34 smithi192 bash[24608]: audit 2024-09-13T22:48:33.908373+0000 mon.b (mon.1) 172 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]: dispatch 2024-09-13T22:48:34.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:34.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:34.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:34.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:34.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:34.727 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:34.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:34.861 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:34.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:34.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:34.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:35.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:35.106 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: cluster 2024-09-13T22:48:33.903336+0000 osd.4 (osd.4) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: cluster 2024-09-13T22:48:33.903387+0000 osd.4 (osd.4) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: audit 2024-09-13T22:48:34.482513+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]': finished 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.487849+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.663453+0000 mgr.y (mgr.14613) 404 : cluster [DBG] pgmap v445: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 1968/62508 objects misplaced (3.148%) 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.820610+0000 osd.5 (osd.5) 251 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.822701+0000 osd.5 (osd.5) 252 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: cluster 2024-09-13T22:48:33.903336+0000 osd.4 (osd.4) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:35.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: cluster 2024-09-13T22:48:33.903387+0000 osd.4 (osd.4) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:35.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: audit 2024-09-13T22:48:34.482513+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]': finished 2024-09-13T22:48:35.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.487849+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-09-13T22:48:35.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.663453+0000 mgr.y (mgr.14613) 404 : cluster [DBG] pgmap v445: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 1968/62508 objects misplaced (3.148%) 2024-09-13T22:48:35.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.820610+0000 osd.5 (osd.5) 251 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:48:35.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:35 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.822701+0000 osd.5 (osd.5) 252 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:48:35.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:35.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:35.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:35.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: cluster 2024-09-13T22:48:33.903336+0000 osd.4 (osd.4) 177 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:35.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: cluster 2024-09-13T22:48:33.903387+0000 osd.4 (osd.4) 178 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:35.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: audit 2024-09-13T22:48:34.482513+0000 mon.a (mon.0) 1226 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 6]}]': finished 2024-09-13T22:48:35.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.487849+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-09-13T22:48:35.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.663453+0000 mgr.y (mgr.14613) 404 : cluster [DBG] pgmap v445: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 1968/62508 objects misplaced (3.148%) 2024-09-13T22:48:35.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.820610+0000 osd.5 (osd.5) 251 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:48:35.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.822701+0000 osd.5 (osd.5) 252 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:48:35.347 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:48:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:48:35.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:35.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:35.498 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:35.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:35.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:35.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:35.755 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:35.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:35.880 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:35.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:35.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:36.014 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:36.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:36.146 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:36.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:36.259 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.243390+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.c scrub starts 2024-09-13T22:48:36.259 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.249402+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.c scrub ok 2024-09-13T22:48:36.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.947051+0000 osd.6 (osd.6) 181 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:48:36.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[22840]: cluster 2024-09-13T22:48:34.950866+0000 osd.6 (osd.6) 182 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:48:36.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[22840]: audit 2024-09-13T22:48:35.343987+0000 mgr.y (mgr.14613) 405 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:36.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[22840]: cluster 2024-09-13T22:48:35.495342+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-09-13T22:48:36.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.243390+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.c scrub starts 2024-09-13T22:48:36.261 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.249402+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.c scrub ok 2024-09-13T22:48:36.262 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.947051+0000 osd.6 (osd.6) 181 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:48:36.262 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[29889]: cluster 2024-09-13T22:48:34.950866+0000 osd.6 (osd.6) 182 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:48:36.262 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[29889]: audit 2024-09-13T22:48:35.343987+0000 mgr.y (mgr.14613) 405 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:36.262 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:36 smithi178 bash[29889]: cluster 2024-09-13T22:48:35.495342+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-09-13T22:48:36.276 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:36.380 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:36.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:36 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.243390+0000 osd.2 (osd.2) 179 : cluster [DBG] 2.c scrub starts 2024-09-13T22:48:36.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:36 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.249402+0000 osd.2 (osd.2) 180 : cluster [DBG] 2.c scrub ok 2024-09-13T22:48:36.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:36 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.947051+0000 osd.6 (osd.6) 181 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:48:36.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:36 smithi192 bash[24608]: cluster 2024-09-13T22:48:34.950866+0000 osd.6 (osd.6) 182 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:48:36.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:36 smithi192 bash[24608]: audit 2024-09-13T22:48:35.343987+0000 mgr.y (mgr.14613) 405 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:36.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:36 smithi192 bash[24608]: cluster 2024-09-13T22:48:35.495342+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-09-13T22:48:36.405 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:36.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:36.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:36.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:36.657 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:36.723 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:48:36.723 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:48:36.723 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200595 8.0 8:0f18704f:::benchmark_data_smithi178_80607_object92550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.695429+0000", 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.029058869000000001, 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.029080000000000002, 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200595 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.724 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695429+0000", 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695429+0000", 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.725 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695431+0000", 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.328e-06 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695436+0000", 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7029999999999997e-06 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695444+0000", 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0560000000000005e-06 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.726 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695450+0000", 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9030000000000001e-06 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.698993+0000", 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0035427940000000002 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.727 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699190+0000", 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00019722500000000001 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,5", 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699366+0000", 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00017645000000000001 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703582+0000", 2024-09-13T22:48:36.728 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0042160849999999996 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703595+0000", 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2537e-05 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200598 8.b 8:d345fbbd:::benchmark_data_smithi178_80607_object92553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.696730+0000", 2024-09-13T22:48:36.729 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.027758092000000002, 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.027831886, 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200598 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696730+0000", 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696730+0000", 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696731+0000", 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.485e-06 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.731 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696736+0000", 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8339999999999998e-06 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696746+0000", 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.8379999999999997e-06 2024-09-13T22:48:36.732 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696753+0000", 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2200000000000003e-06 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696805+0000", 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1990000000000002e-05 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.733 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696874+0000", 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9406000000000006e-05 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2", 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696941+0000", 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6908000000000004e-05 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699091+0000", 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00215007 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.734 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703601+0000", 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0045093219999999996 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200572 8.0 8:034a1a58:::benchmark_data_smithi178_80607_object92527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.665870+0000", 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.058618091999999997, 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.058736875000000001, 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.735 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200572 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.665870+0000", 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.665870+0000", 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.736 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.665870+0000", 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2600000000000002e-07 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.665874+0000", 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.1729999999999999e-06 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.665879+0000", 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1610000000000002e-06 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.737 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.665884+0000", 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8380000000000001e-06 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.685291+0000", 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.019407384999999999 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.685521+0000", 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00022968500000000001 2024-09-13T22:48:36.738 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,5", 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.685589+0000", 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.8740999999999998e-05 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.698870+0000", 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013280513000000001 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.739 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703217+0000", 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0043469010000000002 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200601 8.0 8:0e58c596:::benchmark_data_smithi178_80607_object92556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.699158+0000", 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.025330034000000001, 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.025493923000000002, 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.740 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200601 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.741 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699158+0000", 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699158+0000", 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699159+0000", 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5149999999999999e-06 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.744 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699164+0000", 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7389999999999999e-06 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699174+0000", 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.645e-06 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.699183+0000", 2024-09-13T22:48:36.745 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.9309999999999997e-06 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703868+0000", 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0046850839999999999 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703958+0000", 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.0356e-05 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,5", 2024-09-13T22:48:36.746 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.704044+0000", 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.6038999999999994e-05 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.708475+0000", 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0044312249999999996 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.712568+0000", 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0040923749999999997 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.747 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200581 8.0 8:06d54a86:::benchmark_data_smithi178_80607_object92536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.677523+0000", 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.046964971000000001, 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.047183125999999999, 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200581 2024-09-13T22:48:36.748 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.677523+0000", 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.677523+0000", 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.749 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.677525+0000", 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0959999999999999e-06 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.677531+0000", 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6200000000000004e-06 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.677542+0000", 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1838000000000001e-05 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.750 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.677550+0000", 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2350000000000002e-06 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.689940+0000", 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.012390597999999999 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.690052+0000", 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000111737 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.751 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,5", 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695623+0000", 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0055709269999999998 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703281+0000", 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0076583520000000002 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.703512+0000", 2024-09-13T22:48:36.752 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00023057399999999999 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200608 8.f 8:f93198d5:::benchmark_data_smithi178_80607_object92563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.706948+0000", 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.017539530000000001, 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.017842159999999999, 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.753 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200608 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.706948+0000", 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.754 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.706948+0000", 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.706950+0000", 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.934e-06 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.706955+0000", 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.578e-06 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.755 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.706964+0000", 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3200000000000006e-06 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.706969+0000", 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7899999999999999e-06 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.756 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707027+0000", 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8118999999999997e-05 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707121+0000", 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4085000000000005e-05 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2", 2024-09-13T22:48:36.757 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707200+0000", 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9114000000000002e-05 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.712658+0000", 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0054577130000000003 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.715230+0000", 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0025724340000000002 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.758 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200609 8.f 8:fd8b23ab:::benchmark_data_smithi178_80607_object92564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.707079+0000", 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.017409049999999999, 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.017797568, 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.759 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200609 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707079+0000", 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707079+0000", 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.760 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707080+0000", 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.1100000000000005e-07 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707082+0000", 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.8100000000000002e-06 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707087+0000", 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0100000000000003e-06 2024-09-13T22:48:36.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.707091+0000", 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6049999999999998e-06 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.708952+0000", 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0018605729999999999 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.709024+0000", 2024-09-13T22:48:36.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2532000000000003e-05 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2", 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.709093+0000", 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9373000000000001e-05 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.712668+0000", 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0035742880000000001 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.715330+0000", 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0026622159999999998 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200588 8.f 8:f297c912:::benchmark_data_smithi178_80607_object92543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.684705+0000", 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.039783156, 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.040256333999999998, 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.764 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200588 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.684705+0000", 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.684705+0000", 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.684707+0000", 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.441e-06 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.684714+0000", 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2799999999999998e-06 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.684727+0000", 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2880999999999999e-05 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.684736+0000", 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5099999999999998e-06 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.686611+0000", 2024-09-13T22:48:36.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001875364 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.686680+0000", 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9182000000000003e-05 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2", 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.686747+0000", 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.622e-05 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696367+0000", 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0096204830000000009 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.696569+0000", 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000202234 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200617 8.0 8:08992b9f:::benchmark_data_smithi178_80607_object92572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.723910+0000", 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.00057785699999999998, 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0011311170000000001, 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200617 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.770 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723910+0000", 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723910+0000", 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723912+0000", 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.779e-06 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723916+0000", 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7520000000000004e-06 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723927+0000", 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0030000000000001e-05 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723934+0000", 2024-09-13T22:48:36.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2930000000000003e-06 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.723993+0000", 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9292000000000001e-05 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.724080+0000", 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.6401000000000002e-05 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,5", 2024-09-13T22:48:36.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.724156+0000", 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6037000000000004e-05 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:200590 8.b 8:da72e5c0:::benchmark_data_smithi178_80607_object92545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:48:36.687818+0000", 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.036669761000000002, 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.037296385000000001, 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:48:36.774 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 200590 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687818+0000", 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687818+0000", 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:48:36.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687820+0000", 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5579999999999999e-06 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687824+0000", 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6090000000000003e-06 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687836+0000", 2024-09-13T22:48:36.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2086000000000001e-05 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687841+0000", 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1359999999999996e-06 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687884+0000", 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2630999999999999e-05 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.687953+0000", 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9245999999999994e-05 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2", 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.688016+0000", 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2806999999999997e-05 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695053+0000", 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0070371039999999998 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:48:36.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:48:36.695139+0000", 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5439000000000006e-05 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout: "num_ops": 10 2024-09-13T22:48:36.779 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:48:36.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:36.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:36.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:36.922 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:36.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:37.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:37.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:37.165 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:37.193 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[29889]: cluster 2024-09-13T22:48:35.265195+0000 osd.1 (osd.1) 223 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[29889]: cluster 2024-09-13T22:48:35.265572+0000 osd.2 (osd.2) 181 : cluster [DBG] 5.b scrub starts 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[29889]: cluster 2024-09-13T22:48:35.268066+0000 osd.1 (osd.1) 224 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[29889]: cluster 2024-09-13T22:48:35.269149+0000 osd.2 (osd.2) 182 : cluster [DBG] 5.b scrub ok 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[29889]: cluster 2024-09-13T22:48:36.663911+0000 mgr.y (mgr.14613) 406 : cluster [DBG] pgmap v447: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 1.1 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 1968/52674 objects misplaced (3.736%) 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[29889]: cluster 2024-09-13T22:48:36.784064+0000 osd.5 (osd.5) 253 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[22840]: cluster 2024-09-13T22:48:35.265195+0000 osd.1 (osd.1) 223 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[22840]: cluster 2024-09-13T22:48:35.265572+0000 osd.2 (osd.2) 181 : cluster [DBG] 5.b scrub starts 2024-09-13T22:48:37.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[22840]: cluster 2024-09-13T22:48:35.268066+0000 osd.1 (osd.1) 224 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:48:37.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[22840]: cluster 2024-09-13T22:48:35.269149+0000 osd.2 (osd.2) 182 : cluster [DBG] 5.b scrub ok 2024-09-13T22:48:37.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[22840]: cluster 2024-09-13T22:48:36.663911+0000 mgr.y (mgr.14613) 406 : cluster [DBG] pgmap v447: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 1.1 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 1968/52674 objects misplaced (3.736%) 2024-09-13T22:48:37.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:37 smithi178 bash[22840]: cluster 2024-09-13T22:48:36.784064+0000 osd.5 (osd.5) 253 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:48:37.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:37.333 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:37 smithi192 bash[24608]: cluster 2024-09-13T22:48:35.265195+0000 osd.1 (osd.1) 223 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:48:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:37 smithi192 bash[24608]: cluster 2024-09-13T22:48:35.265572+0000 osd.2 (osd.2) 181 : cluster [DBG] 5.b scrub starts 2024-09-13T22:48:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:37 smithi192 bash[24608]: cluster 2024-09-13T22:48:35.268066+0000 osd.1 (osd.1) 224 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:48:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:37 smithi192 bash[24608]: cluster 2024-09-13T22:48:35.269149+0000 osd.2 (osd.2) 182 : cluster [DBG] 5.b scrub ok 2024-09-13T22:48:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:37 smithi192 bash[24608]: cluster 2024-09-13T22:48:36.663911+0000 mgr.y (mgr.14613) 406 : cluster [DBG] pgmap v447: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 1.1 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 1968/52674 objects misplaced (3.736%) 2024-09-13T22:48:37.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:37 smithi192 bash[24608]: cluster 2024-09-13T22:48:36.784064+0000 osd.5 (osd.5) 253 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:48:37.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:37.457 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:37.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:37.595 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:37.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:37.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:37.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:37.874 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:37.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:37.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:38.010 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:38.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:38.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:38 smithi178 bash[29889]: cluster 2024-09-13T22:48:36.939718+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:48:38.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:38 smithi178 bash[29889]: cluster 2024-09-13T22:48:36.940262+0000 osd.5 (osd.5) 254 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:48:38.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:38 smithi178 bash[29889]: cluster 2024-09-13T22:48:36.942112+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:48:38.125 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:38 smithi178 bash[22840]: cluster 2024-09-13T22:48:36.939718+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:48:38.125 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:38 smithi178 bash[22840]: cluster 2024-09-13T22:48:36.940262+0000 osd.5 (osd.5) 254 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:48:38.125 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:38 smithi178 bash[22840]: cluster 2024-09-13T22:48:36.942112+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:48:38.138 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:38.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:38.265 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:38.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:38.388 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:38.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:38 smithi192 bash[24608]: cluster 2024-09-13T22:48:36.939718+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:48:38.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:38 smithi192 bash[24608]: cluster 2024-09-13T22:48:36.940262+0000 osd.5 (osd.5) 254 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:48:38.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:38 smithi192 bash[24608]: cluster 2024-09-13T22:48:36.942112+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:48:38.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:38.513 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:38.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:38.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:38.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:38.778 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:38.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:38.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:38.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:39.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:39.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:39.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:39.148 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:37.192039+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:48:39.148 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:37.201920+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:48:39.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:37.915128+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:48:39.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:37.917896+0000 osd.6 (osd.6) 186 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:48:39.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:38.169860+0000 osd.7 (osd.7) 237 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:48:39.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:38.173256+0000 osd.7 (osd.7) 238 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:48:39.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:39 smithi192 bash[24608]: cluster 2024-09-13T22:48:38.664532+0000 mgr.y (mgr.14613) 407 : cluster [DBG] pgmap v448: 158 pgs: 1 active+remapped+backfilling, 1 peering, 156 active+clean; 1.0 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 641 op/s; 910/51888 objects misplaced (1.754%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T22:48:39.164 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:39.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:39.294 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:39.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:39.419 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:39.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:37.192039+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:48:39.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:37.201920+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:37.915128+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:37.917896+0000 osd.6 (osd.6) 186 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:38.169860+0000 osd.7 (osd.7) 237 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:38.173256+0000 osd.7 (osd.7) 238 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[29889]: cluster 2024-09-13T22:48:38.664532+0000 mgr.y (mgr.14613) 407 : cluster [DBG] pgmap v448: 158 pgs: 1 active+remapped+backfilling, 1 peering, 156 active+clean; 1.0 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 641 op/s; 910/51888 objects misplaced (1.754%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:37.192039+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:37.201920+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:37.915128+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:37.917896+0000 osd.6 (osd.6) 186 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:38.169860+0000 osd.7 (osd.7) 237 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:38.173256+0000 osd.7 (osd.7) 238 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:48:39.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[22840]: cluster 2024-09-13T22:48:38.664532+0000 mgr.y (mgr.14613) 407 : cluster [DBG] pgmap v448: 158 pgs: 1 active+remapped+backfilling, 1 peering, 156 active+clean; 1.0 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 641 op/s; 910/51888 objects misplaced (1.754%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T22:48:39.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:39.542 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:39.645 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:39.673 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:39.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:39.801 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:39.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:39.831 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T22:48:39.904 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:39.916 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:48:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:48:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:48:39.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:39.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:40.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:40.103 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T22:48:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.104 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:40.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:40.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:38.181087+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:38.188499+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:39.074005+0000 mon.a (mon.0) 1229 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:39.074039+0000 mon.a (mon.0) 1230 : cluster [INF] Cluster is now healthy 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:39.143804+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:39.144739+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: audit 2024-09-13T22:48:39.569701+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? v1:172.21.15.178:0/3801972244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:39.744065+0000 osd.5 (osd.5) 255 : cluster [DBG] 6.d scrub starts 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[22840]: cluster 2024-09-13T22:48:39.747875+0000 osd.5 (osd.5) 256 : cluster [DBG] 6.d scrub ok 2024-09-13T22:48:40.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:38.181087+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:38.188499+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:39.074005+0000 mon.a (mon.0) 1229 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:39.074039+0000 mon.a (mon.0) 1230 : cluster [INF] Cluster is now healthy 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:39.143804+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:39.144739+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: audit 2024-09-13T22:48:39.569701+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? v1:172.21.15.178:0/3801972244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:39.744065+0000 osd.5 (osd.5) 255 : cluster [DBG] 6.d scrub starts 2024-09-13T22:48:40.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:40 smithi178 bash[29889]: cluster 2024-09-13T22:48:39.747875+0000 osd.5 (osd.5) 256 : cluster [DBG] 6.d scrub ok 2024-09-13T22:48:40.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:40.228 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:40.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:40.362 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:38.181087+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:38.188499+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:39.074005+0000 mon.a (mon.0) 1229 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:39.074039+0000 mon.a (mon.0) 1230 : cluster [INF] Cluster is now healthy 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:39.143804+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:39.144739+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:48:40.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: audit 2024-09-13T22:48:39.569701+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? v1:172.21.15.178:0/3801972244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:48:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:39.744065+0000 osd.5 (osd.5) 255 : cluster [DBG] 6.d scrub starts 2024-09-13T22:48:40.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:40 smithi192 bash[24608]: cluster 2024-09-13T22:48:39.747875+0000 osd.5 (osd.5) 256 : cluster [DBG] 6.d scrub ok 2024-09-13T22:48:40.420 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:48:40.421 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:48:40.448 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:48:40.449 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:48:40.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:40.485 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:48:40.485 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:48:40.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:40.522 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:48:40.522 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:48:40.545 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:48:40.545 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:48:40.564 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:48:40.564 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:48:40.591 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:48:40.591 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:48:40.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:40.619 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:40.624 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:48:40.624 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:48:40.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:40.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:40.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:40.889 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:40.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:40.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.030 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:41.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:41.108 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:48:41.132 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:41.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[29889]: audit 2024-09-13T22:48:40.087284+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? v1:172.21.15.178:0/3801972244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:48:41.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[29889]: cluster 2024-09-13T22:48:40.093254+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-09-13T22:48:41.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[29889]: audit 2024-09-13T22:48:40.386717+0000 mon.b (mon.1) 173 : audit [DBG] from='client.? v1:172.21.15.178:0/1208899882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:41.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[29889]: cluster 2024-09-13T22:48:40.665208+0000 mgr.y (mgr.14613) 408 : cluster [DBG] pgmap v450: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 811 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.24k op/s; 910/39501 objects misplaced (2.304%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T22:48:41.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[29889]: audit 2024-09-13T22:48:40.700105+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.178:0/593684808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:48:41.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[29889]: audit 2024-09-13T22:48:40.700614+0000 mon.a (mon.0) 1234 : audit [INF] from='client.25214 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:48:41.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[22840]: audit 2024-09-13T22:48:40.087284+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? v1:172.21.15.178:0/3801972244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:48:41.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[22840]: cluster 2024-09-13T22:48:40.093254+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-09-13T22:48:41.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[22840]: audit 2024-09-13T22:48:40.386717+0000 mon.b (mon.1) 173 : audit [DBG] from='client.? v1:172.21.15.178:0/1208899882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:41.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[22840]: cluster 2024-09-13T22:48:40.665208+0000 mgr.y (mgr.14613) 408 : cluster [DBG] pgmap v450: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 811 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.24k op/s; 910/39501 objects misplaced (2.304%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T22:48:41.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[22840]: audit 2024-09-13T22:48:40.700105+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.178:0/593684808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:48:41.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:41 smithi178 bash[22840]: audit 2024-09-13T22:48:40.700614+0000 mon.a (mon.0) 1234 : audit [INF] from='client.25214 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:48:41.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:41.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:41.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:41.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:41 smithi192 bash[24608]: audit 2024-09-13T22:48:40.087284+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? v1:172.21.15.178:0/3801972244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:48:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:41 smithi192 bash[24608]: cluster 2024-09-13T22:48:40.093254+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-09-13T22:48:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:41 smithi192 bash[24608]: audit 2024-09-13T22:48:40.386717+0000 mon.b (mon.1) 173 : audit [DBG] from='client.? v1:172.21.15.178:0/1208899882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:41 smithi192 bash[24608]: cluster 2024-09-13T22:48:40.665208+0000 mgr.y (mgr.14613) 408 : cluster [DBG] pgmap v450: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 811 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.24k op/s; 910/39501 objects misplaced (2.304%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T22:48:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:41 smithi192 bash[24608]: audit 2024-09-13T22:48:40.700105+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.178:0/593684808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:48:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:41 smithi192 bash[24608]: audit 2024-09-13T22:48:40.700614+0000 mon.a (mon.0) 1234 : audit [INF] from='client.25214 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:48:41.413 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:41.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:41.543 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:41.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:41.670 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:41.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:41.801 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:41.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:41.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:41.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:41.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:42.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:42.064 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:42.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.184 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:42.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: cluster 2024-09-13T22:48:40.283320+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:48:42.235 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: cluster 2024-09-13T22:48:40.290653+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: cluster 2024-09-13T22:48:41.100591+0000 mon.a (mon.0) 1235 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: cluster 2024-09-13T22:48:41.104312+0000 osd.1 (osd.1) 231 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: audit 2024-09-13T22:48:41.105573+0000 mon.a (mon.0) 1236 : audit [INF] from='client.25214 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: cluster 2024-09-13T22:48:41.109240+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[29889]: cluster 2024-09-13T22:48:41.110188+0000 osd.1 (osd.1) 232 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: cluster 2024-09-13T22:48:40.283320+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: cluster 2024-09-13T22:48:40.290653+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: cluster 2024-09-13T22:48:41.100591+0000 mon.a (mon.0) 1235 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:48:42.236 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: cluster 2024-09-13T22:48:41.104312+0000 osd.1 (osd.1) 231 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:48:42.237 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: audit 2024-09-13T22:48:41.105573+0000 mon.a (mon.0) 1236 : audit [INF] from='client.25214 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:48:42.237 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: cluster 2024-09-13T22:48:41.109240+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-09-13T22:48:42.237 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:42 smithi178 bash[22840]: cluster 2024-09-13T22:48:41.110188+0000 osd.1 (osd.1) 232 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:48:42.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: cluster 2024-09-13T22:48:40.283320+0000 osd.2 (osd.2) 183 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:48:42.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: cluster 2024-09-13T22:48:40.290653+0000 osd.2 (osd.2) 184 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:48:42.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: cluster 2024-09-13T22:48:41.100591+0000 mon.a (mon.0) 1235 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:48:42.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: cluster 2024-09-13T22:48:41.104312+0000 osd.1 (osd.1) 231 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:48:42.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: audit 2024-09-13T22:48:41.105573+0000 mon.a (mon.0) 1236 : audit [INF] from='client.25214 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:48:42.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: cluster 2024-09-13T22:48:41.109240+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-09-13T22:48:42.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:42 smithi192 bash[24608]: cluster 2024-09-13T22:48:41.110188+0000 osd.1 (osd.1) 232 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:48:42.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:42.412 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:42.444 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:42.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.574 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:42.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:42.705 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:42.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T22:48:42.839 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:42.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.969 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:42.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:43.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:43.090 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:43.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:43.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:43.317 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:43.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:43.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:43 smithi192 bash[24608]: cluster 2024-09-13T22:48:42.665683+0000 mgr.y (mgr.14613) 409 : cluster [DBG] pgmap v452: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 811 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 713 B/s rd, 0 B/s wr, 612 op/s; 910/39501 objects misplaced (2.304%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T22:48:43.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:43.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:43.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:43.593 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:43.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:43 smithi178 bash[22840]: cluster 2024-09-13T22:48:42.665683+0000 mgr.y (mgr.14613) 409 : cluster [DBG] pgmap v452: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 811 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 713 B/s rd, 0 B/s wr, 612 op/s; 910/39501 objects misplaced (2.304%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T22:48:43.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:43 smithi178 bash[29889]: cluster 2024-09-13T22:48:42.665683+0000 mgr.y (mgr.14613) 409 : cluster [DBG] pgmap v452: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 811 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 713 B/s rd, 0 B/s wr, 612 op/s; 910/39501 objects misplaced (2.304%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T22:48:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:43.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:43.727 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:43.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:43.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:43.854 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:43.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:43.980 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:43.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:44.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:44.101 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:44.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:44.228 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:44.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:44.351 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:44.352 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:44 smithi192 bash[24608]: cluster 2024-09-13T22:48:42.561394+0000 osd.2 (osd.2) 185 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:44.353 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:44 smithi192 bash[24608]: cluster 2024-09-13T22:48:42.561453+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:44.353 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:44 smithi192 bash[24608]: cluster 2024-09-13T22:48:43.122954+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-09-13T22:48:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:44.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:44 smithi178 bash[29889]: cluster 2024-09-13T22:48:42.561394+0000 osd.2 (osd.2) 185 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:44.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:44 smithi178 bash[29889]: cluster 2024-09-13T22:48:42.561453+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:44.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:44 smithi178 bash[29889]: cluster 2024-09-13T22:48:43.122954+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-09-13T22:48:44.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:44 smithi178 bash[22840]: cluster 2024-09-13T22:48:42.561394+0000 osd.2 (osd.2) 185 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:44.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:44 smithi178 bash[22840]: cluster 2024-09-13T22:48:42.561453+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:44.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:44 smithi178 bash[22840]: cluster 2024-09-13T22:48:43.122954+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-09-13T22:48:44.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:44.481 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:44.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:44.609 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:44.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:44.742 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:44.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:44.881 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:44.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:44.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:45.010 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:45.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:45.135 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:45.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:45.259 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:45.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:45.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:48:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:48:45.372 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:45 smithi192 bash[24608]: cluster 2024-09-13T22:48:44.666445+0000 mgr.y (mgr.14613) 410 : cluster [DBG] pgmap v454: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 721 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 930 op/s; 76/35157 objects misplaced (0.216%); 3.4 MiB/s, 55 objects/s recovering 2024-09-13T22:48:45.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:45.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:45 smithi178 bash[29889]: cluster 2024-09-13T22:48:44.666445+0000 mgr.y (mgr.14613) 410 : cluster [DBG] pgmap v454: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 721 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 930 op/s; 76/35157 objects misplaced (0.216%); 3.4 MiB/s, 55 objects/s recovering 2024-09-13T22:48:45.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:45 smithi178 bash[22840]: cluster 2024-09-13T22:48:44.666445+0000 mgr.y (mgr.14613) 410 : cluster [DBG] pgmap v454: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 721 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 930 op/s; 76/35157 objects misplaced (0.216%); 3.4 MiB/s, 55 objects/s recovering 2024-09-13T22:48:45.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:45.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:45.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:45.640 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:45.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:45.771 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:45.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:45.901 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:45.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:46.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:46.029 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:46.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:46.160 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:46.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:46.286 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:46.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:46.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:46 smithi178 bash[22840]: cluster 2024-09-13T22:48:45.135535+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-09-13T22:48:46.400 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:46 smithi178 bash[22840]: audit 2024-09-13T22:48:45.352097+0000 mgr.y (mgr.14613) 411 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:46.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:46 smithi178 bash[29889]: cluster 2024-09-13T22:48:45.135535+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-09-13T22:48:46.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:46 smithi178 bash[29889]: audit 2024-09-13T22:48:45.352097+0000 mgr.y (mgr.14613) 411 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:46.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:46 smithi192 bash[24608]: cluster 2024-09-13T22:48:45.135535+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-09-13T22:48:46.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:46 smithi192 bash[24608]: audit 2024-09-13T22:48:45.352097+0000 mgr.y (mgr.14613) 411 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:46.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:48:46.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:46.544 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:46.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:48:46.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:46.667 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:46.830 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:46.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:46.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:46.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:46.998 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:47.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:47.134 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:47.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:47.265 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:47.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:47.391 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:47.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:47 smithi192 bash[24608]: cluster 2024-09-13T22:48:46.138343+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-09-13T22:48:47.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:47 smithi192 bash[24608]: cluster 2024-09-13T22:48:46.666998+0000 mgr.y (mgr.14613) 412 : cluster [DBG] pgmap v457: 158 pgs: 1 peering, 157 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 849 op/s; 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:48:47.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:47 smithi178 bash[29889]: cluster 2024-09-13T22:48:46.138343+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-09-13T22:48:47.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:47 smithi178 bash[29889]: cluster 2024-09-13T22:48:46.666998+0000 mgr.y (mgr.14613) 412 : cluster [DBG] pgmap v457: 158 pgs: 1 peering, 157 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 849 op/s; 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:48:47.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:47 smithi178 bash[22840]: cluster 2024-09-13T22:48:46.138343+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-09-13T22:48:47.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:47 smithi178 bash[22840]: cluster 2024-09-13T22:48:46.666998+0000 mgr.y (mgr.14613) 412 : cluster [DBG] pgmap v457: 158 pgs: 1 peering, 157 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 849 op/s; 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:48:47.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:47.515 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:47.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:47.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:47.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:47.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:47.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:47.908 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:47.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:48.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:48.039 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:48.042 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:48.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:48.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:48.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:48.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:48 smithi178 bash[29889]: cluster 2024-09-13T22:48:46.338055+0000 osd.3 (osd.3) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:48.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:48 smithi178 bash[29889]: cluster 2024-09-13T22:48:46.338129+0000 osd.3 (osd.3) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:48.280 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:48 smithi178 bash[29889]: cluster 2024-09-13T22:48:47.167306+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-09-13T22:48:48.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:48 smithi178 bash[22840]: cluster 2024-09-13T22:48:46.338055+0000 osd.3 (osd.3) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:48.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:48 smithi178 bash[22840]: cluster 2024-09-13T22:48:46.338129+0000 osd.3 (osd.3) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:48.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:48 smithi178 bash[22840]: cluster 2024-09-13T22:48:47.167306+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-09-13T22:48:48.297 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:48.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:48.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:48 smithi192 bash[24608]: cluster 2024-09-13T22:48:46.338055+0000 osd.3 (osd.3) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:48.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:48 smithi192 bash[24608]: cluster 2024-09-13T22:48:46.338129+0000 osd.3 (osd.3) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:48.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:48 smithi192 bash[24608]: cluster 2024-09-13T22:48:47.167306+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-09-13T22:48:48.422 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:48.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:48.557 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:48.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:48.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:48.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:48.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:48.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:48.950 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:48.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:49.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:49.065 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:49.076 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:49.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:49.199 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:49.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:49.331 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:49.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:49.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:49.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:49.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:49 smithi192 bash[24608]: audit 2024-09-13T22:48:48.507573+0000 mon.a (mon.0) 1242 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:49.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:49 smithi192 bash[24608]: audit 2024-09-13T22:48:48.513187+0000 mon.b (mon.1) 174 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:49.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:49 smithi192 bash[24608]: cluster 2024-09-13T22:48:48.667474+0000 mgr.y (mgr.14613) 413 : cluster [DBG] pgmap v459: 158 pgs: 1 peering, 157 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 851 op/s; 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:48:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:49.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:49.705 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:49.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[29889]: audit 2024-09-13T22:48:48.507573+0000 mon.a (mon.0) 1242 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:49.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[29889]: audit 2024-09-13T22:48:48.513187+0000 mon.b (mon.1) 174 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:49.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[29889]: cluster 2024-09-13T22:48:48.667474+0000 mgr.y (mgr.14613) 413 : cluster [DBG] pgmap v459: 158 pgs: 1 peering, 157 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 851 op/s; 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:48:49.806 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[22840]: audit 2024-09-13T22:48:48.507573+0000 mon.a (mon.0) 1242 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:48:49.807 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[22840]: audit 2024-09-13T22:48:48.513187+0000 mon.b (mon.1) 174 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:48:49.807 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[22840]: cluster 2024-09-13T22:48:48.667474+0000 mgr.y (mgr.14613) 413 : cluster [DBG] pgmap v459: 158 pgs: 1 peering, 157 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 851 op/s; 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:48:49.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:49.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:49.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:49.971 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:49.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:50.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:50.098 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:50.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:48:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:48:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:48:50.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:50.238 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:50.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:50.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:50.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:50.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:50.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:50.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:50.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:50.749 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:50.854 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:50.876 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:50.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:50.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:50.999 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:51.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:51.105 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:51.105 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:48:51.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:51.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:51.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:51.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:51.395 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:51.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:51.526 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:51.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:51.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:51.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[22840]: cluster 2024-09-13T22:48:50.416679+0000 osd.5 (osd.5) 257 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:51.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[22840]: cluster 2024-09-13T22:48:50.416725+0000 osd.5 (osd.5) 258 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:51.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[22840]: cluster 2024-09-13T22:48:50.517924+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-09-13T22:48:51.684 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[22840]: cluster 2024-09-13T22:48:50.667916+0000 mgr.y (mgr.14613) 414 : cluster [DBG] pgmap v461: 158 pgs: 158 active+clean; 245 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.17k op/s; 416 KiB/s, 6 objects/s recovering 2024-09-13T22:48:51.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[29889]: cluster 2024-09-13T22:48:50.416679+0000 osd.5 (osd.5) 257 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:51.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[29889]: cluster 2024-09-13T22:48:50.416725+0000 osd.5 (osd.5) 258 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:51.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[29889]: cluster 2024-09-13T22:48:50.517924+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-09-13T22:48:51.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:51 smithi178 bash[29889]: cluster 2024-09-13T22:48:50.667916+0000 mgr.y (mgr.14613) 414 : cluster [DBG] pgmap v461: 158 pgs: 158 active+clean; 245 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.17k op/s; 416 KiB/s, 6 objects/s recovering 2024-09-13T22:48:51.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:51.763 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:51 smithi192 bash[24608]: cluster 2024-09-13T22:48:50.416679+0000 osd.5 (osd.5) 257 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:51.763 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:51 smithi192 bash[24608]: cluster 2024-09-13T22:48:50.416725+0000 osd.5 (osd.5) 258 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:51.764 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:51 smithi192 bash[24608]: cluster 2024-09-13T22:48:50.517924+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-09-13T22:48:51.764 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:51 smithi192 bash[24608]: cluster 2024-09-13T22:48:50.667916+0000 mgr.y (mgr.14613) 414 : cluster [DBG] pgmap v461: 158 pgs: 158 active+clean; 245 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.17k op/s; 416 KiB/s, 6 objects/s recovering 2024-09-13T22:48:51.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:51.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:51.906 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:51.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:52.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:52.037 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:52.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:52.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:52.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:52.293 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:52.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:52.419 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:52.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:52.552 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:52.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:52.685 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:52.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:52.801 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:52 smithi178 bash[22840]: cluster 2024-09-13T22:48:52.668530+0000 mgr.y (mgr.14613) 415 : cluster [DBG] pgmap v462: 158 pgs: 158 active+clean; 245 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 783 B/s rd, 0 B/s wr, 664 op/s 2024-09-13T22:48:52.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:52 smithi178 bash[29889]: cluster 2024-09-13T22:48:52.668530+0000 mgr.y (mgr.14613) 415 : cluster [DBG] pgmap v462: 158 pgs: 158 active+clean; 245 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 783 B/s rd, 0 B/s wr, 664 op/s 2024-09-13T22:48:52.817 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:52.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:52.946 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:52.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:53.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:53.061 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:52 smithi192 bash[24608]: cluster 2024-09-13T22:48:52.668530+0000 mgr.y (mgr.14613) 415 : cluster [DBG] pgmap v462: 158 pgs: 158 active+clean; 245 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 783 B/s rd, 0 B/s wr, 664 op/s 2024-09-13T22:48:53.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:53.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:53.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:53.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:53.342 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:53.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:53.471 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:53.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:53.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:53.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:53.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:53.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:53.874 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:53.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:53.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:54.024 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:54.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:54.154 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:54.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:54.273 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:54.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:54.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:54.443 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Removed 108042 objects 2024-09-13T22:48:54.443 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Clean up completed and total clean up time :124.616 2024-09-13T22:48:54.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:54.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:54.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:54.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:54.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:54.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:54.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:54.908 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:54 smithi178 bash[29889]: cluster 2024-09-13T22:48:54.669377+0000 mgr.y (mgr.14613) 416 : cluster [DBG] pgmap v463: 158 pgs: 158 active+clean; 172 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 687 op/s 2024-09-13T22:48:54.908 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:54 smithi178 bash[22840]: cluster 2024-09-13T22:48:54.669377+0000 mgr.y (mgr.14613) 416 : cluster [DBG] pgmap v463: 158 pgs: 158 active+clean; 172 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 687 op/s 2024-09-13T22:48:54.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:54.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:55.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:55.039 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:54 smithi192 bash[24608]: cluster 2024-09-13T22:48:54.669377+0000 mgr.y (mgr.14613) 416 : cluster [DBG] pgmap v463: 158 pgs: 158 active+clean; 172 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 687 op/s 2024-09-13T22:48:55.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:55.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:55.183 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:55.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:55.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:55.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:55.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:55.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:55.539 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:48:55.539 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:48:55.563 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:55.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:48:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:48:55.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:55.691 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:55.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:55.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:55.904 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:55 smithi178 bash[29889]: audit 2024-09-13T22:48:55.356094+0000 mgr.y (mgr.14613) 417 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:55.905 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:55 smithi178 bash[22840]: audit 2024-09-13T22:48:55.356094+0000 mgr.y (mgr.14613) 417 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:55.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:55 smithi192 bash[24608]: audit 2024-09-13T22:48:55.356094+0000 mgr.y (mgr.14613) 417 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:48:55.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:55.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:55.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:56.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:56.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:56.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:56.206 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:56.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:56.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:56.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:56.458 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:56.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:56.589 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:56.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:56.719 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:56.812 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T22:48:56.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:56.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:56.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:56 smithi192 bash[24608]: audit 2024-09-13T22:48:56.192635+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? v1:172.21.15.178:0/2023943755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:48:56.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:56 smithi192 bash[24608]: cluster 2024-09-13T22:48:56.216266+0000 osd.1 (osd.1) 233 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:56.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:56 smithi192 bash[24608]: cluster 2024-09-13T22:48:56.216364+0000 osd.1 (osd.1) 234 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:56.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:56 smithi192 bash[24608]: cluster 2024-09-13T22:48:56.670605+0000 mgr.y (mgr.14613) 418 : cluster [DBG] pgmap v464: 158 pgs: 158 active+clean; 7.4 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 968 B/s rd, 0 B/s wr, 856 op/s 2024-09-13T22:48:56.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:56.969 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:48:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:56.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:48:57.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:57.100 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:57.159 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[22840]: audit 2024-09-13T22:48:56.192635+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? v1:172.21.15.178:0/2023943755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:48:57.159 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[22840]: cluster 2024-09-13T22:48:56.216266+0000 osd.1 (osd.1) 233 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:57.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[22840]: cluster 2024-09-13T22:48:56.216364+0000 osd.1 (osd.1) 234 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:57.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[22840]: cluster 2024-09-13T22:48:56.670605+0000 mgr.y (mgr.14613) 418 : cluster [DBG] pgmap v464: 158 pgs: 158 active+clean; 7.4 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 968 B/s rd, 0 B/s wr, 856 op/s 2024-09-13T22:48:57.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[29889]: audit 2024-09-13T22:48:56.192635+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? v1:172.21.15.178:0/2023943755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T22:48:57.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[29889]: cluster 2024-09-13T22:48:56.216266+0000 osd.1 (osd.1) 233 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:48:57.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[29889]: cluster 2024-09-13T22:48:56.216364+0000 osd.1 (osd.1) 234 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:48:57.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:56 smithi178 bash[29889]: cluster 2024-09-13T22:48:56.670605+0000 mgr.y (mgr.14613) 418 : cluster [DBG] pgmap v464: 158 pgs: 158 active+clean; 7.4 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 968 B/s rd, 0 B/s wr, 856 op/s 2024-09-13T22:48:57.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.226 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:57.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:57.291 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:48:57.291 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:48:57.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:57.347 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:48:57.347 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:48:57.352 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:57.369 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:48:57.369 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:48:57.386 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:48:57.387 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:48:57.406 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:48:57.406 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:48:57.428 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:48:57.429 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:48:57.448 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:48:57.449 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:48:57.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:57.470 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:48:57.470 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:48:57.476 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:57.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:57.606 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:57.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:57.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:57.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T22:48:57.853 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:57 smithi192 bash[24608]: audit 2024-09-13T22:48:56.813644+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? v1:172.21.15.178:0/2023943755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T22:48:57.853 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:57 smithi192 bash[24608]: cluster 2024-09-13T22:48:56.818063+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-09-13T22:48:57.853 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:57 smithi192 bash[24608]: audit 2024-09-13T22:48:57.274474+0000 mon.a (mon.0) 1247 : audit [DBG] from='client.? v1:172.21.15.178:0/3453973013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:57.869 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:57.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:57.996 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:57.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:58.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:58.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:57 smithi178 bash[29889]: audit 2024-09-13T22:48:56.813644+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? v1:172.21.15.178:0/2023943755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T22:48:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:57 smithi178 bash[29889]: cluster 2024-09-13T22:48:56.818063+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-09-13T22:48:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:57 smithi178 bash[29889]: audit 2024-09-13T22:48:57.274474+0000 mon.a (mon.0) 1247 : audit [DBG] from='client.? v1:172.21.15.178:0/3453973013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:57 smithi178 bash[22840]: audit 2024-09-13T22:48:56.813644+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? v1:172.21.15.178:0/2023943755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T22:48:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:57 smithi178 bash[22840]: cluster 2024-09-13T22:48:56.818063+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-09-13T22:48:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:57 smithi178 bash[22840]: audit 2024-09-13T22:48:57.274474+0000 mon.a (mon.0) 1247 : audit [DBG] from='client.? v1:172.21.15.178:0/3453973013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:48:58.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:58.252 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:58.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:58.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:58.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:58.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:58.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:58.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:48:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:48:58.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:58.775 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:58.834 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:48:58.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:58.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:58 smithi192 bash[24608]: audit 2024-09-13T22:48:57.833967+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.178:0/892595246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:58.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:58 smithi192 bash[24608]: audit 2024-09-13T22:48:57.834359+0000 mon.a (mon.0) 1248 : audit [INF] from='client.25232 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:58.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:58 smithi192 bash[24608]: cluster 2024-09-13T22:48:58.671305+0000 mgr.y (mgr.14613) 419 : cluster [DBG] pgmap v466: 158 pgs: 158 active+clean; 7.4 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 502 B/s rd, 0 B/s wr, 466 op/s 2024-09-13T22:48:58.903 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:58.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:48:59.006 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:59.033 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:48:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:48:59.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:59.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:58 smithi178 bash[29889]: audit 2024-09-13T22:48:57.833967+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.178:0/892595246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:59.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:58 smithi178 bash[29889]: audit 2024-09-13T22:48:57.834359+0000 mon.a (mon.0) 1248 : audit [INF] from='client.25232 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:59.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:58 smithi178 bash[29889]: cluster 2024-09-13T22:48:58.671305+0000 mgr.y (mgr.14613) 419 : cluster [DBG] pgmap v466: 158 pgs: 158 active+clean; 7.4 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 502 B/s rd, 0 B/s wr, 466 op/s 2024-09-13T22:48:59.147 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:58 smithi178 bash[22840]: audit 2024-09-13T22:48:57.833967+0000 mon.c (mon.2) 46 : audit [INF] from='client.? v1:172.21.15.178:0/892595246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:59.148 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:58 smithi178 bash[22840]: audit 2024-09-13T22:48:57.834359+0000 mon.a (mon.0) 1248 : audit [INF] from='client.25232 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:48:59.148 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:58 smithi178 bash[22840]: cluster 2024-09-13T22:48:58.671305+0000 mgr.y (mgr.14613) 419 : cluster [DBG] pgmap v466: 158 pgs: 158 active+clean; 7.4 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 502 B/s rd, 0 B/s wr, 466 op/s 2024-09-13T22:48:59.160 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:59.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:59.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:48:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:48:59.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T22:48:59.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:59.416 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-13T22:48:59.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-13T22:48:59.453 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:48:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:48:59.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:59.585 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:59.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:59.714 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:48:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:48:59.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:59.845 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:48:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:48:59.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:59.972 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:48:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:48:59.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:00.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:00.088 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:59 smithi192 bash[24608]: cluster 2024-09-13T22:48:58.816451+0000 mon.a (mon.0) 1249 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:00.088 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:59 smithi192 bash[24608]: audit 2024-09-13T22:48:58.826205+0000 mon.a (mon.0) 1250 : audit [INF] from='client.25232 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:00.088 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:48:59 smithi192 bash[24608]: cluster 2024-09-13T22:48:58.833220+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-09-13T22:49:00.102 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:00.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:48:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:49:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[22840]: cluster 2024-09-13T22:48:58.816451+0000 mon.a (mon.0) 1249 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[22840]: audit 2024-09-13T22:48:58.826205+0000 mon.a (mon.0) 1250 : audit [INF] from='client.25232 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[22840]: cluster 2024-09-13T22:48:58.833220+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-09-13T22:49:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[29889]: cluster 2024-09-13T22:48:58.816451+0000 mon.a (mon.0) 1249 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[29889]: audit 2024-09-13T22:48:58.826205+0000 mon.a (mon.0) 1250 : audit [INF] from='client.25232 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:48:59 smithi178 bash[29889]: cluster 2024-09-13T22:48:58.833220+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-09-13T22:49:00.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:00.233 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:00.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:00.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:00.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:00.493 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:00.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:00.625 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:00.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:00.754 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:00.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:00.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:00.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:00.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:00.999 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[29889]: cluster 2024-09-13T22:48:59.372050+0000 osd.6 (osd.6) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:00.999 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[29889]: cluster 2024-09-13T22:48:59.372163+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:00.999 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[29889]: cluster 2024-09-13T22:48:59.923742+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-09-13T22:49:00.999 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[29889]: cluster 2024-09-13T22:49:00.672381+0000 mgr.y (mgr.14613) 420 : cluster [DBG] pgmap v469: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 459 op/s 2024-09-13T22:49:00.999 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[22840]: cluster 2024-09-13T22:48:59.372050+0000 osd.6 (osd.6) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:01.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[22840]: cluster 2024-09-13T22:48:59.372163+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:01.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[22840]: cluster 2024-09-13T22:48:59.923742+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-09-13T22:49:01.000 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:00 smithi178 bash[22840]: cluster 2024-09-13T22:49:00.672381+0000 mgr.y (mgr.14613) 420 : cluster [DBG] pgmap v469: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 459 op/s 2024-09-13T22:49:01.014 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:01.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:01.146 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:01.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:00 smithi192 bash[24608]: cluster 2024-09-13T22:48:59.372050+0000 osd.6 (osd.6) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:01.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:00 smithi192 bash[24608]: cluster 2024-09-13T22:48:59.372163+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:01.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:00 smithi192 bash[24608]: cluster 2024-09-13T22:48:59.923742+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-09-13T22:49:01.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:00 smithi192 bash[24608]: cluster 2024-09-13T22:49:00.672381+0000 mgr.y (mgr.14613) 420 : cluster [DBG] pgmap v469: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 459 op/s 2024-09-13T22:49:01.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:01.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:01.380 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:01.390 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:49:01.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:01.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:01.538 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:01.641 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:01.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:01.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:01.792 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:01.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:01.922 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:01.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:01.972 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:49:02.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:02.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:02.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.178 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:02.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:02.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:02.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:02.437 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:02.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:02.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:02.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.695 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:02.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:02 smithi192 bash[24608]: cluster 2024-09-13T22:49:02.673083+0000 mgr.y (mgr.14613) 421 : cluster [DBG] pgmap v470: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 19 op/s 2024-09-13T22:49:02.826 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:02.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.955 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:02.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:03.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:03.069 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:02 smithi178 bash[29889]: cluster 2024-09-13T22:49:02.673083+0000 mgr.y (mgr.14613) 421 : cluster [DBG] pgmap v470: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 19 op/s 2024-09-13T22:49:03.070 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:02 smithi178 bash[22840]: cluster 2024-09-13T22:49:02.673083+0000 mgr.y (mgr.14613) 421 : cluster [DBG] pgmap v470: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 19 op/s 2024-09-13T22:49:03.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:03.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:03.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:03.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:03.346 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:03.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:03.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:03.503 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:03.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:03.602 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:03.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:03.731 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:03.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:03.862 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:03.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:03 smithi178 bash[22840]: audit 2024-09-13T22:49:03.529191+0000 mon.b (mon.1) 175 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:03 smithi178 bash[29889]: audit 2024-09-13T22:49:03.529191+0000 mon.b (mon.1) 175 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:03.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:03.976 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:03 smithi192 bash[24608]: audit 2024-09-13T22:49:03.529191+0000 mon.b (mon.1) 175 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:03.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:03.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:04.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:04.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:04.252 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:04.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:04.382 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:04.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.513 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:04.522 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:04.522 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:04.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:04.643 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:04.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.772 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:04.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.902 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:04.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:05.005 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:05.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:04 smithi178 bash[22840]: cluster 2024-09-13T22:49:04.674346+0000 mgr.y (mgr.14613) 422 : cluster [DBG] pgmap v471: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 14 op/s 2024-09-13T22:49:05.016 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:04 smithi178 bash[29889]: cluster 2024-09-13T22:49:04.674346+0000 mgr.y (mgr.14613) 422 : cluster [DBG] pgmap v471: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 14 op/s 2024-09-13T22:49:05.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:05.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:04 smithi192 bash[24608]: cluster 2024-09-13T22:49:04.674346+0000 mgr.y (mgr.14613) 422 : cluster [DBG] pgmap v471: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 14 op/s 2024-09-13T22:49:05.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:05.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:05.295 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:05.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:05.404 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:49:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:49:05.427 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:05.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:05.540 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:05.555 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:05.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:05.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:05.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:05.815 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:05.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:05.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:05 smithi178 bash[29889]: audit 2024-09-13T22:49:05.362200+0000 mgr.y (mgr.14613) 423 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:05.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:05 smithi178 bash[22840]: audit 2024-09-13T22:49:05.362200+0000 mgr.y (mgr.14613) 423 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:05.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:06.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:06.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:05 smithi192 bash[24608]: audit 2024-09-13T22:49:05.362200+0000 mgr.y (mgr.14613) 423 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:06.075 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:06.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:06.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:06.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:06.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:06.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:06.449 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:49:06.449 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 20, 2024-09-13T22:49:06.449 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 600, 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104251 8.5 8.2afdc0c5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.890883+0000", 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.55642813899999, 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0150911789999999, 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.450 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104251 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.890883+0000", 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.890883+0000", 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.451 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.890903+0000", 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0094000000000001e-05 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.890931+0000", 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.7378000000000001e-05 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.890949+0000", 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8337000000000001e-05 2024-09-13T22:49:06.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.890958+0000", 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1020000000000003e-06 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.891021+0000", 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2448999999999998e-05 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.453 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.891029+0000", 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7829999999999999e-06 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.905964+0000", 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.014935033 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.905975+0000", 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1003e-05 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.454 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104265 8.5 8.258a9205 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.904199+0000", 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.543112278, 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.001808941, 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.455 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104265 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904199+0000", 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904199+0000", 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.456 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904201+0000", 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9250000000000002e-06 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904219+0000", 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8204000000000001e-05 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904228+0000", 2024-09-13T22:49:06.457 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5629999999999992e-06 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904233+0000", 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.9139999999999999e-06 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904291+0000", 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8057000000000003e-05 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.458 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.904296+0000", 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3499999999999996e-06 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906001+0000", 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0017047269999999 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906008+0000", 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2010000000000002e-06 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.459 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104275 8.5 8.ce6e5185 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.913331+0000", 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.53398041899999, 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99270857599999995, 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.460 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104275 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913331+0000", 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913331+0000", 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.461 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913333+0000", 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2989999999999999e-06 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913356+0000", 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3070000000000001e-05 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.462 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913367+0000", 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0227000000000001e-05 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913374+0000", 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7640000000000005e-06 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913421+0000", 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6527999999999999e-05 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.463 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.913427+0000", 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1979999999999998e-06 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906032+0000", 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99260512099999998 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906040+0000", 2024-09-13T22:49:06.464 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3690000000000001e-06 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104279 8.5 8.f9077575 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.916597+0000", 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.530714402, 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.98947164399999998, 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.465 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104279 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916597+0000", 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916597+0000", 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916599+0000", 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1689999999999999e-06 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916638+0000", 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9167e-05 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.467 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916651+0000", 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2316e-05 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916660+0000", 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.7490000000000004e-06 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916722+0000", 2024-09-13T22:49:06.468 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2725999999999995e-05 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.916731+0000", 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2749999999999995e-06 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906062+0000", 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.98933150999999997 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.469 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906069+0000", 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7320000000000001e-06 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104291 8.5 8.300374a5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.928252+0000", 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.5190595, 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.97785084899999997, 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.470 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104291 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928252+0000", 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928252+0000", 2024-09-13T22:49:06.471 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928253+0000", 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4099999999999997e-07 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928273+0000", 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9689000000000001e-05 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928279+0000", 2024-09-13T22:49:06.472 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4989999999999999e-06 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928285+0000", 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5010000000000004e-06 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928333+0000", 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7954999999999997e-05 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.473 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.928340+0000", 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7619999999999995e-06 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906096+0000", 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.97775547399999996 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906103+0000", 2024-09-13T22:49:06.474 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1280000000000002e-06 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104308 8.5 8.5033b6a5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.949333+0000", 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.49797829400001, 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.95679997999999999, 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.475 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104308 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.949333+0000", 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.476 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.949333+0000", 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.949342+0000", 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5990000000000002e-06 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.949383+0000", 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0839e-05 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.949397+0000", 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4297000000000001e-05 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.949404+0000", 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.99e-06 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.951179+0000", 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001775485 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.951188+0000", 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5750000000000001e-06 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906126+0000", 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.95493839700000005 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906133+0000", 2024-09-13T22:49:06.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.798e-06 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104320 8.5 8.4a0ad1b5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.960646+0000", 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.48666520399999, 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.94551783199999995, 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.480 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104320 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960646+0000", 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.481 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960646+0000", 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960648+0000", 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8700000000000001e-06 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960666+0000", 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7518000000000001e-05 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960674+0000", 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3559999999999993e-06 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960679+0000", 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.9760000000000003e-06 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960724+0000", 2024-09-13T22:49:06.483 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.4647999999999997e-05 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.960729+0000", 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3000000000000001e-06 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906157+0000", 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.94542765500000003 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906164+0000", 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5089999999999996e-06 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104330 8.5 8.fc73ce05 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.971490+0000", 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.47582126699999, 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.93470776899999997, 2024-09-13T22:49:06.485 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104330 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971490+0000", 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.486 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971490+0000", 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971492+0000", 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.883e-06 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971516+0000", 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3515000000000001e-05 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971526+0000", 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0530000000000001e-05 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971534+0000", 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5800000000000003e-06 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971591+0000", 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.7309999999999998e-05 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.971599+0000", 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.1499999999999999e-06 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906190+0000", 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.93459028399999999 2024-09-13T22:49:06.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906198+0000", 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5169999999999999e-06 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104342 8.5 8.71b89c95 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.983476+0000", 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.46383536299999, 2024-09-13T22:49:06.490 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.92275232200000001, 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104342 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.983476+0000", 2024-09-13T22:49:06.491 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.983476+0000", 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.983478+0000", 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0999999999999998e-06 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.983501+0000", 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2317e-05 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.983510+0000", 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7499999999999998e-06 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.983517+0000", 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1570000000000003e-06 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.986748+0000", 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.003230105 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.986753+0000", 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.7790000000000001e-06 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906221+0000", 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.91946781 2024-09-13T22:49:06.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906228+0000", 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3039999999999999e-06 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104355 8.5 8.2413dea5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:43.996529+0000", 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.45078301800001, 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.90972968899999995, 2024-09-13T22:49:06.495 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104355 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996529+0000", 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.496 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996529+0000", 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996530+0000", 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.784e-06 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996552+0000", 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1620999999999998e-05 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996562+0000", 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.9979999999999998e-06 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996570+0000", 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.1990000000000006e-06 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.498 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996635+0000", 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.533e-05 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:43.996644+0000", 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2220000000000002e-06 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906251+0000", 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.90960733599999999 2024-09-13T22:49:06.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906258+0000", 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.199e-06 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104360 8.5 8.a1527235 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:44.001940+0000", 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.44537159000001, 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.90434562500000004, 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.500 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104360 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.001940+0000", 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.001940+0000", 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.001941+0000", 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5319999999999999e-06 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.001968+0000", 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6152999999999998e-05 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.502 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.001979+0000", 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1027e-05 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.001984+0000", 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.4589999999999997e-06 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.002036+0000", 2024-09-13T22:49:06.503 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1709000000000002e-05 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.002046+0000", 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0498e-05 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906281+0000", 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.904234278 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906286+0000", 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.9690000000000002e-06 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:104368 8.5 8.d9ae3f55 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:44.011768+0000", 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 142.43554357100001, 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.89453449200000001, 2024-09-13T22:49:06.505 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 104368 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011768+0000", 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011768+0000", 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011770+0000", 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.747e-06 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011793+0000", 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3073000000000001e-05 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011805+0000", 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1887e-05 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011812+0000", 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3710000000000002e-06 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011868+0000", 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5917e-05 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.508 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.011875+0000", 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3629999999999996e-06 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906299+0000", 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.89442345599999995 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:44.906302+0000", 2024-09-13T22:49:06.509 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6780000000000002e-06 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119747 8.b 8:db3acc4b:::benchmark_data_smithi178_80607_object11702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:02.626131+0000", 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.82118017499999, 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4469614580000001, 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.510 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119747 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626131+0000", 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626131+0000", 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.511 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626133+0000", 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9870000000000002e-06 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626138+0000", 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0799999999999996e-06 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626150+0000", 2024-09-13T22:49:06.512 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1917e-05 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626157+0000", 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1450000000000002e-06 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626225+0000", 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7967999999999997e-05 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626335+0000", 2024-09-13T22:49:06.513 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000109497 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.626452+0000", 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011717 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.628750+0000", 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002297828 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.629322+0000", 2024-09-13T22:49:06.514 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00057230400000000004 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.881649+0000", 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.25232649099999999 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.881662+0000", 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3698e-05 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.515 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.872975+0000", 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99131228800000004 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.872990+0000", 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5651e-05 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073014+0000", 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2000240710000001 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.516 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073093+0000", 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8362999999999995e-05 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119751 8.b 8:d7fde418:::benchmark_data_smithi178_80607_object11706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:02.632240+0000", 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.81507118899999, 2024-09-13T22:49:06.517 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4520012310000001, 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119751 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632240+0000", 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.518 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632240+0000", 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632241+0000", 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0249999999999999e-06 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.519 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632245+0000", 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.2320000000000001e-06 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632250+0000", 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6180000000000003e-06 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632254+0000", 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9469999999999996e-06 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.520 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632303+0000", 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8396000000000002e-05 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632421+0000", 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011796 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:49:06.521 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.632548+0000", 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000127028 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634016+0000", 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001468334 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.636692+0000", 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0026758229999999999 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.522 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.881695+0000", 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.24500365399999999 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.881705+0000", 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1649999999999995e-06 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873061+0000", 2024-09-13T22:49:06.523 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.991356193 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873072+0000", 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0862999999999999e-05 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073385+0000", 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.200313636 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.524 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for missing object", 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073496+0000", 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000110303 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084138+0000", 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010642604999999999 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084242+0000", 2024-09-13T22:49:06.525 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010344900000000001 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119753 8.b 8:db43278e:::benchmark_data_smithi178_80607_object11708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:02.634113+0000", 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.813198775, 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4675905839999999, 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:49:06.526 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119753 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634113+0000", 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.527 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634113+0000", 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634114+0000", 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1880000000000001e-06 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634117+0000", 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.4860000000000002e-06 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.528 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634127+0000", 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1130000000000007e-06 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634131+0000", 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6809999999999997e-06 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634168+0000", 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.7088999999999997e-05 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.529 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634241+0000", 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3067999999999999e-05 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.634311+0000", 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9906999999999993e-05 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.635954+0000", 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0016431869999999999 2024-09-13T22:49:06.530 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.636921+0000", 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00096647599999999997 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.881767+0000", 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.24484640099999999 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.881776+0000", 2024-09-13T22:49:06.531 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3869999999999995e-06 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873105+0000", 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99132889800000001 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873113+0000", 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4810000000000006e-06 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.532 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073556+0000", 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2004424250000001 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for missing object", 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073657+0000", 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000101854 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101622+0000", 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.027964292000000002 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101703+0000", 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.1650999999999995e-05 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.14454.0:2088 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 93919878343936 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:02.869636+0000", 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.577675554, 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0250301079999999, 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.14454", 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/1055532036", 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 2088 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869636+0000", 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.535 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869636+0000", 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869641+0000", 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6870000000000002e-06 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869655+0000", 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4352000000000001e-05 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.536 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869686+0000", 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.1223999999999998e-05 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869708+0000", 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1695000000000002e-05 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869833+0000", 2024-09-13T22:49:06.537 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012543499999999999 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op must wait for map", 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.869845+0000", 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.13e-05 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.882866+0000", 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013021457 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.538 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.882877+0000", 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1001e-05 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.882344+0000", 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99946682099999995 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for degraded object", 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.882776+0000", 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00043240900000000003 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.539 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.893312+0000", 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010535839 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.893399+0000", 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.6858000000000001e-05 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.894666+0000", 2024-09-13T22:49:06.540 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0012670299999999999 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119765 8.b 8:d152d0ed:::benchmark_data_smithi178_80607_object11720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:02.995604+0000", 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.45170735400001, 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1057635870000002, 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.541 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119765 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995604+0000", 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.542 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995604+0000", 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995606+0000", 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2000000000000001e-06 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995612+0000", 2024-09-13T22:49:06.543 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9179999999999999e-06 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995625+0000", 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2265e-05 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995631+0000", 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7159999999999997e-06 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.544 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995693+0000", 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1864000000000003e-05 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:02.995704+0000", 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1029999999999999e-05 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873139+0000", 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.87743444699999995 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.545 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873146+0000", 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6660000000000008e-06 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073703+0000", 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.200556508 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.073822+0000", 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000119429 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.546 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.074256+0000", 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00043417100000000001 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.083763+0000", 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0095068979999999997 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084289+0000", 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00052598599999999999 2024-09-13T22:49:06.547 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.089815+0000", 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0055260559999999997 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101216+0000", 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011400683 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101267+0000", 2024-09-13T22:49:06.548 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1384999999999997e-05 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101368+0000", 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010036500000000001 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119790 8.b 8:dd6baabf:::benchmark_data_smithi178_80607_object11745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:03.039872+0000", 2024-09-13T22:49:06.549 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.407439263, 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0616820150000001, 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119790 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039872+0000", 2024-09-13T22:49:06.550 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039872+0000", 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039874+0000", 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2389999999999999e-06 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.551 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039878+0000", 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.1849999999999997e-06 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039887+0000", 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.9570000000000008e-06 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039892+0000", 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.2320000000000001e-06 2024-09-13T22:49:06.552 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039941+0000", 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.9455e-05 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.039949+0000", 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6040000000000004e-06 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.553 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873173+0000", 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.83322375299999996 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:03.873180+0000", 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.0829999999999998e-06 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.079289+0000", 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.206109388 2024-09-13T22:49:06.554 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.079410+0000", 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000120332 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.079574+0000", 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016462099999999999 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.555 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084101+0000", 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0045267099999999998 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.089871+0000", 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0057703759999999998 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.094762+0000", 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0048904650000000001 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.556 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101409+0000", 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0066473670000000004 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101456+0000", 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7203999999999997e-05 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.557 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.101554+0000", 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.8043999999999998e-05 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119806 8.b 8:dd75b6f8:::benchmark_data_smithi178_80607_object11761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:04.055169+0000", 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 122.39214211399999, 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0522021399999999, 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:06.558 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119806 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055169+0000", 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.559 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055169+0000", 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055171+0000", 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.567e-06 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055175+0000", 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.941e-06 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.560 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055185+0000", 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.5999999999999996e-06 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055192+0000", 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6380000000000004e-06 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055252+0000", 2024-09-13T22:49:06.561 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0318000000000001e-05 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.055261+0000", 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.8189999999999998e-06 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.081544+0000", 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0262831459999999 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.562 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.081621+0000", 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6703000000000001e-05 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.081733+0000", 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000112279 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084241+0000", 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0025072610000000002 2024-09-13T22:49:06.563 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.089923+0000", 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0056819990000000001 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.094806+0000", 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0048831680000000002 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.107190+0000", 2024-09-13T22:49:06.564 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.012383969999999999 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.107250+0000", 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0228e-05 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.107372+0000", 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012150300000000001 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.565 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25037.0:119815 8.b 8:d6479d47:::benchmark_data_smithi178_80607_object11770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:47:04.086121+0000", 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 122.36119099299999, 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0214494140000001, 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25037", 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:06.566 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 119815 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086121+0000", 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086121+0000", 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.567 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086122+0000", 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7570000000000001e-06 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086127+0000", 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.515e-06 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086137+0000", 2024-09-13T22:49:06.568 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.8949999999999993e-06 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086142+0000", 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.2639999999999999e-06 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086202+0000", 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9957000000000002e-05 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.569 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:04.086210+0000", 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2379999999999997e-06 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084328+0000", 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99811763799999997 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084453+0000", 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012551199999999999 2024-09-13T22:49:06.570 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.084596+0000", 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014247000000000001 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.089831+0000", 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0052350629999999999 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.571 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.089970+0000", 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000139161 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.094855+0000", 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0048849000000000002 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.107422+0000", 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.012566598999999999 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.572 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.107470+0000", 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8016999999999998e-05 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:47:05.107570+0000", 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010042799999999999 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:49:06.573 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:49:06.574 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:49:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.575 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:06.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:06.575 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:06.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:06.701 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:06.804 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:06.829 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:06.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:06.962 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:06.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:06.997 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:06 smithi178 bash[29889]: cluster 2024-09-13T22:49:06.675611+0000 mgr.y (mgr.14613) 424 : cluster [DBG] pgmap v472: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:49:06.997 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:06 smithi178 bash[22840]: cluster 2024-09-13T22:49:06.675611+0000 mgr.y (mgr.14613) 424 : cluster [DBG] pgmap v472: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:49:07.065 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:07.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:07.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:06 smithi192 bash[24608]: cluster 2024-09-13T22:49:06.675611+0000 mgr.y (mgr.14613) 424 : cluster [DBG] pgmap v472: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T22:49:07.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.222 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:07.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.351 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:07.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:07.481 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:07.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:07.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:07.844 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:07.870 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:07.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:07.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:08.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:08.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:08.133 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:08.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:08.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:08.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:08.392 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:08.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T22:49:08.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:08.518 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:08.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:08.649 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:08.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:08.778 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:08.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:08.907 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:08.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:09.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:09.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:08 smithi192 bash[24608]: cluster 2024-09-13T22:49:08.676031+0000 mgr.y (mgr.14613) 425 : cluster [DBG] pgmap v473: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-13T22:49:09.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:09.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:09.150 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:08 smithi178 bash[29889]: cluster 2024-09-13T22:49:08.676031+0000 mgr.y (mgr.14613) 425 : cluster [DBG] pgmap v473: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-13T22:49:09.150 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:08 smithi178 bash[22840]: cluster 2024-09-13T22:49:08.676031+0000 mgr.y (mgr.14613) 425 : cluster [DBG] pgmap v473: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-13T22:49:09.166 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:09.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:09.295 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:09.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:09.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:09.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:09.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:09.657 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:09.684 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:09.787 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:09.808 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:09.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:09.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:49:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:49:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:49:09.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:09.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:10.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:10.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:10.131 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_1' removed 2024-09-13T22:49:10.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:10.183 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:10 smithi192 bash[24608]: audit 2024-09-13T22:49:10.067988+0000 mon.c (mon.2) 47 : audit [INF] from='client.? v1:172.21.15.178:0/4181350596' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:49:10.183 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:10 smithi192 bash[24608]: audit 2024-09-13T22:49:10.068269+0000 mon.a (mon.0) 1253 : audit [INF] from='client.25238 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:49:10.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:10.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:10.319 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:10.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:10.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:10 smithi178 bash[29889]: audit 2024-09-13T22:49:10.067988+0000 mon.c (mon.2) 47 : audit [INF] from='client.? v1:172.21.15.178:0/4181350596' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:49:10.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:10 smithi178 bash[29889]: audit 2024-09-13T22:49:10.068269+0000 mon.a (mon.0) 1253 : audit [INF] from='client.25238 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:49:10.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:10 smithi178 bash[22840]: audit 2024-09-13T22:49:10.067988+0000 mon.c (mon.2) 47 : audit [INF] from='client.? v1:172.21.15.178:0/4181350596' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:49:10.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:10 smithi178 bash[22840]: audit 2024-09-13T22:49:10.068269+0000 mon.a (mon.0) 1253 : audit [INF] from='client.25238 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:49:10.435 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:49:10.435 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:49:10.440 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:10.453 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:49:10.454 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:49:10.472 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:49:10.473 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:49:10.489 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:49:10.489 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:49:10.509 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:49:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:49:10.529 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:49:10.529 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:49:10.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:10.551 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:49:10.552 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:49:10.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:10.573 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:49:10.573 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:49:10.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:10.690 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:10.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:10.818 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:10.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:10.948 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:10.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:11.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:11.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:11.138 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T22:49:11.139 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:49:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[29889]: audit 2024-09-13T22:49:10.120263+0000 mon.a (mon.0) 1254 : audit [INF] from='client.25238 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:49:11.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[29889]: cluster 2024-09-13T22:49:10.124790+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[29889]: audit 2024-09-13T22:49:10.222602+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? v1:172.21.15.178:0/851004176' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[29889]: audit 2024-09-13T22:49:10.419661+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.178:0/3069296018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[29889]: audit 2024-09-13T22:49:10.530065+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? v1:172.21.15.178:0/480814168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[29889]: cluster 2024-09-13T22:49:10.676929+0000 mgr.y (mgr.14613) 426 : cluster [DBG] pgmap v475: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[22840]: audit 2024-09-13T22:49:10.120263+0000 mon.a (mon.0) 1254 : audit [INF] from='client.25238 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[22840]: cluster 2024-09-13T22:49:10.124790+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[22840]: audit 2024-09-13T22:49:10.222602+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? v1:172.21.15.178:0/851004176' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[22840]: audit 2024-09-13T22:49:10.419661+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.178:0/3069296018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[22840]: audit 2024-09-13T22:49:10.530065+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? v1:172.21.15.178:0/480814168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:49:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:11 smithi178 bash[22840]: cluster 2024-09-13T22:49:10.676929+0000 mgr.y (mgr.14613) 426 : cluster [DBG] pgmap v475: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:49:11.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:11.207 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:11.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:11.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:11 smithi192 bash[24608]: audit 2024-09-13T22:49:10.120263+0000 mon.a (mon.0) 1254 : audit [INF] from='client.25238 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:49:11.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:11 smithi192 bash[24608]: cluster 2024-09-13T22:49:10.124790+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-09-13T22:49:11.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:11 smithi192 bash[24608]: audit 2024-09-13T22:49:10.222602+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? v1:172.21.15.178:0/851004176' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:49:11.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:11 smithi192 bash[24608]: audit 2024-09-13T22:49:10.419661+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.178:0/3069296018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:11.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:11 smithi192 bash[24608]: audit 2024-09-13T22:49:10.530065+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? v1:172.21.15.178:0/480814168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:49:11.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:11 smithi192 bash[24608]: cluster 2024-09-13T22:49:10.676929+0000 mgr.y (mgr.14613) 426 : cluster [DBG] pgmap v475: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:49:11.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:11.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:11.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:11.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:11.616 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T22:49:11.617 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T22:49:11.617 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-13T22:49:11.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-13T22:49:11.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:11.726 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:11.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:11.854 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:11.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:11.980 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:11.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:12.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:12.110 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:12.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:12.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[29889]: cluster 2024-09-13T22:49:11.135369+0000 mon.a (mon.0) 1258 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:12.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[29889]: audit 2024-09-13T22:49:11.137266+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? v1:172.21.15.178:0/851004176' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:49:12.224 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[29889]: audit 2024-09-13T22:49:11.137368+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.178:0/480814168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[29889]: cluster 2024-09-13T22:49:11.145031+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[29889]: cluster 2024-09-13T22:49:11.179670+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[29889]: cluster 2024-09-13T22:49:11.188380+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[22840]: cluster 2024-09-13T22:49:11.135369+0000 mon.a (mon.0) 1258 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[22840]: audit 2024-09-13T22:49:11.137266+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? v1:172.21.15.178:0/851004176' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[22840]: audit 2024-09-13T22:49:11.137368+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.178:0/480814168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[22840]: cluster 2024-09-13T22:49:11.145031+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[22840]: cluster 2024-09-13T22:49:11.179670+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:49:12.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:12 smithi178 bash[22840]: cluster 2024-09-13T22:49:11.188380+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:49:12.240 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:12.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:12.372 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:12 smithi192 bash[24608]: cluster 2024-09-13T22:49:11.135369+0000 mon.a (mon.0) 1258 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:12 smithi192 bash[24608]: audit 2024-09-13T22:49:11.137266+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? v1:172.21.15.178:0/851004176' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:49:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:12 smithi192 bash[24608]: audit 2024-09-13T22:49:11.137368+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? v1:172.21.15.178:0/480814168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:49:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:12 smithi192 bash[24608]: cluster 2024-09-13T22:49:11.145031+0000 mon.a (mon.0) 1261 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-09-13T22:49:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:12 smithi192 bash[24608]: cluster 2024-09-13T22:49:11.179670+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:49:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:12 smithi192 bash[24608]: cluster 2024-09-13T22:49:11.188380+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:49:12.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:12.503 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:12.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:12.635 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:12.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:12.764 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:12.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:12.894 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:12.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:12.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:13.025 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:13.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:13.154 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:13.241 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:49:13.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:13.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:13.339 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:13 smithi192 bash[24608]: cluster 2024-09-13T22:49:11.968016+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.f scrub starts 2024-09-13T22:49:13.339 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:13 smithi192 bash[24608]: cluster 2024-09-13T22:49:11.973538+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.f scrub ok 2024-09-13T22:49:13.339 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:13 smithi192 bash[24608]: cluster 2024-09-13T22:49:12.677630+0000 mgr.y (mgr.14613) 427 : cluster [DBG] pgmap v477: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:13.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:13.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:13 smithi178 bash[29889]: cluster 2024-09-13T22:49:11.968016+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.f scrub starts 2024-09-13T22:49:13.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:13 smithi178 bash[29889]: cluster 2024-09-13T22:49:11.973538+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.f scrub ok 2024-09-13T22:49:13.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:13 smithi178 bash[29889]: cluster 2024-09-13T22:49:12.677630+0000 mgr.y (mgr.14613) 427 : cluster [DBG] pgmap v477: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:13.403 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:13 smithi178 bash[22840]: cluster 2024-09-13T22:49:11.968016+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.f scrub starts 2024-09-13T22:49:13.403 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:13 smithi178 bash[22840]: cluster 2024-09-13T22:49:11.973538+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.f scrub ok 2024-09-13T22:49:13.403 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:13 smithi178 bash[22840]: cluster 2024-09-13T22:49:12.677630+0000 mgr.y (mgr.14613) 427 : cluster [DBG] pgmap v477: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:13.414 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:13.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:13.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:13.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:13.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:13.715 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T22:49:13.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:49:13.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:13.809 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:13.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:13.936 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:13.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:14.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:14.065 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:14.169 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:14.197 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:14.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:14.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:14.432 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.457 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:14.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:14.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:14.584 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:14.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:14.714 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:14.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:14.844 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:14.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:14 smithi178 bash[29889]: cluster 2024-09-13T22:49:13.906718+0000 osd.7 (osd.7) 241 : cluster [DBG] 3.d scrub starts 2024-09-13T22:49:14.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:14 smithi178 bash[29889]: cluster 2024-09-13T22:49:13.909759+0000 osd.7 (osd.7) 242 : cluster [DBG] 3.d scrub ok 2024-09-13T22:49:14.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:14 smithi178 bash[22840]: cluster 2024-09-13T22:49:13.906718+0000 osd.7 (osd.7) 241 : cluster [DBG] 3.d scrub starts 2024-09-13T22:49:14.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:14 smithi178 bash[22840]: cluster 2024-09-13T22:49:13.909759+0000 osd.7 (osd.7) 242 : cluster [DBG] 3.d scrub ok 2024-09-13T22:49:14.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:14.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:14.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:15.054 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:14 smithi192 bash[24608]: cluster 2024-09-13T22:49:13.906718+0000 osd.7 (osd.7) 241 : cluster [DBG] 3.d scrub starts 2024-09-13T22:49:15.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:14 smithi192 bash[24608]: cluster 2024-09-13T22:49:13.909759+0000 osd.7 (osd.7) 242 : cluster [DBG] 3.d scrub ok 2024-09-13T22:49:15.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:15.102 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:15.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:15.228 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:15.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:15.234 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:49:15.234 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:49:15.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:15.357 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:15.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:49:15.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:15.485 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:15.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:15.599 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:49:15.614 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:15.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:49:15.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:15.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:15.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:15.880 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.147442+0000 osd.6 (osd.6) 189 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:49:15.880 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.149182+0000 osd.6 (osd.6) 190 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:49:15.880 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.399113+0000 osd.1 (osd.1) 235 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:49:15.881 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.400983+0000 osd.1 (osd.1) 236 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:49:15.881 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.427594+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.c scrub starts 2024-09-13T22:49:15.881 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.431527+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.c scrub ok 2024-09-13T22:49:15.881 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: cluster 2024-09-13T22:49:14.678837+0000 mgr.y (mgr.14613) 428 : cluster [DBG] pgmap v478: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T22:49:15.881 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:15 smithi192 bash[24608]: audit 2024-09-13T22:49:15.371833+0000 mgr.y (mgr.14613) 429 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:15.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:15.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:15.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T22:49:16.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.147442+0000 osd.6 (osd.6) 189 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.149182+0000 osd.6 (osd.6) 190 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.399113+0000 osd.1 (osd.1) 235 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.400983+0000 osd.1 (osd.1) 236 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.427594+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.c scrub starts 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.431527+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.c scrub ok 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: cluster 2024-09-13T22:49:14.678837+0000 mgr.y (mgr.14613) 428 : cluster [DBG] pgmap v478: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[29889]: audit 2024-09-13T22:49:15.371833+0000 mgr.y (mgr.14613) 429 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.147442+0000 osd.6 (osd.6) 189 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:49:16.011 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.149182+0000 osd.6 (osd.6) 190 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:49:16.012 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.399113+0000 osd.1 (osd.1) 235 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:49:16.012 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.400983+0000 osd.1 (osd.1) 236 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:49:16.012 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.427594+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.c scrub starts 2024-09-13T22:49:16.012 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.431527+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.c scrub ok 2024-09-13T22:49:16.012 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: cluster 2024-09-13T22:49:14.678837+0000 mgr.y (mgr.14613) 428 : cluster [DBG] pgmap v478: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T22:49:16.012 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:15 smithi178 bash[22840]: audit 2024-09-13T22:49:15.371833+0000 mgr.y (mgr.14613) 429 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:16.025 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:16.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:16.153 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:16.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:16.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:16.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:16.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:16.452 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:16.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:16.538 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:16.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:16.669 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:16.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:16.798 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:16.800 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[29889]: cluster 2024-09-13T22:49:15.906771+0000 osd.7 (osd.7) 243 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:49:16.800 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[29889]: cluster 2024-09-13T22:49:15.907795+0000 osd.7 (osd.7) 244 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:49:16.800 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.304740+0000 osd.3 (osd.3) 227 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:49:16.800 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.306957+0000 osd.3 (osd.3) 228 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:49:16.800 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.679926+0000 mgr.y (mgr.14613) 430 : cluster [DBG] pgmap v479: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:16.801 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[22840]: cluster 2024-09-13T22:49:15.906771+0000 osd.7 (osd.7) 243 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:49:16.801 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[22840]: cluster 2024-09-13T22:49:15.907795+0000 osd.7 (osd.7) 244 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:49:16.801 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.304740+0000 osd.3 (osd.3) 227 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:49:16.801 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.306957+0000 osd.3 (osd.3) 228 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:49:16.801 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:16 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.679926+0000 mgr.y (mgr.14613) 430 : cluster [DBG] pgmap v479: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:16.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:16.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:16 smithi192 bash[24608]: cluster 2024-09-13T22:49:15.906771+0000 osd.7 (osd.7) 243 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:49:16.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:16 smithi192 bash[24608]: cluster 2024-09-13T22:49:15.907795+0000 osd.7 (osd.7) 244 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:49:16.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:16 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.304740+0000 osd.3 (osd.3) 227 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:49:16.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:16 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.306957+0000 osd.3 (osd.3) 228 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:49:16.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:16 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.679926+0000 mgr.y (mgr.14613) 430 : cluster [DBG] pgmap v479: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:16.930 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:16.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:17.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:17.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:17.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:17.189 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:17.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:17.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:17.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:17.467 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:17.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:17.598 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:17.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:17.729 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:17.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:17.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.387526+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:49:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.389919+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:49:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.932485+0000 osd.7 (osd.7) 245 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:49:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:16.934039+0000 osd.7 (osd.7) 246 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:49:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.141728+0000 osd.0 (osd.0) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.141837+0000 osd.0 (osd.0) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:17.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.275421+0000 osd.3 (osd.3) 229 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:49:17.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:17 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.277643+0000 osd.3 (osd.3) 230 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:49:17.962 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:17.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.387526+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.389919+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.932485+0000 osd.7 (osd.7) 245 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:16.934039+0000 osd.7 (osd.7) 246 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.141728+0000 osd.0 (osd.0) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.141837+0000 osd.0 (osd.0) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.275421+0000 osd.3 (osd.3) 229 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.277643+0000 osd.3 (osd.3) 230 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.387526+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.389919+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.932485+0000 osd.7 (osd.7) 245 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:49:17.975 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:16.934039+0000 osd.7 (osd.7) 246 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:49:17.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.141728+0000 osd.0 (osd.0) 229 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:17.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.141837+0000 osd.0 (osd.0) 230 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:17.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.275421+0000 osd.3 (osd.3) 229 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:49:17.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:17 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.277643+0000 osd.3 (osd.3) 230 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:49:17.992 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:17.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:18.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:18.123 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:18.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:18.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:18.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:18.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:18.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:18.497 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:18.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:18.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:18.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:18.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:18.769 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:18.811 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_2' created 2024-09-13T22:49:18.830 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:18 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.708138+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-09-13T22:49:18.831 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:18 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.708138+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-09-13T22:49:18.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:18.890 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:18.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:18.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:18 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.708138+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-09-13T22:49:18.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:19.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:19.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:19.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:19.267 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:19.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:19.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:19.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:19.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:19.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:19.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:19.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:19.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:19.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:19.901 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:49:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:49:19.901 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.956770+0000 osd.7 (osd.7) 247 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:49:19.901 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: cluster 2024-09-13T22:49:17.960491+0000 osd.7 (osd.7) 248 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:49:19.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: audit 2024-09-13T22:49:18.319488+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.178:0/2942708971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-13T22:49:19.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: audit 2024-09-13T22:49:18.541684+0000 mon.b (mon.1) 176 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:19.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: cluster 2024-09-13T22:49:18.680541+0000 mgr.y (mgr.14613) 431 : cluster [DBG] pgmap v481: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:49:19.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: audit 2024-09-13T22:49:18.813491+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? v1:172.21.15.178:0/2942708971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T22:49:19.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: cluster 2024-09-13T22:49:18.816599+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-09-13T22:49:19.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: cluster 2024-09-13T22:49:19.164604+0000 osd.0 (osd.0) 231 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:49:19.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[29889]: cluster 2024-09-13T22:49:19.165928+0000 osd.0 (osd.0) 232 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:49:19.903 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.956770+0000 osd.7 (osd.7) 247 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:49:19.903 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: cluster 2024-09-13T22:49:17.960491+0000 osd.7 (osd.7) 248 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:49:19.903 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: audit 2024-09-13T22:49:18.319488+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.178:0/2942708971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-13T22:49:19.904 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: audit 2024-09-13T22:49:18.541684+0000 mon.b (mon.1) 176 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:19.904 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: cluster 2024-09-13T22:49:18.680541+0000 mgr.y (mgr.14613) 431 : cluster [DBG] pgmap v481: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:49:19.904 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: audit 2024-09-13T22:49:18.813491+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? v1:172.21.15.178:0/2942708971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T22:49:19.904 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: cluster 2024-09-13T22:49:18.816599+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-09-13T22:49:19.904 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: cluster 2024-09-13T22:49:19.164604+0000 osd.0 (osd.0) 231 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:49:19.904 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:19 smithi178 bash[22840]: cluster 2024-09-13T22:49:19.165928+0000 osd.0 (osd.0) 232 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:49:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:19.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:20.018 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:20.039 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:20.079 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-13T22:49:20.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.956770+0000 osd.7 (osd.7) 247 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: cluster 2024-09-13T22:49:17.960491+0000 osd.7 (osd.7) 248 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: audit 2024-09-13T22:49:18.319488+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? v1:172.21.15.178:0/2942708971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: audit 2024-09-13T22:49:18.541684+0000 mon.b (mon.1) 176 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: cluster 2024-09-13T22:49:18.680541+0000 mgr.y (mgr.14613) 431 : cluster [DBG] pgmap v481: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: audit 2024-09-13T22:49:18.813491+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? v1:172.21.15.178:0/2942708971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: cluster 2024-09-13T22:49:18.816599+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: cluster 2024-09-13T22:49:19.164604+0000 osd.0 (osd.0) 231 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:49:20.153 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:19 smithi192 bash[24608]: cluster 2024-09-13T22:49:19.165928+0000 osd.0 (osd.0) 232 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:49:20.169 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:20.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.297 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:20.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.427 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:20.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:20.558 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:49:20.563 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:49:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:20.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:20.566 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:20.567 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:20.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.694 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:20.742 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:49:20.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.822 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:20.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:20.953 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:20.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:21.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: cluster 2024-09-13T22:49:19.254842+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: cluster 2024-09-13T22:49:19.256278+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: cluster 2024-09-13T22:49:19.818649+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.403991+0000 osd.3 (osd.3) 231 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.405498+0000 osd.3 (osd.3) 232 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: audit 2024-09-13T22:49:20.541620+0000 mgr.y (mgr.14613) 432 : audit [DBG] from='client.25268 v1:172.21.15.178:0/3687654780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:49:21.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:20 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.681286+0000 mgr.y (mgr.14613) 433 : cluster [DBG] pgmap v484: 158 pgs: 6 creating+peering, 3 unknown, 149 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T22:49:21.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: cluster 2024-09-13T22:49:19.254842+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: cluster 2024-09-13T22:49:19.256278+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: cluster 2024-09-13T22:49:19.818649+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.403991+0000 osd.3 (osd.3) 231 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.405498+0000 osd.3 (osd.3) 232 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: audit 2024-09-13T22:49:20.541620+0000 mgr.y (mgr.14613) 432 : audit [DBG] from='client.25268 v1:172.21.15.178:0/3687654780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.681286+0000 mgr.y (mgr.14613) 433 : cluster [DBG] pgmap v484: 158 pgs: 6 creating+peering, 3 unknown, 149 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T22:49:21.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: cluster 2024-09-13T22:49:19.254842+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:49:21.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: cluster 2024-09-13T22:49:19.256278+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:49:21.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: cluster 2024-09-13T22:49:19.818649+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-09-13T22:49:21.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.403991+0000 osd.3 (osd.3) 231 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:49:21.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.405498+0000 osd.3 (osd.3) 232 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:49:21.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: audit 2024-09-13T22:49:20.541620+0000 mgr.y (mgr.14613) 432 : audit [DBG] from='client.25268 v1:172.21.15.178:0/3687654780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:49:21.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:20 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.681286+0000 mgr.y (mgr.14613) 433 : cluster [DBG] pgmap v484: 158 pgs: 6 creating+peering, 3 unknown, 149 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T22:49:21.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:21.212 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:21.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:21.342 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:21.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:21.473 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:21.576 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:21.603 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:21.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:21.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:21.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:21.867 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:21.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:21.996 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:21.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:21.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:22.094 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.262716+0000 osd.2 (osd.2) 189 : cluster [DBG] 6.f scrub starts 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.263921+0000 osd.2 (osd.2) 190 : cluster [DBG] 6.f scrub ok 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.815740+0000 mon.a (mon.0) 1267 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.832149+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.973335+0000 osd.7 (osd.7) 249 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:20.975453+0000 osd.7 (osd.7) 250 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:21.192791+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:49:22.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:21 smithi192 bash[24608]: cluster 2024-09-13T22:49:21.195021+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:49:22.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.262716+0000 osd.2 (osd.2) 189 : cluster [DBG] 6.f scrub starts 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.263921+0000 osd.2 (osd.2) 190 : cluster [DBG] 6.f scrub ok 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.815740+0000 mon.a (mon.0) 1267 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.832149+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.973335+0000 osd.7 (osd.7) 249 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:20.975453+0000 osd.7 (osd.7) 250 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:21.192791+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:49:22.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[22840]: cluster 2024-09-13T22:49:21.195021+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.262716+0000 osd.2 (osd.2) 189 : cluster [DBG] 6.f scrub starts 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.263921+0000 osd.2 (osd.2) 190 : cluster [DBG] 6.f scrub ok 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.815740+0000 mon.a (mon.0) 1267 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.832149+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.973335+0000 osd.7 (osd.7) 249 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:20.975453+0000 osd.7 (osd.7) 250 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:21.192791+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:49:22.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:21 smithi178 bash[29889]: cluster 2024-09-13T22:49:21.195021+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:49:22.117 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-13T22:49:19.817624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1835,"num_read_kb":1587,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2304,"ondisk_log_size":2304,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":429,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2044292,"kb_used_data":36468,"kb_used_omap":10,"kb_used_meta":2007669,"kb_avail":654171772,"statfs":{"total":671965249536,"available":669871894528,"internally_reserved":0,"allocated":37343232,"data_stored":34569151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2055853163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.140389"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":382,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174451+0000","last_change":"2024-09-13T22:45:02.514835+0000","last_active":"2024-09-13T22:49:11.174451+0000","last_peered":"2024-09-13T22:49:11.174451+0000","last_clean":"2024-09-13T22:49:11.174451+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:49:11.174451+0000","last_undegraded":"2024-09-13T22:49:11.174451+0000","last_fullsized":"2024-09-13T22:49:11.174451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.514742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.153808+0000","last_change":"2024-09-13T22:47:03.935110+0000","last_active":"2024-09-13T22:49:11.153808+0000","last_peered":"2024-09-13T22:49:11.153808+0000","last_clean":"2024-09-13T22:49:11.153808+0000","last_became_active":"2024-09-13T22:47:03.934779+0000","last_became_peered":"2024-09-13T22:47:03.934779+0000","last_unstale":"2024-09-13T22:49:11.153808+0000","last_undegraded":"2024-09-13T22:49:11.153808+0000","last_fullsized":"2024-09-13T22:49:11.153808+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:54.988990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.988990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"228'25","reported_seq":434,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175335+0000","last_change":"2024-09-13T22:47:03.372631+0000","last_active":"2024-09-13T22:49:11.175335+0000","last_peered":"2024-09-13T22:49:11.175335+0000","last_clean":"2024-09-13T22:49:11.175335+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:49:11.175335+0000","last_undegraded":"2024-09-13T22:49:11.175335+0000","last_fullsized":"2024-09-13T22:49:11.175335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:47:03.372362+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:47:03.372362+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":100,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823424+0000","last_change":"2024-09-13T22:48:14.210156+0000","last_active":"2024-09-13T22:49:18.823424+0000","last_peered":"2024-09-13T22:49:18.823424+0000","last_clean":"2024-09-13T22:49:18.823424+0000","last_became_active":"2024-09-13T22:47:03.898560+0000","last_became_peered":"2024-09-13T22:47:03.898560+0000","last_unstale":"2024-09-13T22:49:18.823424+0000","last_undegraded":"2024-09-13T22:49:18.823424+0000","last_fullsized":"2024-09-13T22:49:18.823424+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.210087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.210087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:34.726534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":395,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175433+0000","last_change":"2024-09-13T22:45:04.482417+0000","last_active":"2024-09-13T22:49:11.175433+0000","last_peered":"2024-09-13T22:49:11.175433+0000","last_clean":"2024-09-13T22:49:11.175433+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:49:11.175433+0000","last_undegraded":"2024-09-13T22:49:11.175433+0000","last_fullsized":"2024-09-13T22:49:11.175433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:04.482214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:45:04.482214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"228'24","reported_seq":435,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.153537+0000","last_change":"2024-09-13T22:46:06.959087+0000","last_active":"2024-09-13T22:49:11.153537+0000","last_peered":"2024-09-13T22:49:11.153537+0000","last_clean":"2024-09-13T22:49:11.153537+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:49:11.153537+0000","last_undegraded":"2024-09-13T22:49:11.153537+0000","last_fullsized":"2024-09-13T22:49:11.153537+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'16","last_scrub_stamp":"2024-09-13T22:46:06.958946+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:46:06.958946+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":398,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820789+0000","last_change":"2024-09-13T22:48:30.135471+0000","last_active":"2024-09-13T22:49:18.820789+0000","last_peered":"2024-09-13T22:49:18.820789+0000","last_clean":"2024-09-13T22:49:18.820789+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:49:18.820789+0000","last_undegraded":"2024-09-13T22:49:18.820789+0000","last_fullsized":"2024-09-13T22:49:18.820789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.135258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.135258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:49.921055+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":100,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.834524+0000","last_change":"2024-09-13T22:48:27.804033+0000","last_active":"2024-09-13T22:49:18.834524+0000","last_peered":"2024-09-13T22:49:18.834524+0000","last_clean":"2024-09-13T22:49:18.834524+0000","last_became_active":"2024-09-13T22:47:03.927913+0000","last_became_peered":"2024-09-13T22:47:03.927913+0000","last_unstale":"2024-09-13T22:49:18.834524+0000","last_undegraded":"2024-09-13T22:49:18.834524+0000","last_fullsized":"2024-09-13T22:49:18.834524+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.803924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.803924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:33.275233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":403,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.711041+0000","last_change":"2024-09-13T22:48:11.350608+0000","last_active":"2024-09-13T22:49:17.711041+0000","last_peered":"2024-09-13T22:49:17.711041+0000","last_clean":"2024-09-13T22:49:17.711041+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:49:17.711041+0000","last_undegraded":"2024-09-13T22:49:17.711041+0000","last_fullsized":"2024-09-13T22:49:17.711041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.350541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.350541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:18.064891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"228'34","reported_seq":481,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836213+0000","last_change":"2024-09-13T22:48:22.907608+0000","last_active":"2024-09-13T22:49:18.836213+0000","last_peered":"2024-09-13T22:49:18.836213+0000","last_clean":"2024-09-13T22:49:18.836213+0000","last_became_active":"2024-09-13T22:47:03.933115+0000","last_became_peered":"2024-09-13T22:47:03.933115+0000","last_unstale":"2024-09-13T22:49:18.836213+0000","last_undegraded":"2024-09-13T22:49:18.836213+0000","last_fullsized":"2024-09-13T22:49:18.836213+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:22.907495+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.907495+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:46.887648+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":393,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709956+0000","last_change":"2024-09-13T22:48:10.365596+0000","last_active":"2024-09-13T22:49:17.709956+0000","last_peered":"2024-09-13T22:49:17.709956+0000","last_clean":"2024-09-13T22:49:17.709956+0000","last_became_active":"2024-09-13T22:47:03.896417+0000","last_became_peered":"2024-09-13T22:47:03.896417+0000","last_unstale":"2024-09-13T22:49:17.709956+0000","last_undegraded":"2024-09-13T22:49:17.709956+0000","last_fullsized":"2024-09-13T22:49:17.709956+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:10.365482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:48:10.365482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:21.770530+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":393,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835242+0000","last_change":"2024-09-13T22:48:03.951130+0000","last_active":"2024-09-13T22:49:18.835242+0000","last_peered":"2024-09-13T22:49:18.835242+0000","last_clean":"2024-09-13T22:49:18.835242+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:49:18.835242+0000","last_undegraded":"2024-09-13T22:49:18.835242+0000","last_fullsized":"2024-09-13T22:49:18.835242+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:03.950989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:48:03.950989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:30.672678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":403,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710715+0000","last_change":"2024-09-13T22:47:03.640700+0000","last_active":"2024-09-13T22:49:17.710715+0000","last_peered":"2024-09-13T22:49:17.710715+0000","last_clean":"2024-09-13T22:49:17.710715+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:49:17.710715+0000","last_undegraded":"2024-09-13T22:49:17.710715+0000","last_fullsized":"2024-09-13T22:49:17.710715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:03.640577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:47:03.640577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"228'18","reported_seq":459,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836358+0000","last_change":"2024-09-13T22:48:05.917260+0000","last_active":"2024-09-13T22:49:18.836358+0000","last_peered":"2024-09-13T22:49:18.836358+0000","last_clean":"2024-09-13T22:49:18.836358+0000","last_became_active":"2024-09-13T22:47:03.937556+0000","last_became_peered":"2024-09-13T22:47:03.937556+0000","last_unstale":"2024-09-13T22:49:18.836358+0000","last_undegraded":"2024-09-13T22:49:18.836358+0000","last_fullsized":"2024-09-13T22:49:18.836358+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:05.917128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:05.917128+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:26.145188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":397,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835086+0000","last_change":"2024-09-13T22:46:47.974936+0000","last_active":"2024-09-13T22:49:18.835086+0000","last_peered":"2024-09-13T22:49:18.835086+0000","last_clean":"2024-09-13T22:49:18.835086+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:49:18.835086+0000","last_undegraded":"2024-09-13T22:49:18.835086+0000","last_fullsized":"2024-09-13T22:49:18.835086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:47.974834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:47.974834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":390,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.150698+0000","last_change":"2024-09-13T22:46:02.912752+0000","last_active":"2024-09-13T22:49:11.150698+0000","last_peered":"2024-09-13T22:49:11.150698+0000","last_clean":"2024-09-13T22:49:11.150698+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:49:11.150698+0000","last_undegraded":"2024-09-13T22:49:11.150698+0000","last_fullsized":"2024-09-13T22:49:11.150698+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:02.912649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.912649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-09-13T22:49:18.819334+0000","last_change":"2024-09-13T22:49:18.819334+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.819334+0000","last_undegraded":"2024-09-13T22:49:18.819334+0000","last_fullsized":"2024-09-13T22:49:18.819334+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":459,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.832073+0000","last_change":"2024-09-13T22:48:36.940636+0000","last_active":"2024-09-13T22:49:18.832073+0000","last_peered":"2024-09-13T22:49:18.832073+0000","last_clean":"2024-09-13T22:49:18.832073+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:49:18.832073+0000","last_undegraded":"2024-09-13T22:49:18.832073+0000","last_fullsized":"2024-09-13T22:49:18.832073+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:36.940280+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:36.940280+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:51.842501+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"228'44","reported_seq":508,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.825574+0000","last_change":"2024-09-13T22:48:11.242327+0000","last_active":"2024-09-13T22:49:18.825574+0000","last_peered":"2024-09-13T22:49:18.825574+0000","last_clean":"2024-09-13T22:49:18.825574+0000","last_became_active":"2024-09-13T22:47:03.918330+0000","last_became_peered":"2024-09-13T22:47:03.918330+0000","last_unstale":"2024-09-13T22:49:18.825574+0000","last_undegraded":"2024-09-13T22:49:18.825574+0000","last_fullsized":"2024-09-13T22:49:18.825574+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:48:11.242204+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.242204+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:38.853596+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":402,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709953+0000","last_change":"2024-09-13T22:47:03.927571+0000","last_active":"2024-09-13T22:49:17.709953+0000","last_peered":"2024-09-13T22:49:17.709953+0000","last_clean":"2024-09-13T22:49:17.709953+0000","last_became_active":"2024-09-13T22:47:03.927239+0000","last_became_peered":"2024-09-13T22:47:03.927239+0000","last_unstale":"2024-09-13T22:49:17.709953+0000","last_undegraded":"2024-09-13T22:49:17.709953+0000","last_fullsized":"2024-09-13T22:49:17.709953+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:46:09.789477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821447+0000","last_change":"2024-09-13T22:47:09.036265+0000","last_active":"2024-09-13T22:49:18.821447+0000","last_peered":"2024-09-13T22:49:18.821447+0000","last_clean":"2024-09-13T22:49:18.821447+0000","last_became_active":"2024-09-13T22:47:03.899126+0000","last_became_peered":"2024-09-13T22:47:03.899126+0000","last_unstale":"2024-09-13T22:49:18.821447+0000","last_undegraded":"2024-09-13T22:49:18.821447+0000","last_fullsized":"2024-09-13T22:49:18.821447+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:09.036143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:47:09.036143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":409,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824385+0000","last_change":"2024-09-13T22:47:03.929036+0000","last_active":"2024-09-13T22:49:18.824385+0000","last_peered":"2024-09-13T22:49:18.824385+0000","last_clean":"2024-09-13T22:49:18.824385+0000","last_became_active":"2024-09-13T22:47:03.928698+0000","last_became_peered":"2024-09-13T22:47:03.928698+0000","last_unstale":"2024-09-13T22:49:18.824385+0000","last_undegraded":"2024-09-13T22:49:18.824385+0000","last_fullsized":"2024-09-13T22:49:18.824385+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:53.000161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:53.000161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":674,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.707946+0000","last_change":"2024-09-13T22:45:59.823771+0000","last_active":"2024-09-13T22:49:17.707946+0000","last_peered":"2024-09-13T22:49:17.707946+0000","last_clean":"2024-09-13T22:49:17.707946+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:49:17.707946+0000","last_undegraded":"2024-09-13T22:49:17.707946+0000","last_fullsized":"2024-09-13T22:49:17.707946+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.823697+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.823697+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"228'19","reported_seq":444,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174740+0000","last_change":"2024-09-13T22:48:13.728920+0000","last_active":"2024-09-13T22:49:11.174740+0000","last_peered":"2024-09-13T22:49:11.174740+0000","last_clean":"2024-09-13T22:49:11.174740+0000","last_became_active":"2024-09-13T22:47:03.889214+0000","last_became_peered":"2024-09-13T22:47:03.889214+0000","last_unstale":"2024-09-13T22:49:11.174740+0000","last_undegraded":"2024-09-13T22:49:11.174740+0000","last_fullsized":"2024-09-13T22:49:11.174740+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:13.725631+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.725631+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.219357+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":407,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.822939+0000","last_change":"2024-09-13T22:47:14.139433+0000","last_active":"2024-09-13T22:49:18.822939+0000","last_peered":"2024-09-13T22:49:18.822939+0000","last_clean":"2024-09-13T22:49:18.822939+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:49:18.822939+0000","last_undegraded":"2024-09-13T22:49:18.822939+0000","last_fullsized":"2024-09-13T22:49:18.822939+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:14.139219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:47:14.139219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":557,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823565+0000","last_change":"2024-09-13T22:48:28.189734+0000","last_active":"2024-09-13T22:49:18.823565+0000","last_peered":"2024-09-13T22:49:18.823565+0000","last_clean":"2024-09-13T22:49:18.823565+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:49:18.823565+0000","last_undegraded":"2024-09-13T22:49:18.823565+0000","last_fullsized":"2024-09-13T22:49:18.823565+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:28.189507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.189507+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.536800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":101,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821625+0000","last_change":"2024-09-13T22:48:26.091094+0000","last_active":"2024-09-13T22:49:18.821625+0000","last_peered":"2024-09-13T22:49:18.821625+0000","last_clean":"2024-09-13T22:49:18.821625+0000","last_became_active":"2024-09-13T22:47:03.901543+0000","last_became_peered":"2024-09-13T22:47:03.901543+0000","last_unstale":"2024-09-13T22:49:18.821625+0000","last_undegraded":"2024-09-13T22:49:18.821625+0000","last_fullsized":"2024-09-13T22:49:18.821625+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.090972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.090972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:32.074163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":459,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836067+0000","last_change":"2024-09-13T22:48:24.860899+0000","last_active":"2024-09-13T22:49:18.836067+0000","last_peered":"2024-09-13T22:49:18.836067+0000","last_clean":"2024-09-13T22:49:18.836067+0000","last_became_active":"2024-09-13T22:45:58.880701+0000","last_became_peered":"2024-09-13T22:45:58.880701+0000","last_unstale":"2024-09-13T22:49:18.836067+0000","last_undegraded":"2024-09-13T22:49:18.836067+0000","last_fullsized":"2024-09-13T22:49:18.836067+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:24.860776+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.860776+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:53.211654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"228'27","reported_seq":442,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174626+0000","last_change":"2024-09-13T22:48:23.929452+0000","last_active":"2024-09-13T22:49:11.174626+0000","last_peered":"2024-09-13T22:49:11.174626+0000","last_clean":"2024-09-13T22:49:11.174626+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:49:11.174626+0000","last_undegraded":"2024-09-13T22:49:11.174626+0000","last_fullsized":"2024-09-13T22:49:11.174626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:48:23.929342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.929342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:43.893724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":415,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820994+0000","last_change":"2024-09-13T22:48:37.212544+0000","last_active":"2024-09-13T22:49:18.820994+0000","last_peered":"2024-09-13T22:49:18.820994+0000","last_clean":"2024-09-13T22:49:18.820994+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:49:18.820994+0000","last_undegraded":"2024-09-13T22:49:18.820994+0000","last_fullsized":"2024-09-13T22:49:18.820994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.201926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.201926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.622463+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"211'13","reported_seq":221,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836565+0000","last_change":"2024-09-13T22:48:08.989818+0000","last_active":"2024-09-13T22:49:18.836565+0000","last_peered":"2024-09-13T22:49:18.836565+0000","last_clean":"2024-09-13T22:49:18.836565+0000","last_became_active":"2024-09-13T22:47:03.901548+0000","last_became_peered":"2024-09-13T22:47:03.901548+0000","last_unstale":"2024-09-13T22:49:18.836565+0000","last_undegraded":"2024-09-13T22:49:18.836565+0000","last_fullsized":"2024-09-13T22:49:18.836565+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:08.989696+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.989696+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:27.865412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":394,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835667+0000","last_change":"2024-09-13T22:48:39.747979+0000","last_active":"2024-09-13T22:49:18.835667+0000","last_peered":"2024-09-13T22:49:18.835667+0000","last_clean":"2024-09-13T22:49:18.835667+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:49:18.835667+0000","last_undegraded":"2024-09-13T22:49:18.835667+0000","last_fullsized":"2024-09-13T22:49:18.835667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.747881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.747881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:56.271576+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"228'18","reported_seq":164,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.819683+0000","last_change":"2024-09-13T22:48:20.191655+0000","last_active":"2024-09-13T22:49:18.819683+0000","last_peered":"2024-09-13T22:49:18.819683+0000","last_clean":"2024-09-13T22:49:18.819683+0000","last_became_active":"2024-09-13T22:47:03.927850+0000","last_became_peered":"2024-09-13T22:47:03.927850+0000","last_unstale":"2024-09-13T22:49:18.819683+0000","last_undegraded":"2024-09-13T22:49:18.819683+0000","last_fullsized":"2024-09-13T22:49:18.819683+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:20.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.191528+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:38.634756+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":396,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708336+0000","last_change":"2024-09-13T22:47:02.999857+0000","last_active":"2024-09-13T22:49:17.708336+0000","last_peered":"2024-09-13T22:49:17.708336+0000","last_clean":"2024-09-13T22:49:17.708336+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:49:17.708336+0000","last_undegraded":"2024-09-13T22:49:17.708336+0000","last_fullsized":"2024-09-13T22:49:17.708336+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:02.999680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:47:02.999680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":207,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709978+0000","last_change":"2024-09-13T22:48:27.194380+0000","last_active":"2024-09-13T22:49:17.709978+0000","last_peered":"2024-09-13T22:49:17.709978+0000","last_clean":"2024-09-13T22:49:17.709978+0000","last_became_active":"2024-09-13T22:47:03.936969+0000","last_became_peered":"2024-09-13T22:47:03.936969+0000","last_unstale":"2024-09-13T22:49:17.709978+0000","last_undegraded":"2024-09-13T22:49:17.709978+0000","last_fullsized":"2024-09-13T22:49:17.709978+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.194219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.194219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:54.254732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":383,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174967+0000","last_change":"2024-09-13T22:45:03.478116+0000","last_active":"2024-09-13T22:49:11.174967+0000","last_peered":"2024-09-13T22:49:11.174967+0000","last_clean":"2024-09-13T22:49:11.174967+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:49:11.174967+0000","last_undegraded":"2024-09-13T22:49:11.174967+0000","last_fullsized":"2024-09-13T22:49:11.174967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.477932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.477932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":447,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175723+0000","last_change":"2024-09-13T22:45:59.360761+0000","last_active":"2024-09-13T22:49:11.175723+0000","last_peered":"2024-09-13T22:49:11.175723+0000","last_clean":"2024-09-13T22:49:11.175723+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:49:11.175723+0000","last_undegraded":"2024-09-13T22:49:11.175723+0000","last_fullsized":"2024-09-13T22:49:11.175723+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.360631+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"228'24","reported_seq":455,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.172850+0000","last_change":"2024-09-13T22:47:03.954215+0000","last_active":"2024-09-13T22:49:11.172850+0000","last_peered":"2024-09-13T22:49:11.172850+0000","last_clean":"2024-09-13T22:49:11.172850+0000","last_became_active":"2024-09-13T22:47:03.926045+0000","last_became_peered":"2024-09-13T22:47:03.926045+0000","last_unstale":"2024-09-13T22:49:11.172850+0000","last_undegraded":"2024-09-13T22:49:11.172850+0000","last_fullsized":"2024-09-13T22:49:11.172850+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:46:56.519330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:56.519330+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.826421+0000","last_change":"2024-09-13T22:48:19.112621+0000","last_active":"2024-09-13T22:49:18.826421+0000","last_peered":"2024-09-13T22:49:18.826421+0000","last_clean":"2024-09-13T22:49:18.826421+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:49:18.826421+0000","last_undegraded":"2024-09-13T22:49:18.826421+0000","last_fullsized":"2024-09-13T22:49:18.826421+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:19.112032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:19.112032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.571193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":396,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835285+0000","last_change":"2024-09-13T22:48:34.822821+0000","last_active":"2024-09-13T22:49:18.835285+0000","last_peered":"2024-09-13T22:49:18.835285+0000","last_clean":"2024-09-13T22:49:18.835285+0000","last_became_active":"2024-09-13T22:47:03.942652+0000","last_became_peered":"2024-09-13T22:47:03.942652+0000","last_unstale":"2024-09-13T22:49:18.835285+0000","last_undegraded":"2024-09-13T22:49:18.835285+0000","last_fullsized":"2024-09-13T22:49:18.835285+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:48:34.822707+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.822707+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.491217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"211'12","reported_seq":547,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175026+0000","last_change":"2024-09-13T22:47:03.895068+0000","last_active":"2024-09-13T22:49:11.175026+0000","last_peered":"2024-09-13T22:49:11.175026+0000","last_clean":"2024-09-13T22:49:11.175026+0000","last_became_active":"2024-09-13T22:47:03.881728+0000","last_became_peered":"2024-09-13T22:47:03.881728+0000","last_unstale":"2024-09-13T22:49:11.175026+0000","last_undegraded":"2024-09-13T22:49:11.175026+0000","last_fullsized":"2024-09-13T22:49:11.175026+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:38.494521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:38.494521+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":384,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708469+0000","last_change":"2024-09-13T22:46:56.706548+0000","last_active":"2024-09-13T22:49:17.708469+0000","last_peered":"2024-09-13T22:49:17.708469+0000","last_clean":"2024-09-13T22:49:17.708469+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:49:17.708469+0000","last_undegraded":"2024-09-13T22:49:17.708469+0000","last_fullsized":"2024-09-13T22:49:17.708469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:56.706439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:56.706439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":417,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823481+0000","last_change":"2024-09-13T22:48:07.145464+0000","last_active":"2024-09-13T22:49:18.823481+0000","last_peered":"2024-09-13T22:49:18.823481+0000","last_clean":"2024-09-13T22:49:18.823481+0000","last_became_active":"2024-09-13T22:47:06.891260+0000","last_became_peered":"2024-09-13T22:47:06.891260+0000","last_unstale":"2024-09-13T22:49:18.823481+0000","last_undegraded":"2024-09-13T22:49:18.823481+0000","last_fullsized":"2024-09-13T22:49:18.823481+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:07.145350+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:07.145350+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"228'22","reported_seq":429,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175954+0000","last_change":"2024-09-13T22:45:01.550550+0000","last_active":"2024-09-13T22:49:11.175954+0000","last_peered":"2024-09-13T22:49:11.175954+0000","last_clean":"2024-09-13T22:49:11.175954+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:49:11.175954+0000","last_undegraded":"2024-09-13T22:49:11.175954+0000","last_fullsized":"2024-09-13T22:49:11.175954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-13T22:45:01.550400+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:45:01.550400+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":419,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.822297+0000","last_change":"2024-09-13T22:47:07.022984+0000","last_active":"2024-09-13T22:49:18.822297+0000","last_peered":"2024-09-13T22:49:18.822297+0000","last_clean":"2024-09-13T22:49:18.822297+0000","last_became_active":"2024-09-13T22:47:03.899279+0000","last_became_peered":"2024-09-13T22:47:03.899279+0000","last_unstale":"2024-09-13T22:49:18.822297+0000","last_undegraded":"2024-09-13T22:49:18.822297+0000","last_fullsized":"2024-09-13T22:49:18.822297+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:47:07.022796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":1072,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:13.526307+0000","last_change":"2024-09-13T22:45:54.489191+0000","last_active":"2024-09-13T22:49:13.526307+0000","last_peered":"2024-09-13T22:49:13.526307+0000","last_clean":"2024-09-13T22:49:13.526307+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:49:13.526307+0000","last_undegraded":"2024-09-13T22:49:13.526307+0000","last_fullsized":"2024-09-13T22:49:13.526307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:45:54.489086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.489086+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":671,"num_read_kb":666,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":96,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174536+0000","last_change":"2024-09-13T22:48:35.511647+0000","last_active":"2024-09-13T22:49:11.174536+0000","last_peered":"2024-09-13T22:49:11.174536+0000","last_clean":"2024-09-13T22:49:11.174536+0000","last_became_active":"2024-09-13T22:48:35.511300+0000","last_became_peered":"2024-09-13T22:48:35.511300+0000","last_unstale":"2024-09-13T22:49:11.174536+0000","last_undegraded":"2024-09-13T22:49:11.174536+0000","last_fullsized":"2024-09-13T22:49:11.174536+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:51.936005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.936005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":89,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174625+0000","last_change":"2024-09-13T22:47:03.902506+0000","last_active":"2024-09-13T22:49:11.174625+0000","last_peered":"2024-09-13T22:49:11.174625+0000","last_clean":"2024-09-13T22:49:11.174625+0000","last_became_active":"2024-09-13T22:47:03.902203+0000","last_became_peered":"2024-09-13T22:47:03.902203+0000","last_unstale":"2024-09-13T22:49:11.174625+0000","last_undegraded":"2024-09-13T22:49:11.174625+0000","last_fullsized":"2024-09-13T22:49:11.174625+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:37.838733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:45:37.838733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":464,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824974+0000","last_change":"2024-09-13T22:48:18.126451+0000","last_active":"2024-09-13T22:49:18.824974+0000","last_peered":"2024-09-13T22:49:18.824974+0000","last_clean":"2024-09-13T22:49:18.824974+0000","last_became_active":"2024-09-13T22:45:58.866066+0000","last_became_peered":"2024-09-13T22:45:58.866066+0000","last_unstale":"2024-09-13T22:49:18.824974+0000","last_undegraded":"2024-09-13T22:49:18.824974+0000","last_fullsized":"2024-09-13T22:49:18.824974+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:18.126322+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:18.126322+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:36.127447+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"228'42","reported_seq":611,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821540+0000","last_change":"2024-09-13T22:48:15.188315+0000","last_active":"2024-09-13T22:49:18.821540+0000","last_peered":"2024-09-13T22:49:18.821540+0000","last_clean":"2024-09-13T22:49:18.821540+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:49:18.821540+0000","last_undegraded":"2024-09-13T22:49:18.821540+0000","last_fullsized":"2024-09-13T22:49:18.821540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'42","last_scrub_stamp":"2024-09-13T22:48:15.188167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.188167+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:25.551678+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":410,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.960640+0000","last_change":"2024-09-13T22:49:17.960697+0000","last_active":"2024-09-13T22:49:17.960640+0000","last_peered":"2024-09-13T22:49:17.960640+0000","last_clean":"2024-09-13T22:49:17.960640+0000","last_became_active":"2024-09-13T22:47:03.907952+0000","last_became_peered":"2024-09-13T22:47:03.907952+0000","last_unstale":"2024-09-13T22:49:17.960640+0000","last_undegraded":"2024-09-13T22:49:17.960640+0000","last_fullsized":"2024-09-13T22:49:17.960640+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.960501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.960501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:36.493584+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.825166+0000","last_change":"2024-09-13T22:48:15.197402+0000","last_active":"2024-09-13T22:49:18.825166+0000","last_peered":"2024-09-13T22:49:18.825166+0000","last_clean":"2024-09-13T22:49:18.825166+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:49:18.825166+0000","last_undegraded":"2024-09-13T22:49:18.825166+0000","last_fullsized":"2024-09-13T22:49:18.825166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.197323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.197323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:27.579983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":395,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820703+0000","last_change":"2024-09-13T22:48:11.214965+0000","last_active":"2024-09-13T22:49:18.820703+0000","last_peered":"2024-09-13T22:49:18.820703+0000","last_clean":"2024-09-13T22:49:18.820703+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:49:18.820703+0000","last_undegraded":"2024-09-13T22:49:18.820703+0000","last_fullsized":"2024-09-13T22:49:18.820703+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.214834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.214834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:28.493150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-09-13T22:49:18.831217+0000","last_change":"2024-09-13T22:49:18.831217+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.831217+0000","last_undegraded":"2024-09-13T22:49:18.831217+0000","last_fullsized":"2024-09-13T22:49:18.831217+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":464,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.825824+0000","last_change":"2024-09-13T22:48:22.045389+0000","last_active":"2024-09-13T22:49:18.825824+0000","last_peered":"2024-09-13T22:49:18.825824+0000","last_clean":"2024-09-13T22:49:18.825824+0000","last_became_active":"2024-09-13T22:45:58.862759+0000","last_became_peered":"2024-09-13T22:45:58.862759+0000","last_unstale":"2024-09-13T22:49:18.825824+0000","last_undegraded":"2024-09-13T22:49:18.825824+0000","last_fullsized":"2024-09-13T22:49:18.825824+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:22.045197+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.045197+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:29.935161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"228'36","reported_seq":590,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836517+0000","last_change":"2024-09-13T22:46:56.049756+0000","last_active":"2024-09-13T22:49:18.836517+0000","last_peered":"2024-09-13T22:49:18.836517+0000","last_clean":"2024-09-13T22:49:18.836517+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:49:18.836517+0000","last_undegraded":"2024-09-13T22:49:18.836517+0000","last_fullsized":"2024-09-13T22:49:18.836517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:46:56.049683+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:46:56.049683+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821649+0000","last_change":"2024-09-13T22:48:35.268201+0000","last_active":"2024-09-13T22:49:18.821649+0000","last_peered":"2024-09-13T22:49:18.821649+0000","last_clean":"2024-09-13T22:49:18.821649+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:49:18.821649+0000","last_undegraded":"2024-09-13T22:49:18.821649+0000","last_fullsized":"2024-09-13T22:49:18.821649+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:48:35.268073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.268073+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:52.247618+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":430,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821373+0000","last_change":"2024-09-13T22:49:14.401085+0000","last_active":"2024-09-13T22:49:18.821373+0000","last_peered":"2024-09-13T22:49:18.821373+0000","last_clean":"2024-09-13T22:49:18.821373+0000","last_became_active":"2024-09-13T22:47:03.911332+0000","last_became_peered":"2024-09-13T22:47:03.911332+0000","last_unstale":"2024-09-13T22:49:18.821373+0000","last_undegraded":"2024-09-13T22:49:18.821373+0000","last_fullsized":"2024-09-13T22:49:18.821373+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:49:14.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.400987+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:26.798020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":553,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.152403+0000","last_change":"2024-09-13T22:48:28.132907+0000","last_active":"2024-09-13T22:49:11.152403+0000","last_peered":"2024-09-13T22:49:11.152403+0000","last_clean":"2024-09-13T22:49:11.152403+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:49:11.152403+0000","last_undegraded":"2024-09-13T22:49:11.152403+0000","last_fullsized":"2024-09-13T22:49:11.152403+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:48:28.132780+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.132780+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:53.044895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":391,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.150938+0000","last_change":"2024-09-13T22:46:55.557651+0000","last_active":"2024-09-13T22:49:11.150938+0000","last_peered":"2024-09-13T22:49:11.150938+0000","last_clean":"2024-09-13T22:49:11.150938+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:49:11.150938+0000","last_undegraded":"2024-09-13T22:49:11.150938+0000","last_fullsized":"2024-09-13T22:49:11.150938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:55.557578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:55.557578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":459,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.834442+0000","last_change":"2024-09-13T22:48:26.832776+0000","last_active":"2024-09-13T22:49:18.834442+0000","last_peered":"2024-09-13T22:49:18.834442+0000","last_clean":"2024-09-13T22:49:18.834442+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:49:18.834442+0000","last_undegraded":"2024-09-13T22:49:18.834442+0000","last_fullsized":"2024-09-13T22:49:18.834442+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.832692+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:56.306610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"228'29","reported_seq":177,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.152582+0000","last_change":"2024-09-13T22:47:03.962082+0000","last_active":"2024-09-13T22:49:11.152582+0000","last_peered":"2024-09-13T22:49:11.152582+0000","last_clean":"2024-09-13T22:49:11.152582+0000","last_became_active":"2024-09-13T22:47:03.933746+0000","last_became_peered":"2024-09-13T22:47:03.933746+0000","last_unstale":"2024-09-13T22:49:11.152582+0000","last_undegraded":"2024-09-13T22:49:11.152582+0000","last_fullsized":"2024-09-13T22:49:11.152582+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'19","last_scrub_stamp":"2024-09-13T22:45:52.914946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:52.914946+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836017+0000","last_change":"2024-09-13T22:48:14.833285+0000","last_active":"2024-09-13T22:49:18.836017+0000","last_peered":"2024-09-13T22:49:18.836017+0000","last_clean":"2024-09-13T22:49:18.836017+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:49:18.836017+0000","last_undegraded":"2024-09-13T22:49:18.836017+0000","last_fullsized":"2024-09-13T22:49:18.836017+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.669505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":401,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.151537+0000","last_change":"2024-09-13T22:47:03.902935+0000","last_active":"2024-09-13T22:49:11.151537+0000","last_peered":"2024-09-13T22:49:11.151537+0000","last_clean":"2024-09-13T22:49:11.151537+0000","last_became_active":"2024-09-13T22:47:03.902594+0000","last_became_peered":"2024-09-13T22:47:03.902594+0000","last_unstale":"2024-09-13T22:49:11.151537+0000","last_undegraded":"2024-09-13T22:49:11.151537+0000","last_fullsized":"2024-09-13T22:49:11.151537+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:44.827803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:45:44.827803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":389,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175170+0000","last_change":"2024-09-13T22:47:03.917191+0000","last_active":"2024-09-13T22:49:11.175170+0000","last_peered":"2024-09-13T22:49:11.175170+0000","last_clean":"2024-09-13T22:49:11.175170+0000","last_became_active":"2024-09-13T22:47:03.916682+0000","last_became_peered":"2024-09-13T22:47:03.916682+0000","last_unstale":"2024-09-13T22:49:11.175170+0000","last_undegraded":"2024-09-13T22:49:11.175170+0000","last_fullsized":"2024-09-13T22:49:11.175170+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.535993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.535993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-09-13T22:49:18.828605+0000","last_change":"2024-09-13T22:49:18.828605+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.828605+0000","last_undegraded":"2024-09-13T22:49:18.828605+0000","last_fullsized":"2024-09-13T22:49:18.828605+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":691,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709070+0000","last_change":"2024-09-13T22:48:38.173367+0000","last_active":"2024-09-13T22:49:17.709070+0000","last_peered":"2024-09-13T22:49:17.709070+0000","last_clean":"2024-09-13T22:49:17.709070+0000","last_became_active":"2024-09-13T22:47:06.894360+0000","last_became_peered":"2024-09-13T22:47:06.894360+0000","last_unstale":"2024-09-13T22:49:17.709070+0000","last_undegraded":"2024-09-13T22:49:17.709070+0000","last_fullsized":"2024-09-13T22:49:17.709070+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:38.173263+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.173263+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:46.545441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"228'38","reported_seq":461,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820790+0000","last_change":"2024-09-13T22:48:17.165438+0000","last_active":"2024-09-13T22:49:18.820790+0000","last_peered":"2024-09-13T22:49:18.820790+0000","last_clean":"2024-09-13T22:49:18.820790+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:49:18.820790+0000","last_undegraded":"2024-09-13T22:49:18.820790+0000","last_fullsized":"2024-09-13T22:49:18.820790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:48:17.165325+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.165325+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.819730+0000","last_change":"2024-09-13T22:47:00.045263+0000","last_active":"2024-09-13T22:49:18.819730+0000","last_peered":"2024-09-13T22:49:18.819730+0000","last_clean":"2024-09-13T22:49:18.819730+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:49:18.819730+0000","last_undegraded":"2024-09-13T22:49:18.819730+0000","last_fullsized":"2024-09-13T22:49:18.819730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:47:00.045058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:47:00.045058+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":415,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823642+0000","last_change":"2024-09-13T22:48:22.142431+0000","last_active":"2024-09-13T22:49:18.823642+0000","last_peered":"2024-09-13T22:49:18.823642+0000","last_clean":"2024-09-13T22:49:18.823642+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:49:18.823642+0000","last_undegraded":"2024-09-13T22:49:18.823642+0000","last_fullsized":"2024-09-13T22:49:18.823642+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:22.142327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.142327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:50.026080+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":392,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710313+0000","last_change":"2024-09-13T22:48:17.369617+0000","last_active":"2024-09-13T22:49:17.710313+0000","last_peered":"2024-09-13T22:49:17.710313+0000","last_clean":"2024-09-13T22:49:17.710313+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:49:17.710313+0000","last_undegraded":"2024-09-13T22:49:17.710313+0000","last_fullsized":"2024-09-13T22:49:17.710313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.369549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.369549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:27.278240+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":468,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821800+0000","last_change":"2024-09-13T22:48:31.174111+0000","last_active":"2024-09-13T22:49:18.821800+0000","last_peered":"2024-09-13T22:49:18.821800+0000","last_clean":"2024-09-13T22:49:18.821800+0000","last_became_active":"2024-09-13T22:47:06.902506+0000","last_became_peered":"2024-09-13T22:47:06.902506+0000","last_unstale":"2024-09-13T22:49:18.821800+0000","last_undegraded":"2024-09-13T22:49:18.821800+0000","last_fullsized":"2024-09-13T22:49:18.821800+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:31.173761+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.802229+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.173761+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:54.370553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"228'28","reported_seq":471,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.188506+0000","last_change":"2024-09-13T22:49:11.188566+0000","last_active":"2024-09-13T22:49:11.188506+0000","last_peered":"2024-09-13T22:49:11.188506+0000","last_clean":"2024-09-13T22:49:11.188506+0000","last_became_active":"2024-09-13T22:47:03.921362+0000","last_became_peered":"2024-09-13T22:47:03.921362+0000","last_unstale":"2024-09-13T22:49:11.188506+0000","last_undegraded":"2024-09-13T22:49:11.188506+0000","last_fullsized":"2024-09-13T22:49:11.188506+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'28","last_scrub_stamp":"2024-09-13T22:49:11.188389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.188389+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:23.207192+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.707721+0000","last_change":"2024-09-13T22:46:54.746202+0000","last_active":"2024-09-13T22:49:17.707721+0000","last_peered":"2024-09-13T22:49:17.707721+0000","last_clean":"2024-09-13T22:49:17.707721+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:49:17.707721+0000","last_undegraded":"2024-09-13T22:49:17.707721+0000","last_fullsized":"2024-09-13T22:49:17.707721+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:54.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:46:54.746095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":398,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.837590+0000","last_change":"2024-09-13T22:46:07.796968+0000","last_active":"2024-09-13T22:49:18.837590+0000","last_peered":"2024-09-13T22:49:18.837590+0000","last_clean":"2024-09-13T22:49:18.837590+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:49:18.837590+0000","last_undegraded":"2024-09-13T22:49:18.837590+0000","last_fullsized":"2024-09-13T22:49:18.837590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:07.796849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:46:07.796849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":395,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821367+0000","last_change":"2024-09-13T22:47:13.085252+0000","last_active":"2024-09-13T22:49:18.821367+0000","last_peered":"2024-09-13T22:49:18.821367+0000","last_clean":"2024-09-13T22:49:18.821367+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:49:18.821367+0000","last_undegraded":"2024-09-13T22:49:18.821367+0000","last_fullsized":"2024-09-13T22:49:18.821367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:13.085159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:47:13.085159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"228'20","reported_seq":421,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174818+0000","last_change":"2024-09-13T22:44:55.563811+0000","last_active":"2024-09-13T22:49:11.174818+0000","last_peered":"2024-09-13T22:49:11.174818+0000","last_clean":"2024-09-13T22:49:11.174818+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:49:11.174818+0000","last_undegraded":"2024-09-13T22:49:11.174818+0000","last_fullsized":"2024-09-13T22:49:11.174818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:44:55.563593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.563593+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":406,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835893+0000","last_change":"2024-09-13T22:46:03.807801+0000","last_active":"2024-09-13T22:49:18.835893+0000","last_peered":"2024-09-13T22:49:18.835893+0000","last_clean":"2024-09-13T22:49:18.835893+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:49:18.835893+0000","last_undegraded":"2024-09-13T22:49:18.835893+0000","last_fullsized":"2024-09-13T22:49:18.835893+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:03.807736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:46:03.807736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":675,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708605+0000","last_change":"2024-09-13T22:48:32.199596+0000","last_active":"2024-09-13T22:49:17.708605+0000","last_peered":"2024-09-13T22:49:17.708605+0000","last_clean":"2024-09-13T22:49:17.708605+0000","last_became_active":"2024-09-13T22:45:53.877322+0000","last_became_peered":"2024-09-13T22:45:53.877322+0000","last_unstale":"2024-09-13T22:49:17.708605+0000","last_undegraded":"2024-09-13T22:49:17.708605+0000","last_fullsized":"2024-09-13T22:49:17.708605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:32.199490+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.199490+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.200419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":396,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709662+0000","last_change":"2024-09-13T22:48:16.361464+0000","last_active":"2024-09-13T22:49:17.709662+0000","last_peered":"2024-09-13T22:49:17.709662+0000","last_clean":"2024-09-13T22:49:17.709662+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:49:17.709662+0000","last_undegraded":"2024-09-13T22:49:17.709662+0000","last_fullsized":"2024-09-13T22:49:17.709662+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:16.361322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:48:16.361322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.692113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":408,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.837584+0000","last_change":"2024-09-13T22:48:26.076140+0000","last_active":"2024-09-13T22:49:18.837584+0000","last_peered":"2024-09-13T22:49:18.837584+0000","last_clean":"2024-09-13T22:49:18.837584+0000","last_became_active":"2024-09-13T22:47:03.910753+0000","last_became_peered":"2024-09-13T22:47:03.910753+0000","last_unstale":"2024-09-13T22:49:18.837584+0000","last_undegraded":"2024-09-13T22:49:18.837584+0000","last_fullsized":"2024-09-13T22:49:18.837584+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.075948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.075948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.525224+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-09-13T22:49:18.830287+0000","last_change":"2024-09-13T22:49:18.830287+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.830287+0000","last_undegraded":"2024-09-13T22:49:18.830287+0000","last_fullsized":"2024-09-13T22:49:18.830287+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"228'31","reported_seq":449,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710837+0000","last_change":"2024-09-13T22:49:13.909885+0000","last_active":"2024-09-13T22:49:17.710837+0000","last_peered":"2024-09-13T22:49:17.710837+0000","last_clean":"2024-09-13T22:49:17.710837+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:49:17.710837+0000","last_undegraded":"2024-09-13T22:49:17.710837+0000","last_fullsized":"2024-09-13T22:49:17.710837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'31","last_scrub_stamp":"2024-09-13T22:49:13.909765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:13.909765+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:32.029113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823340+0000","last_change":"2024-09-13T22:48:34.249596+0000","last_active":"2024-09-13T22:49:18.823340+0000","last_peered":"2024-09-13T22:49:18.823340+0000","last_clean":"2024-09-13T22:49:18.823340+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:49:18.823340+0000","last_undegraded":"2024-09-13T22:49:18.823340+0000","last_fullsized":"2024-09-13T22:49:18.823340+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:34.249408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.249408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.152002+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":407,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820302+0000","last_change":"2024-09-13T22:48:35.269277+0000","last_active":"2024-09-13T22:49:18.820302+0000","last_peered":"2024-09-13T22:49:18.820302+0000","last_clean":"2024-09-13T22:49:18.820302+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:49:18.820302+0000","last_undegraded":"2024-09-13T22:49:18.820302+0000","last_fullsized":"2024-09-13T22:49:18.820302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:35.269156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.269156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:48.954038+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":384,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708485+0000","last_change":"2024-09-13T22:46:10.833710+0000","last_active":"2024-09-13T22:49:17.708485+0000","last_peered":"2024-09-13T22:49:17.708485+0000","last_clean":"2024-09-13T22:49:17.708485+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:49:17.708485+0000","last_undegraded":"2024-09-13T22:49:17.708485+0000","last_fullsized":"2024-09-13T22:49:17.708485+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:10.833635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:46:10.833635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-09-13T22:49:18.830057+0000","last_change":"2024-09-13T22:49:18.830057+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.830057+0000","last_undegraded":"2024-09-13T22:49:18.830057+0000","last_fullsized":"2024-09-13T22:49:18.830057+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"228'20","reported_seq":440,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836473+0000","last_change":"2024-09-13T22:49:14.431757+0000","last_active":"2024-09-13T22:49:18.836473+0000","last_peered":"2024-09-13T22:49:18.836473+0000","last_clean":"2024-09-13T22:49:18.836473+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:49:18.836473+0000","last_undegraded":"2024-09-13T22:49:18.836473+0000","last_fullsized":"2024-09-13T22:49:18.836473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:49:14.431540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.431540+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.128652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.822826+0000","last_change":"2024-09-13T22:48:23.112220+0000","last_active":"2024-09-13T22:49:18.822826+0000","last_peered":"2024-09-13T22:49:18.822826+0000","last_clean":"2024-09-13T22:49:18.822826+0000","last_became_active":"2024-09-13T22:47:03.889309+0000","last_became_peered":"2024-09-13T22:47:03.889309+0000","last_unstale":"2024-09-13T22:49:18.822826+0000","last_undegraded":"2024-09-13T22:49:18.822826+0000","last_fullsized":"2024-09-13T22:49:18.822826+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:23.112116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.112116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:48.980030+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.822051+0000","last_change":"2024-09-13T22:47:03.890403+0000","last_active":"2024-09-13T22:49:18.822051+0000","last_peered":"2024-09-13T22:49:18.822051+0000","last_clean":"2024-09-13T22:49:18.822051+0000","last_became_active":"2024-09-13T22:47:03.889556+0000","last_became_peered":"2024-09-13T22:47:03.889556+0000","last_unstale":"2024-09-13T22:49:18.822051+0000","last_undegraded":"2024-09-13T22:49:18.822051+0000","last_fullsized":"2024-09-13T22:49:18.822051+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:53.986569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:53.986569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.163387+0000","last_change":"2024-09-13T22:48:11.570730+0000","last_active":"2024-09-13T22:49:11.163387+0000","last_peered":"2024-09-13T22:49:11.163387+0000","last_clean":"2024-09-13T22:49:11.163387+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:49:11.163387+0000","last_undegraded":"2024-09-13T22:49:11.163387+0000","last_fullsized":"2024-09-13T22:49:11.163387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.570502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.570502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:33.459976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T22:49:18.811427+0000","last_change":"2024-09-13T22:49:18.811427+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.811427+0000","last_undegraded":"2024-09-13T22:49:18.811427+0000","last_fullsized":"2024-09-13T22:49:18.811427+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"228'34","reported_seq":472,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710213+0000","last_change":"2024-09-13T22:49:11.973786+0000","last_active":"2024-09-13T22:49:17.710213+0000","last_peered":"2024-09-13T22:49:17.710213+0000","last_clean":"2024-09-13T22:49:17.710213+0000","last_became_active":"2024-09-13T22:47:03.911810+0000","last_became_peered":"2024-09-13T22:47:03.911810+0000","last_unstale":"2024-09-13T22:49:17.710213+0000","last_undegraded":"2024-09-13T22:49:17.710213+0000","last_fullsized":"2024-09-13T22:49:17.710213+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:49:11.973550+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.973550+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:22.620534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":415,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823892+0000","last_change":"2024-09-13T22:48:17.191331+0000","last_active":"2024-09-13T22:49:18.823892+0000","last_peered":"2024-09-13T22:49:18.823892+0000","last_clean":"2024-09-13T22:49:18.823892+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:49:18.823892+0000","last_undegraded":"2024-09-13T22:49:18.823892+0000","last_fullsized":"2024-09-13T22:49:18.823892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.191239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.191239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.201488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"211'12","reported_seq":557,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708415+0000","last_change":"2024-09-13T22:47:03.966726+0000","last_active":"2024-09-13T22:49:17.708415+0000","last_peered":"2024-09-13T22:49:17.708415+0000","last_clean":"2024-09-13T22:49:17.708415+0000","last_became_active":"2024-09-13T22:47:03.942956+0000","last_became_peered":"2024-09-13T22:47:03.942956+0000","last_unstale":"2024-09-13T22:49:17.708415+0000","last_undegraded":"2024-09-13T22:49:17.708415+0000","last_fullsized":"2024-09-13T22:49:17.708415+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:46:55.715383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:55.715383+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":394,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.834801+0000","last_change":"2024-09-13T22:46:04.822159+0000","last_active":"2024-09-13T22:49:18.834801+0000","last_peered":"2024-09-13T22:49:18.834801+0000","last_clean":"2024-09-13T22:49:18.834801+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:49:18.834801+0000","last_undegraded":"2024-09-13T22:49:18.834801+0000","last_fullsized":"2024-09-13T22:49:18.834801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:04.822089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:04.822089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-09-13T22:49:18.819337+0000","last_change":"2024-09-13T22:49:18.819337+0000","last_active":"2024-09-13T22:49:18.811427+0000","last_peered":"2024-09-13T22:49:18.811427+0000","last_clean":"2024-09-13T22:49:18.811427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T22:49:18.819337+0000","last_undegraded":"2024-09-13T22:49:18.819337+0000","last_fullsized":"2024-09-13T22:49:18.819337+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"228'19","reported_seq":449,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710786+0000","last_change":"2024-09-13T22:47:03.955949+0000","last_active":"2024-09-13T22:49:17.710786+0000","last_peered":"2024-09-13T22:49:17.710786+0000","last_clean":"2024-09-13T22:49:17.710786+0000","last_became_active":"2024-09-13T22:47:03.925193+0000","last_became_peered":"2024-09-13T22:47:03.925193+0000","last_unstale":"2024-09-13T22:49:17.710786+0000","last_undegraded":"2024-09-13T22:49:17.710786+0000","last_fullsized":"2024-09-13T22:49:17.710786+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:46:57.700461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:57.700461+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":206,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.153727+0000","last_change":"2024-09-13T22:47:03.928773+0000","last_active":"2024-09-13T22:49:11.153727+0000","last_peered":"2024-09-13T22:49:11.153727+0000","last_clean":"2024-09-13T22:49:11.153727+0000","last_became_active":"2024-09-13T22:47:03.899714+0000","last_became_peered":"2024-09-13T22:47:03.899714+0000","last_unstale":"2024-09-13T22:49:11.153727+0000","last_undegraded":"2024-09-13T22:49:11.153727+0000","last_fullsized":"2024-09-13T22:49:11.153727+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:46:02.057013+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.057013+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":407,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.819578+0000","last_change":"2024-09-13T22:48:40.290789+0000","last_active":"2024-09-13T22:49:18.819578+0000","last_peered":"2024-09-13T22:49:18.819578+0000","last_clean":"2024-09-13T22:49:18.819578+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:49:18.819578+0000","last_undegraded":"2024-09-13T22:49:18.819578+0000","last_fullsized":"2024-09-13T22:49:18.819578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:40.290665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:48:40.290665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:07.600216+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":391,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174477+0000","last_change":"2024-09-13T22:48:17.839741+0000","last_active":"2024-09-13T22:49:11.174477+0000","last_peered":"2024-09-13T22:49:11.174477+0000","last_clean":"2024-09-13T22:49:11.174477+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:49:11.174477+0000","last_undegraded":"2024-09-13T22:49:11.174477+0000","last_fullsized":"2024-09-13T22:49:11.174477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.839516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.839516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:28.555266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"228'24","reported_seq":453,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708699+0000","last_change":"2024-09-13T22:48:24.168534+0000","last_active":"2024-09-13T22:49:17.708699+0000","last_peered":"2024-09-13T22:49:17.708699+0000","last_clean":"2024-09-13T22:49:17.708699+0000","last_became_active":"2024-09-13T22:47:03.943714+0000","last_became_peered":"2024-09-13T22:47:03.943714+0000","last_unstale":"2024-09-13T22:49:17.708699+0000","last_undegraded":"2024-09-13T22:49:17.708699+0000","last_fullsized":"2024-09-13T22:49:17.708699+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.168396+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:30.026966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.819795+0000","last_change":"2024-09-13T22:46:51.976469+0000","last_active":"2024-09-13T22:49:18.819795+0000","last_peered":"2024-09-13T22:49:18.819795+0000","last_clean":"2024-09-13T22:49:18.819795+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:49:18.819795+0000","last_undegraded":"2024-09-13T22:49:18.819795+0000","last_fullsized":"2024-09-13T22:49:18.819795+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:51.976346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:46:51.976346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":394,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175866+0000","last_change":"2024-09-13T22:48:15.782055+0000","last_active":"2024-09-13T22:49:11.175866+0000","last_peered":"2024-09-13T22:49:11.175866+0000","last_clean":"2024-09-13T22:49:11.175866+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:49:11.175866+0000","last_undegraded":"2024-09-13T22:49:11.175866+0000","last_fullsized":"2024-09-13T22:49:11.175866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.778860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.778860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:31.851147+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":416,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820708+0000","last_change":"2024-09-13T22:48:24.158752+0000","last_active":"2024-09-13T22:49:18.820708+0000","last_peered":"2024-09-13T22:49:18.820708+0000","last_clean":"2024-09-13T22:49:18.820708+0000","last_became_active":"2024-09-13T22:47:03.916650+0000","last_became_peered":"2024-09-13T22:47:03.916650+0000","last_unstale":"2024-09-13T22:49:18.820708+0000","last_undegraded":"2024-09-13T22:49:18.820708+0000","last_fullsized":"2024-09-13T22:49:18.820708+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.158640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.696146+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"235'10","reported_seq":431,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.823877+0000","last_change":"2024-09-13T22:48:31.992271+0000","last_active":"2024-09-13T22:49:18.823877+0000","last_peered":"2024-09-13T22:49:18.823877+0000","last_clean":"2024-09-13T22:49:18.823877+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:49:18.823877+0000","last_undegraded":"2024-09-13T22:49:18.823877+0000","last_fullsized":"2024-09-13T22:49:18.823877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'10","last_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.992124+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.848412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":428,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824930+0000","last_change":"2024-09-13T22:48:30.082529+0000","last_active":"2024-09-13T22:49:18.824930+0000","last_peered":"2024-09-13T22:49:18.824930+0000","last_clean":"2024-09-13T22:49:18.824930+0000","last_became_active":"2024-09-13T22:47:03.927967+0000","last_became_peered":"2024-09-13T22:47:03.927967+0000","last_unstale":"2024-09-13T22:49:18.824930+0000","last_undegraded":"2024-09-13T22:49:18.824930+0000","last_fullsized":"2024-09-13T22:49:18.824930+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.082437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.082437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.482334+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":397,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836649+0000","last_change":"2024-09-13T22:46:08.851682+0000","last_active":"2024-09-13T22:49:18.836649+0000","last_peered":"2024-09-13T22:49:18.836649+0000","last_clean":"2024-09-13T22:49:18.836649+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:49:18.836649+0000","last_undegraded":"2024-09-13T22:49:18.836649+0000","last_fullsized":"2024-09-13T22:49:18.836649+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:08.851579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.851579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":389,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710386+0000","last_change":"2024-09-13T22:47:03.921324+0000","last_active":"2024-09-13T22:49:17.710386+0000","last_peered":"2024-09-13T22:49:17.710386+0000","last_clean":"2024-09-13T22:49:17.710386+0000","last_became_active":"2024-09-13T22:47:03.919704+0000","last_became_peered":"2024-09-13T22:47:03.919704+0000","last_unstale":"2024-09-13T22:49:17.710386+0000","last_undegraded":"2024-09-13T22:49:17.710386+0000","last_fullsized":"2024-09-13T22:49:17.710386+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:06.698355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:06.698355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"228'19","reported_seq":448,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709465+0000","last_change":"2024-09-13T22:48:18.321148+0000","last_active":"2024-09-13T22:49:17.709465+0000","last_peered":"2024-09-13T22:49:17.709465+0000","last_clean":"2024-09-13T22:49:17.709465+0000","last_became_active":"2024-09-13T22:47:03.935139+0000","last_became_peered":"2024-09-13T22:47:03.935139+0000","last_unstale":"2024-09-13T22:49:17.709465+0000","last_undegraded":"2024-09-13T22:49:17.709465+0000","last_fullsized":"2024-09-13T22:49:17.709465+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:18.321070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:18.321070+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.490331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835751+0000","last_change":"2024-09-13T22:48:15.804011+0000","last_active":"2024-09-13T22:49:18.835751+0000","last_peered":"2024-09-13T22:49:18.835751+0000","last_clean":"2024-09-13T22:49:18.835751+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:49:18.835751+0000","last_undegraded":"2024-09-13T22:49:18.835751+0000","last_fullsized":"2024-09-13T22:49:18.835751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.803899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.803899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.474325+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":558,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.835964+0000","last_change":"2024-09-13T22:48:04.929694+0000","last_active":"2024-09-13T22:49:18.835964+0000","last_peered":"2024-09-13T22:49:18.835964+0000","last_clean":"2024-09-13T22:49:18.835964+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:49:18.835964+0000","last_undegraded":"2024-09-13T22:49:18.835964+0000","last_fullsized":"2024-09-13T22:49:18.835964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:04.929561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:04.929561+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:28.339642+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":390,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.159022+0000","last_change":"2024-09-13T22:46:53.530343+0000","last_active":"2024-09-13T22:49:11.159022+0000","last_peered":"2024-09-13T22:49:11.159022+0000","last_clean":"2024-09-13T22:49:11.159022+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:49:11.159022+0000","last_undegraded":"2024-09-13T22:49:11.159022+0000","last_fullsized":"2024-09-13T22:49:11.159022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:53.530241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:53.530241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"228'15","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.179801+0000","last_change":"2024-09-13T22:46:47.409756+0000","last_active":"2024-09-13T22:49:11.179801+0000","last_peered":"2024-09-13T22:49:11.179801+0000","last_clean":"2024-09-13T22:49:11.179801+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:49:11.179801+0000","last_undegraded":"2024-09-13T22:49:11.179801+0000","last_fullsized":"2024-09-13T22:49:11.179801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:46:47.409619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:47.409619+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.153043+0000","last_change":"2024-09-13T22:48:04.615427+0000","last_active":"2024-09-13T22:49:11.153043+0000","last_peered":"2024-09-13T22:49:11.153043+0000","last_clean":"2024-09-13T22:49:11.153043+0000","last_became_active":"2024-09-13T22:47:03.915370+0000","last_became_peered":"2024-09-13T22:47:03.915370+0000","last_unstale":"2024-09-13T22:49:11.153043+0000","last_undegraded":"2024-09-13T22:49:11.153043+0000","last_fullsized":"2024-09-13T22:49:11.153043+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:04.615328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:04.615328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":543,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175182+0000","last_change":"2024-09-13T22:45:48.537059+0000","last_active":"2024-09-13T22:49:11.175182+0000","last_peered":"2024-09-13T22:49:11.175182+0000","last_clean":"2024-09-13T22:49:11.175182+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:49:11.175182+0000","last_undegraded":"2024-09-13T22:49:11.175182+0000","last_fullsized":"2024-09-13T22:49:11.175182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:48.536952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:48.536952+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":100,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824239+0000","last_change":"2024-09-13T22:48:08.144978+0000","last_active":"2024-09-13T22:49:18.824239+0000","last_peered":"2024-09-13T22:49:18.824239+0000","last_clean":"2024-09-13T22:49:18.824239+0000","last_became_active":"2024-09-13T22:47:03.900992+0000","last_became_peered":"2024-09-13T22:47:03.900992+0000","last_unstale":"2024-09-13T22:49:18.824239+0000","last_undegraded":"2024-09-13T22:49:18.824239+0000","last_fullsized":"2024-09-13T22:49:18.824239+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:08.144897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.144897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.239175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"228'15","reported_seq":440,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710834+0000","last_change":"2024-09-13T22:48:21.238575+0000","last_active":"2024-09-13T22:49:17.710834+0000","last_peered":"2024-09-13T22:49:17.710834+0000","last_clean":"2024-09-13T22:49:17.710834+0000","last_became_active":"2024-09-13T22:47:03.896530+0000","last_became_peered":"2024-09-13T22:47:03.896530+0000","last_unstale":"2024-09-13T22:49:17.710834+0000","last_undegraded":"2024-09-13T22:49:17.710834+0000","last_fullsized":"2024-09-13T22:49:17.710834+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:21.238464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:21.238464+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:31.819323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824896+0000","last_change":"2024-09-13T22:48:36.942296+0000","last_active":"2024-09-13T22:49:18.824896+0000","last_peered":"2024-09-13T22:49:18.824896+0000","last_clean":"2024-09-13T22:49:18.824896+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:49:18.824896+0000","last_undegraded":"2024-09-13T22:49:18.824896+0000","last_fullsized":"2024-09-13T22:49:18.824896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:36.942121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:36.942121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:57.238255+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":386,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174634+0000","last_change":"2024-09-13T22:45:00.534688+0000","last_active":"2024-09-13T22:49:11.174634+0000","last_peered":"2024-09-13T22:49:11.174634+0000","last_clean":"2024-09-13T22:49:11.174634+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:49:11.174634+0000","last_undegraded":"2024-09-13T22:49:11.174634+0000","last_fullsized":"2024-09-13T22:49:11.174634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:00.534596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:45:00.534596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":390,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.150452+0000","last_change":"2024-09-13T22:48:33.071864+0000","last_active":"2024-09-13T22:49:11.150452+0000","last_peered":"2024-09-13T22:49:11.150452+0000","last_clean":"2024-09-13T22:49:11.150452+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:49:11.150452+0000","last_undegraded":"2024-09-13T22:49:11.150452+0000","last_fullsized":"2024-09-13T22:49:11.150452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:33.071751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:33.071751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:45.377810+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"228'20","reported_seq":159,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.710462+0000","last_change":"2024-09-13T22:48:20.275253+0000","last_active":"2024-09-13T22:49:17.710462+0000","last_peered":"2024-09-13T22:49:17.710462+0000","last_clean":"2024-09-13T22:49:17.710462+0000","last_became_active":"2024-09-13T22:47:03.932022+0000","last_became_peered":"2024-09-13T22:47:03.932022+0000","last_unstale":"2024-09-13T22:49:17.710462+0000","last_undegraded":"2024-09-13T22:49:17.710462+0000","last_fullsized":"2024-09-13T22:49:17.710462+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:48:20.275130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.275130+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:39.506678+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.822745+0000","last_change":"2024-09-13T22:49:16.390793+0000","last_active":"2024-09-13T22:49:18.822745+0000","last_peered":"2024-09-13T22:49:18.822745+0000","last_clean":"2024-09-13T22:49:18.822745+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:49:18.822745+0000","last_undegraded":"2024-09-13T22:49:18.822745+0000","last_fullsized":"2024-09-13T22:49:18.822745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.389928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:03.849281+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.389928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.083913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":406,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821222+0000","last_change":"2024-09-13T22:48:38.188618+0000","last_active":"2024-09-13T22:49:18.821222+0000","last_peered":"2024-09-13T22:49:18.821222+0000","last_clean":"2024-09-13T22:49:18.821222+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:49:18.821222+0000","last_undegraded":"2024-09-13T22:49:18.821222+0000","last_fullsized":"2024-09-13T22:49:18.821222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:38.188506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.188506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:02.439391+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":382,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174427+0000","last_change":"2024-09-13T22:44:45.461864+0000","last_active":"2024-09-13T22:49:11.174427+0000","last_peered":"2024-09-13T22:49:11.174427+0000","last_clean":"2024-09-13T22:49:11.174427+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:49:11.174427+0000","last_undegraded":"2024-09-13T22:49:11.174427+0000","last_fullsized":"2024-09-13T22:49:11.174427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:44:45.461720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:44:45.461720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"228'12","reported_seq":424,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.153619+0000","last_change":"2024-09-13T22:48:32.051004+0000","last_active":"2024-09-13T22:49:11.153619+0000","last_peered":"2024-09-13T22:49:11.153619+0000","last_clean":"2024-09-13T22:49:11.153619+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:49:11.153619+0000","last_undegraded":"2024-09-13T22:49:11.153619+0000","last_fullsized":"2024-09-13T22:49:11.153619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12","last_scrub_stamp":"2024-09-13T22:48:32.050878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.050878+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:32.659695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.837102+0000","last_change":"2024-09-13T22:46:02.798750+0000","last_active":"2024-09-13T22:49:18.837102+0000","last_peered":"2024-09-13T22:49:18.837102+0000","last_clean":"2024-09-13T22:49:18.837102+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:49:18.837102+0000","last_undegraded":"2024-09-13T22:49:18.837102+0000","last_fullsized":"2024-09-13T22:49:18.837102+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:02.798641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.798641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":419,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824012+0000","last_change":"2024-09-13T22:49:14.149288+0000","last_active":"2024-09-13T22:49:18.824012+0000","last_peered":"2024-09-13T22:49:18.824012+0000","last_clean":"2024-09-13T22:49:18.824012+0000","last_became_active":"2024-09-13T22:47:03.935103+0000","last_became_peered":"2024-09-13T22:47:03.935103+0000","last_unstale":"2024-09-13T22:49:18.824012+0000","last_undegraded":"2024-09-13T22:49:18.824012+0000","last_fullsized":"2024-09-13T22:49:18.824012+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:14.149187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.149187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:37.441459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":397,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.708530+0000","last_change":"2024-09-13T22:49:15.907904+0000","last_active":"2024-09-13T22:49:17.708530+0000","last_peered":"2024-09-13T22:49:17.708530+0000","last_clean":"2024-09-13T22:49:17.708530+0000","last_became_active":"2024-09-13T22:47:03.896472+0000","last_became_peered":"2024-09-13T22:47:03.896472+0000","last_unstale":"2024-09-13T22:49:17.708530+0000","last_undegraded":"2024-09-13T22:49:17.708530+0000","last_fullsized":"2024-09-13T22:49:17.708530+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:15.907800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:15.907800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.113950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"228'15","reported_seq":426,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.836873+0000","last_change":"2024-09-13T22:46:06.750745+0000","last_active":"2024-09-13T22:49:18.836873+0000","last_peered":"2024-09-13T22:49:18.836873+0000","last_clean":"2024-09-13T22:49:18.836873+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:49:18.836873+0000","last_undegraded":"2024-09-13T22:49:18.836873+0000","last_fullsized":"2024-09-13T22:49:18.836873+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:46:06.750407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:06.750407+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.826297+0000","last_change":"2024-09-13T22:48:13.617965+0000","last_active":"2024-09-13T22:49:18.826297+0000","last_peered":"2024-09-13T22:49:18.826297+0000","last_clean":"2024-09-13T22:49:18.826297+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:49:18.826297+0000","last_undegraded":"2024-09-13T22:49:18.826297+0000","last_fullsized":"2024-09-13T22:49:18.826297+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:13.617680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.617680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.137219+0000","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":401,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.707769+0000","last_change":"2024-09-13T22:49:16.934196+0000","last_active":"2024-09-13T22:49:17.707769+0000","last_peered":"2024-09-13T22:49:17.707769+0000","last_clean":"2024-09-13T22:49:17.707769+0000","last_became_active":"2024-09-13T22:47:03.935739+0000","last_became_peered":"2024-09-13T22:47:03.935739+0000","last_unstale":"2024-09-13T22:49:17.707769+0000","last_undegraded":"2024-09-13T22:49:17.707769+0000","last_fullsized":"2024-09-13T22:49:17.707769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.934047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.934047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.592545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":390,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174736+0000","last_change":"2024-09-13T22:45:40.486348+0000","last_active":"2024-09-13T22:49:11.174736+0000","last_peered":"2024-09-13T22:49:11.174736+0000","last_clean":"2024-09-13T22:49:11.174736+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:49:11.174736+0000","last_undegraded":"2024-09-13T22:49:11.174736+0000","last_fullsized":"2024-09-13T22:49:11.174736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:40.486247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:45:40.486247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":393,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T22:49:17.709387+0000","last_change":"2024-09-13T22:48:29.166658+0000","last_active":"2024-09-13T22:49:17.709387+0000","last_peered":"2024-09-13T22:49:17.709387+0000","last_clean":"2024-09-13T22:49:17.709387+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:49:17.709387+0000","last_undegraded":"2024-09-13T22:49:17.709387+0000","last_fullsized":"2024-09-13T22:49:17.709387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:48:29.166588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:29.166588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:50.874694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"228'34","reported_seq":482,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.820867+0000","last_change":"2024-09-13T22:48:28.140759+0000","last_active":"2024-09-13T22:49:18.820867+0000","last_peered":"2024-09-13T22:49:18.820867+0000","last_clean":"2024-09-13T22:49:18.820867+0000","last_became_active":"2024-09-13T22:47:03.904504+0000","last_became_peered":"2024-09-13T22:47:03.904504+0000","last_unstale":"2024-09-13T22:49:18.820867+0000","last_undegraded":"2024-09-13T22:49:18.820867+0000","last_fullsized":"2024-09-13T22:49:18.820867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:28.140653+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.140653+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:52.903926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":419,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.832700+0000","last_change":"2024-09-13T22:48:12.849274+0000","last_active":"2024-09-13T22:49:18.832700+0000","last_peered":"2024-09-13T22:49:18.832700+0000","last_clean":"2024-09-13T22:49:18.832700+0000","last_became_active":"2024-09-13T22:47:03.935641+0000","last_became_peered":"2024-09-13T22:47:03.935641+0000","last_unstale":"2024-09-13T22:49:18.832700+0000","last_undegraded":"2024-09-13T22:49:18.832700+0000","last_fullsized":"2024-09-13T22:49:18.832700+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:12.849206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:48:12.849206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.788041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"211'13","reported_seq":579,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.826231+0000","last_change":"2024-09-13T22:48:34.950988+0000","last_active":"2024-09-13T22:49:18.826231+0000","last_peered":"2024-09-13T22:49:18.826231+0000","last_clean":"2024-09-13T22:49:18.826231+0000","last_became_active":"2024-09-13T22:47:03.935719+0000","last_became_peered":"2024-09-13T22:47:03.935719+0000","last_unstale":"2024-09-13T22:49:18.826231+0000","last_undegraded":"2024-09-13T22:49:18.826231+0000","last_fullsized":"2024-09-13T22:49:18.826231+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:34.950872+0000","last_deep_scrub":"172'12","last_deep_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.950872+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:55.568584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":408,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821557+0000","last_change":"2024-09-13T22:48:39.144813+0000","last_active":"2024-09-13T22:49:18.821557+0000","last_peered":"2024-09-13T22:49:18.821557+0000","last_clean":"2024-09-13T22:49:18.821557+0000","last_became_active":"2024-09-13T22:47:03.901711+0000","last_became_peered":"2024-09-13T22:47:03.901711+0000","last_unstale":"2024-09-13T22:49:18.821557+0000","last_undegraded":"2024-09-13T22:49:18.821557+0000","last_fullsized":"2024-09-13T22:49:18.821557+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.144743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.144743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:09.061290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"228'15","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174522+0000","last_change":"2024-09-13T22:48:24.914421+0000","last_active":"2024-09-13T22:49:11.174522+0000","last_peered":"2024-09-13T22:49:11.174522+0000","last_clean":"2024-09-13T22:49:11.174522+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:49:11.174522+0000","last_undegraded":"2024-09-13T22:49:11.174522+0000","last_fullsized":"2024-09-13T22:49:11.174522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:24.913947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.913947+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:43.347124+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":395,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174868+0000","last_change":"2024-09-13T22:44:58.543023+0000","last_active":"2024-09-13T22:49:11.174868+0000","last_peered":"2024-09-13T22:49:11.174868+0000","last_clean":"2024-09-13T22:49:11.174868+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:49:11.174868+0000","last_undegraded":"2024-09-13T22:49:11.174868+0000","last_fullsized":"2024-09-13T22:49:11.174868+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:58.542781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.542781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.172268+0000","last_change":"2024-09-13T22:48:30.083820+0000","last_active":"2024-09-13T22:49:11.172268+0000","last_peered":"2024-09-13T22:49:11.172268+0000","last_clean":"2024-09-13T22:49:11.172268+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:49:11.172268+0000","last_undegraded":"2024-09-13T22:49:11.172268+0000","last_fullsized":"2024-09-13T22:49:11.172268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.083568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:59.748166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":100,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.824532+0000","last_change":"2024-09-13T22:48:16.354115+0000","last_active":"2024-09-13T22:49:18.824532+0000","last_peered":"2024-09-13T22:49:18.824532+0000","last_clean":"2024-09-13T22:49:18.824532+0000","last_became_active":"2024-09-13T22:47:03.926169+0000","last_became_peered":"2024-09-13T22:47:03.926169+0000","last_unstale":"2024-09-13T22:49:18.824532+0000","last_undegraded":"2024-09-13T22:49:18.824532+0000","last_fullsized":"2024-09-13T22:49:18.824532+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:16.353783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:16.353783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:31.111988+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":427,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.826019+0000","last_change":"2024-09-13T22:48:37.918024+0000","last_active":"2024-09-13T22:49:18.826019+0000","last_peered":"2024-09-13T22:49:18.826019+0000","last_clean":"2024-09-13T22:49:18.826019+0000","last_became_active":"2024-09-13T22:47:03.902900+0000","last_became_peered":"2024-09-13T22:47:03.902900+0000","last_unstale":"2024-09-13T22:49:18.826019+0000","last_undegraded":"2024-09-13T22:49:18.826019+0000","last_fullsized":"2024-09-13T22:49:18.826019+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.917904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.917904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:06.438554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"228'9","reported_seq":433,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.152758+0000","last_change":"2024-09-13T22:47:03.928761+0000","last_active":"2024-09-13T22:49:11.152758+0000","last_peered":"2024-09-13T22:49:11.152758+0000","last_clean":"2024-09-13T22:49:11.152758+0000","last_became_active":"2024-09-13T22:47:03.915408+0000","last_became_peered":"2024-09-13T22:47:03.915408+0000","last_unstale":"2024-09-13T22:49:11.152758+0000","last_undegraded":"2024-09-13T22:49:11.152758+0000","last_fullsized":"2024-09-13T22:49:11.152758+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:46:58.546268+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:46:58.546268+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":412,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.822661+0000","last_change":"2024-09-13T22:48:41.110319+0000","last_active":"2024-09-13T22:49:18.822661+0000","last_peered":"2024-09-13T22:49:18.822661+0000","last_clean":"2024-09-13T22:49:18.822661+0000","last_became_active":"2024-09-13T22:47:03.902134+0000","last_became_peered":"2024-09-13T22:47:03.902134+0000","last_unstale":"2024-09-13T22:49:18.822661+0000","last_undegraded":"2024-09-13T22:49:18.822661+0000","last_fullsized":"2024-09-13T22:49:18.822661+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:41.110195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:48:41.110195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:03.599329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":392,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175843+0000","last_change":"2024-09-13T22:45:51.542656+0000","last_active":"2024-09-13T22:49:11.175843+0000","last_peered":"2024-09-13T22:49:11.175843+0000","last_clean":"2024-09-13T22:49:11.175843+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:49:11.175843+0000","last_undegraded":"2024-09-13T22:49:11.175843+0000","last_fullsized":"2024-09-13T22:49:11.175843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:45:51.542373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.542373+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.174851+0000","last_change":"2024-09-13T22:48:28.089740+0000","last_active":"2024-09-13T22:49:11.174851+0000","last_peered":"2024-09-13T22:49:11.174851+0000","last_clean":"2024-09-13T22:49:11.174851+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:49:11.174851+0000","last_undegraded":"2024-09-13T22:49:11.174851+0000","last_fullsized":"2024-09-13T22:49:11.174851+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:28.089548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.089548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:38.980036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"228'19","reported_seq":409,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T22:49:18.821257+0000","last_change":"2024-09-13T22:47:03.904471+0000","last_active":"2024-09-13T22:49:18.821257+0000","last_peered":"2024-09-13T22:49:18.821257+0000","last_clean":"2024-09-13T22:49:18.821257+0000","last_became_active":"2024-09-13T22:47:03.891133+0000","last_became_peered":"2024-09-13T22:47:03.891133+0000","last_unstale":"2024-09-13T22:49:18.821257+0000","last_undegraded":"2024-09-13T22:49:18.821257+0000","last_fullsized":"2024-09-13T22:49:18.821257+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.028593+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":89,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T22:49:11.175057+0000","last_change":"2024-09-13T22:47:03.917147+0000","last_active":"2024-09-13T22:49:11.175057+0000","last_peered":"2024-09-13T22:49:11.175057+0000","last_clean":"2024-09-13T22:49:11.175057+0000","last_became_active":"2024-09-13T22:47:03.916657+0000","last_became_peered":"2024-09-13T22:47:03.916657+0000","last_unstale":"2024-09-13T22:49:11.175057+0000","last_undegraded":"2024-09-13T22:49:11.175057+0000","last_fullsized":"2024-09-13T22:49:11.175057+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:50.912196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:45:50.912196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":2},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":671,"num_read_kb":666,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808980,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230816,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93514336,"statfs":{"total":95995035648,"available":95758680064,"internally_reserved":0,"allocated":1126400,"data_stored":726918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136031,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93456368,"statfs":{"total":95995035648,"available":95699320832,"internally_reserved":0,"allocated":782336,"data_stored":397246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528490,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262996,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482156,"statfs":{"total":95995035648,"available":95725727744,"internally_reserved":0,"allocated":1376256,"data_stored":987778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888244,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084477,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365708,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379444,"statfs":{"total":95995035648,"available":95620550656,"internally_reserved":0,"allocated":1363968,"data_stored":981754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378826,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284876,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460276,"statfs":{"total":95995035648,"available":95703322624,"internally_reserved":0,"allocated":1040384,"data_stored":660520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322316,"kb_used_data":30132,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93422836,"statfs":{"total":95995035648,"available":95664984064,"internally_reserved":0,"allocated":30855168,"data_stored":30421998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93456356,"statfs":{"total":95995035648,"available":95699308544,"internally_reserved":0,"allocated":798720,"data_stored":392937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:49:22.125 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:49:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.127 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:22.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:22.242 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:49:22.243 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:49:22.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:22.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:22.370 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:22.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:22.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:22.623 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:22.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.754 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:22.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:21.442765+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:21.447379+0000 osd.5 (osd.5) 262 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:21.454068+0000 osd.1 (osd.1) 239 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:21.457497+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.147783+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.149756+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:49:22.869 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.682012+0000 mgr.y (mgr.14613) 434 : cluster [DBG] pgmap v486: 158 pgs: 6 creating+peering, 3 unknown, 149 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:21.442765+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:21.447379+0000 osd.5 (osd.5) 262 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:21.454068+0000 osd.1 (osd.1) 239 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:21.457497+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.147783+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.149756+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:49:22.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:22 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.682012+0000 mgr.y (mgr.14613) 434 : cluster [DBG] pgmap v486: 158 pgs: 6 creating+peering, 3 unknown, 149 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:49:22.885 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:22.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:22.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:22.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:21.442765+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:49:22.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:21.447379+0000 osd.5 (osd.5) 262 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:49:22.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:21.454068+0000 osd.1 (osd.1) 239 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:49:22.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:21.457497+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:49:22.999 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.147783+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:49:22.999 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.149756+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:49:22.999 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:22 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.682012+0000 mgr.y (mgr.14613) 434 : cluster [DBG] pgmap v486: 158 pgs: 6 creating+peering, 3 unknown, 149 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:49:23.012 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:23.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T22:49:23.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:23.221 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:49:23.222 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:49:23.240 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:49:23.240 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:49:23.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.258 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:49:23.258 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:49:23.261 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:23.293 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:49:23.293 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:49:23.310 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:49:23.311 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:49:23.342 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:49:23.342 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:49:23.362 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:49:23.362 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:49:23.364 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:23.386 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:49:23.386 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:49:23.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:23.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:23.629 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:23.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:23.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:23.854 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:23.875 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:23.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:23.929 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:49:23.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.995 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:23.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:24.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:24.119 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:24.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.280440+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-13T22:49:24.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.282333+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-13T22:49:24.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.418734+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:49:24.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.420046+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:49:24.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.973757+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:49:24.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:22.976102+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.082598+0000 mon.b (mon.1) 177 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.191583+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.178:0/696210720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:23.309586+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:23.309644+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:23.372924+0000 osd.3 (osd.3) 233 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: cluster 2024-09-13T22:49:23.375840+0000 osd.3 (osd.3) 234 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.410251+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? v1:172.21.15.178:0/1859337982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.657857+0000 mon.a (mon.0) 1270 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.664081+0000 mon.a (mon.0) 1271 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.675743+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[29889]: audit 2024-09-13T22:49:23.683435+0000 mon.a (mon.0) 1273 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.280440+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.282333+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.418734+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.420046+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.973757+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:22.976102+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.082598+0000 mon.b (mon.1) 177 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.191583+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.178:0/696210720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:24.122 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:23.309586+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:23.309644+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:23.372924+0000 osd.3 (osd.3) 233 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: cluster 2024-09-13T22:49:23.375840+0000 osd.3 (osd.3) 234 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.410251+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? v1:172.21.15.178:0/1859337982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.657857+0000 mon.a (mon.0) 1270 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.664081+0000 mon.a (mon.0) 1271 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.675743+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.123 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:23 smithi178 bash[22840]: audit 2024-09-13T22:49:23.683435+0000 mon.a (mon.0) 1273 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:24.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.280440+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.a scrub starts 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.282333+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.a scrub ok 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.418734+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.420046+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.973757+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:22.976102+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:49:24.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.082598+0000 mon.b (mon.1) 177 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.191583+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.178:0/696210720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:23.309586+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:23.309644+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:23.372924+0000 osd.3 (osd.3) 233 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: cluster 2024-09-13T22:49:23.375840+0000 osd.3 (osd.3) 234 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.410251+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? v1:172.21.15.178:0/1859337982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.657857+0000 mon.a (mon.0) 1270 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.664081+0000 mon.a (mon.0) 1271 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.675743+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:23 smithi192 bash[24608]: audit 2024-09-13T22:49:23.683435+0000 mon.a (mon.0) 1273 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:24.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:24.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:24.375 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:24.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:24.500 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:24.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:24.628 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:24.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:24.758 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:24.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:24.889 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:24.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:24.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:25.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:25.121 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:25.147 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:25.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: cluster 2024-09-13T22:49:23.919707+0000 mon.a (mon.0) 1274 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:49:25.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:23.931183+0000 mon.a (mon.0) 1275 : audit [INF] from='client.? v1:172.21.15.178:0/1859337982' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: cluster 2024-09-13T22:49:23.934336+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.332775+0000 mon.a (mon.0) 1277 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.333468+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.334788+0000 mon.b (mon.1) 178 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.334947+0000 mon.b (mon.1) 179 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.335336+0000 mon.a (mon.0) 1279 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.336109+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.337344+0000 mon.b (mon.1) 180 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.337608+0000 mon.a (mon.0) 1281 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.337719+0000 mon.b (mon.1) 181 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.338534+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.339660+0000 mon.b (mon.1) 182 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.340282+0000 mon.b (mon.1) 183 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.340392+0000 mon.a (mon.0) 1283 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.341510+0000 mon.a (mon.0) 1284 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: cephadm 2024-09-13T22:49:24.342148+0000 mgr.y (mgr.14613) 435 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 3292M 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.342244+0000 mon.b (mon.1) 184 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: cephadm 2024-09-13T22:49:24.343398+0000 mgr.y (mgr.14613) 436 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 3292M 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.343462+0000 mon.b (mon.1) 185 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.348016+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.356926+0000 mon.a (mon.0) 1286 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.366025+0000 mon.b (mon.1) 186 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.367178+0000 mon.b (mon.1) 187 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: audit 2024-09-13T22:49:24.373801+0000 mon.a (mon.0) 1287 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:24 smithi192 bash[24608]: cluster 2024-09-13T22:49:24.682824+0000 mgr.y (mgr.14613) 437 : cluster [DBG] pgmap v488: 158 pgs: 2 creating+peering, 156 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 872 B/s rd, 0 op/s 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: cluster 2024-09-13T22:49:23.919707+0000 mon.a (mon.0) 1274 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:23.931183+0000 mon.a (mon.0) 1275 : audit [INF] from='client.? v1:172.21.15.178:0/1859337982' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: cluster 2024-09-13T22:49:23.934336+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.332775+0000 mon.a (mon.0) 1277 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.333468+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.334788+0000 mon.b (mon.1) 178 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.334947+0000 mon.b (mon.1) 179 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.335336+0000 mon.a (mon.0) 1279 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.336109+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.337344+0000 mon.b (mon.1) 180 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.337608+0000 mon.a (mon.0) 1281 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.337719+0000 mon.b (mon.1) 181 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.338534+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.339660+0000 mon.b (mon.1) 182 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.340282+0000 mon.b (mon.1) 183 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.340392+0000 mon.a (mon.0) 1283 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.341510+0000 mon.a (mon.0) 1284 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: cluster 2024-09-13T22:49:23.919707+0000 mon.a (mon.0) 1274 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:23.931183+0000 mon.a (mon.0) 1275 : audit [INF] from='client.? v1:172.21.15.178:0/1859337982' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: cluster 2024-09-13T22:49:23.934336+0000 mon.a (mon.0) 1276 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.332775+0000 mon.a (mon.0) 1277 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.333468+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.334788+0000 mon.b (mon.1) 178 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.334947+0000 mon.b (mon.1) 179 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.335336+0000 mon.a (mon.0) 1279 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.336109+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.337344+0000 mon.b (mon.1) 180 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.337608+0000 mon.a (mon.0) 1281 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.337719+0000 mon.b (mon.1) 181 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.338534+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.339660+0000 mon.b (mon.1) 182 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.340282+0000 mon.b (mon.1) 183 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.340392+0000 mon.a (mon.0) 1283 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.341510+0000 mon.a (mon.0) 1284 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: cephadm 2024-09-13T22:49:24.342148+0000 mgr.y (mgr.14613) 435 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 3292M 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.342244+0000 mon.b (mon.1) 184 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: cephadm 2024-09-13T22:49:24.343398+0000 mgr.y (mgr.14613) 436 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 3292M 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.343462+0000 mon.b (mon.1) 185 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.348016+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.356926+0000 mon.a (mon.0) 1286 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.366025+0000 mon.b (mon.1) 186 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.367178+0000 mon.b (mon.1) 187 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: audit 2024-09-13T22:49:24.373801+0000 mon.a (mon.0) 1287 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[29889]: cluster 2024-09-13T22:49:24.682824+0000 mgr.y (mgr.14613) 437 : cluster [DBG] pgmap v488: 158 pgs: 2 creating+peering, 156 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 872 B/s rd, 0 op/s 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: cephadm 2024-09-13T22:49:24.342148+0000 mgr.y (mgr.14613) 435 : cephadm [INF] Adjusting osd_memory_target on smithi178 to 3292M 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.342244+0000 mon.b (mon.1) 184 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: cephadm 2024-09-13T22:49:24.343398+0000 mgr.y (mgr.14613) 436 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 3292M 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.343462+0000 mon.b (mon.1) 185 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.348016+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.356926+0000 mon.a (mon.0) 1286 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.366025+0000 mon.b (mon.1) 186 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:49:25.184 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.367178+0000 mon.b (mon.1) 187 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:49:25.185 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: audit 2024-09-13T22:49:24.373801+0000 mon.a (mon.0) 1287 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:25.185 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:24 smithi178 bash[22840]: cluster 2024-09-13T22:49:24.682824+0000 mgr.y (mgr.14613) 437 : cluster [DBG] pgmap v488: 158 pgs: 2 creating+peering, 156 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 872 B/s rd, 0 op/s 2024-09-13T22:49:25.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:25.276 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:25.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:25.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:25.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:25.531 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:25.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:25.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:49:25.661 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:25.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:25.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:25.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: cluster 2024-09-13T22:49:24.263805+0000 osd.2 (osd.2) 193 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: cluster 2024-09-13T22:49:24.271338+0000 osd.2 (osd.2) 194 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: cluster 2024-09-13T22:49:24.434605+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: cluster 2024-09-13T22:49:24.436022+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: cluster 2024-09-13T22:49:25.090435+0000 osd.0 (osd.0) 237 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: cluster 2024-09-13T22:49:25.093971+0000 osd.0 (osd.0) 238 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:49:25.925 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[22840]: audit 2024-09-13T22:49:25.379831+0000 mgr.y (mgr.14613) 438 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:25.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:25.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:26.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: cluster 2024-09-13T22:49:24.263805+0000 osd.2 (osd.2) 193 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: cluster 2024-09-13T22:49:24.271338+0000 osd.2 (osd.2) 194 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: cluster 2024-09-13T22:49:24.434605+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: cluster 2024-09-13T22:49:24.436022+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: cluster 2024-09-13T22:49:25.090435+0000 osd.0 (osd.0) 237 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: cluster 2024-09-13T22:49:25.093971+0000 osd.0 (osd.0) 238 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:49:26.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:25 smithi192 bash[24608]: audit 2024-09-13T22:49:25.379831+0000 mgr.y (mgr.14613) 438 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:26.055 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:26.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: cluster 2024-09-13T22:49:24.263805+0000 osd.2 (osd.2) 193 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: cluster 2024-09-13T22:49:24.271338+0000 osd.2 (osd.2) 194 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: cluster 2024-09-13T22:49:24.434605+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: cluster 2024-09-13T22:49:24.436022+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: cluster 2024-09-13T22:49:25.090435+0000 osd.0 (osd.0) 237 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: cluster 2024-09-13T22:49:25.093971+0000 osd.0 (osd.0) 238 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:49:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:25 smithi178 bash[29889]: audit 2024-09-13T22:49:25.379831+0000 mgr.y (mgr.14613) 438 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:26.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:26.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:26.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:26.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:26.447 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:26.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:26.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:26.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:49:26.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:26.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:26.720 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:26.809 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:26.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:26.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:26.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:26.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:27.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:27.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:27.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:26 smithi178 bash[22840]: cluster 2024-09-13T22:49:25.443979+0000 osd.1 (osd.1) 241 : cluster [DBG] 2.a scrub starts 2024-09-13T22:49:27.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:26 smithi178 bash[22840]: cluster 2024-09-13T22:49:25.445953+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.a scrub ok 2024-09-13T22:49:27.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:26 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.683709+0000 mgr.y (mgr.14613) 439 : cluster [DBG] pgmap v489: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:49:27.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:26 smithi178 bash[29889]: cluster 2024-09-13T22:49:25.443979+0000 osd.1 (osd.1) 241 : cluster [DBG] 2.a scrub starts 2024-09-13T22:49:27.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:26 smithi178 bash[29889]: cluster 2024-09-13T22:49:25.445953+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.a scrub ok 2024-09-13T22:49:27.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:26 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.683709+0000 mgr.y (mgr.14613) 439 : cluster [DBG] pgmap v489: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:49:27.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.201 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:26 smithi192 bash[24608]: cluster 2024-09-13T22:49:25.443979+0000 osd.1 (osd.1) 241 : cluster [DBG] 2.a scrub starts 2024-09-13T22:49:27.201 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:26 smithi192 bash[24608]: cluster 2024-09-13T22:49:25.445953+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.a scrub ok 2024-09-13T22:49:27.201 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:26 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.683709+0000 mgr.y (mgr.14613) 439 : cluster [DBG] pgmap v489: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:49:27.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:49:27.218 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:27.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.349 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:27.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:27.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.609 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:27.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:27.739 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:27.740 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:27.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:27.852 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:49:27.869 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:27.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:27.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:28.000 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:28.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:28.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.227592+0000 osd.2 (osd.2) 195 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:49:28.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.230014+0000 osd.2 (osd.2) 196 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:49:28.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.455115+0000 osd.1 (osd.1) 243 : cluster [DBG] 5.c scrub starts 2024-09-13T22:49:28.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.457222+0000 osd.1 (osd.1) 244 : cluster [DBG] 5.c scrub ok 2024-09-13T22:49:28.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.957478+0000 osd.7 (osd.7) 255 : cluster [DBG] 2.b scrub starts 2024-09-13T22:49:28.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[22840]: cluster 2024-09-13T22:49:26.959775+0000 osd.7 (osd.7) 256 : cluster [DBG] 2.b scrub ok 2024-09-13T22:49:28.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.227592+0000 osd.2 (osd.2) 195 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:49:28.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.230014+0000 osd.2 (osd.2) 196 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:49:28.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.455115+0000 osd.1 (osd.1) 243 : cluster [DBG] 5.c scrub starts 2024-09-13T22:49:28.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.457222+0000 osd.1 (osd.1) 244 : cluster [DBG] 5.c scrub ok 2024-09-13T22:49:28.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.957478+0000 osd.7 (osd.7) 255 : cluster [DBG] 2.b scrub starts 2024-09-13T22:49:28.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:27 smithi178 bash[29889]: cluster 2024-09-13T22:49:26.959775+0000 osd.7 (osd.7) 256 : cluster [DBG] 2.b scrub ok 2024-09-13T22:49:28.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:28.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:28.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:28.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:28.374 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:27 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.227592+0000 osd.2 (osd.2) 195 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:49:28.375 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:27 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.230014+0000 osd.2 (osd.2) 196 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:49:28.375 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:27 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.455115+0000 osd.1 (osd.1) 243 : cluster [DBG] 5.c scrub starts 2024-09-13T22:49:28.375 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:27 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.457222+0000 osd.1 (osd.1) 244 : cluster [DBG] 5.c scrub ok 2024-09-13T22:49:28.375 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:27 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.957478+0000 osd.7 (osd.7) 255 : cluster [DBG] 2.b scrub starts 2024-09-13T22:49:28.375 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:27 smithi192 bash[24608]: cluster 2024-09-13T22:49:26.959775+0000 osd.7 (osd.7) 256 : cluster [DBG] 2.b scrub ok 2024-09-13T22:49:28.391 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:28.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:28.521 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:28.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:28.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:28.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:28.781 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:28.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:28.912 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:28.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:28.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:29.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:29.042 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:29.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:29.156 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:28 smithi178 bash[22840]: cluster 2024-09-13T22:49:27.967678+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:49:29.157 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:28 smithi178 bash[22840]: cluster 2024-09-13T22:49:27.968951+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:49:29.157 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:28 smithi178 bash[22840]: cluster 2024-09-13T22:49:28.684510+0000 mgr.y (mgr.14613) 440 : cluster [DBG] pgmap v490: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:49:29.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:28 smithi178 bash[29889]: cluster 2024-09-13T22:49:27.967678+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:49:29.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:28 smithi178 bash[29889]: cluster 2024-09-13T22:49:27.968951+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:49:29.157 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:28 smithi178 bash[29889]: cluster 2024-09-13T22:49:28.684510+0000 mgr.y (mgr.14613) 440 : cluster [DBG] pgmap v490: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:49:29.172 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:29.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:29.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:28 smithi192 bash[24608]: cluster 2024-09-13T22:49:27.967678+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:49:29.287 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:28 smithi192 bash[24608]: cluster 2024-09-13T22:49:27.968951+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:49:29.287 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:28 smithi192 bash[24608]: cluster 2024-09-13T22:49:28.684510+0000 mgr.y (mgr.14613) 440 : cluster [DBG] pgmap v490: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:49:29.300 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:29.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:29.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:29.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:29.534 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:49:29.534 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:23488 7.1 7:850ab3f9:::benchmark_data_smithi178_64772_object23487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:18.465804+0000", 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 491.06966849899999, 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36562665, 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23488 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465804+0000", 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465804+0000", 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465806+0000", 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5469999999999999e-06 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465869+0000", 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2069000000000005e-05 2024-09-13T22:49:29.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465884+0000", 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5108e-05 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465892+0000", 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9759999999999995e-06 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.465953+0000", 2024-09-13T22:49:29.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1078e-05 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.466045+0000", 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2062e-05 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.466131+0000", 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6131999999999995e-05 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467123+0000", 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00099208200000000003 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467351+0000", 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022803600000000001 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.831170+0000", 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.363819267 2024-09-13T22:49:29.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.831317+0000", 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014654499999999999 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.831431+0000", 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113748 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:23497 7.1 7:8b48dc47:::benchmark_data_smithi178_64772_object23496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:18.471380+0000", 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 491.06409201700001, 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36288741400000002, 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23497 2024-09-13T22:49:29.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471380+0000", 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471380+0000", 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471382+0000", 2024-09-13T22:49:29.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0020000000000001e-06 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471442+0000", 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9549999999999997e-05 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471453+0000", 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0851e-05 2024-09-13T22:49:29.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471460+0000", 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3579999999999997e-06 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471500+0000", 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9609e-05 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471588+0000", 2024-09-13T22:49:29.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8125000000000001e-05 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.471676+0000", 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8212000000000002e-05 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.472691+0000", 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010149720000000001 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.473336+0000", 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064471099999999996 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.833980+0000", 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36064401800000001 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.834148+0000", 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016836100000000001 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.834268+0000", 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119645 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:61929 7.1 7:8c57732f:::benchmark_data_smithi178_64772_object61928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T22:49:29.548 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:47.899087+0000", 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 461.63638496300001, 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30580340099999997, 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61929 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899087+0000", 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899087+0000", 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899089+0000", 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.855e-06 2024-09-13T22:49:29.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899152+0000", 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799000000000006e-05 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899161+0000", 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2580000000000001e-06 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899167+0000", 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3319999999999999e-06 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899219+0000", 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2182000000000003e-05 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899347+0000", 2024-09-13T22:49:29.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012842800000000001 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.899486+0000", 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013823500000000001 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.901429+0000", 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001943382 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.901926+0000", 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049681900000000001 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903754+0000", 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018287030000000001 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204760+0000", 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30100569399999999 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204891+0000", 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013071400000000001 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24767.0:61929 7.1 e126/116 ondisk, result = 0)", 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:47.903675+0000", 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 461.63179739499998, 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30123481000000002, 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.555 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903675+0000", 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903675+0000", 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903677+0000", 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8300000000000001e-06 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903685+0000", 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4509999999999992e-06 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903696+0000", 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.094e-05 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903704+0000", 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2530000000000004e-06 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903747+0000", 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2833999999999997e-05 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.903751+0000", 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1529999999999999e-06 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204910+0000", 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30115834899999999 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24767.0:61930 7.9 e126/116 7:92ab1bce:::benchmark_data_smithi178_64772_object61929:head v 126'3884, mlcod=126'3880)", 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:47.910399+0000", 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 461.62507319999997, 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.310940781, 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.559 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.910399+0000", 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.910399+0000", 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.910401+0000", 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5400000000000001e-06 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.910443+0000", 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2654999999999997e-05 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.205108+0000", 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.294664604 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.205126+0000", 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8376000000000002e-05 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.205370+0000", 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024317300000000001 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.205420+0000", 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0868999999999997e-05 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.221226+0000", 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015805212999999999 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.221340+0000", 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114351 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.564 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24767.0:61939 7.3 e126/116 7:c03d222f:::benchmark_data_smithi178_64772_object61938:head v 126'3857, mlcod=126'3855)", 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:47.911928+0000", 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 461.62354423599999, 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.301171725, 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.911928+0000", 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.911928+0000", 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.911930+0000", 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4050000000000001e-06 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.911977+0000", 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7125e-05 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204640+0000", 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29266373200000001 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204668+0000", 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7869e-05 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204871+0000", 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000203053 2024-09-13T22:49:29.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204925+0000", 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3736000000000002e-05 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.213000+0000", 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080745250000000008 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.213100+0000", 2024-09-13T22:49:29.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010028 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24767.0:61942 7.8 e126/116 7:14c8f2b0:::benchmark_data_smithi178_64772_object61941:head v 126'3876, mlcod=126'3874)", 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:47.912953+0000", 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 461.62251943199999, 2024-09-13T22:49:29.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.31221275900000001, 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.912953+0000", 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.912953+0000", 2024-09-13T22:49:29.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.912954+0000", 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4339999999999999e-06 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:47.913054+0000", 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9747000000000006e-05 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204608+0000", 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29155388999999998 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.204640+0000", 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2286999999999998e-05 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.205093+0000", 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045283600000000003 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.205151+0000", 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7911999999999997e-05 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.225082+0000", 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019930962 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:48.225166+0000", 2024-09-13T22:49:29.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3690999999999995e-05 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:135684 7.1 7:849616d7:::benchmark_data_smithi178_64772_object29931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:43:18.051869+0000", 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 371.483603679, 2024-09-13T22:49:29.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34361546500000001, 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 135684 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051869+0000", 2024-09-13T22:49:29.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051869+0000", 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051870+0000", 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6980000000000001e-06 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051875+0000", 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1479999999999999e-06 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051884+0000", 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3989999999999993e-06 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051890+0000", 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7520000000000002e-06 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.051930+0000", 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0459e-05 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.052004+0000", 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4220000000000004e-05 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.052076+0000", 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-05 2024-09-13T22:49:29.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.054645+0000", 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025684029999999999 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.054872+0000", 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022728700000000001 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.395326+0000", 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34045377300000002 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.395418+0000", 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2429999999999994e-05 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:43:18.395484+0000", 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6136000000000005e-05 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.581 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:172735 7.1 7:8feb6226:::benchmark_data_smithi178_64772_object66982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:00.438169+0000", 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 329.09730318200002, 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26062966799999998, 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172735 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438169+0000", 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438169+0000", 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438172+0000", 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6249999999999999e-06 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438177+0000", 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4589999999999997e-06 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438190+0000", 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2573e-05 2024-09-13T22:49:29.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438198+0000", 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6459999999999993e-06 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438257+0000", 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9373999999999998e-05 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438379+0000", 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121945 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.438501+0000", 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012225800000000001 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.440839+0000", 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023375710000000001 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.441516+0000", 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067757099999999997 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.698588+0000", 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25707191400000001 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.698705+0000", 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011728200000000001 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:00.698799+0000", 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3449999999999995e-05 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:196396 7.1 7:8c2b9306:::benchmark_data_smithi178_64772_object90643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:31.326469+0000", 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 298.20900384399999, 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39449398699999999, 2024-09-13T22:49:29.588 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 196396 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326469+0000", 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326469+0000", 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326471+0000", 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1840000000000002e-06 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326476+0000", 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2510000000000002e-06 2024-09-13T22:49:29.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326488+0000", 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1606e-05 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326493+0000", 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4600000000000002e-06 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326544+0000", 2024-09-13T22:49:29.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1408e-05 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326647+0000", 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102138 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.326729+0000", 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2825999999999996e-05 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.328931+0000", 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022018099999999998 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333461+0000", 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045299880000000004 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.720659+0000", 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.38719797 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.720877+0000", 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000217932 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.720963+0000", 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5414000000000006e-05 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:196403 7.1 7:8bfd8be5:::benchmark_data_smithi178_64772_object90650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T22:49:29.594 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:31.333535+0000", 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 298.201937026, 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.387813772, 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 196403 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333535+0000", 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333535+0000", 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333537+0000", 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0059999999999999e-06 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333542+0000", 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6859999999999997e-06 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333552+0000", 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0355e-05 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333559+0000", 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6390000000000003e-06 2024-09-13T22:49:29.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333606+0000", 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7101999999999997e-05 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333709+0000", 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102983 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:49:29.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.333783+0000", 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3759999999999999e-05 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.335742+0000", 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019588990000000001 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.336023+0000", 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028129099999999997 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.721085+0000", 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.38506176600000003 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.721264+0000", 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017923900000000001 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:31.721349+0000", 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5045999999999999e-05 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.600 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:91341 8.2 8:42bf45b5:::benchmark_data_smithi178_80607_object91340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:46:29.390312+0000", 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 180.14516081400001, 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.269941767, 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.601 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 91341 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390312+0000", 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390312+0000", 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390314+0000", 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.278e-06 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390365+0000", 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1307000000000002e-05 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390378+0000", 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2595999999999999e-05 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390385+0000", 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5170000000000002e-06 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390452+0000", 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7016999999999999e-05 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390574+0000", 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121482 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,5", 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.390708+0000", 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134265 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.395283+0000", 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004575155 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.399155+0000", 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038718709999999998 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.660180+0000", 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.261024494 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.660222+0000", 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2157000000000003e-05 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:29.660253+0000", 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1628000000000003e-05 2024-09-13T22:49:29.606 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119756 8.5 8:a0634a38:::benchmark_data_smithi178_80607_object11711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:02.645055+0000", 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 146.890417544, 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3719272899999999, 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119756 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645055+0000", 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645055+0000", 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645057+0000", 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0729999999999999e-06 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645062+0000", 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0590000000000002e-06 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645074+0000", 2024-09-13T22:49:29.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2478e-05 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645081+0000", 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4180000000000002e-06 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645132+0000", 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.092e-05 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645223+0000", 2024-09-13T22:49:29.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-05 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3,6,7", 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.645303+0000", 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0137999999999997e-05 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.647140+0000", 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001837247 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.647217+0000", 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7075000000000003e-05 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.648160+0000", 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00094297000000000001 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.648406+0000", 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024578099999999998 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.892460+0000", 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24405389 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.892474+0000", 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4012999999999999e-05 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.016855+0000", 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1243811379999999 2024-09-13T22:49:29.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.016982+0000", 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012739000000000001 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119761 8.5 8:abe9ee50:::benchmark_data_smithi178_80607_object11716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:02.989556+0000", 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 146.54591668500001, 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.036404288, 2024-09-13T22:49:29.614 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119761 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989556+0000", 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989556+0000", 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989559+0000", 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.41e-06 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989570+0000", 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0773e-05 2024-09-13T22:49:29.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989596+0000", 2024-09-13T22:49:29.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5870000000000001e-05 2024-09-13T22:49:29.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989614+0000", 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8587e-05 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989729+0000", 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114671 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.989753+0000", 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4349999999999999e-05 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.017517+0000", 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0277640539999999 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.017871+0000", 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000353326 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.018230+0000", 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000358907 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.024953+0000", 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006723428 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025083+0000", 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013021300000000001 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025195+0000", 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111641 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025255+0000", 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9833999999999997e-05 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025569+0000", 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031453900000000002 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025683+0000", 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113513 2024-09-13T22:49:29.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025960+0000", 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027717200000000001 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119764 8.5 8:a1aab702:::benchmark_data_smithi178_80607_object11719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:02.996505+0000", 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 146.53896699200001, 2024-09-13T22:49:29.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.030484548, 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119764 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996505+0000", 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996505+0000", 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996510+0000", 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4569999999999998e-06 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996523+0000", 2024-09-13T22:49:29.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3536999999999999e-05 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996563+0000", 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9981999999999998e-05 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996585+0000", 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1270000000000001e-05 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996765+0000", 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017991700000000001 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.996791+0000", 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6047e-05 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.021555+0000", 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0247641860000001 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.021830+0000", 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000275178 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.022123+0000", 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000293326 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025297+0000", 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031737139999999998 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025340+0000", 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2704000000000003e-05 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.025979+0000", 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063945400000000002 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.026635+0000", 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00065580499999999997 2024-09-13T22:49:29.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.026690+0000", 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4614999999999998e-05 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.026758+0000", 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8113000000000003e-05 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.026990+0000", 2024-09-13T22:49:29.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023224300000000001 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:190465 8.5 8:a77ba6aa:::benchmark_data_smithi178_80607_object82420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:48:25.394631+0000", 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 64.140841864999999, 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67167511199999996, 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.633 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 190465 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394631+0000", 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394631+0000", 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394633+0000", 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.802e-06 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394640+0000", 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7730000000000002e-06 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394656+0000", 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5889999999999999e-05 2024-09-13T22:49:29.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394664+0000", 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7770000000000002e-06 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394717+0000", 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3610999999999999e-05 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394808+0000", 2024-09-13T22:49:29.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0870000000000002e-05 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394932+0000", 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012340699999999999 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.398972+0000", 2024-09-13T22:49:29.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004040493 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.399640+0000", 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000668215 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.399866+0000", 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000225472 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.065542+0000", 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66567600199999999 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.065971+0000", 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00042878700000000003 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.066061+0000", 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0665e-05 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.066306+0000", 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024434800000000003 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:198594 8.5 8:aaf33b74:::benchmark_data_smithi178_80607_object90549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:48:34.327876+0000", 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.207596352000003, 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.248699328, 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.640 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 198594 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.327876+0000", 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.327876+0000", 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.327877+0000", 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.04e-06 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.327881+0000", 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2420000000000002e-06 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.327892+0000", 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1059e-05 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.327899+0000", 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.251e-06 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.332721+0000", 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048220490000000001 2024-09-13T22:49:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.332857+0000", 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136804 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.332992+0000", 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013457200000000001 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.336780+0000", 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037884170000000001 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.337109+0000", 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032812 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.337816+0000", 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00070757600000000002 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.341141+0000", 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003324642 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.576371+0000", 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23523033199999999 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.576438+0000", 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6759000000000002e-05 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:34.576575+0000", 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137465 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:199108 8.5 8:a55c15bc:::benchmark_data_smithi178_80607_object91063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:48:35.012997+0000", 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 54.522475053999997, 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27877702500000001, 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.647 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199108 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.012997+0000", 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.012997+0000", 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.012999+0000", 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1229999999999998e-06 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.013005+0000", 2024-09-13T22:49:29.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.575e-06 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.013020+0000", 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4554e-05 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.013027+0000", 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5259999999999996e-06 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.013074+0000", 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6558999999999999e-05 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.013161+0000", 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7455999999999997e-05 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.013240+0000", 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8666999999999997e-05 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017237+0000", 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039972469999999998 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.023015+0000", 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057780339999999996 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.025116+0000", 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021008210000000001 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.271955+0000", 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24683867400000001 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.291497+0000", 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019541995 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.291546+0000", 2024-09-13T22:49:29.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9619e-05 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.291774+0000", 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000228175 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:199112 8.5 8:a6c5e879:::benchmark_data_smithi178_80607_object91067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:48:35.017397+0000", 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 54.518075281999998, 2024-09-13T22:49:29.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27485130600000002, 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199112 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017397+0000", 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017397+0000", 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017399+0000", 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6139999999999999e-06 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017404+0000", 2024-09-13T22:49:29.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017413+0000", 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9789999999999999e-06 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017418+0000", 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017455+0000", 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6869000000000002e-05 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017588+0000", 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132874 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.017745+0000", 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015763100000000001 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.022140+0000", 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043944190000000001 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.023190+0000", 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010501320000000001 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.025158+0000", 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001967744 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.272148+0000", 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.246989922 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.291880+0000", 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019731955999999998 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.291978+0000", 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8828999999999999e-05 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.292248+0000", 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000270086 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:199120 8.5 8:a75114a1:::benchmark_data_smithi178_80607_object91075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:48:35.026765+0000", 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 54.508707325000003, 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26581132800000001, 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199120 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026765+0000", 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026765+0000", 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026767+0000", 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3020000000000002e-06 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026774+0000", 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3520000000000003e-06 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026790+0000", 2024-09-13T22:49:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5991e-05 2024-09-13T22:49:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026798+0000", 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5459999999999992e-06 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026861+0000", 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3224000000000003e-05 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.026988+0000", 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012693399999999999 2024-09-13T22:49:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.027117+0000", 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000128472 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.031671+0000", 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045542120000000002 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.031734+0000", 2024-09-13T22:49:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2535999999999999e-05 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.031754+0000", 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0313e-05 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.272282+0000", 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.240527775 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.292344+0000", 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020062112 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.292406+0000", 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2234000000000001e-05 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:35.292576+0000", 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017032500000000001 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:29.669 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:49:29.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:29.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.826 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:29.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:29.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:29.954 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-13T22:49:29.954 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:29.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:30.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:30.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:29 smithi192 bash[24608]: cluster 2024-09-13T22:49:28.368407+0000 osd.5 (osd.5) 267 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:49:30.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:29 smithi192 bash[24608]: cluster 2024-09-13T22:49:28.371102+0000 osd.5 (osd.5) 268 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:49:30.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:29 smithi192 bash[24608]: cluster 2024-09-13T22:49:28.467123+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:49:30.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:29 smithi192 bash[24608]: cluster 2024-09-13T22:49:28.469340+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:49:30.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:29 smithi192 bash[24608]: audit 2024-09-13T22:49:29.911279+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.178:0/3967608010' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:49:30.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:29 smithi192 bash[24608]: audit 2024-09-13T22:49:29.911833+0000 mon.a (mon.0) 1288 : audit [INF] from='client.25286 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:49:30.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:30.179 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:49:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:49:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[29889]: cluster 2024-09-13T22:49:28.368407+0000 osd.5 (osd.5) 267 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:49:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[29889]: cluster 2024-09-13T22:49:28.371102+0000 osd.5 (osd.5) 268 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:49:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[29889]: cluster 2024-09-13T22:49:28.467123+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:49:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[29889]: cluster 2024-09-13T22:49:28.469340+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:49:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[29889]: audit 2024-09-13T22:49:29.911279+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.178:0/3967608010' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:49:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[29889]: audit 2024-09-13T22:49:29.911833+0000 mon.a (mon.0) 1288 : audit [INF] from='client.25286 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:49:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[22840]: cluster 2024-09-13T22:49:28.368407+0000 osd.5 (osd.5) 267 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:49:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[22840]: cluster 2024-09-13T22:49:28.371102+0000 osd.5 (osd.5) 268 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:49:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[22840]: cluster 2024-09-13T22:49:28.467123+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:49:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[22840]: cluster 2024-09-13T22:49:28.469340+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:49:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[22840]: audit 2024-09-13T22:49:29.911279+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.178:0/3967608010' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:49:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:29 smithi178 bash[22840]: audit 2024-09-13T22:49:29.911833+0000 mon.a (mon.0) 1288 : audit [INF] from='client.25286 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:49:30.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:30.213 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:30.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:30.335 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:30.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T22:49:30.438 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:30.463 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:30.523 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:49:30.525 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:49:30.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:30.573 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[50858]: logger=cleanup t=2024-09-13T22:49:30.313212985Z level=info msg="Completed cleanup jobs" duration=331.812173ms 2024-09-13T22:49:30.584 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:30.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:30.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:30.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:30.841 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:30.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:30.975 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:30.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: cluster 2024-09-13T22:49:29.114022+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.2 deep-scrub starts 2024-09-13T22:49:30.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: cluster 2024-09-13T22:49:29.115652+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.2 deep-scrub ok 2024-09-13T22:49:30.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: cluster 2024-09-13T22:49:29.892181+0000 osd.7 (osd.7) 259 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:49:30.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: cluster 2024-09-13T22:49:29.894294+0000 osd.7 (osd.7) 260 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:49:30.976 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: audit 2024-09-13T22:49:29.952134+0000 mon.a (mon.0) 1289 : audit [INF] from='client.25286 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: cluster 2024-09-13T22:49:29.958753+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: audit 2024-09-13T22:49:30.511066+0000 mgr.y (mgr.14613) 441 : audit [DBG] from='client.25292 v1:172.21.15.178:0/504562198' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[29889]: cluster 2024-09-13T22:49:30.685565+0000 mgr.y (mgr.14613) 442 : cluster [DBG] pgmap v492: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: cluster 2024-09-13T22:49:29.114022+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.2 deep-scrub starts 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: cluster 2024-09-13T22:49:29.115652+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.2 deep-scrub ok 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: cluster 2024-09-13T22:49:29.892181+0000 osd.7 (osd.7) 259 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: cluster 2024-09-13T22:49:29.894294+0000 osd.7 (osd.7) 260 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:49:30.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: audit 2024-09-13T22:49:29.952134+0000 mon.a (mon.0) 1289 : audit [INF] from='client.25286 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:49:30.978 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: cluster 2024-09-13T22:49:29.958753+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-09-13T22:49:30.978 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: audit 2024-09-13T22:49:30.511066+0000 mgr.y (mgr.14613) 441 : audit [DBG] from='client.25292 v1:172.21.15.178:0/504562198' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:49:30.978 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:30 smithi178 bash[22840]: cluster 2024-09-13T22:49:30.685565+0000 mgr.y (mgr.14613) 442 : cluster [DBG] pgmap v492: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:49:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:30.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:31.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:31.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: cluster 2024-09-13T22:49:29.114022+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.2 deep-scrub starts 2024-09-13T22:49:31.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: cluster 2024-09-13T22:49:29.115652+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.2 deep-scrub ok 2024-09-13T22:49:31.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: cluster 2024-09-13T22:49:29.892181+0000 osd.7 (osd.7) 259 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:49:31.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: cluster 2024-09-13T22:49:29.894294+0000 osd.7 (osd.7) 260 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:49:31.093 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: audit 2024-09-13T22:49:29.952134+0000 mon.a (mon.0) 1289 : audit [INF] from='client.25286 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:49:31.093 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: cluster 2024-09-13T22:49:29.958753+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-09-13T22:49:31.093 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: audit 2024-09-13T22:49:30.511066+0000 mgr.y (mgr.14613) 441 : audit [DBG] from='client.25292 v1:172.21.15.178:0/504562198' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:49:31.093 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:30 smithi192 bash[24608]: cluster 2024-09-13T22:49:30.685565+0000 mgr.y (mgr.14613) 442 : cluster [DBG] pgmap v492: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:49:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:31.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:31.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:31.374 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:31.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:31.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:31.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:31.634 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:31.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:31.764 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:31.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:31.895 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:31.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:31.912 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-13T22:49:29.952917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1845,"num_read_kb":1597,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2304,"ondisk_log_size":2304,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2025416,"kb_used_data":17616,"kb_used_omap":10,"kb_used_meta":2007669,"kb_avail":654190648,"statfs":{"total":671965249536,"available":669891223552,"internally_reserved":0,"allocated":18038784,"data_stored":15110417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2055853163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.127175"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":392,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266346+0000","last_change":"2024-09-13T22:45:02.514835+0000","last_active":"2024-09-13T22:49:24.266346+0000","last_peered":"2024-09-13T22:49:24.266346+0000","last_clean":"2024-09-13T22:49:24.266346+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:49:24.266346+0000","last_undegraded":"2024-09-13T22:49:24.266346+0000","last_fullsized":"2024-09-13T22:49:24.266346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:45:02.514742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":433,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933882+0000","last_change":"2024-09-13T22:49:21.195224+0000","last_active":"2024-09-13T22:49:23.933882+0000","last_peered":"2024-09-13T22:49:23.933882+0000","last_clean":"2024-09-13T22:49:23.933882+0000","last_became_active":"2024-09-13T22:47:03.934779+0000","last_became_peered":"2024-09-13T22:47:03.934779+0000","last_unstale":"2024-09-13T22:49:23.933882+0000","last_undegraded":"2024-09-13T22:49:23.933882+0000","last_fullsized":"2024-09-13T22:49:23.933882+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:21.195029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.195029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:41.877653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"228'25","reported_seq":444,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266819+0000","last_change":"2024-09-13T22:47:03.372631+0000","last_active":"2024-09-13T22:49:24.266819+0000","last_peered":"2024-09-13T22:49:24.266819+0000","last_clean":"2024-09-13T22:49:24.266819+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:49:24.266819+0000","last_undegraded":"2024-09-13T22:49:24.266819+0000","last_fullsized":"2024-09-13T22:49:24.266819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:47:03.372362+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:47:03.372362+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":107,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383804+0000","last_change":"2024-09-13T22:48:14.210156+0000","last_active":"2024-09-13T22:49:24.383804+0000","last_peered":"2024-09-13T22:49:24.383804+0000","last_clean":"2024-09-13T22:49:24.383804+0000","last_became_active":"2024-09-13T22:47:03.898560+0000","last_became_peered":"2024-09-13T22:47:03.898560+0000","last_unstale":"2024-09-13T22:49:24.383804+0000","last_undegraded":"2024-09-13T22:49:24.383804+0000","last_fullsized":"2024-09-13T22:49:24.383804+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.210087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.210087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:34.726534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":413,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266936+0000","last_change":"2024-09-13T22:49:16.307170+0000","last_active":"2024-09-13T22:49:24.266936+0000","last_peered":"2024-09-13T22:49:24.266936+0000","last_clean":"2024-09-13T22:49:24.266936+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:49:24.266936+0000","last_undegraded":"2024-09-13T22:49:24.266936+0000","last_fullsized":"2024-09-13T22:49:24.266936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.306966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.306966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:22.675921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"228'24","reported_seq":453,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933633+0000","last_change":"2024-09-13T22:49:22.149867+0000","last_active":"2024-09-13T22:49:23.933633+0000","last_peered":"2024-09-13T22:49:23.933633+0000","last_clean":"2024-09-13T22:49:23.933633+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:49:23.933633+0000","last_undegraded":"2024-09-13T22:49:23.933633+0000","last_fullsized":"2024-09-13T22:49:23.933633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:49:22.149762+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.149762+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:40.311825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":404,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382030+0000","last_change":"2024-09-13T22:48:30.135471+0000","last_active":"2024-09-13T22:49:24.382030+0000","last_peered":"2024-09-13T22:49:24.382030+0000","last_clean":"2024-09-13T22:49:24.382030+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:49:24.382030+0000","last_undegraded":"2024-09-13T22:49:24.382030+0000","last_fullsized":"2024-09-13T22:49:24.382030+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.135258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.135258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:49.921055+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":106,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941727+0000","last_change":"2024-09-13T22:48:27.804033+0000","last_active":"2024-09-13T22:49:23.941727+0000","last_peered":"2024-09-13T22:49:23.941727+0000","last_clean":"2024-09-13T22:49:23.941727+0000","last_became_active":"2024-09-13T22:47:03.927913+0000","last_became_peered":"2024-09-13T22:47:03.927913+0000","last_unstale":"2024-09-13T22:49:23.941727+0000","last_undegraded":"2024-09-13T22:49:23.941727+0000","last_fullsized":"2024-09-13T22:49:23.941727+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.803924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.803924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:33.275233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":411,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385576+0000","last_change":"2024-09-13T22:48:11.350608+0000","last_active":"2024-09-13T22:49:24.385576+0000","last_peered":"2024-09-13T22:49:24.385576+0000","last_clean":"2024-09-13T22:49:24.385576+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:49:24.385576+0000","last_undegraded":"2024-09-13T22:49:24.385576+0000","last_fullsized":"2024-09-13T22:49:24.385576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.350541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.350541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"228'34","reported_seq":487,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943130+0000","last_change":"2024-09-13T22:48:22.907608+0000","last_active":"2024-09-13T22:49:23.943130+0000","last_peered":"2024-09-13T22:49:23.943130+0000","last_clean":"2024-09-13T22:49:23.943130+0000","last_became_active":"2024-09-13T22:47:03.933115+0000","last_became_peered":"2024-09-13T22:47:03.933115+0000","last_unstale":"2024-09-13T22:49:23.943130+0000","last_undegraded":"2024-09-13T22:49:23.943130+0000","last_fullsized":"2024-09-13T22:49:23.943130+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:22.907495+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.907495+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:46.887648+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":401,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384209+0000","last_change":"2024-09-13T22:48:10.365596+0000","last_active":"2024-09-13T22:49:24.384209+0000","last_peered":"2024-09-13T22:49:24.384209+0000","last_clean":"2024-09-13T22:49:24.384209+0000","last_became_active":"2024-09-13T22:47:03.896417+0000","last_became_peered":"2024-09-13T22:47:03.896417+0000","last_unstale":"2024-09-13T22:49:24.384209+0000","last_undegraded":"2024-09-13T22:49:24.384209+0000","last_fullsized":"2024-09-13T22:49:24.384209+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:10.365482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:48:10.365482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":399,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941799+0000","last_change":"2024-09-13T22:48:03.951130+0000","last_active":"2024-09-13T22:49:23.941799+0000","last_peered":"2024-09-13T22:49:23.941799+0000","last_clean":"2024-09-13T22:49:23.941799+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:49:23.941799+0000","last_undegraded":"2024-09-13T22:49:23.941799+0000","last_fullsized":"2024-09-13T22:49:23.941799+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:03.950989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:48:03.950989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:30.672678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":411,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385722+0000","last_change":"2024-09-13T22:47:03.640700+0000","last_active":"2024-09-13T22:49:24.385722+0000","last_peered":"2024-09-13T22:49:24.385722+0000","last_clean":"2024-09-13T22:49:24.385722+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:49:24.385722+0000","last_undegraded":"2024-09-13T22:49:24.385722+0000","last_fullsized":"2024-09-13T22:49:24.385722+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:03.640577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:47:03.640577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"228'18","reported_seq":465,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943912+0000","last_change":"2024-09-13T22:48:05.917260+0000","last_active":"2024-09-13T22:49:23.943912+0000","last_peered":"2024-09-13T22:49:23.943912+0000","last_clean":"2024-09-13T22:49:23.943912+0000","last_became_active":"2024-09-13T22:47:03.937556+0000","last_became_peered":"2024-09-13T22:47:03.937556+0000","last_unstale":"2024-09-13T22:49:23.943912+0000","last_undegraded":"2024-09-13T22:49:23.943912+0000","last_fullsized":"2024-09-13T22:49:23.943912+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:05.917128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:05.917128+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:26.145188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":403,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942387+0000","last_change":"2024-09-13T22:46:47.974936+0000","last_active":"2024-09-13T22:49:23.942387+0000","last_peered":"2024-09-13T22:49:23.942387+0000","last_clean":"2024-09-13T22:49:23.942387+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:49:23.942387+0000","last_undegraded":"2024-09-13T22:49:23.942387+0000","last_fullsized":"2024-09-13T22:49:23.942387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:47.974834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:47.974834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933193+0000","last_change":"2024-09-13T22:46:02.912752+0000","last_active":"2024-09-13T22:49:23.933193+0000","last_peered":"2024-09-13T22:49:23.933193+0000","last_clean":"2024-09-13T22:49:23.933193+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:49:23.933193+0000","last_undegraded":"2024-09-13T22:49:23.933193+0000","last_fullsized":"2024-09-13T22:49:23.933193+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:02.912649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.912649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.381613+0000","last_change":"2024-09-13T22:49:19.908624+0000","last_active":"2024-09-13T22:49:24.381613+0000","last_peered":"2024-09-13T22:49:24.381613+0000","last_clean":"2024-09-13T22:49:24.381613+0000","last_became_active":"2024-09-13T22:49:19.907655+0000","last_became_peered":"2024-09-13T22:49:19.907655+0000","last_unstale":"2024-09-13T22:49:24.381613+0000","last_undegraded":"2024-09-13T22:49:24.381613+0000","last_fullsized":"2024-09-13T22:49:24.381613+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:26.840714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":465,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941326+0000","last_change":"2024-09-13T22:48:36.940636+0000","last_active":"2024-09-13T22:49:23.941326+0000","last_peered":"2024-09-13T22:49:23.941326+0000","last_clean":"2024-09-13T22:49:23.941326+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:49:23.941326+0000","last_undegraded":"2024-09-13T22:49:23.941326+0000","last_fullsized":"2024-09-13T22:49:23.941326+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:36.940280+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:36.940280+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:51.842501+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"228'44","reported_seq":514,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384171+0000","last_change":"2024-09-13T22:48:11.242327+0000","last_active":"2024-09-13T22:49:24.384171+0000","last_peered":"2024-09-13T22:49:24.384171+0000","last_clean":"2024-09-13T22:49:24.384171+0000","last_became_active":"2024-09-13T22:47:03.918330+0000","last_became_peered":"2024-09-13T22:47:03.918330+0000","last_unstale":"2024-09-13T22:49:24.384171+0000","last_undegraded":"2024-09-13T22:49:24.384171+0000","last_fullsized":"2024-09-13T22:49:24.384171+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:48:11.242204+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.242204+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:38.853596+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":418,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:26.959927+0000","last_change":"2024-09-13T22:49:26.959984+0000","last_active":"2024-09-13T22:49:26.959927+0000","last_peered":"2024-09-13T22:49:26.959927+0000","last_clean":"2024-09-13T22:49:26.959927+0000","last_became_active":"2024-09-13T22:47:03.927239+0000","last_became_peered":"2024-09-13T22:47:03.927239+0000","last_unstale":"2024-09-13T22:49:26.959927+0000","last_undegraded":"2024-09-13T22:49:26.959927+0000","last_fullsized":"2024-09-13T22:49:26.959927+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:26.959784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.959784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:28.018024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":419,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:26.457379+0000","last_change":"2024-09-13T22:49:26.457433+0000","last_active":"2024-09-13T22:49:26.457379+0000","last_peered":"2024-09-13T22:49:26.457379+0000","last_clean":"2024-09-13T22:49:26.457379+0000","last_became_active":"2024-09-13T22:47:03.899126+0000","last_became_peered":"2024-09-13T22:47:03.899126+0000","last_unstale":"2024-09-13T22:49:26.457379+0000","last_undegraded":"2024-09-13T22:49:26.457379+0000","last_fullsized":"2024-09-13T22:49:26.457379+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:26.457236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.457236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:48.937660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":423,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.265166+0000","last_change":"2024-09-13T22:49:20.264042+0000","last_active":"2024-09-13T22:49:24.265166+0000","last_peered":"2024-09-13T22:49:24.265166+0000","last_clean":"2024-09-13T22:49:24.265166+0000","last_became_active":"2024-09-13T22:47:03.928698+0000","last_became_peered":"2024-09-13T22:47:03.928698+0000","last_unstale":"2024-09-13T22:49:24.265166+0000","last_undegraded":"2024-09-13T22:49:24.265166+0000","last_fullsized":"2024-09-13T22:49:24.265166+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:20.263925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.263925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:28.134290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266239+0000","last_change":"2024-09-13T22:49:19.909371+0000","last_active":"2024-09-13T22:49:24.266239+0000","last_peered":"2024-09-13T22:49:24.266239+0000","last_clean":"2024-09-13T22:49:24.266239+0000","last_became_active":"2024-09-13T22:49:19.908686+0000","last_became_peered":"2024-09-13T22:49:19.908686+0000","last_unstale":"2024-09-13T22:49:24.266239+0000","last_undegraded":"2024-09-13T22:49:24.266239+0000","last_fullsized":"2024-09-13T22:49:24.266239+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:41.030497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":690,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384131+0000","last_change":"2024-09-13T22:49:20.975665+0000","last_active":"2024-09-13T22:49:24.384131+0000","last_peered":"2024-09-13T22:49:24.384131+0000","last_clean":"2024-09-13T22:49:24.384131+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:49:24.384131+0000","last_undegraded":"2024-09-13T22:49:24.384131+0000","last_fullsized":"2024-09-13T22:49:24.384131+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:20.975463+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.975463+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:30.138844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"228'19","reported_seq":454,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266975+0000","last_change":"2024-09-13T22:48:13.728920+0000","last_active":"2024-09-13T22:49:24.266975+0000","last_peered":"2024-09-13T22:49:24.266975+0000","last_clean":"2024-09-13T22:49:24.266975+0000","last_became_active":"2024-09-13T22:47:03.889214+0000","last_became_peered":"2024-09-13T22:47:03.889214+0000","last_unstale":"2024-09-13T22:49:24.266975+0000","last_undegraded":"2024-09-13T22:49:24.266975+0000","last_fullsized":"2024-09-13T22:49:24.266975+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:13.725631+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.725631+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.219357+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":421,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:25.446103+0000","last_change":"2024-09-13T22:49:25.446162+0000","last_active":"2024-09-13T22:49:25.446103+0000","last_peered":"2024-09-13T22:49:25.446103+0000","last_clean":"2024-09-13T22:49:25.446103+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:49:25.446103+0000","last_undegraded":"2024-09-13T22:49:25.446103+0000","last_fullsized":"2024-09-13T22:49:25.446103+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:25.445965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:25.445965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:50.875335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"82'11","reported_seq":565,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.265117+0000","last_change":"2024-09-13T22:48:28.189734+0000","last_active":"2024-09-13T22:49:24.265117+0000","last_peered":"2024-09-13T22:49:24.265117+0000","last_clean":"2024-09-13T22:49:24.265117+0000","last_became_active":"2024-09-13T22:37:44.878630+0000","last_became_peered":"2024-09-13T22:37:44.878630+0000","last_unstale":"2024-09-13T22:49:24.265117+0000","last_undegraded":"2024-09-13T22:49:24.265117+0000","last_fullsized":"2024-09-13T22:49:24.265117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:28.189507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.189507+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.536800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":107,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382797+0000","last_change":"2024-09-13T22:48:26.091094+0000","last_active":"2024-09-13T22:49:24.382797+0000","last_peered":"2024-09-13T22:49:24.382797+0000","last_clean":"2024-09-13T22:49:24.382797+0000","last_became_active":"2024-09-13T22:47:03.901543+0000","last_became_peered":"2024-09-13T22:47:03.901543+0000","last_unstale":"2024-09-13T22:49:24.382797+0000","last_undegraded":"2024-09-13T22:49:24.382797+0000","last_fullsized":"2024-09-13T22:49:24.382797+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.090972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.090972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:32.074163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266192+0000","last_change":"2024-09-13T22:49:19.826930+0000","last_active":"2024-09-13T22:49:24.266192+0000","last_peered":"2024-09-13T22:49:24.266192+0000","last_clean":"2024-09-13T22:49:24.266192+0000","last_became_active":"2024-09-13T22:49:19.826580+0000","last_became_peered":"2024-09-13T22:49:19.826580+0000","last_unstale":"2024-09-13T22:49:24.266192+0000","last_undegraded":"2024-09-13T22:49:24.266192+0000","last_fullsized":"2024-09-13T22:49:24.266192+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:19.090894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":465,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941834+0000","last_change":"2024-09-13T22:48:24.860899+0000","last_active":"2024-09-13T22:49:23.941834+0000","last_peered":"2024-09-13T22:49:23.941834+0000","last_clean":"2024-09-13T22:49:23.941834+0000","last_became_active":"2024-09-13T22:45:58.880701+0000","last_became_peered":"2024-09-13T22:45:58.880701+0000","last_unstale":"2024-09-13T22:49:23.941834+0000","last_undegraded":"2024-09-13T22:49:23.941834+0000","last_fullsized":"2024-09-13T22:49:23.941834+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:24.860776+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.860776+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:53.211654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"228'27","reported_seq":452,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267267+0000","last_change":"2024-09-13T22:48:23.929452+0000","last_active":"2024-09-13T22:49:24.267267+0000","last_peered":"2024-09-13T22:49:24.267267+0000","last_clean":"2024-09-13T22:49:24.267267+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:49:24.267267+0000","last_undegraded":"2024-09-13T22:49:24.267267+0000","last_fullsized":"2024-09-13T22:49:24.267267+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:48:23.929342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.929342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:43.893724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":421,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382163+0000","last_change":"2024-09-13T22:48:37.212544+0000","last_active":"2024-09-13T22:49:24.382163+0000","last_peered":"2024-09-13T22:49:24.382163+0000","last_clean":"2024-09-13T22:49:24.382163+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:49:24.382163+0000","last_undegraded":"2024-09-13T22:49:24.382163+0000","last_fullsized":"2024-09-13T22:49:24.382163+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.201926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.201926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.622463+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"211'13","reported_seq":229,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943237+0000","last_change":"2024-09-13T22:48:08.989818+0000","last_active":"2024-09-13T22:49:23.943237+0000","last_peered":"2024-09-13T22:49:23.943237+0000","last_clean":"2024-09-13T22:49:23.943237+0000","last_became_active":"2024-09-13T22:47:03.901548+0000","last_became_peered":"2024-09-13T22:47:03.901548+0000","last_unstale":"2024-09-13T22:49:23.943237+0000","last_undegraded":"2024-09-13T22:49:23.943237+0000","last_fullsized":"2024-09-13T22:49:23.943237+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:08.989696+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.989696+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:27.865412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942954+0000","last_change":"2024-09-13T22:48:39.747979+0000","last_active":"2024-09-13T22:49:23.942954+0000","last_peered":"2024-09-13T22:49:23.942954+0000","last_clean":"2024-09-13T22:49:23.942954+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:49:23.942954+0000","last_undegraded":"2024-09-13T22:49:23.942954+0000","last_fullsized":"2024-09-13T22:49:23.942954+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.747881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.747881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:56.271576+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383935+0000","last_change":"2024-09-13T22:49:19.839310+0000","last_active":"2024-09-13T22:49:24.383935+0000","last_peered":"2024-09-13T22:49:24.383935+0000","last_clean":"2024-09-13T22:49:24.383935+0000","last_became_active":"2024-09-13T22:49:19.838528+0000","last_became_peered":"2024-09-13T22:49:19.838528+0000","last_unstale":"2024-09-13T22:49:24.383935+0000","last_undegraded":"2024-09-13T22:49:24.383935+0000","last_fullsized":"2024-09-13T22:49:24.383935+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:29.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"228'18","reported_seq":170,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.264166+0000","last_change":"2024-09-13T22:48:20.191655+0000","last_active":"2024-09-13T22:49:24.264166+0000","last_peered":"2024-09-13T22:49:24.264166+0000","last_clean":"2024-09-13T22:49:24.264166+0000","last_became_active":"2024-09-13T22:47:03.927850+0000","last_became_peered":"2024-09-13T22:47:03.927850+0000","last_unstale":"2024-09-13T22:49:24.264166+0000","last_undegraded":"2024-09-13T22:49:24.264166+0000","last_fullsized":"2024-09-13T22:49:24.264166+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:20.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.191528+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:38.634756+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":404,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384077+0000","last_change":"2024-09-13T22:47:02.999857+0000","last_active":"2024-09-13T22:49:24.384077+0000","last_peered":"2024-09-13T22:49:24.384077+0000","last_clean":"2024-09-13T22:49:24.384077+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:49:24.384077+0000","last_undegraded":"2024-09-13T22:49:24.384077+0000","last_fullsized":"2024-09-13T22:49:24.384077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:47:02.999680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:47:02.999680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385303+0000","last_change":"2024-09-13T22:48:27.194380+0000","last_active":"2024-09-13T22:49:24.385303+0000","last_peered":"2024-09-13T22:49:24.385303+0000","last_clean":"2024-09-13T22:49:24.385303+0000","last_became_active":"2024-09-13T22:47:03.936969+0000","last_became_peered":"2024-09-13T22:47:03.936969+0000","last_unstale":"2024-09-13T22:49:24.385303+0000","last_undegraded":"2024-09-13T22:49:24.385303+0000","last_fullsized":"2024-09-13T22:49:24.385303+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.194219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.194219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:54.254732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":393,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266010+0000","last_change":"2024-09-13T22:45:03.478116+0000","last_active":"2024-09-13T22:49:24.266010+0000","last_peered":"2024-09-13T22:49:24.266010+0000","last_clean":"2024-09-13T22:49:24.266010+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:49:24.266010+0000","last_undegraded":"2024-09-13T22:49:24.266010+0000","last_fullsized":"2024-09-13T22:49:24.266010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.477932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.477932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266803+0000","last_change":"2024-09-13T22:49:19.838339+0000","last_active":"2024-09-13T22:49:24.266803+0000","last_peered":"2024-09-13T22:49:24.266803+0000","last_clean":"2024-09-13T22:49:24.266803+0000","last_became_active":"2024-09-13T22:49:19.837978+0000","last_became_peered":"2024-09-13T22:49:19.837978+0000","last_unstale":"2024-09-13T22:49:24.266803+0000","last_undegraded":"2024-09-13T22:49:24.266803+0000","last_fullsized":"2024-09-13T22:49:24.266803+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:20.931950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":457,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267315+0000","last_change":"2024-09-13T22:45:59.360761+0000","last_active":"2024-09-13T22:49:24.267315+0000","last_peered":"2024-09-13T22:49:24.267315+0000","last_clean":"2024-09-13T22:49:24.267315+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:49:24.267315+0000","last_undegraded":"2024-09-13T22:49:24.267315+0000","last_fullsized":"2024-09-13T22:49:24.267315+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.360631+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"228'24","reported_seq":465,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935453+0000","last_change":"2024-09-13T22:47:03.954215+0000","last_active":"2024-09-13T22:49:23.935453+0000","last_peered":"2024-09-13T22:49:23.935453+0000","last_clean":"2024-09-13T22:49:23.935453+0000","last_became_active":"2024-09-13T22:47:03.926045+0000","last_became_peered":"2024-09-13T22:47:03.926045+0000","last_unstale":"2024-09-13T22:49:23.935453+0000","last_undegraded":"2024-09-13T22:49:23.935453+0000","last_fullsized":"2024-09-13T22:49:23.935453+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'12","last_scrub_stamp":"2024-09-13T22:46:56.519330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:56.519330+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":428,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385229+0000","last_change":"2024-09-13T22:48:19.112621+0000","last_active":"2024-09-13T22:49:24.385229+0000","last_peered":"2024-09-13T22:49:24.385229+0000","last_clean":"2024-09-13T22:49:24.385229+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:49:24.385229+0000","last_undegraded":"2024-09-13T22:49:24.385229+0000","last_fullsized":"2024-09-13T22:49:24.385229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:19.112032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:19.112032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.571193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":402,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941783+0000","last_change":"2024-09-13T22:48:34.822821+0000","last_active":"2024-09-13T22:49:23.941783+0000","last_peered":"2024-09-13T22:49:23.941783+0000","last_clean":"2024-09-13T22:49:23.941783+0000","last_became_active":"2024-09-13T22:47:03.942652+0000","last_became_peered":"2024-09-13T22:47:03.942652+0000","last_unstale":"2024-09-13T22:49:23.941783+0000","last_undegraded":"2024-09-13T22:49:23.941783+0000","last_fullsized":"2024-09-13T22:49:23.941783+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:48:34.822707+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.822707+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.491217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"211'12","reported_seq":559,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267002+0000","last_change":"2024-09-13T22:47:03.895068+0000","last_active":"2024-09-13T22:49:24.267002+0000","last_peered":"2024-09-13T22:49:24.267002+0000","last_clean":"2024-09-13T22:49:24.267002+0000","last_became_active":"2024-09-13T22:47:03.881728+0000","last_became_peered":"2024-09-13T22:47:03.881728+0000","last_unstale":"2024-09-13T22:49:24.267002+0000","last_undegraded":"2024-09-13T22:49:24.267002+0000","last_fullsized":"2024-09-13T22:49:24.267002+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:38.494521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:38.494521+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":392,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384387+0000","last_change":"2024-09-13T22:46:56.706548+0000","last_active":"2024-09-13T22:49:24.384387+0000","last_peered":"2024-09-13T22:49:24.384387+0000","last_clean":"2024-09-13T22:49:24.384387+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:49:24.384387+0000","last_undegraded":"2024-09-13T22:49:24.384387+0000","last_fullsized":"2024-09-13T22:49:24.384387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:56.706439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:56.706439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385285+0000","last_change":"2024-09-13T22:49:19.844311+0000","last_active":"2024-09-13T22:49:24.385285+0000","last_peered":"2024-09-13T22:49:24.385285+0000","last_clean":"2024-09-13T22:49:24.385285+0000","last_became_active":"2024-09-13T22:49:19.843295+0000","last_became_peered":"2024-09-13T22:49:19.843295+0000","last_unstale":"2024-09-13T22:49:24.385285+0000","last_undegraded":"2024-09-13T22:49:24.385285+0000","last_fullsized":"2024-09-13T22:49:24.385285+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:39.644765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":431,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:26.230162+0000","last_change":"2024-09-13T22:49:26.230216+0000","last_active":"2024-09-13T22:49:26.230162+0000","last_peered":"2024-09-13T22:49:26.230162+0000","last_clean":"2024-09-13T22:49:26.230162+0000","last_became_active":"2024-09-13T22:47:06.891260+0000","last_became_peered":"2024-09-13T22:47:06.891260+0000","last_unstale":"2024-09-13T22:49:26.230162+0000","last_undegraded":"2024-09-13T22:49:26.230162+0000","last_fullsized":"2024-09-13T22:49:26.230162+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:26.230025+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.230025+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:31.321578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"228'22","reported_seq":447,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267350+0000","last_change":"2024-09-13T22:49:23.376019+0000","last_active":"2024-09-13T22:49:24.267350+0000","last_peered":"2024-09-13T22:49:24.267350+0000","last_clean":"2024-09-13T22:49:24.267350+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:49:24.267350+0000","last_undegraded":"2024-09-13T22:49:24.267350+0000","last_fullsized":"2024-09-13T22:49:24.267350+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:49:23.375848+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:49:23.375848+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:25.230518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":425,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383080+0000","last_change":"2024-09-13T22:47:07.022984+0000","last_active":"2024-09-13T22:49:24.383080+0000","last_peered":"2024-09-13T22:49:24.383080+0000","last_clean":"2024-09-13T22:49:24.383080+0000","last_became_active":"2024-09-13T22:47:03.899279+0000","last_became_peered":"2024-09-13T22:47:03.899279+0000","last_unstale":"2024-09-13T22:49:24.383080+0000","last_undegraded":"2024-09-13T22:49:24.383080+0000","last_fullsized":"2024-09-13T22:49:24.383080+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:47:07.022796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":1092,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267151+0000","last_change":"2024-09-13T22:45:54.489191+0000","last_active":"2024-09-13T22:49:24.267151+0000","last_peered":"2024-09-13T22:49:24.267151+0000","last_clean":"2024-09-13T22:49:24.267151+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:49:24.267151+0000","last_undegraded":"2024-09-13T22:49:24.267151+0000","last_fullsized":"2024-09-13T22:49:24.267151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:45:54.489086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.489086+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":681,"num_read_kb":676,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":106,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266443+0000","last_change":"2024-09-13T22:48:35.511647+0000","last_active":"2024-09-13T22:49:24.266443+0000","last_peered":"2024-09-13T22:49:24.266443+0000","last_clean":"2024-09-13T22:49:24.266443+0000","last_became_active":"2024-09-13T22:48:35.511300+0000","last_became_peered":"2024-09-13T22:48:35.511300+0000","last_unstale":"2024-09-13T22:49:24.266443+0000","last_undegraded":"2024-09-13T22:49:24.266443+0000","last_fullsized":"2024-09-13T22:49:24.266443+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:51.936005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.936005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":99,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266352+0000","last_change":"2024-09-13T22:47:03.902506+0000","last_active":"2024-09-13T22:49:24.266352+0000","last_peered":"2024-09-13T22:49:24.266352+0000","last_clean":"2024-09-13T22:49:24.266352+0000","last_became_active":"2024-09-13T22:47:03.902203+0000","last_became_peered":"2024-09-13T22:47:03.902203+0000","last_unstale":"2024-09-13T22:49:24.266352+0000","last_undegraded":"2024-09-13T22:49:24.266352+0000","last_fullsized":"2024-09-13T22:49:24.266352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:37.838733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:45:37.838733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384774+0000","last_change":"2024-09-13T22:49:19.841243+0000","last_active":"2024-09-13T22:49:24.384774+0000","last_peered":"2024-09-13T22:49:24.384774+0000","last_clean":"2024-09-13T22:49:24.384774+0000","last_became_active":"2024-09-13T22:49:19.840309+0000","last_became_peered":"2024-09-13T22:49:19.840309+0000","last_unstale":"2024-09-13T22:49:24.384774+0000","last_undegraded":"2024-09-13T22:49:24.384774+0000","last_fullsized":"2024-09-13T22:49:24.384774+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:48.689623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":470,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384933+0000","last_change":"2024-09-13T22:48:18.126451+0000","last_active":"2024-09-13T22:49:24.384933+0000","last_peered":"2024-09-13T22:49:24.384933+0000","last_clean":"2024-09-13T22:49:24.384933+0000","last_became_active":"2024-09-13T22:45:58.866066+0000","last_became_peered":"2024-09-13T22:45:58.866066+0000","last_unstale":"2024-09-13T22:49:24.384933+0000","last_undegraded":"2024-09-13T22:49:24.384933+0000","last_fullsized":"2024-09-13T22:49:24.384933+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:18.126322+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:18.126322+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:36.127447+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"228'42","reported_seq":619,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382533+0000","last_change":"2024-09-13T22:48:15.188315+0000","last_active":"2024-09-13T22:49:24.382533+0000","last_peered":"2024-09-13T22:49:24.382533+0000","last_clean":"2024-09-13T22:49:24.382533+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:49:24.382533+0000","last_undegraded":"2024-09-13T22:49:24.382533+0000","last_fullsized":"2024-09-13T22:49:24.382533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'42","last_scrub_stamp":"2024-09-13T22:48:15.188167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.188167+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":418,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385625+0000","last_change":"2024-09-13T22:49:17.960697+0000","last_active":"2024-09-13T22:49:24.385625+0000","last_peered":"2024-09-13T22:49:24.385625+0000","last_clean":"2024-09-13T22:49:24.385625+0000","last_became_active":"2024-09-13T22:47:03.907952+0000","last_became_peered":"2024-09-13T22:47:03.907952+0000","last_unstale":"2024-09-13T22:49:24.385625+0000","last_undegraded":"2024-09-13T22:49:24.385625+0000","last_fullsized":"2024-09-13T22:49:24.385625+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.960501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.960501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:36.493584+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":420,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383998+0000","last_change":"2024-09-13T22:48:15.197402+0000","last_active":"2024-09-13T22:49:24.383998+0000","last_peered":"2024-09-13T22:49:24.383998+0000","last_clean":"2024-09-13T22:49:24.383998+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:49:24.383998+0000","last_undegraded":"2024-09-13T22:49:24.383998+0000","last_fullsized":"2024-09-13T22:49:24.383998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.197323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.197323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:27.579983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":401,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.381898+0000","last_change":"2024-09-13T22:48:11.214965+0000","last_active":"2024-09-13T22:49:24.381898+0000","last_peered":"2024-09-13T22:49:24.381898+0000","last_clean":"2024-09-13T22:49:24.381898+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:49:24.381898+0000","last_undegraded":"2024-09-13T22:49:24.381898+0000","last_fullsized":"2024-09-13T22:49:24.381898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.214834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.214834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:28.493150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943502+0000","last_change":"2024-09-13T22:49:19.836085+0000","last_active":"2024-09-13T22:49:23.943502+0000","last_peered":"2024-09-13T22:49:23.943502+0000","last_clean":"2024-09-13T22:49:23.943502+0000","last_became_active":"2024-09-13T22:49:19.834656+0000","last_became_peered":"2024-09-13T22:49:19.834656+0000","last_unstale":"2024-09-13T22:49:23.943502+0000","last_undegraded":"2024-09-13T22:49:23.943502+0000","last_fullsized":"2024-09-13T22:49:23.943502+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:47.368272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":470,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384723+0000","last_change":"2024-09-13T22:48:22.045389+0000","last_active":"2024-09-13T22:49:24.384723+0000","last_peered":"2024-09-13T22:49:24.384723+0000","last_clean":"2024-09-13T22:49:24.384723+0000","last_became_active":"2024-09-13T22:45:58.862759+0000","last_became_peered":"2024-09-13T22:45:58.862759+0000","last_unstale":"2024-09-13T22:49:24.384723+0000","last_undegraded":"2024-09-13T22:49:24.384723+0000","last_fullsized":"2024-09-13T22:49:24.384723+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:22.045197+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.045197+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:29.935161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"228'36","reported_seq":598,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.944438+0000","last_change":"2024-09-13T22:46:56.049756+0000","last_active":"2024-09-13T22:49:23.944438+0000","last_peered":"2024-09-13T22:49:23.944438+0000","last_clean":"2024-09-13T22:49:23.944438+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:49:23.944438+0000","last_undegraded":"2024-09-13T22:49:23.944438+0000","last_fullsized":"2024-09-13T22:49:23.944438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'24","last_scrub_stamp":"2024-09-13T22:46:56.049683+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:46:56.049683+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":429,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382590+0000","last_change":"2024-09-13T22:48:35.268201+0000","last_active":"2024-09-13T22:49:24.382590+0000","last_peered":"2024-09-13T22:49:24.382590+0000","last_clean":"2024-09-13T22:49:24.382590+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:49:24.382590+0000","last_undegraded":"2024-09-13T22:49:24.382590+0000","last_fullsized":"2024-09-13T22:49:24.382590+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:48:35.268073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.268073+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:52.247618+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":436,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382439+0000","last_change":"2024-09-13T22:49:14.401085+0000","last_active":"2024-09-13T22:49:24.382439+0000","last_peered":"2024-09-13T22:49:24.382439+0000","last_clean":"2024-09-13T22:49:24.382439+0000","last_became_active":"2024-09-13T22:47:03.911332+0000","last_became_peered":"2024-09-13T22:47:03.911332+0000","last_unstale":"2024-09-13T22:49:24.382439+0000","last_undegraded":"2024-09-13T22:49:24.382439+0000","last_fullsized":"2024-09-13T22:49:24.382439+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:49:14.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.400987+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:26.798020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":565,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933602+0000","last_change":"2024-09-13T22:48:28.132907+0000","last_active":"2024-09-13T22:49:23.933602+0000","last_peered":"2024-09-13T22:49:23.933602+0000","last_clean":"2024-09-13T22:49:23.933602+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:49:23.933602+0000","last_undegraded":"2024-09-13T22:49:23.933602+0000","last_fullsized":"2024-09-13T22:49:23.933602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:48:28.132780+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.132780+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:53.044895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":401,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933064+0000","last_change":"2024-09-13T22:46:55.557651+0000","last_active":"2024-09-13T22:49:23.933064+0000","last_peered":"2024-09-13T22:49:23.933064+0000","last_clean":"2024-09-13T22:49:23.933064+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:49:23.933064+0000","last_undegraded":"2024-09-13T22:49:23.933064+0000","last_fullsized":"2024-09-13T22:49:23.933064+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:55.557578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:55.557578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935349+0000","last_change":"2024-09-13T22:49:19.836077+0000","last_active":"2024-09-13T22:49:23.935349+0000","last_peered":"2024-09-13T22:49:23.935349+0000","last_clean":"2024-09-13T22:49:23.935349+0000","last_became_active":"2024-09-13T22:49:19.835836+0000","last_became_peered":"2024-09-13T22:49:19.835836+0000","last_unstale":"2024-09-13T22:49:23.935349+0000","last_undegraded":"2024-09-13T22:49:23.935349+0000","last_fullsized":"2024-09-13T22:49:23.935349+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:37.710222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":465,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941503+0000","last_change":"2024-09-13T22:48:26.832776+0000","last_active":"2024-09-13T22:49:23.941503+0000","last_peered":"2024-09-13T22:49:23.941503+0000","last_clean":"2024-09-13T22:49:23.941503+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:49:23.941503+0000","last_undegraded":"2024-09-13T22:49:23.941503+0000","last_fullsized":"2024-09-13T22:49:23.941503+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.832692+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:56.306610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"228'29","reported_seq":195,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:25.094047+0000","last_change":"2024-09-13T22:49:25.094108+0000","last_active":"2024-09-13T22:49:25.094047+0000","last_peered":"2024-09-13T22:49:25.094047+0000","last_clean":"2024-09-13T22:49:25.094047+0000","last_became_active":"2024-09-13T22:47:03.933746+0000","last_became_peered":"2024-09-13T22:47:03.933746+0000","last_unstale":"2024-09-13T22:49:25.094047+0000","last_undegraded":"2024-09-13T22:49:25.094047+0000","last_fullsized":"2024-09-13T22:49:25.094047+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'29","last_scrub_stamp":"2024-09-13T22:49:25.093976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:25.093976+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:28.770973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":420,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942767+0000","last_change":"2024-09-13T22:48:14.833285+0000","last_active":"2024-09-13T22:49:23.942767+0000","last_peered":"2024-09-13T22:49:23.942767+0000","last_clean":"2024-09-13T22:49:23.942767+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:49:23.942767+0000","last_undegraded":"2024-09-13T22:49:23.942767+0000","last_fullsized":"2024-09-13T22:49:23.942767+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.669505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":419,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.934778+0000","last_change":"2024-09-13T22:49:19.166008+0000","last_active":"2024-09-13T22:49:23.934778+0000","last_peered":"2024-09-13T22:49:23.934778+0000","last_clean":"2024-09-13T22:49:23.934778+0000","last_became_active":"2024-09-13T22:47:03.902594+0000","last_became_peered":"2024-09-13T22:47:03.902594+0000","last_unstale":"2024-09-13T22:49:23.934778+0000","last_undegraded":"2024-09-13T22:49:23.934778+0000","last_fullsized":"2024-09-13T22:49:23.934778+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:19.165932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:49:19.165932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:26.685203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":399,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266789+0000","last_change":"2024-09-13T22:47:03.917191+0000","last_active":"2024-09-13T22:49:24.266789+0000","last_peered":"2024-09-13T22:49:24.266789+0000","last_clean":"2024-09-13T22:49:24.266789+0000","last_became_active":"2024-09-13T22:47:03.916682+0000","last_became_peered":"2024-09-13T22:47:03.916682+0000","last_unstale":"2024-09-13T22:49:24.266789+0000","last_undegraded":"2024-09-13T22:49:24.266789+0000","last_fullsized":"2024-09-13T22:49:24.266789+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.535993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.535993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942528+0000","last_change":"2024-09-13T22:49:19.844519+0000","last_active":"2024-09-13T22:49:23.942528+0000","last_peered":"2024-09-13T22:49:23.942528+0000","last_clean":"2024-09-13T22:49:23.942528+0000","last_became_active":"2024-09-13T22:49:19.840996+0000","last_became_peered":"2024-09-13T22:49:19.840996+0000","last_unstale":"2024-09-13T22:49:23.942528+0000","last_undegraded":"2024-09-13T22:49:23.942528+0000","last_fullsized":"2024-09-13T22:49:23.942528+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.561445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":699,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384372+0000","last_change":"2024-09-13T22:48:38.173367+0000","last_active":"2024-09-13T22:49:24.384372+0000","last_peered":"2024-09-13T22:49:24.384372+0000","last_clean":"2024-09-13T22:49:24.384372+0000","last_became_active":"2024-09-13T22:47:06.894360+0000","last_became_peered":"2024-09-13T22:47:06.894360+0000","last_unstale":"2024-09-13T22:49:24.384372+0000","last_undegraded":"2024-09-13T22:49:24.384372+0000","last_fullsized":"2024-09-13T22:49:24.384372+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:38.173263+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.173263+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:46.545441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"228'38","reported_seq":467,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.381875+0000","last_change":"2024-09-13T22:48:17.165438+0000","last_active":"2024-09-13T22:49:24.381875+0000","last_peered":"2024-09-13T22:49:24.381875+0000","last_clean":"2024-09-13T22:49:24.381875+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:49:24.381875+0000","last_undegraded":"2024-09-13T22:49:24.381875+0000","last_fullsized":"2024-09-13T22:49:24.381875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:48:17.165325+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-09-13T22:40:28.022233+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.165325+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":437,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.271396+0000","last_change":"2024-09-13T22:49:24.271420+0000","last_active":"2024-09-13T22:49:24.271396+0000","last_peered":"2024-09-13T22:49:24.271396+0000","last_clean":"2024-09-13T22:49:24.271396+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:49:24.271396+0000","last_undegraded":"2024-09-13T22:49:24.271396+0000","last_fullsized":"2024-09-13T22:49:24.271396+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:49:24.271342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:24.271342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:38.148279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":421,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.265454+0000","last_change":"2024-09-13T22:48:22.142431+0000","last_active":"2024-09-13T22:49:24.265454+0000","last_peered":"2024-09-13T22:49:24.265454+0000","last_clean":"2024-09-13T22:49:24.265454+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:49:24.265454+0000","last_undegraded":"2024-09-13T22:49:24.265454+0000","last_fullsized":"2024-09-13T22:49:24.265454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:22.142327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.142327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:50.026080+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385461+0000","last_change":"2024-09-13T22:48:17.369617+0000","last_active":"2024-09-13T22:49:24.385461+0000","last_peered":"2024-09-13T22:49:24.385461+0000","last_clean":"2024-09-13T22:49:24.385461+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:49:24.385461+0000","last_undegraded":"2024-09-13T22:49:24.385461+0000","last_fullsized":"2024-09-13T22:49:24.385461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.369549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.369549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:27.278240+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933414+0000","last_change":"2024-09-13T22:49:19.908928+0000","last_active":"2024-09-13T22:49:23.933414+0000","last_peered":"2024-09-13T22:49:23.933414+0000","last_clean":"2024-09-13T22:49:23.933414+0000","last_became_active":"2024-09-13T22:49:19.908470+0000","last_became_peered":"2024-09-13T22:49:19.908470+0000","last_unstale":"2024-09-13T22:49:23.933414+0000","last_undegraded":"2024-09-13T22:49:23.933414+0000","last_fullsized":"2024-09-13T22:49:23.933414+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:47.328099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":474,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382583+0000","last_change":"2024-09-13T22:48:31.174111+0000","last_active":"2024-09-13T22:49:24.382583+0000","last_peered":"2024-09-13T22:49:24.382583+0000","last_clean":"2024-09-13T22:49:24.382583+0000","last_became_active":"2024-09-13T22:47:06.902506+0000","last_became_peered":"2024-09-13T22:47:06.902506+0000","last_unstale":"2024-09-13T22:49:24.382583+0000","last_undegraded":"2024-09-13T22:49:24.382583+0000","last_fullsized":"2024-09-13T22:49:24.382583+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:31.173761+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.802229+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.173761+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:54.370553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"228'28","reported_seq":481,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935585+0000","last_change":"2024-09-13T22:49:11.188566+0000","last_active":"2024-09-13T22:49:23.935585+0000","last_peered":"2024-09-13T22:49:23.935585+0000","last_clean":"2024-09-13T22:49:23.935585+0000","last_became_active":"2024-09-13T22:47:03.921362+0000","last_became_peered":"2024-09-13T22:47:03.921362+0000","last_unstale":"2024-09-13T22:49:23.935585+0000","last_undegraded":"2024-09-13T22:49:23.935585+0000","last_fullsized":"2024-09-13T22:49:23.935585+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'28","last_scrub_stamp":"2024-09-13T22:49:11.188389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.188389+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:23.207192+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":187,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383713+0000","last_change":"2024-09-13T22:46:54.746202+0000","last_active":"2024-09-13T22:49:24.383713+0000","last_peered":"2024-09-13T22:49:24.383713+0000","last_clean":"2024-09-13T22:49:24.383713+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:49:24.383713+0000","last_undegraded":"2024-09-13T22:49:24.383713+0000","last_fullsized":"2024-09-13T22:49:24.383713+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:54.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:46:54.746095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":412,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.944545+0000","last_change":"2024-09-13T22:49:21.447561+0000","last_active":"2024-09-13T22:49:23.944545+0000","last_peered":"2024-09-13T22:49:23.944545+0000","last_clean":"2024-09-13T22:49:23.944545+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:49:23.944545+0000","last_undegraded":"2024-09-13T22:49:23.944545+0000","last_fullsized":"2024-09-13T22:49:23.944545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:21.447387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.447387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:27.672741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":409,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:28.469484+0000","last_change":"2024-09-13T22:49:28.469536+0000","last_active":"2024-09-13T22:49:28.469484+0000","last_peered":"2024-09-13T22:49:28.469484+0000","last_clean":"2024-09-13T22:49:28.469484+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:49:28.469484+0000","last_undegraded":"2024-09-13T22:49:28.469484+0000","last_fullsized":"2024-09-13T22:49:28.469484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:28.469350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:49:28.469350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.153355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267109+0000","last_change":"2024-09-13T22:49:19.832503+0000","last_active":"2024-09-13T22:49:24.267109+0000","last_peered":"2024-09-13T22:49:24.267109+0000","last_clean":"2024-09-13T22:49:24.267109+0000","last_became_active":"2024-09-13T22:49:19.831731+0000","last_became_peered":"2024-09-13T22:49:19.831731+0000","last_unstale":"2024-09-13T22:49:24.267109+0000","last_undegraded":"2024-09-13T22:49:24.267109+0000","last_fullsized":"2024-09-13T22:49:24.267109+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:23.705901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"228'20","reported_seq":431,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266446+0000","last_change":"2024-09-13T22:44:55.563811+0000","last_active":"2024-09-13T22:49:24.266446+0000","last_peered":"2024-09-13T22:49:24.266446+0000","last_clean":"2024-09-13T22:49:24.266446+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:49:24.266446+0000","last_undegraded":"2024-09-13T22:49:24.266446+0000","last_fullsized":"2024-09-13T22:49:24.266446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'10","last_scrub_stamp":"2024-09-13T22:44:55.563593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:44:55.563593+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":420,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942557+0000","last_change":"2024-09-13T22:49:22.420187+0000","last_active":"2024-09-13T22:49:23.942557+0000","last_peered":"2024-09-13T22:49:23.942557+0000","last_clean":"2024-09-13T22:49:23.942557+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:49:23.942557+0000","last_undegraded":"2024-09-13T22:49:23.942557+0000","last_fullsized":"2024-09-13T22:49:23.942557+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:35.501613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":683,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384255+0000","last_change":"2024-09-13T22:48:32.199596+0000","last_active":"2024-09-13T22:49:24.384255+0000","last_peered":"2024-09-13T22:49:24.384255+0000","last_clean":"2024-09-13T22:49:24.384255+0000","last_became_active":"2024-09-13T22:45:53.877322+0000","last_became_peered":"2024-09-13T22:45:53.877322+0000","last_unstale":"2024-09-13T22:49:24.384255+0000","last_undegraded":"2024-09-13T22:49:24.384255+0000","last_fullsized":"2024-09-13T22:49:24.384255+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:32.199490+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.199490+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.200419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":404,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385561+0000","last_change":"2024-09-13T22:48:16.361464+0000","last_active":"2024-09-13T22:49:24.385561+0000","last_peered":"2024-09-13T22:49:24.385561+0000","last_clean":"2024-09-13T22:49:24.385561+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:49:24.385561+0000","last_undegraded":"2024-09-13T22:49:24.385561+0000","last_fullsized":"2024-09-13T22:49:24.385561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:16.361322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:48:16.361322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.692113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":414,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.944477+0000","last_change":"2024-09-13T22:48:26.076140+0000","last_active":"2024-09-13T22:49:23.944477+0000","last_peered":"2024-09-13T22:49:23.944477+0000","last_clean":"2024-09-13T22:49:23.944477+0000","last_became_active":"2024-09-13T22:47:03.910753+0000","last_became_peered":"2024-09-13T22:47:03.910753+0000","last_unstale":"2024-09-13T22:49:23.944477+0000","last_undegraded":"2024-09-13T22:49:23.944477+0000","last_fullsized":"2024-09-13T22:49:23.944477+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.075948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.075948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.525224+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.944285+0000","last_change":"2024-09-13T22:49:19.844752+0000","last_active":"2024-09-13T22:49:23.944285+0000","last_peered":"2024-09-13T22:49:23.944285+0000","last_clean":"2024-09-13T22:49:23.944285+0000","last_became_active":"2024-09-13T22:49:19.841262+0000","last_became_peered":"2024-09-13T22:49:19.841262+0000","last_unstale":"2024-09-13T22:49:23.944285+0000","last_undegraded":"2024-09-13T22:49:23.944285+0000","last_fullsized":"2024-09-13T22:49:23.944285+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:35.216519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"228'31","reported_seq":457,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385561+0000","last_change":"2024-09-13T22:49:13.909885+0000","last_active":"2024-09-13T22:49:24.385561+0000","last_peered":"2024-09-13T22:49:24.385561+0000","last_clean":"2024-09-13T22:49:24.385561+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:49:24.385561+0000","last_undegraded":"2024-09-13T22:49:24.385561+0000","last_fullsized":"2024-09-13T22:49:24.385561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'31","last_scrub_stamp":"2024-09-13T22:49:13.909765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:13.909765+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:32.029113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":429,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.265387+0000","last_change":"2024-09-13T22:48:34.249596+0000","last_active":"2024-09-13T22:49:24.265387+0000","last_peered":"2024-09-13T22:49:24.265387+0000","last_clean":"2024-09-13T22:49:24.265387+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:49:24.265387+0000","last_undegraded":"2024-09-13T22:49:24.265387+0000","last_fullsized":"2024-09-13T22:49:24.265387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:34.249408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.249408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.152002+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":413,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.264161+0000","last_change":"2024-09-13T22:48:35.269277+0000","last_active":"2024-09-13T22:49:24.264161+0000","last_peered":"2024-09-13T22:49:24.264161+0000","last_clean":"2024-09-13T22:49:24.264161+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:49:24.264161+0000","last_undegraded":"2024-09-13T22:49:24.264161+0000","last_fullsized":"2024-09-13T22:49:24.264161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:35.269156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.269156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:48.954038+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384162+0000","last_change":"2024-09-13T22:49:22.976336+0000","last_active":"2024-09-13T22:49:24.384162+0000","last_peered":"2024-09-13T22:49:24.384162+0000","last_clean":"2024-09-13T22:49:24.384162+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:49:24.384162+0000","last_undegraded":"2024-09-13T22:49:24.384162+0000","last_fullsized":"2024-09-13T22:49:24.384162+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.976109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.976109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:48.810377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943525+0000","last_change":"2024-09-13T22:49:19.838975+0000","last_active":"2024-09-13T22:49:23.943525+0000","last_peered":"2024-09-13T22:49:23.943525+0000","last_clean":"2024-09-13T22:49:23.943525+0000","last_became_active":"2024-09-13T22:49:19.835650+0000","last_became_peered":"2024-09-13T22:49:19.835650+0000","last_unstale":"2024-09-13T22:49:23.943525+0000","last_undegraded":"2024-09-13T22:49:23.943525+0000","last_fullsized":"2024-09-13T22:49:23.943525+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:32.500889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"228'20","reported_seq":446,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943552+0000","last_change":"2024-09-13T22:49:14.431757+0000","last_active":"2024-09-13T22:49:23.943552+0000","last_peered":"2024-09-13T22:49:23.943552+0000","last_clean":"2024-09-13T22:49:23.943552+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:49:23.943552+0000","last_undegraded":"2024-09-13T22:49:23.943552+0000","last_fullsized":"2024-09-13T22:49:23.943552+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:49:14.431540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.431540+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.128652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":427,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383352+0000","last_change":"2024-09-13T22:48:23.112220+0000","last_active":"2024-09-13T22:49:24.383352+0000","last_peered":"2024-09-13T22:49:24.383352+0000","last_clean":"2024-09-13T22:49:24.383352+0000","last_became_active":"2024-09-13T22:47:03.889309+0000","last_became_peered":"2024-09-13T22:47:03.889309+0000","last_unstale":"2024-09-13T22:49:24.383352+0000","last_undegraded":"2024-09-13T22:49:24.383352+0000","last_fullsized":"2024-09-13T22:49:24.383352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:23.112116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.112116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:48.980030+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":427,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.264601+0000","last_change":"2024-09-13T22:49:22.282613+0000","last_active":"2024-09-13T22:49:24.264601+0000","last_peered":"2024-09-13T22:49:24.264601+0000","last_clean":"2024-09-13T22:49:24.264601+0000","last_became_active":"2024-09-13T22:47:03.889556+0000","last_became_peered":"2024-09-13T22:47:03.889556+0000","last_unstale":"2024-09-13T22:49:24.264601+0000","last_undegraded":"2024-09-13T22:49:24.264601+0000","last_fullsized":"2024-09-13T22:49:24.264601+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.282342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.282342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:31.911548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":409,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935520+0000","last_change":"2024-09-13T22:48:11.570730+0000","last_active":"2024-09-13T22:49:23.935520+0000","last_peered":"2024-09-13T22:49:23.935520+0000","last_clean":"2024-09-13T22:49:23.935520+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:49:23.935520+0000","last_undegraded":"2024-09-13T22:49:23.935520+0000","last_fullsized":"2024-09-13T22:49:23.935520+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.570502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.570502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:33.459976+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.934466+0000","last_change":"2024-09-13T22:49:19.838725+0000","last_active":"2024-09-13T22:49:23.934466+0000","last_peered":"2024-09-13T22:49:23.934466+0000","last_clean":"2024-09-13T22:49:23.934466+0000","last_became_active":"2024-09-13T22:49:19.838415+0000","last_became_peered":"2024-09-13T22:49:19.838415+0000","last_unstale":"2024-09-13T22:49:23.934466+0000","last_undegraded":"2024-09-13T22:49:23.934466+0000","last_fullsized":"2024-09-13T22:49:23.934466+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:30.186446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"228'34","reported_seq":480,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385359+0000","last_change":"2024-09-13T22:49:11.973786+0000","last_active":"2024-09-13T22:49:24.385359+0000","last_peered":"2024-09-13T22:49:24.385359+0000","last_clean":"2024-09-13T22:49:24.385359+0000","last_became_active":"2024-09-13T22:47:03.911810+0000","last_became_peered":"2024-09-13T22:47:03.911810+0000","last_unstale":"2024-09-13T22:49:24.385359+0000","last_undegraded":"2024-09-13T22:49:24.385359+0000","last_fullsized":"2024-09-13T22:49:24.385359+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:49:11.973550+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.973550+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:22.620534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":421,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266101+0000","last_change":"2024-09-13T22:48:17.191331+0000","last_active":"2024-09-13T22:49:24.266101+0000","last_peered":"2024-09-13T22:49:24.266101+0000","last_clean":"2024-09-13T22:49:24.266101+0000","last_became_active":"2024-09-13T22:37:40.740470+0000","last_became_peered":"2024-09-13T22:37:40.740470+0000","last_unstale":"2024-09-13T22:49:24.266101+0000","last_undegraded":"2024-09-13T22:49:24.266101+0000","last_fullsized":"2024-09-13T22:49:24.266101+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.191239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.191239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.201488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"211'12","reported_seq":567,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384523+0000","last_change":"2024-09-13T22:47:03.966726+0000","last_active":"2024-09-13T22:49:24.384523+0000","last_peered":"2024-09-13T22:49:24.384523+0000","last_clean":"2024-09-13T22:49:24.384523+0000","last_became_active":"2024-09-13T22:47:03.942956+0000","last_became_peered":"2024-09-13T22:47:03.942956+0000","last_unstale":"2024-09-13T22:49:24.384523+0000","last_undegraded":"2024-09-13T22:49:24.384523+0000","last_fullsized":"2024-09-13T22:49:24.384523+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:46:55.715383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:55.715383+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942049+0000","last_change":"2024-09-13T22:46:04.822159+0000","last_active":"2024-09-13T22:49:23.942049+0000","last_peered":"2024-09-13T22:49:23.942049+0000","last_clean":"2024-09-13T22:49:23.942049+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:49:23.942049+0000","last_undegraded":"2024-09-13T22:49:23.942049+0000","last_fullsized":"2024-09-13T22:49:23.942049+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:04.822089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:04.822089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382192+0000","last_change":"2024-09-13T22:49:19.829474+0000","last_active":"2024-09-13T22:49:24.382192+0000","last_peered":"2024-09-13T22:49:24.382192+0000","last_clean":"2024-09-13T22:49:24.382192+0000","last_became_active":"2024-09-13T22:49:19.829175+0000","last_became_peered":"2024-09-13T22:49:19.829175+0000","last_unstale":"2024-09-13T22:49:24.382192+0000","last_undegraded":"2024-09-13T22:49:24.382192+0000","last_fullsized":"2024-09-13T22:49:24.382192+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:28.988027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"228'19","reported_seq":457,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385092+0000","last_change":"2024-09-13T22:47:03.955949+0000","last_active":"2024-09-13T22:49:24.385092+0000","last_peered":"2024-09-13T22:49:24.385092+0000","last_clean":"2024-09-13T22:49:24.385092+0000","last_became_active":"2024-09-13T22:47:03.925193+0000","last_became_peered":"2024-09-13T22:47:03.925193+0000","last_unstale":"2024-09-13T22:49:24.385092+0000","last_undegraded":"2024-09-13T22:49:24.385092+0000","last_fullsized":"2024-09-13T22:49:24.385092+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:46:57.700461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:57.700461+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":217,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933800+0000","last_change":"2024-09-13T22:47:03.928773+0000","last_active":"2024-09-13T22:49:23.933800+0000","last_peered":"2024-09-13T22:49:23.933800+0000","last_clean":"2024-09-13T22:49:23.933800+0000","last_became_active":"2024-09-13T22:47:03.899714+0000","last_became_peered":"2024-09-13T22:47:03.899714+0000","last_unstale":"2024-09-13T22:49:23.933800+0000","last_undegraded":"2024-09-13T22:49:23.933800+0000","last_fullsized":"2024-09-13T22:49:23.933800+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:46:02.057013+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:46:02.057013+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":413,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.264066+0000","last_change":"2024-09-13T22:48:40.290789+0000","last_active":"2024-09-13T22:49:24.264066+0000","last_peered":"2024-09-13T22:49:24.264066+0000","last_clean":"2024-09-13T22:49:24.264066+0000","last_became_active":"2024-09-13T22:37:44.866432+0000","last_became_peered":"2024-09-13T22:37:44.866432+0000","last_unstale":"2024-09-13T22:49:24.264066+0000","last_undegraded":"2024-09-13T22:49:24.264066+0000","last_fullsized":"2024-09-13T22:49:24.264066+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:40.290665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:48:40.290665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:07.600216+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":401,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267225+0000","last_change":"2024-09-13T22:48:17.839741+0000","last_active":"2024-09-13T22:49:24.267225+0000","last_peered":"2024-09-13T22:49:24.267225+0000","last_clean":"2024-09-13T22:49:24.267225+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:49:24.267225+0000","last_undegraded":"2024-09-13T22:49:24.267225+0000","last_fullsized":"2024-09-13T22:49:24.267225+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.839516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.839516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:28.555266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"228'24","reported_seq":462,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384659+0000","last_change":"2024-09-13T22:48:24.168534+0000","last_active":"2024-09-13T22:49:24.384659+0000","last_peered":"2024-09-13T22:49:24.384659+0000","last_clean":"2024-09-13T22:49:24.384659+0000","last_became_active":"2024-09-13T22:47:03.943714+0000","last_became_peered":"2024-09-13T22:47:03.943714+0000","last_unstale":"2024-09-13T22:49:24.384659+0000","last_undegraded":"2024-09-13T22:49:24.384659+0000","last_fullsized":"2024-09-13T22:49:24.384659+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.168396+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:30.026966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":429,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.264226+0000","last_change":"2024-09-13T22:49:19.256364+0000","last_active":"2024-09-13T22:49:24.264226+0000","last_peered":"2024-09-13T22:49:24.264226+0000","last_clean":"2024-09-13T22:49:24.264226+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:49:24.264226+0000","last_undegraded":"2024-09-13T22:49:24.264226+0000","last_fullsized":"2024-09-13T22:49:24.264226+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:19.256282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:49:19.256282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:43.438849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":404,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267254+0000","last_change":"2024-09-13T22:48:15.782055+0000","last_active":"2024-09-13T22:49:24.267254+0000","last_peered":"2024-09-13T22:49:24.267254+0000","last_clean":"2024-09-13T22:49:24.267254+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:49:24.267254+0000","last_undegraded":"2024-09-13T22:49:24.267254+0000","last_fullsized":"2024-09-13T22:49:24.267254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.778860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.778860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:31.851147+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":423,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.264208+0000","last_change":"2024-09-13T22:48:24.158752+0000","last_active":"2024-09-13T22:49:24.264208+0000","last_peered":"2024-09-13T22:49:24.264208+0000","last_clean":"2024-09-13T22:49:24.264208+0000","last_became_active":"2024-09-13T22:47:03.916650+0000","last_became_peered":"2024-09-13T22:47:03.916650+0000","last_unstale":"2024-09-13T22:49:24.264208+0000","last_undegraded":"2024-09-13T22:49:24.264208+0000","last_fullsized":"2024-09-13T22:49:24.264208+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.158640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.696146+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"235'10","reported_seq":438,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383875+0000","last_change":"2024-09-13T22:48:31.992271+0000","last_active":"2024-09-13T22:49:24.383875+0000","last_peered":"2024-09-13T22:49:24.383875+0000","last_clean":"2024-09-13T22:49:24.383875+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:49:24.383875+0000","last_undegraded":"2024-09-13T22:49:24.383875+0000","last_fullsized":"2024-09-13T22:49:24.383875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'10","last_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.992124+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.848412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":434,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385221+0000","last_change":"2024-09-13T22:48:30.082529+0000","last_active":"2024-09-13T22:49:24.385221+0000","last_peered":"2024-09-13T22:49:24.385221+0000","last_clean":"2024-09-13T22:49:24.385221+0000","last_became_active":"2024-09-13T22:47:03.927967+0000","last_became_peered":"2024-09-13T22:47:03.927967+0000","last_unstale":"2024-09-13T22:49:24.385221+0000","last_undegraded":"2024-09-13T22:49:24.385221+0000","last_fullsized":"2024-09-13T22:49:24.385221+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.082437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.082437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.482334+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":403,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.943317+0000","last_change":"2024-09-13T22:46:08.851682+0000","last_active":"2024-09-13T22:49:23.943317+0000","last_peered":"2024-09-13T22:49:23.943317+0000","last_clean":"2024-09-13T22:49:23.943317+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:49:23.943317+0000","last_undegraded":"2024-09-13T22:49:23.943317+0000","last_fullsized":"2024-09-13T22:49:23.943317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:08.851579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:46:08.851579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":405,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:27.969058+0000","last_change":"2024-09-13T22:49:27.969112+0000","last_active":"2024-09-13T22:49:27.969058+0000","last_peered":"2024-09-13T22:49:27.969058+0000","last_clean":"2024-09-13T22:49:27.969058+0000","last_became_active":"2024-09-13T22:47:03.919704+0000","last_became_peered":"2024-09-13T22:47:03.919704+0000","last_unstale":"2024-09-13T22:49:27.969058+0000","last_undegraded":"2024-09-13T22:49:27.969058+0000","last_fullsized":"2024-09-13T22:49:27.969058+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:27.968957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:27.968957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:56.872280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"228'19","reported_seq":457,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384925+0000","last_change":"2024-09-13T22:48:18.321148+0000","last_active":"2024-09-13T22:49:24.384925+0000","last_peered":"2024-09-13T22:49:24.384925+0000","last_clean":"2024-09-13T22:49:24.384925+0000","last_became_active":"2024-09-13T22:47:03.935139+0000","last_became_peered":"2024-09-13T22:47:03.935139+0000","last_unstale":"2024-09-13T22:49:24.384925+0000","last_undegraded":"2024-09-13T22:49:24.384925+0000","last_fullsized":"2024-09-13T22:49:24.384925+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:18.321070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:18.321070+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.490331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":420,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.942623+0000","last_change":"2024-09-13T22:48:15.804011+0000","last_active":"2024-09-13T22:49:23.942623+0000","last_peered":"2024-09-13T22:49:23.942623+0000","last_clean":"2024-09-13T22:49:23.942623+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:49:23.942623+0000","last_undegraded":"2024-09-13T22:49:23.942623+0000","last_fullsized":"2024-09-13T22:49:23.942623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.803899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.803899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:42.474325+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":566,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.944820+0000","last_change":"2024-09-13T22:48:04.929694+0000","last_active":"2024-09-13T22:49:23.944820+0000","last_peered":"2024-09-13T22:49:23.944820+0000","last_clean":"2024-09-13T22:49:23.944820+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:49:23.944820+0000","last_undegraded":"2024-09-13T22:49:23.944820+0000","last_fullsized":"2024-09-13T22:49:23.944820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:04.929561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:04.929561+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:28.339642+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935238+0000","last_change":"2024-09-13T22:46:53.530343+0000","last_active":"2024-09-13T22:49:23.935238+0000","last_peered":"2024-09-13T22:49:23.935238+0000","last_clean":"2024-09-13T22:49:23.935238+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:49:23.935238+0000","last_undegraded":"2024-09-13T22:49:23.935238+0000","last_fullsized":"2024-09-13T22:49:23.935238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:53.530241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:46:53.530241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"228'15","reported_seq":434,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935133+0000","last_change":"2024-09-13T22:46:47.409756+0000","last_active":"2024-09-13T22:49:23.935133+0000","last_peered":"2024-09-13T22:49:23.935133+0000","last_clean":"2024-09-13T22:49:23.935133+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:49:23.935133+0000","last_undegraded":"2024-09-13T22:49:23.935133+0000","last_fullsized":"2024-09-13T22:49:23.935133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-09-13T22:46:47.409619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:46:47.409619+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":427,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.934082+0000","last_change":"2024-09-13T22:48:04.615427+0000","last_active":"2024-09-13T22:49:23.934082+0000","last_peered":"2024-09-13T22:49:23.934082+0000","last_clean":"2024-09-13T22:49:23.934082+0000","last_became_active":"2024-09-13T22:47:03.915370+0000","last_became_peered":"2024-09-13T22:47:03.915370+0000","last_unstale":"2024-09-13T22:49:23.934082+0000","last_undegraded":"2024-09-13T22:49:23.934082+0000","last_fullsized":"2024-09-13T22:49:23.934082+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:04.615328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:04.615328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":555,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267396+0000","last_change":"2024-09-13T22:45:48.537059+0000","last_active":"2024-09-13T22:49:24.267396+0000","last_peered":"2024-09-13T22:49:24.267396+0000","last_clean":"2024-09-13T22:49:24.267396+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:49:24.267396+0000","last_undegraded":"2024-09-13T22:49:24.267396+0000","last_fullsized":"2024-09-13T22:49:24.267396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:48.536952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:48.536952+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":107,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.265116+0000","last_change":"2024-09-13T22:48:08.144978+0000","last_active":"2024-09-13T22:49:24.265116+0000","last_peered":"2024-09-13T22:49:24.265116+0000","last_clean":"2024-09-13T22:49:24.265116+0000","last_became_active":"2024-09-13T22:47:03.900992+0000","last_became_peered":"2024-09-13T22:47:03.900992+0000","last_unstale":"2024-09-13T22:49:24.265116+0000","last_undegraded":"2024-09-13T22:49:24.265116+0000","last_fullsized":"2024-09-13T22:49:24.265116+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:08.144897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.144897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.239175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"228'15","reported_seq":448,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385515+0000","last_change":"2024-09-13T22:48:21.238575+0000","last_active":"2024-09-13T22:49:24.385515+0000","last_peered":"2024-09-13T22:49:24.385515+0000","last_clean":"2024-09-13T22:49:24.385515+0000","last_became_active":"2024-09-13T22:47:03.896530+0000","last_became_peered":"2024-09-13T22:47:03.896530+0000","last_unstale":"2024-09-13T22:49:24.385515+0000","last_undegraded":"2024-09-13T22:49:24.385515+0000","last_fullsized":"2024-09-13T22:49:24.385515+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:21.238464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:21.238464+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:31.819323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":428,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384412+0000","last_change":"2024-09-13T22:48:36.942296+0000","last_active":"2024-09-13T22:49:24.384412+0000","last_peered":"2024-09-13T22:49:24.384412+0000","last_clean":"2024-09-13T22:49:24.384412+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:49:24.384412+0000","last_undegraded":"2024-09-13T22:49:24.384412+0000","last_fullsized":"2024-09-13T22:49:24.384412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:36.942121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:36.942121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:57.238255+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":404,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266868+0000","last_change":"2024-09-13T22:49:17.277840+0000","last_active":"2024-09-13T22:49:24.266868+0000","last_peered":"2024-09-13T22:49:24.266868+0000","last_clean":"2024-09-13T22:49:24.266868+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:49:24.266868+0000","last_undegraded":"2024-09-13T22:49:24.266868+0000","last_fullsized":"2024-09-13T22:49:24.266868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.277656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.277656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.471913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933285+0000","last_change":"2024-09-13T22:48:33.071864+0000","last_active":"2024-09-13T22:49:23.933285+0000","last_peered":"2024-09-13T22:49:23.933285+0000","last_clean":"2024-09-13T22:49:23.933285+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:49:23.933285+0000","last_undegraded":"2024-09-13T22:49:23.933285+0000","last_fullsized":"2024-09-13T22:49:23.933285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:33.071751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:33.071751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:45.377810+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"228'20","reported_seq":168,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385618+0000","last_change":"2024-09-13T22:48:20.275253+0000","last_active":"2024-09-13T22:49:24.385618+0000","last_peered":"2024-09-13T22:49:24.385618+0000","last_clean":"2024-09-13T22:49:24.385618+0000","last_became_active":"2024-09-13T22:47:03.932022+0000","last_became_peered":"2024-09-13T22:47:03.932022+0000","last_unstale":"2024-09-13T22:49:24.385618+0000","last_undegraded":"2024-09-13T22:49:24.385618+0000","last_fullsized":"2024-09-13T22:49:24.385618+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:48:20.275130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.275130+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:39.506678+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":420,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.383301+0000","last_change":"2024-09-13T22:49:16.390793+0000","last_active":"2024-09-13T22:49:24.383301+0000","last_peered":"2024-09-13T22:49:24.383301+0000","last_clean":"2024-09-13T22:49:24.383301+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:49:24.383301+0000","last_undegraded":"2024-09-13T22:49:24.383301+0000","last_fullsized":"2024-09-13T22:49:24.383301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.389928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:03.849281+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.389928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.083913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":413,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382319+0000","last_change":"2024-09-13T22:48:38.188618+0000","last_active":"2024-09-13T22:49:24.382319+0000","last_peered":"2024-09-13T22:49:24.382319+0000","last_clean":"2024-09-13T22:49:24.382319+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:49:24.382319+0000","last_undegraded":"2024-09-13T22:49:24.382319+0000","last_fullsized":"2024-09-13T22:49:24.382319+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:38.188506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.188506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:02.439391+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266444+0000","last_change":"2024-09-13T22:49:20.405583+0000","last_active":"2024-09-13T22:49:24.266444+0000","last_peered":"2024-09-13T22:49:24.266444+0000","last_clean":"2024-09-13T22:49:24.266444+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:49:24.266444+0000","last_undegraded":"2024-09-13T22:49:24.266444+0000","last_fullsized":"2024-09-13T22:49:24.266444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:20.405502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.405502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:41.635537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"228'12","reported_seq":434,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933717+0000","last_change":"2024-09-13T22:48:32.051004+0000","last_active":"2024-09-13T22:49:23.933717+0000","last_peered":"2024-09-13T22:49:23.933717+0000","last_clean":"2024-09-13T22:49:23.933717+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:49:23.933717+0000","last_undegraded":"2024-09-13T22:49:23.933717+0000","last_fullsized":"2024-09-13T22:49:23.933717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12","last_scrub_stamp":"2024-09-13T22:48:32.050878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.050878+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:32.659695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":419,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:28.371266+0000","last_change":"2024-09-13T22:49:28.371324+0000","last_active":"2024-09-13T22:49:28.371266+0000","last_peered":"2024-09-13T22:49:28.371266+0000","last_clean":"2024-09-13T22:49:28.371266+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:49:28.371266+0000","last_undegraded":"2024-09-13T22:49:28.371266+0000","last_fullsized":"2024-09-13T22:49:28.371266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:28.371115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:49:28.371115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:36.022385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":426,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384130+0000","last_change":"2024-09-13T22:49:14.149288+0000","last_active":"2024-09-13T22:49:24.384130+0000","last_peered":"2024-09-13T22:49:24.384130+0000","last_clean":"2024-09-13T22:49:24.384130+0000","last_became_active":"2024-09-13T22:47:03.935103+0000","last_became_peered":"2024-09-13T22:47:03.935103+0000","last_unstale":"2024-09-13T22:49:24.384130+0000","last_undegraded":"2024-09-13T22:49:24.384130+0000","last_fullsized":"2024-09-13T22:49:24.384130+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:14.149187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.149187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:37.441459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":406,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384156+0000","last_change":"2024-09-13T22:49:15.907904+0000","last_active":"2024-09-13T22:49:24.384156+0000","last_peered":"2024-09-13T22:49:24.384156+0000","last_clean":"2024-09-13T22:49:24.384156+0000","last_became_active":"2024-09-13T22:47:03.896472+0000","last_became_peered":"2024-09-13T22:47:03.896472+0000","last_unstale":"2024-09-13T22:49:24.384156+0000","last_undegraded":"2024-09-13T22:49:24.384156+0000","last_fullsized":"2024-09-13T22:49:24.384156+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:15.907800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:15.907800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.113950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"228'15","reported_seq":440,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.436079+0000","last_change":"2024-09-13T22:49:24.436104+0000","last_active":"2024-09-13T22:49:24.436079+0000","last_peered":"2024-09-13T22:49:24.436079+0000","last_clean":"2024-09-13T22:49:24.436079+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:49:24.436079+0000","last_undegraded":"2024-09-13T22:49:24.436079+0000","last_fullsized":"2024-09-13T22:49:24.436079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:49:24.436026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:24.436026+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.631379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":427,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385143+0000","last_change":"2024-09-13T22:48:13.617965+0000","last_active":"2024-09-13T22:49:24.385143+0000","last_peered":"2024-09-13T22:49:24.385143+0000","last_clean":"2024-09-13T22:49:24.385143+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:49:24.385143+0000","last_undegraded":"2024-09-13T22:49:24.385143+0000","last_fullsized":"2024-09-13T22:49:24.385143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:13.617680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.617680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:41.137219+0000","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":410,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384540+0000","last_change":"2024-09-13T22:49:16.934196+0000","last_active":"2024-09-13T22:49:24.384540+0000","last_peered":"2024-09-13T22:49:24.384540+0000","last_clean":"2024-09-13T22:49:24.384540+0000","last_became_active":"2024-09-13T22:47:03.935739+0000","last_became_peered":"2024-09-13T22:47:03.935739+0000","last_unstale":"2024-09-13T22:49:24.384540+0000","last_undegraded":"2024-09-13T22:49:24.384540+0000","last_fullsized":"2024-09-13T22:49:24.384540+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.934047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.934047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.592545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266010+0000","last_change":"2024-09-13T22:45:40.486348+0000","last_active":"2024-09-13T22:49:24.266010+0000","last_peered":"2024-09-13T22:49:24.266010+0000","last_clean":"2024-09-13T22:49:24.266010+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:49:24.266010+0000","last_undegraded":"2024-09-13T22:49:24.266010+0000","last_fullsized":"2024-09-13T22:49:24.266010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:40.486247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:45:40.486247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":401,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385474+0000","last_change":"2024-09-13T22:48:29.166658+0000","last_active":"2024-09-13T22:49:24.385474+0000","last_peered":"2024-09-13T22:49:24.385474+0000","last_clean":"2024-09-13T22:49:24.385474+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:49:24.385474+0000","last_undegraded":"2024-09-13T22:49:24.385474+0000","last_fullsized":"2024-09-13T22:49:24.385474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:48:29.166588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:29.166588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:50.874694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"228'34","reported_seq":488,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382015+0000","last_change":"2024-09-13T22:48:28.140759+0000","last_active":"2024-09-13T22:49:24.382015+0000","last_peered":"2024-09-13T22:49:24.382015+0000","last_clean":"2024-09-13T22:49:24.382015+0000","last_became_active":"2024-09-13T22:47:03.904504+0000","last_became_peered":"2024-09-13T22:47:03.904504+0000","last_unstale":"2024-09-13T22:49:24.382015+0000","last_undegraded":"2024-09-13T22:49:24.382015+0000","last_fullsized":"2024-09-13T22:49:24.382015+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:28.140653+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.140653+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:52.903926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":425,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.941668+0000","last_change":"2024-09-13T22:48:12.849274+0000","last_active":"2024-09-13T22:49:23.941668+0000","last_peered":"2024-09-13T22:49:23.941668+0000","last_clean":"2024-09-13T22:49:23.941668+0000","last_became_active":"2024-09-13T22:47:03.935641+0000","last_became_peered":"2024-09-13T22:47:03.935641+0000","last_unstale":"2024-09-13T22:49:23.941668+0000","last_undegraded":"2024-09-13T22:49:23.941668+0000","last_fullsized":"2024-09-13T22:49:23.941668+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:12.849206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:48:12.849206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:37.788041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"211'13","reported_seq":587,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.385174+0000","last_change":"2024-09-13T22:48:34.950988+0000","last_active":"2024-09-13T22:49:24.385174+0000","last_peered":"2024-09-13T22:49:24.385174+0000","last_clean":"2024-09-13T22:49:24.385174+0000","last_became_active":"2024-09-13T22:47:03.935719+0000","last_became_peered":"2024-09-13T22:47:03.935719+0000","last_unstale":"2024-09-13T22:49:24.385174+0000","last_undegraded":"2024-09-13T22:49:24.385174+0000","last_fullsized":"2024-09-13T22:49:24.385174+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:34.950872+0000","last_deep_scrub":"172'12","last_deep_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.950872+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:55.568584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":414,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382425+0000","last_change":"2024-09-13T22:48:39.144813+0000","last_active":"2024-09-13T22:49:24.382425+0000","last_peered":"2024-09-13T22:49:24.382425+0000","last_clean":"2024-09-13T22:49:24.382425+0000","last_became_active":"2024-09-13T22:47:03.901711+0000","last_became_peered":"2024-09-13T22:47:03.901711+0000","last_unstale":"2024-09-13T22:49:24.382425+0000","last_undegraded":"2024-09-13T22:49:24.382425+0000","last_fullsized":"2024-09-13T22:49:24.382425+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.144743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.144743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:09.061290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"228'15","reported_seq":433,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267256+0000","last_change":"2024-09-13T22:48:24.914421+0000","last_active":"2024-09-13T22:49:24.267256+0000","last_peered":"2024-09-13T22:49:24.267256+0000","last_clean":"2024-09-13T22:49:24.267256+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:49:24.267256+0000","last_undegraded":"2024-09-13T22:49:24.267256+0000","last_fullsized":"2024-09-13T22:49:24.267256+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:24.913947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.913947+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:43.347124+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":405,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266742+0000","last_change":"2024-09-13T22:44:58.543023+0000","last_active":"2024-09-13T22:49:24.266742+0000","last_peered":"2024-09-13T22:49:24.266742+0000","last_clean":"2024-09-13T22:49:24.266742+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:49:24.266742+0000","last_undegraded":"2024-09-13T22:49:24.266742+0000","last_fullsized":"2024-09-13T22:49:24.266742+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:58.542781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.542781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":413,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.935369+0000","last_change":"2024-09-13T22:48:30.083820+0000","last_active":"2024-09-13T22:49:23.935369+0000","last_peered":"2024-09-13T22:49:23.935369+0000","last_clean":"2024-09-13T22:49:23.935369+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:49:23.935369+0000","last_undegraded":"2024-09-13T22:49:23.935369+0000","last_fullsized":"2024-09-13T22:49:23.935369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.083568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:59.748166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":106,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384830+0000","last_change":"2024-09-13T22:48:16.354115+0000","last_active":"2024-09-13T22:49:24.384830+0000","last_peered":"2024-09-13T22:49:24.384830+0000","last_clean":"2024-09-13T22:49:24.384830+0000","last_became_active":"2024-09-13T22:47:03.926169+0000","last_became_peered":"2024-09-13T22:47:03.926169+0000","last_unstale":"2024-09-13T22:49:24.384830+0000","last_undegraded":"2024-09-13T22:49:24.384830+0000","last_fullsized":"2024-09-13T22:49:24.384830+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:16.353783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:16.353783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:31.111988+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":433,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.384388+0000","last_change":"2024-09-13T22:48:37.918024+0000","last_active":"2024-09-13T22:49:24.384388+0000","last_peered":"2024-09-13T22:49:24.384388+0000","last_clean":"2024-09-13T22:49:24.384388+0000","last_became_active":"2024-09-13T22:47:03.902900+0000","last_became_peered":"2024-09-13T22:47:03.902900+0000","last_unstale":"2024-09-13T22:49:24.384388+0000","last_undegraded":"2024-09-13T22:49:24.384388+0000","last_fullsized":"2024-09-13T22:49:24.384388+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.917904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.917904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:06.438554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"228'9","reported_seq":443,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:23.933906+0000","last_change":"2024-09-13T22:47:03.928761+0000","last_active":"2024-09-13T22:49:23.933906+0000","last_peered":"2024-09-13T22:49:23.933906+0000","last_clean":"2024-09-13T22:49:23.933906+0000","last_became_active":"2024-09-13T22:47:03.915408+0000","last_became_peered":"2024-09-13T22:47:03.915408+0000","last_unstale":"2024-09-13T22:49:23.933906+0000","last_undegraded":"2024-09-13T22:49:23.933906+0000","last_fullsized":"2024-09-13T22:49:23.933906+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:46:58.546268+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:46:58.546268+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":418,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382758+0000","last_change":"2024-09-13T22:48:41.110319+0000","last_active":"2024-09-13T22:49:24.382758+0000","last_peered":"2024-09-13T22:49:24.382758+0000","last_clean":"2024-09-13T22:49:24.382758+0000","last_became_active":"2024-09-13T22:47:03.902134+0000","last_became_peered":"2024-09-13T22:47:03.902134+0000","last_unstale":"2024-09-13T22:49:24.382758+0000","last_undegraded":"2024-09-13T22:49:24.382758+0000","last_fullsized":"2024-09-13T22:49:24.382758+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:41.110195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:48:41.110195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:03.599329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":402,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.266048+0000","last_change":"2024-09-13T22:45:51.542656+0000","last_active":"2024-09-13T22:49:24.266048+0000","last_peered":"2024-09-13T22:49:24.266048+0000","last_clean":"2024-09-13T22:49:24.266048+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:49:24.266048+0000","last_undegraded":"2024-09-13T22:49:24.266048+0000","last_fullsized":"2024-09-13T22:49:24.266048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:45:51.542373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.542373+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":412,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267133+0000","last_change":"2024-09-13T22:48:28.089740+0000","last_active":"2024-09-13T22:49:24.267133+0000","last_peered":"2024-09-13T22:49:24.267133+0000","last_clean":"2024-09-13T22:49:24.267133+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:49:24.267133+0000","last_undegraded":"2024-09-13T22:49:24.267133+0000","last_fullsized":"2024-09-13T22:49:24.267133+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:28.089548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.089548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:49:38.980036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"228'19","reported_seq":423,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.382293+0000","last_change":"2024-09-13T22:49:21.457818+0000","last_active":"2024-09-13T22:49:24.382293+0000","last_peered":"2024-09-13T22:49:24.382293+0000","last_clean":"2024-09-13T22:49:24.382293+0000","last_became_active":"2024-09-13T22:47:03.891133+0000","last_became_peered":"2024-09-13T22:47:03.891133+0000","last_unstale":"2024-09-13T22:49:24.382293+0000","last_undegraded":"2024-09-13T22:49:24.382293+0000","last_fullsized":"2024-09-13T22:49:24.382293+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:49:21.457510+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.457510+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:47.529341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":99,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T22:49:24.267404+0000","last_change":"2024-09-13T22:47:03.917147+0000","last_active":"2024-09-13T22:49:24.267404+0000","last_peered":"2024-09-13T22:49:24.267404+0000","last_clean":"2024-09-13T22:49:24.267404+0000","last_became_active":"2024-09-13T22:47:03.916657+0000","last_became_peered":"2024-09-13T22:47:03.916657+0000","last_unstale":"2024-09-13T22:49:24.267404+0000","last_undegraded":"2024-09-13T22:49:24.267404+0000","last_fullsized":"2024-09-13T22:49:24.267404+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:50.912196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:45:50.912196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":681,"num_read_kb":676,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808982,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230860,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93514292,"statfs":{"total":95995035648,"available":95758635008,"internally_reserved":0,"allocated":1175552,"data_stored":754951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136033,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93456324,"statfs":{"total":95995035648,"available":95699275776,"internally_reserved":0,"allocated":831488,"data_stored":425279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528492,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":263040,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482112,"statfs":{"total":95995035648,"available":95725682688,"internally_reserved":0,"allocated":1425408,"data_stored":1015811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084479,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365764,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379388,"statfs":{"total":95995035648,"available":95620493312,"internally_reserved":0,"allocated":1425408,"data_stored":1016296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378828,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284920,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460232,"statfs":{"total":95995035648,"available":95703277568,"internally_reserved":0,"allocated":1089536,"data_stored":688553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303144,"kb_used_data":10968,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93442008,"statfs":{"total":95995035648,"available":95684616192,"internally_reserved":0,"allocated":11231232,"data_stored":10782048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738592,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93456292,"statfs":{"total":95995035648,"available":95699243008,"internally_reserved":0,"allocated":860160,"data_stored":427479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:49:31.979 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T22:49:31.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:49:31.980 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T22:49:31.980 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.416638 2024-09-13T22:49:31.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 0.41663832260396405 2024-09-13T22:49:31.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:32.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:32.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:32.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:31 smithi192 bash[24608]: cluster 2024-09-13T22:49:30.936348+0000 osd.7 (osd.7) 261 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:49:32.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:31 smithi192 bash[24608]: cluster 2024-09-13T22:49:30.942244+0000 osd.7 (osd.7) 262 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:49:32.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:31 smithi192 bash[24608]: cluster 2024-09-13T22:49:30.952561+0000 mon.a (mon.0) 1291 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:49:32.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:31 smithi192 bash[24608]: cluster 2024-09-13T22:49:30.952610+0000 mon.a (mon.0) 1292 : cluster [INF] Cluster is now healthy 2024-09-13T22:49:32.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:31 smithi192 bash[24608]: cluster 2024-09-13T22:49:31.101143+0000 osd.6 (osd.6) 193 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:49:32.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:31 smithi192 bash[24608]: cluster 2024-09-13T22:49:31.103005+0000 osd.6 (osd.6) 194 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:49:32.154 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:32.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[29889]: cluster 2024-09-13T22:49:30.936348+0000 osd.7 (osd.7) 261 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:49:32.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[29889]: cluster 2024-09-13T22:49:30.942244+0000 osd.7 (osd.7) 262 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:49:32.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[29889]: cluster 2024-09-13T22:49:30.952561+0000 mon.a (mon.0) 1291 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[29889]: cluster 2024-09-13T22:49:30.952610+0000 mon.a (mon.0) 1292 : cluster [INF] Cluster is now healthy 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[29889]: cluster 2024-09-13T22:49:31.101143+0000 osd.6 (osd.6) 193 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[29889]: cluster 2024-09-13T22:49:31.103005+0000 osd.6 (osd.6) 194 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[22840]: cluster 2024-09-13T22:49:30.936348+0000 osd.7 (osd.7) 261 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[22840]: cluster 2024-09-13T22:49:30.942244+0000 osd.7 (osd.7) 262 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[22840]: cluster 2024-09-13T22:49:30.952561+0000 mon.a (mon.0) 1291 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[22840]: cluster 2024-09-13T22:49:30.952610+0000 mon.a (mon.0) 1292 : cluster [INF] Cluster is now healthy 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[22840]: cluster 2024-09-13T22:49:31.101143+0000 osd.6 (osd.6) 193 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:49:32.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:31 smithi178 bash[22840]: cluster 2024-09-13T22:49:31.103005+0000 osd.6 (osd.6) 194 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:49:32.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:32.270 DEBUG:teuthology.orchestra.run.smithi178:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-13T22:49:32.279 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:32.281 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T22:49:32.281 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T22:49:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:32.315 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:32.318 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:hints = 1 2024-09-13T22:49:32.318 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T22:49:32.318 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_96665 2024-09-13T22:49:32.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:49:32.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-13T22:49:32.387 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:32.420 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:32.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:32.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:32.657 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:32.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:32.745 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:32.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:32.809 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:32.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:32.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:32.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:33.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:33.052 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:31.478951+0000 osd.1 (osd.1) 247 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:49:33.052 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:31.481503+0000 osd.1 (osd.1) 248 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:49:33.052 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:31.957272+0000 osd.7 (osd.7) 263 : cluster [DBG] 3.e scrub starts 2024-09-13T22:49:33.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:31.961797+0000 osd.7 (osd.7) 264 : cluster [DBG] 3.e scrub ok 2024-09-13T22:49:33.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:32.132579+0000 osd.0 (osd.0) 239 : cluster [DBG] 2.f scrub starts 2024-09-13T22:49:33.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:32.135924+0000 osd.0 (osd.0) 240 : cluster [DBG] 2.f scrub ok 2024-09-13T22:49:33.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:32.387901+0000 osd.5 (osd.5) 269 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:49:33.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:32.392903+0000 osd.5 (osd.5) 270 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:49:33.053 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:32 smithi192 bash[24608]: cluster 2024-09-13T22:49:32.685917+0000 mgr.y (mgr.14613) 443 : cluster [DBG] pgmap v493: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:33.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:33.175 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:33.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:33.239 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:31.478951+0000 osd.1 (osd.1) 247 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:31.481503+0000 osd.1 (osd.1) 248 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:31.957272+0000 osd.7 (osd.7) 263 : cluster [DBG] 3.e scrub starts 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:31.961797+0000 osd.7 (osd.7) 264 : cluster [DBG] 3.e scrub ok 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:32.132579+0000 osd.0 (osd.0) 239 : cluster [DBG] 2.f scrub starts 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:32.135924+0000 osd.0 (osd.0) 240 : cluster [DBG] 2.f scrub ok 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:32.387901+0000 osd.5 (osd.5) 269 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:32.392903+0000 osd.5 (osd.5) 270 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:49:33.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[22840]: cluster 2024-09-13T22:49:32.685917+0000 mgr.y (mgr.14613) 443 : cluster [DBG] pgmap v493: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:33.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:31.478951+0000 osd.1 (osd.1) 247 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:49:33.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:31.481503+0000 osd.1 (osd.1) 248 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:49:33.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:31.957272+0000 osd.7 (osd.7) 263 : cluster [DBG] 3.e scrub starts 2024-09-13T22:49:33.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:31.961797+0000 osd.7 (osd.7) 264 : cluster [DBG] 3.e scrub ok 2024-09-13T22:49:33.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:32.132579+0000 osd.0 (osd.0) 239 : cluster [DBG] 2.f scrub starts 2024-09-13T22:49:33.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:32.135924+0000 osd.0 (osd.0) 240 : cluster [DBG] 2.f scrub ok 2024-09-13T22:49:33.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:32.387901+0000 osd.5 (osd.5) 269 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:49:33.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:32.392903+0000 osd.5 (osd.5) 270 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:49:33.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:32 smithi178 bash[29889]: cluster 2024-09-13T22:49:32.685917+0000 mgr.y (mgr.14613) 443 : cluster [DBG] pgmap v493: 158 pgs: 158 active+clean; 583 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:49:33.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:33.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 1 16 2866 2850 178.115 178.125 0.00752078 0.00558802 2024-09-13T22:49:33.332 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:33.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:33.462 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:33.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:33.595 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:33.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:33.723 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:33.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:33.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:33.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:33.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:33.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:34.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:34.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.237 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:34.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:34.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 2 16 5551 5535 172.956 167.812 0.00388827 0.00576963 2024-09-13T22:49:34.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:34.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:34.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:34.502 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:34.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:34.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:34.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:34.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:34.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:34.877 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[29889]: cluster 2024-09-13T22:49:33.360907+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.a scrub starts 2024-09-13T22:49:34.877 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[29889]: cluster 2024-09-13T22:49:33.362386+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.a scrub ok 2024-09-13T22:49:34.877 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[29889]: audit 2024-09-13T22:49:33.554852+0000 mon.a (mon.0) 1293 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[29889]: audit 2024-09-13T22:49:33.559150+0000 mon.b (mon.1) 188 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[29889]: cluster 2024-09-13T22:49:34.080650+0000 osd.0 (osd.0) 241 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[29889]: cluster 2024-09-13T22:49:34.085271+0000 osd.0 (osd.0) 242 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[22840]: cluster 2024-09-13T22:49:33.360907+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.a scrub starts 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[22840]: cluster 2024-09-13T22:49:33.362386+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.a scrub ok 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[22840]: audit 2024-09-13T22:49:33.554852+0000 mon.a (mon.0) 1293 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[22840]: audit 2024-09-13T22:49:33.559150+0000 mon.b (mon.1) 188 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[22840]: cluster 2024-09-13T22:49:34.080650+0000 osd.0 (osd.0) 241 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:49:34.878 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:34 smithi178 bash[22840]: cluster 2024-09-13T22:49:34.085271+0000 osd.0 (osd.0) 242 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:49:34.887 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:34.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:34.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:34 smithi192 bash[24608]: cluster 2024-09-13T22:49:33.360907+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.a scrub starts 2024-09-13T22:49:34.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:34 smithi192 bash[24608]: cluster 2024-09-13T22:49:33.362386+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.a scrub ok 2024-09-13T22:49:34.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:34 smithi192 bash[24608]: audit 2024-09-13T22:49:33.554852+0000 mon.a (mon.0) 1293 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:34.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:34 smithi192 bash[24608]: audit 2024-09-13T22:49:33.559150+0000 mon.b (mon.1) 188 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:34.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:34 smithi192 bash[24608]: cluster 2024-09-13T22:49:34.080650+0000 osd.0 (osd.0) 241 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:49:34.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:34 smithi192 bash[24608]: cluster 2024-09-13T22:49:34.085271+0000 osd.0 (osd.0) 242 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:49:34.990 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:35.017 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:35.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:35.144 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:49:35.145 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:35.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:35.273 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:35.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 3 16 7723 7707 160.55 135.75 0.00913624 0.00621706 2024-09-13T22:49:35.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:35.400 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:35.504 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:35.531 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:35.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:35.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:49:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:49:35.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:35 smithi192 bash[24608]: cluster 2024-09-13T22:49:34.686712+0000 mgr.y (mgr.14613) 444 : cluster [DBG] pgmap v494: 158 pgs: 158 active+clean; 115 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 11 MiB/s wr, 184 op/s 2024-09-13T22:49:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:35.768 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:35.778 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:35 smithi178 bash[29889]: cluster 2024-09-13T22:49:34.686712+0000 mgr.y (mgr.14613) 444 : cluster [DBG] pgmap v494: 158 pgs: 158 active+clean; 115 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 11 MiB/s wr, 184 op/s 2024-09-13T22:49:35.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:35 smithi178 bash[22840]: cluster 2024-09-13T22:49:34.686712+0000 mgr.y (mgr.14613) 444 : cluster [DBG] pgmap v494: 158 pgs: 158 active+clean; 115 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 11 MiB/s wr, 184 op/s 2024-09-13T22:49:35.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:35.805 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:35.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:35.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:35.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:36.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:36.063 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:36.166 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:36.194 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:36.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:36.319 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:36.320 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 4 16 9647 9631 150.471 120.25 0.0036171 0.00654156 2024-09-13T22:49:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:36.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:36.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:36.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:36.582 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:36.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[29889]: cluster 2024-09-13T22:49:35.371747+0000 osd.3 (osd.3) 235 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[29889]: cluster 2024-09-13T22:49:35.374314+0000 osd.3 (osd.3) 236 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[29889]: audit 2024-09-13T22:49:35.388528+0000 mgr.y (mgr.14613) 445 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[29889]: cluster 2024-09-13T22:49:36.079202+0000 osd.0 (osd.0) 243 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[29889]: cluster 2024-09-13T22:49:36.082950+0000 osd.0 (osd.0) 244 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[22840]: cluster 2024-09-13T22:49:35.371747+0000 osd.3 (osd.3) 235 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[22840]: cluster 2024-09-13T22:49:35.374314+0000 osd.3 (osd.3) 236 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[22840]: audit 2024-09-13T22:49:35.388528+0000 mgr.y (mgr.14613) 445 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[22840]: cluster 2024-09-13T22:49:36.079202+0000 osd.0 (osd.0) 243 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:49:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:36 smithi178 bash[22840]: cluster 2024-09-13T22:49:36.082950+0000 osd.0 (osd.0) 244 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:49:36.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:36.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:36.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:36.839 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:36 smithi192 bash[24608]: cluster 2024-09-13T22:49:35.371747+0000 osd.3 (osd.3) 235 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:49:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:36 smithi192 bash[24608]: cluster 2024-09-13T22:49:35.374314+0000 osd.3 (osd.3) 236 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:49:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:36 smithi192 bash[24608]: audit 2024-09-13T22:49:35.388528+0000 mgr.y (mgr.14613) 445 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:36 smithi192 bash[24608]: cluster 2024-09-13T22:49:36.079202+0000 osd.0 (osd.0) 243 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:49:36.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:36 smithi192 bash[24608]: cluster 2024-09-13T22:49:36.082950+0000 osd.0 (osd.0) 244 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:49:36.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:36.970 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:36.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:37.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:37.107 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:37.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:37.228 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:37.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 5 16 10819 10803 135.026 73.25 0.00716171 0.0072342 2024-09-13T22:49:37.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:37.360 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:37.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:37.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:37.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:37.628 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:37.731 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:37.751 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:37.769 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:37 smithi178 bash[29889]: cluster 2024-09-13T22:49:36.687774+0000 mgr.y (mgr.14613) 446 : cluster [DBG] pgmap v495: 158 pgs: 158 active+clean; 378 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 604 op/s 2024-09-13T22:49:37.769 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:37 smithi178 bash[29889]: cluster 2024-09-13T22:49:36.789600+0000 osd.1 (osd.1) 249 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:49:37.769 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:37 smithi178 bash[29889]: cluster 2024-09-13T22:49:36.839467+0000 osd.1 (osd.1) 250 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:49:37.770 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:37 smithi178 bash[22840]: cluster 2024-09-13T22:49:36.687774+0000 mgr.y (mgr.14613) 446 : cluster [DBG] pgmap v495: 158 pgs: 158 active+clean; 378 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 604 op/s 2024-09-13T22:49:37.770 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:37 smithi178 bash[22840]: cluster 2024-09-13T22:49:36.789600+0000 osd.1 (osd.1) 249 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:49:37.770 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:37 smithi178 bash[22840]: cluster 2024-09-13T22:49:36.839467+0000 osd.1 (osd.1) 250 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:49:37.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:37.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:37 smithi192 bash[24608]: cluster 2024-09-13T22:49:36.687774+0000 mgr.y (mgr.14613) 446 : cluster [DBG] pgmap v495: 158 pgs: 158 active+clean; 378 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 604 op/s 2024-09-13T22:49:37.933 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:37 smithi192 bash[24608]: cluster 2024-09-13T22:49:36.789600+0000 osd.1 (osd.1) 249 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:49:37.933 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:37 smithi192 bash[24608]: cluster 2024-09-13T22:49:36.839467+0000 osd.1 (osd.1) 250 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:49:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:37.934 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:37.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:38.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:38.068 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:38.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:38.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:38.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:38.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 6 16 12308 12292 128.031 93.0625 0.0112732 0.00780206 2024-09-13T22:49:38.339 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:38.443 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:38.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:38.555 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:49:38.555 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:49:38.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.555 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:12782 9.7 9:e8265a94:::benchmark_data_smithi178_96665_object12781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:49:38.557657+0000", 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.001193829, 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012293810000000001, 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 12782 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.556 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557657+0000", 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557657+0000", 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:38.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557659+0000", 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.773e-06 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557723+0000", 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4319999999999994e-05 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557737+0000", 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3745e-05 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557745+0000", 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4630000000000001e-06 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557798+0000", 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2861000000000001e-05 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557898+0000", 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9974000000000002e-05 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,7", 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.557990+0000", 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1603999999999999e-05 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:12784 9.7 9:e5537526:::benchmark_data_smithi178_96665_object12783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:49:38.558259+0000", 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00059197500000000001, 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00071542899999999996, 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:38.560 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 12784 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558259+0000", 2024-09-13T22:49:38.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558259+0000", 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558260+0000", 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.497e-06 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:38.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558331+0000", 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0554000000000002e-05 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558341+0000", 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0681e-05 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558348+0000", 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9630000000000001e-06 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558639+0000", 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029072499999999998 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558768+0000", 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000129288 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,7", 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.558909+0000", 2024-09-13T22:49:38.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014047300000000001 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:12776 9.a 9:59870010:::benchmark_data_smithi178_96665_object12775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:49:38.552497+0000", 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0063540560000000003, 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0065594599999999996, 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:49:38.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 12776 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552497+0000", 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:38.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552497+0000", 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552498+0000", 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6139999999999999e-06 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552613+0000", 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011484000000000001 2024-09-13T22:49:38.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552624+0000", 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0686999999999999e-05 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552629+0000", 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6840000000000001e-06 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:38.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552683+0000", 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4020000000000001e-05 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.552832+0000", 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014896899999999999 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.553011+0000", 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000178903 2024-09-13T22:49:38.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.553802+0000", 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00079099699999999999 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:49:38.555159+0000", 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013570030000000001 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:38.571 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-09-13T22:49:38.571 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 3 2024-09-13T22:49:38.571 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:49:38.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:38.607 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:38.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:38 smithi192 bash[24608]: cluster 2024-09-13T22:49:37.426129+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:49:38.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:38 smithi192 bash[24608]: cluster 2024-09-13T22:49:37.429468+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:49:38.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:38 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.010222+0000 osd.6 (osd.6) 195 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:49:38.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:38 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.014216+0000 osd.6 (osd.6) 196 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:49:38.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:38 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.137652+0000 osd.0 (osd.0) 245 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:49:38.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:38 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.143753+0000 osd.0 (osd.0) 246 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:49:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:38.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:38.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:38.743 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:49:38.744 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:49:38.774 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:49:38.775 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:49:38.820 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:49:38.821 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:49:38.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[22840]: cluster 2024-09-13T22:49:37.426129+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[22840]: cluster 2024-09-13T22:49:37.429468+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.010222+0000 osd.6 (osd.6) 195 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.014216+0000 osd.6 (osd.6) 196 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.137652+0000 osd.0 (osd.0) 245 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.143753+0000 osd.0 (osd.0) 246 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[29889]: cluster 2024-09-13T22:49:37.426129+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[29889]: cluster 2024-09-13T22:49:37.429468+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:49:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.010222+0000 osd.6 (osd.6) 195 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:49:38.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.014216+0000 osd.6 (osd.6) 196 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:49:38.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.137652+0000 osd.0 (osd.0) 245 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:49:38.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:38 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.143753+0000 osd.0 (osd.0) 246 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:49:38.860 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:38.861 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:49:38.861 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:49:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:38.895 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:49:38.895 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:49:38.935 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:49:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:49:38.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:38.975 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:49:38.975 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:49:38.985 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:38.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:39.010 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:49:39.011 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:49:39.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:39.120 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:39.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:39.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 7 16 14510 14494 129.4 137.625 0.00404019 0.00772093 2024-09-13T22:49:39.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:39.378 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:39.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:39.505 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:39.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.633 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:39.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.688260+0000 mgr.y (mgr.14613) 447 : cluster [DBG] pgmap v496: 158 pgs: 158 active+clean; 378 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 604 op/s 2024-09-13T22:49:39.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[22840]: audit 2024-09-13T22:49:38.699921+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.178:0/2125261501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:39.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[22840]: cluster 2024-09-13T22:49:39.103043+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:49:39.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[22840]: cluster 2024-09-13T22:49:39.109360+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:49:39.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.747 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:39 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.688260+0000 mgr.y (mgr.14613) 447 : cluster [DBG] pgmap v496: 158 pgs: 158 active+clean; 378 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 604 op/s 2024-09-13T22:49:39.747 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:39 smithi192 bash[24608]: audit 2024-09-13T22:49:38.699921+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.178:0/2125261501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:39.747 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:39 smithi192 bash[24608]: cluster 2024-09-13T22:49:39.103043+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:49:39.747 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:39 smithi192 bash[24608]: cluster 2024-09-13T22:49:39.109360+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:49:39.762 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:39.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.890 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:39.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:39.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T22:49:39.958 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:49:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:49:39.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.688260+0000 mgr.y (mgr.14613) 447 : cluster [DBG] pgmap v496: 158 pgs: 158 active+clean; 378 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 604 op/s 2024-09-13T22:49:39.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[29889]: audit 2024-09-13T22:49:38.699921+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.178:0/2125261501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:39.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[29889]: cluster 2024-09-13T22:49:39.103043+0000 osd.0 (osd.0) 247 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:49:39.958 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:39 smithi178 bash[29889]: cluster 2024-09-13T22:49:39.109360+0000 osd.0 (osd.0) 248 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:49:39.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:40.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:40.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:40.155 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:40.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:40.287 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:40.319 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 8 16 16705 16689 130.372 137.188 0.00547548 0.00766198 2024-09-13T22:49:40.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:40.428 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:40.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:40.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:40.684 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:40.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:40.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:40.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:40 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.842948+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:49:40.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:40 smithi192 bash[24608]: cluster 2024-09-13T22:49:38.845289+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:49:40.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:40.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:40 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.842948+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:49:40.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:40 smithi178 bash[22840]: cluster 2024-09-13T22:49:38.845289+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:49:40.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:40 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.842948+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:49:40.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:40 smithi178 bash[29889]: cluster 2024-09-13T22:49:38.845289+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:49:40.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:40.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:41.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:41.074 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:41.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:41.205 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:41.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:41.320 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 9 16 18655 18639 129.426 121.875 0.00310967 0.00771902 2024-09-13T22:49:41.332 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:41.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:41.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:41.588 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:41.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.716 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:41.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:39.879706+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:39.881744+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:40.198275+0000 osd.5 (osd.5) 273 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:40.203224+0000 osd.5 (osd.5) 274 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:40.689212+0000 mgr.y (mgr.14613) 448 : cluster [DBG] pgmap v497: 158 pgs: 158 active+clean; 940 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:40.701770+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[22840]: cluster 2024-09-13T22:49:40.725751+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T22:49:41.836 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:39.879706+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:49:41.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:39.881744+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:49:41.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:40.198275+0000 osd.5 (osd.5) 273 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:49:41.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:40.203224+0000 osd.5 (osd.5) 274 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:49:41.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:40.689212+0000 mgr.y (mgr.14613) 448 : cluster [DBG] pgmap v497: 158 pgs: 158 active+clean; 940 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-13T22:49:41.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:40.701770+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T22:49:41.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:41 smithi178 bash[29889]: cluster 2024-09-13T22:49:40.725751+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T22:49:41.848 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:41.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:39.879706+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:49:41.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:39.881744+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:49:41.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:40.198275+0000 osd.5 (osd.5) 273 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:49:41.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:40.203224+0000 osd.5 (osd.5) 274 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:49:41.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:40.689212+0000 mgr.y (mgr.14613) 448 : cluster [DBG] pgmap v497: 158 pgs: 158 active+clean; 940 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-13T22:49:41.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:40.701770+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T22:49:41.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:41 smithi192 bash[24608]: cluster 2024-09-13T22:49:40.725751+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T22:49:41.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.986 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:41.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:42.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:42.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:42.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:42.255 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:42.320 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 10 16 20480 20464 127.889 114.062 0.0032894 0.00781258 2024-09-13T22:49:42.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:42.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:42.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:42.605 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:49:42.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:42.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.752 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:42 smithi192 bash[24608]: audit 2024-09-13T22:49:41.633062+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? v1:172.21.15.178:0/2364473592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:49:42.752 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:42 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.158272+0000 osd.0 (osd.0) 249 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:49:42.752 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:42 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.161800+0000 osd.0 (osd.0) 250 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:49:42.764 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:42.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.894 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:42.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:42.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:42 smithi178 bash[29889]: audit 2024-09-13T22:49:41.633062+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? v1:172.21.15.178:0/2364473592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:49:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:42 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.158272+0000 osd.0 (osd.0) 249 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:49:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:42 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.161800+0000 osd.0 (osd.0) 250 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:49:42.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:42 smithi178 bash[22840]: audit 2024-09-13T22:49:41.633062+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? v1:172.21.15.178:0/2364473592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:49:42.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:42 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.158272+0000 osd.0 (osd.0) 249 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:49:42.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:42 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.161800+0000 osd.0 (osd.0) 250 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:49:42.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:43.028 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:43.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:43.153 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:43.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:43.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:43.320 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 11 16 22521 22505 127.858 127.562 0.00996507 0.00781406 2024-09-13T22:49:43.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:43.401 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:43.504 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:43.537 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:43.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:43.664 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:43.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:43.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:43.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:43.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:43 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.174183+0000 osd.5 (osd.5) 275 : cluster [DBG] 5.1b deep-scrub starts 2024-09-13T22:49:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:43 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.177423+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.1b deep-scrub ok 2024-09-13T22:49:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:43 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.593758+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:43 smithi192 bash[24608]: audit 2024-09-13T22:49:42.603572+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? v1:172.21.15.178:0/2364473592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:49:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:43 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.609149+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-09-13T22:49:43.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:43 smithi192 bash[24608]: cluster 2024-09-13T22:49:42.689644+0000 mgr.y (mgr.14613) 449 : cluster [DBG] pgmap v499: 158 pgs: 158 active+clean; 940 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T22:49:43.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:43.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:43.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.174183+0000 osd.5 (osd.5) 275 : cluster [DBG] 5.1b deep-scrub starts 2024-09-13T22:49:43.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.177423+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.1b deep-scrub ok 2024-09-13T22:49:43.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.593758+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:43.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[29889]: audit 2024-09-13T22:49:42.603572+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? v1:172.21.15.178:0/2364473592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:49:43.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.609149+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[29889]: cluster 2024-09-13T22:49:42.689644+0000 mgr.y (mgr.14613) 449 : cluster [DBG] pgmap v499: 158 pgs: 158 active+clean; 940 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.174183+0000 osd.5 (osd.5) 275 : cluster [DBG] 5.1b deep-scrub starts 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.177423+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.1b deep-scrub ok 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.593758+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[22840]: audit 2024-09-13T22:49:42.603572+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? v1:172.21.15.178:0/2364473592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.609149+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-09-13T22:49:43.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:43 smithi178 bash[22840]: cluster 2024-09-13T22:49:42.689644+0000 mgr.y (mgr.14613) 449 : cluster [DBG] pgmap v499: 158 pgs: 158 active+clean; 940 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T22:49:44.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:44.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:44.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:44.180 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:44.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:44.306 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:44.320 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 12 16 24151 24135 125.692 101.875 0.00870256 0.00794959 2024-09-13T22:49:44.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:44.434 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:44.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:44.560 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:44.612 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.2 primary-affinity to 0.416638 (8273042) 2024-09-13T22:49:44.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:44.673 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:44 smithi192 bash[24608]: audit 2024-09-13T22:49:44.471351+0000 mon.a (mon.0) 1298 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]: dispatch 2024-09-13T22:49:44.674 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:44 smithi192 bash[24608]: audit 2024-09-13T22:49:44.472722+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.178:0/428979138' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]: dispatch 2024-09-13T22:49:44.690 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:44.756 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:49:44.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:44.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:44.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:44.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:44 smithi178 bash[22840]: audit 2024-09-13T22:49:44.471351+0000 mon.a (mon.0) 1298 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]: dispatch 2024-09-13T22:49:44.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:44 smithi178 bash[22840]: audit 2024-09-13T22:49:44.472722+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.178:0/428979138' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]: dispatch 2024-09-13T22:49:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:44 smithi178 bash[29889]: audit 2024-09-13T22:49:44.471351+0000 mon.a (mon.0) 1298 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]: dispatch 2024-09-13T22:49:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:44 smithi178 bash[29889]: audit 2024-09-13T22:49:44.472722+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.178:0/428979138' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]: dispatch 2024-09-13T22:49:44.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:44.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:45.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:45.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:45.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:49:45.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:45.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:45.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:45.321 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 13 16 26107 26091 125.419 122.25 0.0122724 0.0079681 2024-09-13T22:49:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:45.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:45.476 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:45.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:45.590 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:49:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:49:45.603 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:45.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:45.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:45 smithi178 bash[29889]: audit 2024-09-13T22:49:44.607409+0000 mon.a (mon.0) 1299 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]': finished 2024-09-13T22:49:45.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:45 smithi178 bash[29889]: cluster 2024-09-13T22:49:44.611097+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-09-13T22:49:45.718 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:45 smithi178 bash[29889]: cluster 2024-09-13T22:49:44.690007+0000 mgr.y (mgr.14613) 450 : cluster [DBG] pgmap v501: 158 pgs: 158 active+clean; 1.2 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-13T22:49:45.718 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:45 smithi178 bash[22840]: audit 2024-09-13T22:49:44.607409+0000 mon.a (mon.0) 1299 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]': finished 2024-09-13T22:49:45.718 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:45 smithi178 bash[22840]: cluster 2024-09-13T22:49:44.611097+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-09-13T22:49:45.718 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:45 smithi178 bash[22840]: cluster 2024-09-13T22:49:44.690007+0000 mgr.y (mgr.14613) 450 : cluster [DBG] pgmap v501: 158 pgs: 158 active+clean; 1.2 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-13T22:49:45.733 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:45.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:45.863 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:45.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:45 smithi192 bash[24608]: audit 2024-09-13T22:49:44.607409+0000 mon.a (mon.0) 1299 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.41663832260396405}]': finished 2024-09-13T22:49:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:45 smithi192 bash[24608]: cluster 2024-09-13T22:49:44.611097+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-09-13T22:49:45.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:45 smithi192 bash[24608]: cluster 2024-09-13T22:49:44.690007+0000 mgr.y (mgr.14613) 450 : cluster [DBG] pgmap v501: 158 pgs: 158 active+clean; 1.2 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-13T22:49:45.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:45.986 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:45.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:46.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:46.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:46.221 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:46.249 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:46.321 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 14 16 27716 27700 123.643 100.562 0.00414924 0.00808206 2024-09-13T22:49:46.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:46.377 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:46.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:46.507 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:46.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:46.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:46.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:46.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:46.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:46.892 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:46.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:46.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:46 smithi192 bash[24608]: audit 2024-09-13T22:49:45.398025+0000 mgr.y (mgr.14613) 451 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:46.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:46 smithi192 bash[24608]: cluster 2024-09-13T22:49:45.629692+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-09-13T22:49:46.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:46 smithi178 bash[22840]: audit 2024-09-13T22:49:45.398025+0000 mgr.y (mgr.14613) 451 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:46.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:46 smithi178 bash[22840]: cluster 2024-09-13T22:49:45.629692+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-09-13T22:49:46.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:46 smithi178 bash[29889]: audit 2024-09-13T22:49:45.398025+0000 mgr.y (mgr.14613) 451 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:46.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:46 smithi178 bash[29889]: cluster 2024-09-13T22:49:45.629692+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-09-13T22:49:46.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.020 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:47.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:47.142 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:47.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:47.321 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 15 16 28938 28922 120.491 76.375 0.00466812 0.00829224 2024-09-13T22:49:47.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:47.403 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:47.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:47.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:47.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:47.794 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:47.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:47.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:47 smithi192 bash[24608]: cluster 2024-09-13T22:49:46.690527+0000 mgr.y (mgr.14613) 452 : cluster [DBG] pgmap v503: 158 pgs: 4 peering, 154 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-13T22:49:47.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:47.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:47.929 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:47 smithi178 bash[22840]: cluster 2024-09-13T22:49:46.690527+0000 mgr.y (mgr.14613) 452 : cluster [DBG] pgmap v503: 158 pgs: 4 peering, 154 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-13T22:49:47.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:47 smithi178 bash[29889]: cluster 2024-09-13T22:49:46.690527+0000 mgr.y (mgr.14613) 452 : cluster [DBG] pgmap v503: 158 pgs: 4 peering, 154 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-13T22:49:48.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:48.056 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:48.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:48.182 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:48.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:48.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:49:48.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:48.322 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 16 16 30754 30738 120.047 113.5 0.0101396 0.00832221 2024-09-13T22:49:48.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:48.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:48.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:48.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:48.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:48.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:48 smithi192 bash[24608]: cluster 2024-09-13T22:49:47.634300+0000 mon.a (mon.0) 1302 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T22:49:48.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:48 smithi192 bash[24608]: audit 2024-09-13T22:49:48.576758+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:48.677 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:48 smithi192 bash[24608]: audit 2024-09-13T22:49:48.580912+0000 mon.b (mon.1) 190 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:48.691 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:48.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:48.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:48.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:48.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:48 smithi178 bash[22840]: cluster 2024-09-13T22:49:47.634300+0000 mon.a (mon.0) 1302 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T22:49:48.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:48 smithi178 bash[22840]: audit 2024-09-13T22:49:48.576758+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:48.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:48 smithi178 bash[22840]: audit 2024-09-13T22:49:48.580912+0000 mon.b (mon.1) 190 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:48.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:48 smithi178 bash[29889]: cluster 2024-09-13T22:49:47.634300+0000 mon.a (mon.0) 1302 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-13T22:49:48.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:48 smithi178 bash[29889]: audit 2024-09-13T22:49:48.576758+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:49:48.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:48 smithi178 bash[29889]: audit 2024-09-13T22:49:48.580912+0000 mon.b (mon.1) 190 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:49:48.944 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:48.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:49.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:49.073 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:49.148 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:49:49.148 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:49:49.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:49.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:49.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:49.322 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 17 16 32487 32471 119.356 108.312 0.0117587 0.00837095 2024-09-13T22:49:49.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:49.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:49.462 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:49.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:49.591 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:49.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:49.716 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:49.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:49.830 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:49:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:49:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:49:49.830 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:49 smithi178 bash[29889]: cluster 2024-09-13T22:49:48.690978+0000 mgr.y (mgr.14613) 453 : cluster [DBG] pgmap v504: 158 pgs: 4 peering, 154 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 841 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T22:49:49.831 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:49 smithi178 bash[22840]: cluster 2024-09-13T22:49:48.690978+0000 mgr.y (mgr.14613) 453 : cluster [DBG] pgmap v504: 158 pgs: 4 peering, 154 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 841 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T22:49:49.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:49.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:49.962 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:49 smithi192 bash[24608]: cluster 2024-09-13T22:49:48.690978+0000 mgr.y (mgr.14613) 453 : cluster [DBG] pgmap v504: 158 pgs: 4 peering, 154 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 841 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T22:49:49.978 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:49.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:50.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:50.108 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:50.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:50.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:50.322 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 18 16 34238 34222 118.804 109.438 0.00592381 0.00841081 2024-09-13T22:49:50.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:50.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T22:49:50.365 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:50.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:50.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:50.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:50.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:50.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:50.764 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:50.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:50.901 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:50.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:51.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:51.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:50 smithi178 bash[29889]: cluster 2024-09-13T22:49:50.691993+0000 mgr.y (mgr.14613) 454 : cluster [DBG] pgmap v505: 158 pgs: 158 active+clean; 2.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-13T22:49:51.014 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:50 smithi178 bash[22840]: cluster 2024-09-13T22:49:50.691993+0000 mgr.y (mgr.14613) 454 : cluster [DBG] pgmap v505: 158 pgs: 158 active+clean; 2.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-13T22:49:51.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:51.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:51.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:50 smithi192 bash[24608]: cluster 2024-09-13T22:49:50.691993+0000 mgr.y (mgr.14613) 454 : cluster [DBG] pgmap v505: 158 pgs: 158 active+clean; 2.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-13T22:49:51.160 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:51.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:51.293 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:51.322 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 19 16 35920 35904 118.084 105.125 0.00454775 0.00846218 2024-09-13T22:49:51.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:51.419 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:51.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:51.546 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:51.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:51.679 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:51.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:51.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:51 smithi192 bash[24608]: cluster 2024-09-13T22:49:50.743431+0000 mon.a (mon.0) 1304 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T22:49:51.807 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:51.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:51.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:51 smithi178 bash[29889]: cluster 2024-09-13T22:49:50.743431+0000 mon.a (mon.0) 1304 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T22:49:51.921 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:51 smithi178 bash[22840]: cluster 2024-09-13T22:49:50.743431+0000 mon.a (mon.0) 1304 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-13T22:49:51.939 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:51.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:52.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:52.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:52.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:52.198 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:52.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:52.324 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:49:52.325161+0000 min lat: 0.00136126 max lat: 0.323978 avg lat: 0.00853978 2024-09-13T22:49:52.324 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:49:52.324 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 20 16 37474 37458 117.036 97.125 0.0083441 0.00853978 2024-09-13T22:49:52.326 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:52.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:52.452 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:52.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:52.591 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:52.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:52.720 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:52.743 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:49:52.743 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:49:52.743 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T22:49:52.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T22:49:52.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:52.834 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:52 smithi178 bash[22840]: cluster 2024-09-13T22:49:52.692447+0000 mgr.y (mgr.14613) 455 : cluster [DBG] pgmap v506: 158 pgs: 158 active+clean; 2.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:49:52.835 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:52 smithi178 bash[29889]: cluster 2024-09-13T22:49:52.692447+0000 mgr.y (mgr.14613) 455 : cluster [DBG] pgmap v506: 158 pgs: 158 active+clean; 2.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:49:52.854 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:52.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:52.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:52.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:53.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:53.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:53.104 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:53.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:52 smithi192 bash[24608]: cluster 2024-09-13T22:49:52.692447+0000 mgr.y (mgr.14613) 455 : cluster [DBG] pgmap v506: 158 pgs: 158 active+clean; 2.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-13T22:49:53.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:53.228 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:53.323 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 21 16 38427 38411 114.299 59.5625 0.419414 0.00871359 2024-09-13T22:49:53.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:53.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:53.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:53.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:53.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:53.634 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:53.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:53.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:53.898 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:53.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:54.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.026 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:54.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.158 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:54.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.289 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:54.323 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 22 16 39864 39848 113.185 89.8125 0.00641879 0.00882943 2024-09-13T22:49:54.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.422 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:54.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:54.543 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:54.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:54.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:54.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.802 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:54.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:54.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:54.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:55.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:55.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:54 smithi192 bash[24608]: cluster 2024-09-13T22:49:53.502154+0000 osd.5 (osd.5) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:55.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:54 smithi192 bash[24608]: cluster 2024-09-13T22:49:53.502212+0000 osd.5 (osd.5) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:55.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:54 smithi192 bash[24608]: cluster 2024-09-13T22:49:53.883992+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:55.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:54 smithi192 bash[24608]: cluster 2024-09-13T22:49:53.884078+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:55.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:54 smithi192 bash[24608]: cluster 2024-09-13T22:49:53.933428+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2024-09-13T22:49:55.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:54 smithi192 bash[24608]: cluster 2024-09-13T22:49:54.692920+0000 mgr.y (mgr.14613) 456 : cluster [DBG] pgmap v508: 158 pgs: 158 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 564 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-13T22:49:55.049 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:55.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:55.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[22840]: cluster 2024-09-13T22:49:53.502154+0000 osd.5 (osd.5) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:55.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[22840]: cluster 2024-09-13T22:49:53.502212+0000 osd.5 (osd.5) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:55.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[22840]: cluster 2024-09-13T22:49:53.883992+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:55.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[22840]: cluster 2024-09-13T22:49:53.884078+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[22840]: cluster 2024-09-13T22:49:53.933428+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[22840]: cluster 2024-09-13T22:49:54.692920+0000 mgr.y (mgr.14613) 456 : cluster [DBG] pgmap v508: 158 pgs: 158 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 564 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[29889]: cluster 2024-09-13T22:49:53.502154+0000 osd.5 (osd.5) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[29889]: cluster 2024-09-13T22:49:53.502212+0000 osd.5 (osd.5) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[29889]: cluster 2024-09-13T22:49:53.883992+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[29889]: cluster 2024-09-13T22:49:53.884078+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[29889]: cluster 2024-09-13T22:49:53.933428+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2024-09-13T22:49:55.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:54 smithi178 bash[29889]: cluster 2024-09-13T22:49:54.692920+0000 mgr.y (mgr.14613) 456 : cluster [DBG] pgmap v508: 158 pgs: 158 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 564 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-13T22:49:55.173 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:49:55.188 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:55.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:55.323 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 23 16 41290 41274 112.139 89.125 0.0032574 0.00891173 2024-09-13T22:49:55.327 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:55.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:55.448 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:55.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:55.577 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:55.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:55.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:49:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:49:55.706 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:55.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:55.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:55.954 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:55.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:56.008 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:55 smithi192 bash[24608]: audit 2024-09-13T22:49:55.408653+0000 mgr.y (mgr.14613) 457 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:56.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:56.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:56.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:56.205 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:56.262 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:49:56.262 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:49:56.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:56.315 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:49:56.316 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:49:56.319 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:55 smithi178 bash[29889]: audit 2024-09-13T22:49:55.408653+0000 mgr.y (mgr.14613) 457 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:56.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:55 smithi178 bash[22840]: audit 2024-09-13T22:49:55.408653+0000 mgr.y (mgr.14613) 457 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:49:56.324 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 24 16 42583 42567 110.834 80.8125 0.0054533 0.00901767 2024-09-13T22:49:56.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:56.353 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:49:56.353 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:49:56.388 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:49:56.389 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:49:56.420 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:49:56.421 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:49:56.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:56.447 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:49:56.447 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:49:56.459 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:56.475 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:49:56.475 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:49:56.506 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:49:56.506 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:49:56.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:56.600 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:56.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:56.721 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:56.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:56.849 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:56.941 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:49:56.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:56.970 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:56.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:57.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:57.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:56 smithi192 bash[24608]: audit 2024-09-13T22:49:56.224625+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.178:0/3625234104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:57.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:56 smithi192 bash[24608]: cluster 2024-09-13T22:49:56.693505+0000 mgr.y (mgr.14613) 458 : cluster [DBG] pgmap v509: 158 pgs: 158 active+clean; 2.5 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T22:49:57.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:56 smithi192 bash[24608]: audit 2024-09-13T22:49:56.936055+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? v1:172.21.15.178:0/2950205994' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:49:57.096 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:57.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:57.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:56 smithi178 bash[29889]: audit 2024-09-13T22:49:56.224625+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.178:0/3625234104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:57.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:56 smithi178 bash[29889]: cluster 2024-09-13T22:49:56.693505+0000 mgr.y (mgr.14613) 458 : cluster [DBG] pgmap v509: 158 pgs: 158 active+clean; 2.5 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T22:49:57.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:56 smithi178 bash[29889]: audit 2024-09-13T22:49:56.936055+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? v1:172.21.15.178:0/2950205994' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:49:57.210 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:56 smithi178 bash[22840]: audit 2024-09-13T22:49:56.224625+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.178:0/3625234104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:49:57.211 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:56 smithi178 bash[22840]: cluster 2024-09-13T22:49:56.693505+0000 mgr.y (mgr.14613) 458 : cluster [DBG] pgmap v509: 158 pgs: 158 active+clean; 2.5 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T22:49:57.211 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:56 smithi178 bash[22840]: audit 2024-09-13T22:49:56.936055+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? v1:172.21.15.178:0/2950205994' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:49:57.233 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:57.323 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 25 16 43520 43504 108.743 58.5625 0.0129908 0.00918991 2024-09-13T22:49:57.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.365 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:57.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:57.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:57.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:57.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:57.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:57.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:57.752 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:57.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:57.897 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:49:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:57.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:49:58.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:58.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:49:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:49:58.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:58.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[29889]: cluster 2024-09-13T22:49:56.938828+0000 mon.a (mon.0) 1307 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:58.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[29889]: audit 2024-09-13T22:49:56.943535+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.178:0/2950205994' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[29889]: cluster 2024-09-13T22:49:56.948893+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e260: 8 total, 8 up, 7 in 2024-09-13T22:49:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[29889]: cluster 2024-09-13T22:49:57.931745+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e261: 8 total, 8 up, 7 in 2024-09-13T22:49:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[22840]: cluster 2024-09-13T22:49:56.938828+0000 mon.a (mon.0) 1307 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[22840]: audit 2024-09-13T22:49:56.943535+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.178:0/2950205994' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[22840]: cluster 2024-09-13T22:49:56.948893+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e260: 8 total, 8 up, 7 in 2024-09-13T22:49:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[22840]: cluster 2024-09-13T22:49:57.931745+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e261: 8 total, 8 up, 7 in 2024-09-13T22:49:58.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:58 smithi192 bash[24608]: cluster 2024-09-13T22:49:56.938828+0000 mon.a (mon.0) 1307 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:49:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:58 smithi192 bash[24608]: audit 2024-09-13T22:49:56.943535+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.178:0/2950205994' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:49:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:58 smithi192 bash[24608]: cluster 2024-09-13T22:49:56.948893+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e260: 8 total, 8 up, 7 in 2024-09-13T22:49:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:58 smithi192 bash[24608]: cluster 2024-09-13T22:49:57.931745+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e261: 8 total, 8 up, 7 in 2024-09-13T22:49:58.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:49:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:49:58.323 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 26 16 44685 44669 107.36 72.8125 0.0907077 0.00928675 2024-09-13T22:49:58.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:58.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:49:58.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:58.530 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:58.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:58.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:23490 7.0 7:01cf0e7e:::benchmark_data_smithi178_64772_object23489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-13T22:49:58.762 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:18.467563+0000", 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 520.296275487, 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35839699699999999, 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23490 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467563+0000", 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467563+0000", 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467565+0000", 2024-09-13T22:49:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.756e-06 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467633+0000", 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8287999999999994e-05 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467661+0000", 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7744000000000001e-05 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467669+0000", 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.977e-06 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467710+0000", 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1112000000000002e-05 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467813+0000", 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102951 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.467900+0000", 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7072999999999998e-05 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.468613+0000", 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00071297299999999999 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.469605+0000", 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00099249300000000011 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.825674+0000", 2024-09-13T22:49:58.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35606887199999998 2024-09-13T22:49:58.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.825805+0000", 2024-09-13T22:49:58.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000130748 2024-09-13T22:49:58.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.825960+0000", 2024-09-13T22:49:58.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015500999999999999 2024-09-13T22:49:58.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.771 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.772 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.772 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:23502 7.0 7:0eb8c4bc:::benchmark_data_smithi178_64772_object23501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-13T22:49:58.772 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:41:18.474625+0000", 2024-09-13T22:49:58.772 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 520.28921293500002, 2024-09-13T22:49:58.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35159708499999998, 2024-09-13T22:49:58.773 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.773 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.773 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.773 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.773 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:58.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:58.774 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23502 2024-09-13T22:49:58.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.774 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474625+0000", 2024-09-13T22:49:58.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474625+0000", 2024-09-13T22:49:58.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474627+0000", 2024-09-13T22:49:58.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.542e-06 2024-09-13T22:49:58.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474696+0000", 2024-09-13T22:49:58.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9435000000000002e-05 2024-09-13T22:49:58.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474708+0000", 2024-09-13T22:49:58.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1542999999999999e-05 2024-09-13T22:49:58.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474716+0000", 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3550000000000005e-06 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474763+0000", 2024-09-13T22:49:58.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7048000000000001e-05 2024-09-13T22:49:58.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474870+0000", 2024-09-13T22:49:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010668 2024-09-13T22:49:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-09-13T22:49:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.474958+0000", 2024-09-13T22:49:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7711000000000001e-05 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.475732+0000", 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00077461999999999995 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.476091+0000", 2024-09-13T22:49:58.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035836700000000001 2024-09-13T22:49:58.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.826053+0000", 2024-09-13T22:49:58.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34996211300000002 2024-09-13T22:49:58.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.826145+0000", 2024-09-13T22:49:58.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2608999999999994e-05 2024-09-13T22:49:58.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:41:18.826223+0000", 2024-09-13T22:49:58.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7062000000000002e-05 2024-09-13T22:49:58.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.786 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.786 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24767.0:189698 7.b e165/116 7:db044c2c:::benchmark_data_smithi178_64772_object83945:head v 165'15698, mlcod=165'15692)", 2024-09-13T22:49:58.787 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:23.326580+0000", 2024-09-13T22:49:58.787 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 335.43725798999998, 2024-09-13T22:49:58.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.38755634999999999, 2024-09-13T22:49:58.787 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.787 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.788 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.788 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.326580+0000", 2024-09-13T22:49:58.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.326580+0000", 2024-09-13T22:49:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.326584+0000", 2024-09-13T22:49:58.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3739999999999998e-06 2024-09-13T22:49:58.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.326600+0000", 2024-09-13T22:49:58.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6299000000000001e-05 2024-09-13T22:49:58.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.326626+0000", 2024-09-13T22:49:58.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5496000000000001e-05 2024-09-13T22:49:58.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.326644+0000", 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8519999999999999e-05 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698275+0000", 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37163050199999997 2024-09-13T22:49:58.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698612+0000", 2024-09-13T22:49:58.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033741400000000002 2024-09-13T22:49:58.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.713986+0000", 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015374214000000001 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.714137+0000", 2024-09-13T22:49:58.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000150531 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24767.0:189700 7.e e165/116 7:76360c71:::benchmark_data_smithi178_64772_object83947:head v 165'11787, mlcod=165'11785)", 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:23.330306+0000", 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 335.43353273299999, 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37632971399999998, 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.796 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330306+0000", 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330306+0000", 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330317+0000", 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0923e-05 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330332+0000", 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5387e-05 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698196+0000", 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.367863622 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698232+0000", 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6386e-05 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698470+0000", 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000237879 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698530+0000", 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0278999999999997e-05 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.706444+0000", 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0079134359999999994 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.706635+0000", 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000191802 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.800 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24767.0:189701 7.b e165/116 7:d7620514:::benchmark_data_smithi178_64772_object83948:head v 165'15699, mlcod=165'15692)", 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:23.330878+0000", 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 335.43296041999997, 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.38338837799999997, 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330878+0000", 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330878+0000", 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330881+0000", 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1130000000000001e-06 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.330895+0000", 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4192999999999999e-05 2024-09-13T22:49:58.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698195+0000", 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36729950700000003 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.698232+0000", 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7258999999999997e-05 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.703704+0000", 2024-09-13T22:49:58.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054724389999999999 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.703736+0000", 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1090000000000002e-05 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.714149+0000", 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010412967 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:23.714266+0000", 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011781 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24767.0:195077 7.0 7:0bf3a157:::benchmark_data_smithi178_64772_object89324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:44:29.476789+0000", 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 329.28704969799998, 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.384112605, 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.805 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24767", 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 195077 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476789+0000", 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476789+0000", 2024-09-13T22:49:58.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476792+0000", 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1379999999999999e-06 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476801+0000", 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7919999999999998e-06 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476824+0000", 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3737e-05 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476836+0000", 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1632e-05 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.476949+0000", 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113416 2024-09-13T22:49:58.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.477202+0000", 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025239499999999998 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.477439+0000", 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023692400000000001 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.480674+0000", 2024-09-13T22:49:58.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003235553 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.481394+0000", 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00072008200000000004 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.860795+0000", 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37940043299999998 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.860838+0000", 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2724e-05 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:44:29.860901+0000", 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3779e-05 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.811 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15720 8.d 8:bad8a164:::benchmark_data_smithi178_80607_object15719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.897679+0000", 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.86615980900001, 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69822529, 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15720 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897679+0000", 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897679+0000", 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897680+0000", 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.201e-06 2024-09-13T22:49:58.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897749+0000", 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9377999999999999e-05 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897761+0000", 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1622e-05 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897768+0000", 2024-09-13T22:49:58.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8759999999999999e-06 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.897865+0000", 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7592999999999999e-05 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899304+0000", 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014391930000000001 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899520+0000", 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021548699999999999 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.903810+0000", 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042897660000000004 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.132967+0000", 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.229157742 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.595763+0000", 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46279550200000003 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.595854+0000", 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1446999999999994e-05 2024-09-13T22:49:58.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.595904+0000", 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9483e-05 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15724 8.d 8:b65b1368:::benchmark_data_smithi178_80607_object15723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.818 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.899123+0000", 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.86471515900001, 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69855785699999995, 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15724 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899123+0000", 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899123+0000", 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899128+0000", 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4939999999999997e-06 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899181+0000", 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3671999999999997e-05 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899196+0000", 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.417e-05 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899205+0000", 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3729999999999999e-06 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900596+0000", 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013915189999999999 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900806+0000", 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020990000000000001 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.900989+0000", 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000182139 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.905439+0000", 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044506709999999998 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.133092+0000", 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22765264099999999 2024-09-13T22:49:58.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.597578+0000", 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46448661499999999 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.597632+0000", 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3953999999999999e-05 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.597681+0000", 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8708999999999997e-05 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15725 8.d 8:b8445dc3:::benchmark_data_smithi178_80607_object15724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.899220+0000", 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.864618382, 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69898660400000001, 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.825 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15725 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899220+0000", 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899220+0000", 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899221+0000", 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.096e-06 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899379+0000", 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015772299999999999 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899389+0000", 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0254999999999999e-05 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.899397+0000", 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8120000000000008e-06 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.902648+0000", 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003250822 2024-09-13T22:49:58.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.902795+0000", 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147603 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.902984+0000", 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000188512 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.905444+0000", 2024-09-13T22:49:58.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024597310000000002 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.133183+0000", 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.227738947 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598144+0000", 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46496159599999998 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598180+0000", 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6353999999999998e-05 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598207+0000", 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6152999999999998e-05 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15739 8.8 8:15aec11f:::benchmark_data_smithi178_80607_object15738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.904240+0000", 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.85959883800001, 2024-09-13T22:49:58.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69193515999999999, 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15739 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904240+0000", 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904240+0000", 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904242+0000", 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5979999999999999e-06 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904301+0000", 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8622999999999998e-05 2024-09-13T22:49:58.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904319+0000", 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8128e-05 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904330+0000", 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0963e-05 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904405+0000", 2024-09-13T22:49:58.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5339999999999994e-05 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904527+0000", 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121784 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,7", 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904674+0000", 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014690300000000001 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.905509+0000", 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00083487100000000005 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.905781+0000", 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000272293 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596077+0000", 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69029582099999998 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596134+0000", 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6966999999999999e-05 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.596175+0000", 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0868999999999998e-05 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15740 8.d 8:bc28c5dc:::benchmark_data_smithi178_80607_object15739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.837 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:25.904358+0000", 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.85948005, 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69399002399999998, 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15740 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904358+0000", 2024-09-13T22:49:58.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904358+0000", 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904360+0000", 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1990000000000001e-06 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904404+0000", 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3965999999999999e-05 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904415+0000", 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1596999999999999e-05 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904424+0000", 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5629999999999992e-06 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904550+0000", 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012676699999999999 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904691+0000", 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000140199 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.904840+0000", 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014912200000000001 2024-09-13T22:49:58.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:25.905669+0000", 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00082971499999999997 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.133246+0000", 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22757639599999999 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598270+0000", 2024-09-13T22:49:58.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46502417000000001 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598322+0000", 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1459000000000003e-05 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598348+0000", 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6871000000000001e-05 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15743 8.d 8:b5718b56:::benchmark_data_smithi178_80607_object15742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:26.131008+0000", 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.63283064799998, 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.467425706, 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15743 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131008+0000", 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131008+0000", 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131013+0000", 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1370000000000001e-06 2024-09-13T22:49:58.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131075+0000", 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1898999999999998e-05 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131126+0000", 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1384999999999997e-05 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131143+0000", 2024-09-13T22:49:58.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7297000000000001e-05 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131276+0000", 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013300399999999999 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131516+0000", 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023960299999999999 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.131766+0000", 2024-09-13T22:49:58.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024946199999999999 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.133272+0000", 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001506332 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.134585+0000", 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013133439999999999 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598382+0000", 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46379679899999998 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598412+0000", 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9856999999999999e-05 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598433+0000", 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1586999999999998e-05 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.849 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:15747 8.d 8:bc318304:::benchmark_data_smithi178_80607_object15746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:45:26.136563+0000", 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 272.62727495399997, 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46212077099999999, 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.850 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15747 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136563+0000", 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136563+0000", 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136567+0000", 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5659999999999998e-06 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136629+0000", 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2069000000000005e-05 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136658+0000", 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9016000000000001e-05 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136673+0000", 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4653e-05 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.136786+0000", 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011312200000000001 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.137114+0000", 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032765099999999998 2024-09-13T22:49:58.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.137350+0000", 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023640999999999999 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.138735+0000", 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001385156 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.138786+0000", 2024-09-13T22:49:58.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1075999999999997e-05 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598496+0000", 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45970989400000001 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598607+0000", 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111248 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:45:26.598684+0000", 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:63785 8.d 8:b7eafa44:::benchmark_data_smithi178_80607_object63784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:46:00.537252+0000", 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 238.226585976, 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35774764599999997, 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63785 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537252+0000", 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537252+0000", 2024-09-13T22:49:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537254+0000", 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5400000000000001e-06 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537315+0000", 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0900000000000003e-05 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537325+0000", 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0040999999999999e-05 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537332+0000", 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3200000000000002e-06 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537389+0000", 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6969999999999998e-05 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537518+0000", 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012867 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.537671+0000", 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015348299999999999 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.539932+0000", 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022603739999999999 2024-09-13T22:49:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.539974+0000", 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1798999999999997e-05 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.894902+0000", 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35492850599999998 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.894954+0000", 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2336999999999997e-05 2024-09-13T22:49:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:46:00.895000+0000", 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5705999999999999e-05 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14454.0:2089 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93919878339328 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:02.870175+0000", 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 175.89366319600001, 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0526293250000001, 2024-09-13T22:49:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14454", 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/1055532036", 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2089 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870175+0000", 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870175+0000", 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870179+0000", 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.309e-06 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870188+0000", 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3440000000000007e-06 2024-09-13T22:49:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870209+0000", 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1098e-05 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870219+0000", 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5689999999999994e-06 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870292+0000", 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3633999999999993e-05 2024-09-13T22:49:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870298+0000", 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0800000000000002e-06 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.884573+0000", 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014274902000000001 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.884585+0000", 2024-09-13T22:49:58.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1874e-05 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.911824+0000", 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.027238936 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.912206+0000", 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038173100000000001 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.914244+0000", 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020380900000000002 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.914473+0000", 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022925699999999999 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.922805+0000", 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0083315009999999998 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119763 8.4 8:248c2d46:::benchmark_data_smithi178_80607_object11718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:02.993075+0000", 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 175.77076368100001, 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0157326630000001, 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119763 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993075+0000", 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993075+0000", 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993079+0000", 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1799999999999998e-06 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993091+0000", 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1996e-05 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993120+0000", 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8963999999999999e-05 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993135+0000", 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5146e-05 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993253+0000", 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011846200000000001 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.993271+0000", 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7136e-05 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.903754+0000", 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91048370700000003 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.903787+0000", 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2348999999999998e-05 2024-09-13T22:49:58.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.992793+0000", 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.089006819 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.992914+0000", 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120321 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,6", 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:04.993280+0000", 2024-09-13T22:49:58.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036629700000000002 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.000549+0000", 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072688309999999999 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008334+0000", 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0077847589999999996 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008346+0000", 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2653000000000001e-05 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008549+0000", 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000202876 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008655+0000", 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105843 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008807+0000", 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015232399999999999 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119775 8.4 8:235102e3:::benchmark_data_smithi178_80607_object11730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:03.018505+0000", 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 175.745333684, 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9941815470000002, 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.878 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119775 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018505+0000", 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018505+0000", 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018508+0000", 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0599999999999999e-06 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018519+0000", 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0728e-05 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018540+0000", 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1628999999999999e-05 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018562+0000", 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1838999999999999e-05 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018715+0000", 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015322400000000001 2024-09-13T22:49:58.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.018741+0000", 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5761999999999998e-05 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.903957+0000", 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88521613499999996 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.903981+0000", 2024-09-13T22:49:58.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4147000000000001e-05 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.000697+0000", 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0967157569999999 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.001045+0000", 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034769500000000002 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,6", 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.001373+0000", 2024-09-13T22:49:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032865800000000001 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008351+0000", 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069771900000000003 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008379+0000", 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8031999999999999e-05 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008431+0000", 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2410000000000001e-05 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.012463+0000", 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040317800000000004 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.012539+0000", 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5754999999999996e-05 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.012686+0000", 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147746 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119778 8.4 8:20d19d6a:::benchmark_data_smithi178_80607_object11733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:03.019237+0000", 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 175.744601475, 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.993742031, 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.886 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119778 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019237+0000", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019237+0000", 2024-09-13T22:49:58.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019245+0000", 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.303e-06 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019255+0000", 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8740000000000007e-06 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019280+0000", 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5048000000000001e-05 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019296+0000", 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6163000000000001e-05 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019401+0000", 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010516100000000001 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.019416+0000", 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4960000000000001e-05 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.904048+0000", 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.884631748 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.904071+0000", 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2682999999999999e-05 2024-09-13T22:49:58.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.004861+0000", 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.100789947 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.005026+0000", 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016517699999999999 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,6", 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.005140+0000", 2024-09-13T22:49:58.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114096 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008387+0000", 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032468229999999998 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008893+0000", 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00050565099999999997 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.008973+0000", 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0782e-05 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.012772+0000", 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037984609999999999 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.012840+0000", 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8032000000000002e-05 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.012979+0000", 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013912199999999999 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:119794 8.4 8:2506cd94:::benchmark_data_smithi178_80607_object11749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:03.055202+0000", 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 175.70863619100001, 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.96826335, 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.894 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 119794 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055202+0000", 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055202+0000", 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055206+0000", 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5159999999999999e-06 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055217+0000", 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0927000000000001e-05 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055242+0000", 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5823e-05 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055257+0000", 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4766000000000001e-05 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055454+0000", 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019633699999999999 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.055470+0000", 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5919000000000001e-05 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.904141+0000", 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.84867164699999997 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.904163+0000", 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1880000000000001e-05 2024-09-13T22:49:58.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.009321+0000", 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.10515827 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.009497+0000", 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000175639 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,6", 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.009696+0000", 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019855900000000001 2024-09-13T22:49:58.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.022971+0000", 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013275255 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.023048+0000", 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6829000000000006e-05 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.023126+0000", 2024-09-13T22:49:58.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8640000000000006e-05 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.023185+0000", 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8832000000000002e-05 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.023284+0000", 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8942000000000002e-05 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:05.023466+0000", 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000181569 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25037.0:190463 8.8 8:1bfe82cb:::benchmark_data_smithi178_80607_object82418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:48:25.392188+0000", 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 93.371650478999996, 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67686436299999997, 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:49:58.902 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25037", 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/125950278", 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 190463 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392188+0000", 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392188+0000", 2024-09-13T22:49:58.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392189+0000", 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9699999999999995e-07 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392191+0000", 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7989999999999998e-06 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:49:58.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392198+0000", 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7070000000000003e-06 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392202+0000", 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9330000000000003e-06 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392239+0000", 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7366000000000001e-05 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392316+0000", 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6075000000000006e-05 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,7", 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.392381+0000", 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5916999999999999e-05 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394317+0000", 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001935656 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:25.394476+0000", 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001593 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.068759+0000", 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67428246700000005 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.068951+0000", 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019213800000000001 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:49:58.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:48:26.069052+0000", 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101308 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:49:58.908 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:49:58.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:58.934 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:58.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:59.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:59.068 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:49:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:49:59.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:59.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[29889]: cluster 2024-09-13T22:49:58.693927+0000 mgr.y (mgr.14613) 459 : cluster [DBG] pgmap v512: 158 pgs: 158 active+clean; 2.5 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-13T22:49:59.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:58 smithi178 bash[22840]: cluster 2024-09-13T22:49:58.693927+0000 mgr.y (mgr.14613) 459 : cluster [DBG] pgmap v512: 158 pgs: 158 active+clean; 2.5 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-13T22:49:59.206 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:59.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:59.324 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 27 16 45850 45834 106.081 72.8125 0.00912744 0.0094205 2024-09-13T22:49:59.338 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:58 smithi192 bash[24608]: cluster 2024-09-13T22:49:58.693927+0000 mgr.y (mgr.14613) 459 : cluster [DBG] pgmap v512: 158 pgs: 158 active+clean; 2.5 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-13T22:49:59.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:59.467 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:59.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:59.595 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:49:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:49:59.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:59.725 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:49:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:49:59.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T22:49:59.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:49:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:49:59.858 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:49:59.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:59.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:49:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:49:59.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:00.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:00.097 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:49:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:50:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[29889]: cluster 2024-09-13T22:49:58.726383+0000 osd.3 (osd.3) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[29889]: cluster 2024-09-13T22:49:58.726437+0000 osd.3 (osd.3) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[29889]: cluster 2024-09-13T22:49:58.961979+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e262: 8 total, 8 up, 7 in 2024-09-13T22:50:00.097 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[22840]: cluster 2024-09-13T22:49:58.726383+0000 osd.3 (osd.3) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:00.098 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[22840]: cluster 2024-09-13T22:49:58.726437+0000 osd.3 (osd.3) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:00.098 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:49:59 smithi178 bash[22840]: cluster 2024-09-13T22:49:58.961979+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e262: 8 total, 8 up, 7 in 2024-09-13T22:50:00.109 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:00.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:00.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:00.323 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 28 16 47053 47037 104.977 75.1875 0.145116 0.00950953 2024-09-13T22:50:00.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:00.369 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:00.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:59 smithi192 bash[24608]: cluster 2024-09-13T22:49:58.726383+0000 osd.3 (osd.3) 239 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:59 smithi192 bash[24608]: cluster 2024-09-13T22:49:58.726437+0000 osd.3 (osd.3) 240 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:00.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:49:59 smithi192 bash[24608]: cluster 2024-09-13T22:49:58.961979+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e262: 8 total, 8 up, 7 in 2024-09-13T22:50:00.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:00.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:00.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:00.625 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:00.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:00.752 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:00.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:00.886 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:00.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:00.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:00.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:50:00.997 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:00 smithi192 bash[24608]: cluster 2024-09-13T22:50:00.000056+0000 mon.a (mon.0) 1312 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-13T22:50:00.997 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:00 smithi192 bash[24608]: cluster 2024-09-13T22:50:00.000105+0000 mon.a (mon.0) 1313 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T22:50:01.011 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:01.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:01.159 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:01.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:01.284 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:01.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:01.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 29 16 48479 48463 104.422 89.125 0.0131177 0.00957045 2024-09-13T22:50:01.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:01.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:01.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:50:01.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:01.526 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:00 smithi178 bash[22840]: cluster 2024-09-13T22:50:00.000056+0000 mon.a (mon.0) 1312 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-13T22:50:01.526 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:00 smithi178 bash[22840]: cluster 2024-09-13T22:50:00.000105+0000 mon.a (mon.0) 1313 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T22:50:01.526 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:00 smithi178 bash[29889]: cluster 2024-09-13T22:50:00.000056+0000 mon.a (mon.0) 1312 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-13T22:50:01.526 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:00 smithi178 bash[29889]: cluster 2024-09-13T22:50:00.000105+0000 mon.a (mon.0) 1313 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T22:50:01.542 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:01.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:01.665 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:01.767 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:01.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:01.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:01.934 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:01.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:02.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:02.058 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:02.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:02.162 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:01 smithi178 bash[29889]: cluster 2024-09-13T22:50:00.695561+0000 mgr.y (mgr.14613) 460 : cluster [DBG] pgmap v514: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-13T22:50:02.162 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:01 smithi178 bash[22840]: cluster 2024-09-13T22:50:00.695561+0000 mgr.y (mgr.14613) 460 : cluster [DBG] pgmap v514: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-13T22:50:02.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:02.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:02.321 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:02.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 30 16 49757 49741 103.603 79.875 0.0130066 0.0096458 2024-09-13T22:50:02.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:01 smithi192 bash[24608]: cluster 2024-09-13T22:50:00.695561+0000 mgr.y (mgr.14613) 460 : cluster [DBG] pgmap v514: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-13T22:50:02.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:02.455 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:02.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:02.580 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:02.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:02.697 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[22840]: debug 2024-09-13T22:50:02.666+0000 7f8b8e05d640 -1 mon.a@0(leader).osd e262 definitely_dead 0 2024-09-13T22:50:02.711 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:02.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:02.841 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:02.936 INFO:teuthology.orchestra.run.smithi178.stderr:marked in osd.4. 2024-09-13T22:50:02.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:02.973 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:02.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:03.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:03.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:02 smithi192 bash[24608]: audit 2024-09-13T22:50:02.670506+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T22:50:03.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:02 smithi192 bash[24608]: audit 2024-09-13T22:50:02.672253+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.178:0/3363524379' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T22:50:03.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:02 smithi192 bash[24608]: audit 2024-09-13T22:50:02.932171+0000 mon.a (mon.0) 1315 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T22:50:03.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:02 smithi192 bash[24608]: cluster 2024-09-13T22:50:02.938232+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-13T22:50:03.094 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:03.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:03.220 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:03.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:03.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 31 16 50643 50627 102.047 55.375 0.00237341 0.00968924 2024-09-13T22:50:03.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[29889]: audit 2024-09-13T22:50:02.670506+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[29889]: audit 2024-09-13T22:50:02.672253+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.178:0/3363524379' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[29889]: audit 2024-09-13T22:50:02.932171+0000 mon.a (mon.0) 1315 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[29889]: cluster 2024-09-13T22:50:02.938232+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[22840]: audit 2024-09-13T22:50:02.670506+0000 mon.a (mon.0) 1314 : audit [DBG] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[22840]: audit 2024-09-13T22:50:02.672253+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.178:0/3363524379' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[22840]: audit 2024-09-13T22:50:02.932171+0000 mon.a (mon.0) 1315 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T22:50:03.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:02 smithi178 bash[22840]: cluster 2024-09-13T22:50:02.938232+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-13T22:50:03.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:03.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:03.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:03.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:03.609 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:03.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:03.737 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:03.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:03.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:03.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:03.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:04.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:04.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:04.132 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:04.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:04.261 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:04.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 32 16 50643 50627 98.8586 0 - 0.00968924 2024-09-13T22:50:04.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:04.390 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:04.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:04.514 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T22:50:04.517 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:04.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:04.630 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:50:04.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:04 smithi192 bash[24608]: cluster 2024-09-13T22:50:02.696034+0000 mgr.y (mgr.14613) 461 : cluster [DBG] pgmap v515: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 987 op/s 2024-09-13T22:50:04.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:04 smithi192 bash[24608]: audit 2024-09-13T22:50:03.609614+0000 mon.a (mon.0) 1317 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:04.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:04 smithi192 bash[24608]: audit 2024-09-13T22:50:03.614145+0000 mon.b (mon.1) 192 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:04.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:04 smithi192 bash[24608]: cluster 2024-09-13T22:50:03.938075+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-13T22:50:04.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:04.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:04.777 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:04.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:04.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:04.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[29889]: cluster 2024-09-13T22:50:02.696034+0000 mgr.y (mgr.14613) 461 : cluster [DBG] pgmap v515: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 987 op/s 2024-09-13T22:50:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[29889]: audit 2024-09-13T22:50:03.609614+0000 mon.a (mon.0) 1317 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[29889]: audit 2024-09-13T22:50:03.614145+0000 mon.b (mon.1) 192 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[29889]: cluster 2024-09-13T22:50:03.938075+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-13T22:50:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[22840]: cluster 2024-09-13T22:50:02.696034+0000 mgr.y (mgr.14613) 461 : cluster [DBG] pgmap v515: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 987 op/s 2024-09-13T22:50:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[22840]: audit 2024-09-13T22:50:03.609614+0000 mon.a (mon.0) 1317 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[22840]: audit 2024-09-13T22:50:03.614145+0000 mon.b (mon.1) 192 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:04 smithi178 bash[22840]: cluster 2024-09-13T22:50:03.938075+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-13T22:50:05.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:05.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:05.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:05.164 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:05.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:05.294 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:05.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 33 16 50667 50651 95.9087 0.75 0.00380385 0.010144 2024-09-13T22:50:05.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:05.427 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:05.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:05.570 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:05.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:05.704 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:05.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:05.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:50:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:50:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.840 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:05.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:05.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:05.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:05.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:06.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:06.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:05 smithi192 bash[24608]: cluster 2024-09-13T22:50:04.697017+0000 mgr.y (mgr.14613) 462 : cluster [DBG] pgmap v518: 158 pgs: 39 peering, 2 unknown, 117 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 341 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T22:50:06.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:05 smithi192 bash[24608]: cluster 2024-09-13T22:50:04.944086+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-13T22:50:06.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:05 smithi192 bash[24608]: audit 2024-09-13T22:50:05.419734+0000 mgr.y (mgr.14613) 463 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:06.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:05 smithi192 bash[24608]: cluster 2024-09-13T22:50:05.607198+0000 mon.a (mon.0) 1320 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-09-13T22:50:06.110 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:06.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[29889]: cluster 2024-09-13T22:50:04.697017+0000 mgr.y (mgr.14613) 462 : cluster [DBG] pgmap v518: 158 pgs: 39 peering, 2 unknown, 117 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 341 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[29889]: cluster 2024-09-13T22:50:04.944086+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[29889]: audit 2024-09-13T22:50:05.419734+0000 mgr.y (mgr.14613) 463 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[29889]: cluster 2024-09-13T22:50:05.607198+0000 mon.a (mon.0) 1320 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[22840]: cluster 2024-09-13T22:50:04.697017+0000 mgr.y (mgr.14613) 462 : cluster [DBG] pgmap v518: 158 pgs: 39 peering, 2 unknown, 117 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 341 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[22840]: cluster 2024-09-13T22:50:04.944086+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[22840]: audit 2024-09-13T22:50:05.419734+0000 mgr.y (mgr.14613) 463 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:06.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:05 smithi178 bash[22840]: cluster 2024-09-13T22:50:05.607198+0000 mon.a (mon.0) 1320 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-09-13T22:50:06.238 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:06.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 34 16 51816 51800 95.1999 71.8125 0.00766375 0.0104967 2024-09-13T22:50:06.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:06.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:06.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:06.507 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:06.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:06.643 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:06.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:06.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:06.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:06.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:06.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:07.009 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:07.009 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:07.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:07.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:07.141 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:06 smithi178 bash[22840]: cluster 2024-09-13T22:50:05.950318+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-13T22:50:07.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:06 smithi178 bash[29889]: cluster 2024-09-13T22:50:05.950318+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-13T22:50:07.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:07.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:07.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:07.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:06 smithi192 bash[24608]: cluster 2024-09-13T22:50:05.950318+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-13T22:50:07.303 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:07.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 35 16 52956 52940 94.5155 71.25 0.0159165 0.0105755 2024-09-13T22:50:07.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:07.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:07.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:07.570 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:07.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:07.703 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:07.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:07.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:07.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:07.964 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:50:07.964 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15408.0:53643 9.1 e267/251 9:81807851:::benchmark_data_smithi178_96665_object53642:head v 267'3343, mlcod=267'3340)", 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:07.959202+0000", 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0043902430000000003, 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044217550000000003, 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959202+0000", 2024-09-13T22:50:07.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959202+0000", 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959204+0000", 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7010000000000001e-06 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959272+0000", 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7972999999999994e-05 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959293+0000", 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1120000000000001e-05 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959303+0000", 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2220000000000008e-06 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959493+0000", 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000190223 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:07.959543+0000", 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0677999999999998e-05 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-09-13T22:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-09-13T22:50:07.969 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:50:07.969 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:07.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:07 smithi192 bash[24608]: cluster 2024-09-13T22:50:06.697663+0000 mgr.y (mgr.14613) 464 : cluster [DBG] pgmap v521: 158 pgs: 1 active+recovering+undersized+remapped, 39 peering, 2 remapped+peering, 2 unknown, 114 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 69 MiB/s wr, 1.10k op/s; 3216/152592 objects misplaced (2.108%); 32 KiB/s, 7 objects/s recovering 2024-09-13T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:07 smithi192 bash[24608]: cluster 2024-09-13T22:50:06.956192+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-13T22:50:08.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:08.094 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:08.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:08.215 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:08.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:07 smithi178 bash[29889]: cluster 2024-09-13T22:50:06.697663+0000 mgr.y (mgr.14613) 464 : cluster [DBG] pgmap v521: 158 pgs: 1 active+recovering+undersized+remapped, 39 peering, 2 remapped+peering, 2 unknown, 114 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 69 MiB/s wr, 1.10k op/s; 3216/152592 objects misplaced (2.108%); 32 KiB/s, 7 objects/s recovering 2024-09-13T22:50:08.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:07 smithi178 bash[29889]: cluster 2024-09-13T22:50:06.956192+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-13T22:50:08.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:07 smithi178 bash[22840]: cluster 2024-09-13T22:50:06.697663+0000 mgr.y (mgr.14613) 464 : cluster [DBG] pgmap v521: 158 pgs: 1 active+recovering+undersized+remapped, 39 peering, 2 remapped+peering, 2 unknown, 114 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 69 MiB/s wr, 1.10k op/s; 3216/152592 objects misplaced (2.108%); 32 KiB/s, 7 objects/s recovering 2024-09-13T22:50:08.317 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:07 smithi178 bash[22840]: cluster 2024-09-13T22:50:06.956192+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-13T22:50:08.317 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:08.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 36 15 53920 53905 93.5653 60.3125 0.0275809 0.0106782 2024-09-13T22:50:08.346 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:08.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:08.484 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:08.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.660 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:08.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:08.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:08.790 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:08.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:08.929 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:08.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:09.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:09.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:09.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:09.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:09 smithi178 bash[29889]: cluster 2024-09-13T22:50:08.960366+0000 mon.a (mon.0) 1323 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-09-13T22:50:09.173 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:09 smithi178 bash[22840]: cluster 2024-09-13T22:50:08.960366+0000 mon.a (mon.0) 1323 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-09-13T22:50:09.189 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:09.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:09.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:09 smithi192 bash[24608]: cluster 2024-09-13T22:50:08.960366+0000 mon.a (mon.0) 1323 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-09-13T22:50:09.313 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:09.326 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 37 16 54922 54906 92.7274 62.5625 0.00272187 0.010739 2024-09-13T22:50:09.416 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:09.438 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:09.515 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:50:09.515 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:50:09.516 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-09-13T22:50:09.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:50:09.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:09.565 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:09.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T22:50:09.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:09.709 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:09.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:09.851 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:09.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:50:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:50:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:50:09.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:09.986 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:09.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:10.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:10.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:10 smithi178 bash[29889]: cluster 2024-09-13T22:50:08.698734+0000 mgr.y (mgr.14613) 465 : cluster [DBG] pgmap v523: 158 pgs: 1 active+recovering+undersized+remapped, 23 peering, 2 remapped+peering, 1 unknown, 131 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 27 MiB/s wr, 432 op/s; 3216/152592 objects misplaced (2.108%); 27 KiB/s, 11 objects/s recovering 2024-09-13T22:50:10.098 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:10 smithi178 bash[22840]: cluster 2024-09-13T22:50:08.698734+0000 mgr.y (mgr.14613) 465 : cluster [DBG] pgmap v523: 158 pgs: 1 active+recovering+undersized+remapped, 23 peering, 2 remapped+peering, 1 unknown, 131 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 27 MiB/s wr, 432 op/s; 3216/152592 objects misplaced (2.108%); 27 KiB/s, 11 objects/s recovering 2024-09-13T22:50:10.112 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:10.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:10 smithi192 bash[24608]: cluster 2024-09-13T22:50:08.698734+0000 mgr.y (mgr.14613) 465 : cluster [DBG] pgmap v523: 158 pgs: 1 active+recovering+undersized+remapped, 23 peering, 2 remapped+peering, 1 unknown, 131 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 27 MiB/s wr, 432 op/s; 3216/152592 objects misplaced (2.108%); 27 KiB/s, 11 objects/s recovering 2024-09-13T22:50:10.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:10.246 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:10.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 38 16 55736 55720 91.626 50.875 0.00569457 0.0109076 2024-09-13T22:50:10.350 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:10.378 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:10.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:10.507 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:10.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:10.635 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:10.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:10.764 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:10.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:10.903 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:10.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:11.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:11.041 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:11.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:11.167 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:11.274 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:11.298 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:11.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 39 16 56876 56860 91.1034 71.25 0.00816731 0.0109705 2024-09-13T22:50:11.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:11.439 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:11.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:11.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:11.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:11.693 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:11.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:11.823 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:11.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:11.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:11.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:12.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:12.098 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:12.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:12.232 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:12.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:50:12.329598+0000 min lat: 0.00136126 max lat: 2.39712 avg lat: 0.0110467 2024-09-13T22:50:12.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:50:12.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 40 16 57934 57918 90.4789 66.125 0.00486016 0.0110467 2024-09-13T22:50:12.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:12 smithi178 bash[29889]: cluster 2024-09-13T22:50:10.701607+0000 mgr.y (mgr.14613) 466 : cluster [DBG] pgmap v524: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 8914/166011 objects misplaced (5.370%); 5.5 MiB/s, 105 objects/s recovering 2024-09-13T22:50:12.359 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:12 smithi178 bash[22840]: cluster 2024-09-13T22:50:10.701607+0000 mgr.y (mgr.14613) 466 : cluster [DBG] pgmap v524: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 8914/166011 objects misplaced (5.370%); 5.5 MiB/s, 105 objects/s recovering 2024-09-13T22:50:12.374 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:12.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:12 smithi192 bash[24608]: cluster 2024-09-13T22:50:10.701607+0000 mgr.y (mgr.14613) 466 : cluster [DBG] pgmap v524: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 8914/166011 objects misplaced (5.370%); 5.5 MiB/s, 105 objects/s recovering 2024-09-13T22:50:12.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:12.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:12.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:12.644 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:12.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:12.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:12.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:12.909 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:12.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:13.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:13.046 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:13.061 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:50:13.061 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:50:13.105 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:50:13.105 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:50:13.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:13.155 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:50:13.156 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: cluster 2024-09-13T22:50:11.276487+0000 osd.6 (osd.6) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: cluster 2024-09-13T22:50:11.276555+0000 osd.6 (osd.6) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: cluster 2024-09-13T22:50:12.033369+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: cluster 2024-09-13T22:50:12.108364+0000 osd.2 (osd.2) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: cluster 2024-09-13T22:50:12.108427+0000 osd.2 (osd.2) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: cluster 2024-09-13T22:50:12.943906+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[29889]: audit 2024-09-13T22:50:13.024156+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? v1:172.21.15.178:0/3844458335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: cluster 2024-09-13T22:50:11.276487+0000 osd.6 (osd.6) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:13.165 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: cluster 2024-09-13T22:50:11.276555+0000 osd.6 (osd.6) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:13.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: cluster 2024-09-13T22:50:12.033369+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-13T22:50:13.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: cluster 2024-09-13T22:50:12.108364+0000 osd.2 (osd.2) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:13.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: cluster 2024-09-13T22:50:12.108427+0000 osd.2 (osd.2) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:13.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: cluster 2024-09-13T22:50:12.943906+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-13T22:50:13.166 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:13 smithi178 bash[22840]: audit 2024-09-13T22:50:13.024156+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? v1:172.21.15.178:0/3844458335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:50:13.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:13.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:50:13.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:50:13.222 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:50:13.222 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:50:13.257 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:50:13.257 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:50:13.283 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:50:13.283 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:50:13.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.299 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: cluster 2024-09-13T22:50:11.276487+0000 osd.6 (osd.6) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:13.299 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: cluster 2024-09-13T22:50:11.276555+0000 osd.6 (osd.6) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:13.300 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: cluster 2024-09-13T22:50:12.033369+0000 mon.a (mon.0) 1324 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-13T22:50:13.300 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: cluster 2024-09-13T22:50:12.108364+0000 osd.2 (osd.2) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:13.300 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: cluster 2024-09-13T22:50:12.108427+0000 osd.2 (osd.2) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:13.300 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: cluster 2024-09-13T22:50:12.943906+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-13T22:50:13.300 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:13 smithi192 bash[24608]: audit 2024-09-13T22:50:13.024156+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? v1:172.21.15.178:0/3844458335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:50:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.320 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:13.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 41 16 58900 58884 89.7446 60.375 0.0268478 0.0111365 2024-09-13T22:50:13.329 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:50:13.329 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:50:13.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:13.445 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:13.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.584 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:13.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:13.718 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:13.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.844 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:13.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:13.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:14.049 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:50:14.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:14.107 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:14 smithi192 bash[24608]: cluster 2024-09-13T22:50:12.702041+0000 mgr.y (mgr.14613) 467 : cluster [DBG] pgmap v526: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 758 B/s rd, 41 MiB/s wr, 663 op/s; 8914/166011 objects misplaced (5.370%); 4.9 MiB/s, 89 objects/s recovering 2024-09-13T22:50:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:14 smithi192 bash[24608]: audit 2024-09-13T22:50:13.373684+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.178:0/2233092912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:50:14.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:14 smithi192 bash[24608]: audit 2024-09-13T22:50:13.374708+0000 mon.a (mon.0) 1326 : audit [INF] from='client.25343 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:50:14.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:14.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:14.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 42 15 60005 59990 89.2535 69.125 0.0167598 0.0111992 2024-09-13T22:50:14.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:14.376 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:14.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:14 smithi178 bash[22840]: cluster 2024-09-13T22:50:12.702041+0000 mgr.y (mgr.14613) 467 : cluster [DBG] pgmap v526: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 758 B/s rd, 41 MiB/s wr, 663 op/s; 8914/166011 objects misplaced (5.370%); 4.9 MiB/s, 89 objects/s recovering 2024-09-13T22:50:14.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:14 smithi178 bash[22840]: audit 2024-09-13T22:50:13.373684+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.178:0/2233092912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:50:14.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:14 smithi178 bash[22840]: audit 2024-09-13T22:50:13.374708+0000 mon.a (mon.0) 1326 : audit [INF] from='client.25343 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:50:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:14 smithi178 bash[29889]: cluster 2024-09-13T22:50:12.702041+0000 mgr.y (mgr.14613) 467 : cluster [DBG] pgmap v526: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 758 B/s rd, 41 MiB/s wr, 663 op/s; 8914/166011 objects misplaced (5.370%); 4.9 MiB/s, 89 objects/s recovering 2024-09-13T22:50:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:14 smithi178 bash[29889]: audit 2024-09-13T22:50:13.373684+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.178:0/2233092912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:50:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:14 smithi178 bash[29889]: audit 2024-09-13T22:50:13.374708+0000 mon.a (mon.0) 1326 : audit [INF] from='client.25343 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:50:14.472 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:50:14.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:14.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:14.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:14.637 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:14.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:14.763 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:14.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:14.891 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:14.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:14.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:15.015 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:15.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.149 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:15.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:15 smithi178 bash[22840]: cluster 2024-09-13T22:50:14.046089+0000 mon.a (mon.0) 1327 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:50:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:15 smithi178 bash[22840]: audit 2024-09-13T22:50:14.047080+0000 mon.a (mon.0) 1328 : audit [INF] from='client.25343 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:50:15.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:15 smithi178 bash[22840]: cluster 2024-09-13T22:50:14.051235+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-13T22:50:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:15 smithi178 bash[29889]: cluster 2024-09-13T22:50:14.046089+0000 mon.a (mon.0) 1327 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:50:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:15 smithi178 bash[29889]: audit 2024-09-13T22:50:14.047080+0000 mon.a (mon.0) 1328 : audit [INF] from='client.25343 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:50:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:15 smithi178 bash[29889]: cluster 2024-09-13T22:50:14.051235+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-13T22:50:15.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:15.278 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:15.327 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 43 16 60917 60901 88.5019 56.9375 0.011337 0.0112933 2024-09-13T22:50:15.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:15.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:15 smithi192 bash[24608]: cluster 2024-09-13T22:50:14.046089+0000 mon.a (mon.0) 1327 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:50:15.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:15 smithi192 bash[24608]: audit 2024-09-13T22:50:14.047080+0000 mon.a (mon.0) 1328 : audit [INF] from='client.25343 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:50:15.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:15 smithi192 bash[24608]: cluster 2024-09-13T22:50:14.051235+0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-13T22:50:15.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:15.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:15.544 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:15.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:15.678 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:50:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:50:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:15.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:15.814 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:15.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:15.939 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:15.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:16.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:16.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:16.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:16.199 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:16.301 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:16.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:16 smithi178 bash[22840]: cluster 2024-09-13T22:50:14.702568+0000 mgr.y (mgr.14613) 468 : cluster [DBG] pgmap v529: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 8603/172626 objects misplaced (4.984%); 8.7 MiB/s, 147 objects/s recovering 2024-09-13T22:50:16.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:16 smithi178 bash[22840]: audit 2024-09-13T22:50:15.427814+0000 mgr.y (mgr.14613) 469 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:16.316 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:16 smithi178 bash[29889]: cluster 2024-09-13T22:50:14.702568+0000 mgr.y (mgr.14613) 468 : cluster [DBG] pgmap v529: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 8603/172626 objects misplaced (4.984%); 8.7 MiB/s, 147 objects/s recovering 2024-09-13T22:50:16.317 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:16 smithi178 bash[29889]: audit 2024-09-13T22:50:15.427814+0000 mgr.y (mgr.14613) 469 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:16.328 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 44 16 61918 61902 87.9108 62.5625 0.0119089 0.0113689 2024-09-13T22:50:16.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:16.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:16 smithi192 bash[24608]: cluster 2024-09-13T22:50:14.702568+0000 mgr.y (mgr.14613) 468 : cluster [DBG] pgmap v529: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 8603/172626 objects misplaced (4.984%); 8.7 MiB/s, 147 objects/s recovering 2024-09-13T22:50:16.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:16 smithi192 bash[24608]: audit 2024-09-13T22:50:15.427814+0000 mgr.y (mgr.14613) 469 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:16.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:16.463 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:16.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:16.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:16.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:16.715 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:16.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:16.845 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:16.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:16.980 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:16.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:17.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:17.103 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:17.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:17.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.225 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:17.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:17.328 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 45 16 62741 62725 87.1003 51.4375 0.00685983 0.0114753 2024-09-13T22:50:17.353 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:17.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:17.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.620 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:17.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:17.756 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:17.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.886 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:17.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:17.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:18.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:18.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:18.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:18 smithi192 bash[24608]: cluster 2024-09-13T22:50:16.281649+0000 osd.6 (osd.6) 199 : cluster [DBG] 3.a scrub starts 2024-09-13T22:50:18.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:18 smithi192 bash[24608]: cluster 2024-09-13T22:50:16.286379+0000 osd.6 (osd.6) 200 : cluster [DBG] 3.a scrub ok 2024-09-13T22:50:18.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:18 smithi192 bash[24608]: cluster 2024-09-13T22:50:16.595114+0000 osd.2 (osd.2) 199 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:50:18.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:18 smithi192 bash[24608]: cluster 2024-09-13T22:50:16.597641+0000 osd.2 (osd.2) 200 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:50:18.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:18 smithi192 bash[24608]: cluster 2024-09-13T22:50:16.703683+0000 mgr.y (mgr.14613) 470 : cluster [DBG] pgmap v530: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 854 op/s; 8269/181377 objects misplaced (4.559%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T22:50:18.140 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:18 smithi192 bash[24608]: cluster 2024-09-13T22:50:17.940531+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-13T22:50:18.157 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:18.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:18.274 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[29889]: cluster 2024-09-13T22:50:16.281649+0000 osd.6 (osd.6) 199 : cluster [DBG] 3.a scrub starts 2024-09-13T22:50:18.274 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[29889]: cluster 2024-09-13T22:50:16.286379+0000 osd.6 (osd.6) 200 : cluster [DBG] 3.a scrub ok 2024-09-13T22:50:18.274 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[29889]: cluster 2024-09-13T22:50:16.595114+0000 osd.2 (osd.2) 199 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:50:18.274 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[29889]: cluster 2024-09-13T22:50:16.597641+0000 osd.2 (osd.2) 200 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:50:18.274 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[29889]: cluster 2024-09-13T22:50:16.703683+0000 mgr.y (mgr.14613) 470 : cluster [DBG] pgmap v530: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 854 op/s; 8269/181377 objects misplaced (4.559%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T22:50:18.274 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[29889]: cluster 2024-09-13T22:50:17.940531+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-13T22:50:18.275 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[22840]: cluster 2024-09-13T22:50:16.281649+0000 osd.6 (osd.6) 199 : cluster [DBG] 3.a scrub starts 2024-09-13T22:50:18.275 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[22840]: cluster 2024-09-13T22:50:16.286379+0000 osd.6 (osd.6) 200 : cluster [DBG] 3.a scrub ok 2024-09-13T22:50:18.275 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[22840]: cluster 2024-09-13T22:50:16.595114+0000 osd.2 (osd.2) 199 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:50:18.275 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[22840]: cluster 2024-09-13T22:50:16.597641+0000 osd.2 (osd.2) 200 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:50:18.275 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[22840]: cluster 2024-09-13T22:50:16.703683+0000 mgr.y (mgr.14613) 470 : cluster [DBG] pgmap v530: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 854 op/s; 8269/181377 objects misplaced (4.559%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T22:50:18.275 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:18 smithi178 bash[22840]: cluster 2024-09-13T22:50:17.940531+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-13T22:50:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.305 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:18.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:18.328 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 46 16 63795 63779 86.6388 65.875 0.00753886 0.011535 2024-09-13T22:50:18.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:18.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:18.467 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:18.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:18.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:18.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:18.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:18.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:18.818 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:18.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:50:18.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:18.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:18.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:19.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:19.101 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:19.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:19.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:19.328 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 47 16 64669 64653 85.9576 54.625 0.0133796 0.011628 2024-09-13T22:50:19.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:19.354 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:19 smithi192 bash[24608]: cluster 2024-09-13T22:50:17.887612+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:19.355 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:19 smithi192 bash[24608]: cluster 2024-09-13T22:50:17.887679+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:19.355 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:19 smithi192 bash[24608]: audit 2024-09-13T22:50:18.640847+0000 mon.b (mon.1) 194 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:19.372 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:19.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:19.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:19.548 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:50:19.549 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:50:19.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:19.632 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:19.653 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[29889]: cluster 2024-09-13T22:50:17.887612+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:19.654 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[29889]: cluster 2024-09-13T22:50:17.887679+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:19.654 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[29889]: audit 2024-09-13T22:50:18.640847+0000 mon.b (mon.1) 194 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:19.654 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[22840]: cluster 2024-09-13T22:50:17.887612+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:19.654 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[22840]: cluster 2024-09-13T22:50:17.887679+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:19.654 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[22840]: audit 2024-09-13T22:50:18.640847+0000 mon.b (mon.1) 194 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:19.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:19.761 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:19.866 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:19.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:19.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:50:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:50:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:50:19.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:20.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:50:20.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:20.155 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:20.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:20.279 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:20.329 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 48 15 65669 65654 85.4691 62.5625 0.013992 0.0116959 2024-09-13T22:50:20.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:20.411 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:18.244008+0000 osd.6 (osd.6) 201 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:18.246742+0000 osd.6 (osd.6) 202 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:18.655135+0000 osd.2 (osd.2) 201 : cluster [DBG] 2.c scrub starts 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:18.656927+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.c scrub ok 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:18.704501+0000 mgr.y (mgr.14613) 471 : cluster [DBG] pgmap v532: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 854 op/s; 8269/181377 objects misplaced (4.559%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:19.092775+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[29889]: cluster 2024-09-13T22:50:19.094494+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:18.244008+0000 osd.6 (osd.6) 201 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:18.246742+0000 osd.6 (osd.6) 202 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:50:20.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:18.655135+0000 osd.2 (osd.2) 201 : cluster [DBG] 2.c scrub starts 2024-09-13T22:50:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:18.656927+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.c scrub ok 2024-09-13T22:50:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:18.704501+0000 mgr.y (mgr.14613) 471 : cluster [DBG] pgmap v532: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 854 op/s; 8269/181377 objects misplaced (4.559%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T22:50:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:19.092775+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:50:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:20 smithi178 bash[22840]: cluster 2024-09-13T22:50:19.094494+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:50:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:18.244008+0000 osd.6 (osd.6) 201 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:50:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:18.246742+0000 osd.6 (osd.6) 202 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:50:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:18.655135+0000 osd.2 (osd.2) 201 : cluster [DBG] 2.c scrub starts 2024-09-13T22:50:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:18.656927+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.c scrub ok 2024-09-13T22:50:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:18.704501+0000 mgr.y (mgr.14613) 471 : cluster [DBG] pgmap v532: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 854 op/s; 8269/181377 objects misplaced (4.559%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T22:50:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:19.092775+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:50:20.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:20 smithi192 bash[24608]: cluster 2024-09-13T22:50:19.094494+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:50:20.514 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:20.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:20.643 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:20.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:20.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:20.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:20.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:20.950 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:20.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:21.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T22:50:21.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:21.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:21.223 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:21.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:21.329 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 49 15 66710 66695 85.0526 65.0625 0.00781056 0.0117522 2024-09-13T22:50:21.352 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:21.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:21.485 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:21.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:21.618 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:21.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:21.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:21.854 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:21.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:21.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:22.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:22.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:22.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:22.152 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:22.179 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:22 smithi178 bash[22840]: cluster 2024-09-13T22:50:20.261018+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:50:22.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:22 smithi178 bash[22840]: cluster 2024-09-13T22:50:20.262199+0000 osd.6 (osd.6) 204 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:50:22.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:22 smithi178 bash[22840]: cluster 2024-09-13T22:50:20.711647+0000 mgr.y (mgr.14613) 472 : cluster [DBG] pgmap v533: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7607/195591 objects misplaced (3.889%); 11 MiB/s, 168 objects/s recovering 2024-09-13T22:50:22.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:22 smithi178 bash[29889]: cluster 2024-09-13T22:50:20.261018+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:50:22.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:22 smithi178 bash[29889]: cluster 2024-09-13T22:50:20.262199+0000 osd.6 (osd.6) 204 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:50:22.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:22 smithi178 bash[29889]: cluster 2024-09-13T22:50:20.711647+0000 mgr.y (mgr.14613) 472 : cluster [DBG] pgmap v533: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7607/195591 objects misplaced (3.889%); 11 MiB/s, 168 objects/s recovering 2024-09-13T22:50:22.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:22.286 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:50:22.287 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:22.291 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:50:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:22.329 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 50 16 67700 67684 84.5877 61.8125 0.0108571 0.0118156 2024-09-13T22:50:22.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:22 smithi192 bash[24608]: cluster 2024-09-13T22:50:20.261018+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:50:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:22 smithi192 bash[24608]: cluster 2024-09-13T22:50:20.262199+0000 osd.6 (osd.6) 204 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:50:22.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:22 smithi192 bash[24608]: cluster 2024-09-13T22:50:20.711647+0000 mgr.y (mgr.14613) 472 : cluster [DBG] pgmap v533: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7607/195591 objects misplaced (3.889%); 11 MiB/s, 168 objects/s recovering 2024-09-13T22:50:22.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:22.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:22.552 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:22.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:22.683 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:22.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:22.814 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:22.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:22.945 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:22.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:23.055 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:23.087 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:23.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:23.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:23.319 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[29889]: cluster 2024-09-13T22:50:21.277069+0000 osd.6 (osd.6) 205 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[29889]: cluster 2024-09-13T22:50:21.280823+0000 osd.6 (osd.6) 206 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[29889]: cluster 2024-09-13T22:50:21.660463+0000 osd.2 (osd.2) 203 : cluster [DBG] 5.b scrub starts 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[29889]: cluster 2024-09-13T22:50:21.662604+0000 osd.2 (osd.2) 204 : cluster [DBG] 5.b scrub ok 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[29889]: audit 2024-09-13T22:50:22.263401+0000 mgr.y (mgr.14613) 473 : audit [DBG] from='client.25349 v1:172.21.15.178:0/702382568' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[22840]: cluster 2024-09-13T22:50:21.277069+0000 osd.6 (osd.6) 205 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[22840]: cluster 2024-09-13T22:50:21.280823+0000 osd.6 (osd.6) 206 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[22840]: cluster 2024-09-13T22:50:21.660463+0000 osd.2 (osd.2) 203 : cluster [DBG] 5.b scrub starts 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[22840]: cluster 2024-09-13T22:50:21.662604+0000 osd.2 (osd.2) 204 : cluster [DBG] 5.b scrub ok 2024-09-13T22:50:23.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:23 smithi178 bash[22840]: audit 2024-09-13T22:50:22.263401+0000 mgr.y (mgr.14613) 473 : audit [DBG] from='client.25349 v1:172.21.15.178:0/702382568' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:50:23.329 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 51 16 68739 68723 84.2023 64.9375 0.0186274 0.0118664 2024-09-13T22:50:23.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:23.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:23 smithi192 bash[24608]: cluster 2024-09-13T22:50:21.277069+0000 osd.6 (osd.6) 205 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:50:23.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:23 smithi192 bash[24608]: cluster 2024-09-13T22:50:21.280823+0000 osd.6 (osd.6) 206 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:50:23.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:23 smithi192 bash[24608]: cluster 2024-09-13T22:50:21.660463+0000 osd.2 (osd.2) 203 : cluster [DBG] 5.b scrub starts 2024-09-13T22:50:23.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:23 smithi192 bash[24608]: cluster 2024-09-13T22:50:21.662604+0000 osd.2 (osd.2) 204 : cluster [DBG] 5.b scrub ok 2024-09-13T22:50:23.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:23 smithi192 bash[24608]: audit 2024-09-13T22:50:22.263401+0000 mgr.y (mgr.14613) 473 : audit [DBG] from='client.25349 v1:172.21.15.178:0/702382568' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:50:23.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:23.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:23.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:23.589 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:23.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:23.722 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:23.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:23.869 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:23.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:23.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:24.000 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:24.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:24.143 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:24.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:24.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:24 smithi192 bash[24608]: cluster 2024-09-13T22:50:22.712074+0000 mgr.y (mgr.14613) 474 : cluster [DBG] pgmap v534: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 885 op/s; 7607/195591 objects misplaced (3.889%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T22:50:24.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:24.329 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 52 16 69489 69473 83.4843 46.875 0.018804 0.0119723 2024-09-13T22:50:24.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:24.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:24 smithi178 bash[29889]: cluster 2024-09-13T22:50:22.712074+0000 mgr.y (mgr.14613) 474 : cluster [DBG] pgmap v534: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 885 op/s; 7607/195591 objects misplaced (3.889%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T22:50:24.374 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:24 smithi178 bash[22840]: cluster 2024-09-13T22:50:22.712074+0000 mgr.y (mgr.14613) 474 : cluster [DBG] pgmap v534: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 885 op/s; 7607/195591 objects misplaced (3.889%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T22:50:24.375 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-09-13T22:50:20.710920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4260305435,"num_objects":65197,"num_object_clones":0,"num_object_copies":195591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7607,"num_objects_unfound":0,"num_objects_dirty":65197,"num_whiteouts":0,"num_read":1897,"num_read_kb":1649,"num_write":65720,"num_write_kb":4159917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2043,"num_bytes_recovered":120393906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67317,"ondisk_log_size":67317,"up":474,"acting":471,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14922080,"kb_used_data":12467984,"kb_used_omap":12,"kb_used_meta":2453939,"kb_avail":735039136,"statfs":{"total":767960285184,"available":752680075264,"internally_reserved":0,"allocated":12767215616,"data_stored":12763235413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2512834096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":646184960,"num_objects":9860,"num_object_clones":0,"num_object_copies":29580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1307,"num_objects_unfound":0,"num_objects_dirty":9860,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9860,"num_write_kb":631040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":85655552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.772918"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958266+0000","last_change":"2024-09-13T22:49:37.429809+0000","last_active":"2024-09-13T22:50:17.958266+0000","last_peered":"2024-09-13T22:50:17.958266+0000","last_clean":"2024-09-13T22:50:17.958266+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:50:17.958266+0000","last_undegraded":"2024-09-13T22:50:17.958266+0000","last_fullsized":"2024-09-13T22:50:17.958266+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:37.429474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:49:37.429474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:52.228026+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":474,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.944281+0000","last_change":"2024-09-13T22:50:03.973780+0000","last_active":"2024-09-13T22:50:17.944281+0000","last_peered":"2024-09-13T22:50:17.944281+0000","last_clean":"2024-09-13T22:50:17.944281+0000","last_became_active":"2024-09-13T22:50:03.973150+0000","last_became_peered":"2024-09-13T22:50:03.973150+0000","last_unstale":"2024-09-13T22:50:17.944281+0000","last_undegraded":"2024-09-13T22:50:17.944281+0000","last_fullsized":"2024-09-13T22:50:17.944281+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:21.195029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.195029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.497830+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"228'25","reported_seq":479,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958683+0000","last_change":"2024-09-13T22:47:03.372631+0000","last_active":"2024-09-13T22:50:17.958683+0000","last_peered":"2024-09-13T22:50:17.958683+0000","last_clean":"2024-09-13T22:50:17.958683+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:50:17.958683+0000","last_undegraded":"2024-09-13T22:50:17.958683+0000","last_fullsized":"2024-09-13T22:50:17.958683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:47:03.372362+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:47:03.372362+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.968687+0000","last_change":"2024-09-13T22:50:03.983646+0000","last_active":"2024-09-13T22:50:17.968687+0000","last_peered":"2024-09-13T22:50:17.968687+0000","last_clean":"2024-09-13T22:50:17.968687+0000","last_became_active":"2024-09-13T22:50:03.982074+0000","last_became_peered":"2024-09-13T22:50:03.982074+0000","last_unstale":"2024-09-13T22:50:17.968687+0000","last_undegraded":"2024-09-13T22:50:17.968687+0000","last_fullsized":"2024-09-13T22:50:17.968687+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.210087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.210087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":448,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958732+0000","last_change":"2024-09-13T22:49:16.307170+0000","last_active":"2024-09-13T22:50:17.958732+0000","last_peered":"2024-09-13T22:50:17.958732+0000","last_clean":"2024-09-13T22:50:17.958732+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:50:17.958732+0000","last_undegraded":"2024-09-13T22:50:17.958732+0000","last_fullsized":"2024-09-13T22:50:17.958732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.306966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.306966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:22.675921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"228'24","reported_seq":488,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943997+0000","last_change":"2024-09-13T22:49:22.149867+0000","last_active":"2024-09-13T22:50:17.943997+0000","last_peered":"2024-09-13T22:50:17.943997+0000","last_clean":"2024-09-13T22:50:17.943997+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:50:17.943997+0000","last_undegraded":"2024-09-13T22:50:17.943997+0000","last_fullsized":"2024-09-13T22:50:17.943997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:49:22.149762+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.149762+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:40.311825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.956081+0000","last_change":"2024-09-13T22:48:30.135471+0000","last_active":"2024-09-13T22:50:17.956081+0000","last_peered":"2024-09-13T22:50:17.956081+0000","last_clean":"2024-09-13T22:50:17.956081+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:50:17.956081+0000","last_undegraded":"2024-09-13T22:50:17.956081+0000","last_fullsized":"2024-09-13T22:50:17.956081+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.135258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.135258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969268+0000","last_change":"2024-09-13T22:50:04.001279+0000","last_active":"2024-09-13T22:50:17.969268+0000","last_peered":"2024-09-13T22:50:17.969268+0000","last_clean":"2024-09-13T22:50:17.969268+0000","last_became_active":"2024-09-13T22:50:04.000951+0000","last_became_peered":"2024-09-13T22:50:04.000951+0000","last_unstale":"2024-09-13T22:50:17.969268+0000","last_undegraded":"2024-09-13T22:50:17.969268+0000","last_fullsized":"2024-09-13T22:50:17.969268+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.803924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.803924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.959087+0000","last_change":"2024-09-13T22:48:11.350608+0000","last_active":"2024-09-13T22:50:17.959087+0000","last_peered":"2024-09-13T22:50:17.959087+0000","last_clean":"2024-09-13T22:50:17.959087+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:50:17.959087+0000","last_undegraded":"2024-09-13T22:50:17.959087+0000","last_fullsized":"2024-09-13T22:50:17.959087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.350541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.350541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"228'34","reported_seq":544,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950209+0000","last_change":"2024-09-13T22:50:04.023774+0000","last_active":"2024-09-13T22:50:17.950209+0000","last_peered":"2024-09-13T22:50:17.950209+0000","last_clean":"2024-09-13T22:50:17.950209+0000","last_became_active":"2024-09-13T22:50:03.990784+0000","last_became_peered":"2024-09-13T22:50:03.990784+0000","last_unstale":"2024-09-13T22:50:17.950209+0000","last_undegraded":"2024-09-13T22:50:17.950209+0000","last_fullsized":"2024-09-13T22:50:17.950209+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:22.907495+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.907495+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":442,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958489+0000","last_change":"2024-09-13T22:50:03.995918+0000","last_active":"2024-09-13T22:50:17.958489+0000","last_peered":"2024-09-13T22:50:17.958489+0000","last_clean":"2024-09-13T22:50:17.958489+0000","last_became_active":"2024-09-13T22:50:03.995043+0000","last_became_peered":"2024-09-13T22:50:03.995043+0000","last_unstale":"2024-09-13T22:50:17.958489+0000","last_undegraded":"2024-09-13T22:50:17.958489+0000","last_fullsized":"2024-09-13T22:50:17.958489+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:10.365482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:48:10.365482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":434,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.948275+0000","last_change":"2024-09-13T22:48:03.951130+0000","last_active":"2024-09-13T22:50:17.948275+0000","last_peered":"2024-09-13T22:50:17.948275+0000","last_clean":"2024-09-13T22:50:17.948275+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:50:17.948275+0000","last_undegraded":"2024-09-13T22:50:17.948275+0000","last_fullsized":"2024-09-13T22:50:17.948275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:03.950989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:48:03.950989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":454,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957601+0000","last_change":"2024-09-13T22:49:39.881881+0000","last_active":"2024-09-13T22:50:17.957601+0000","last_peered":"2024-09-13T22:50:17.957601+0000","last_clean":"2024-09-13T22:50:17.957601+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:50:17.957601+0000","last_undegraded":"2024-09-13T22:50:17.957601+0000","last_fullsized":"2024-09-13T22:50:17.957601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:39.881753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:39.881753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:07.204865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"228'18","reported_seq":518,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950782+0000","last_change":"2024-09-13T22:50:04.009597+0000","last_active":"2024-09-13T22:50:17.950782+0000","last_peered":"2024-09-13T22:50:17.950782+0000","last_clean":"2024-09-13T22:50:17.950782+0000","last_became_active":"2024-09-13T22:50:03.977915+0000","last_became_peered":"2024-09-13T22:50:03.977915+0000","last_unstale":"2024-09-13T22:50:17.950782+0000","last_undegraded":"2024-09-13T22:50:17.950782+0000","last_fullsized":"2024-09-13T22:50:17.950782+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:05.917128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:05.917128+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950082+0000","last_change":"2024-09-13T22:49:42.177553+0000","last_active":"2024-09-13T22:50:17.950082+0000","last_peered":"2024-09-13T22:50:17.950082+0000","last_clean":"2024-09-13T22:50:17.950082+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:50:17.950082+0000","last_undegraded":"2024-09-13T22:50:17.950082+0000","last_fullsized":"2024-09-13T22:50:17.950082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T22:49:42.177430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:58.765423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943458+0000","last_change":"2024-09-13T22:49:34.085378+0000","last_active":"2024-09-13T22:50:17.943458+0000","last_peered":"2024-09-13T22:50:17.943458+0000","last_clean":"2024-09-13T22:50:17.943458+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:50:17.943458+0000","last_undegraded":"2024-09-13T22:50:17.943458+0000","last_fullsized":"2024-09-13T22:50:17.943458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:34.085275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:34.085275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:51.064682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"271'4136","reported_seq":4186,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:19.739405+0000","last_change":"2024-09-13T22:49:19.908624+0000","last_active":"2024-09-13T22:50:19.739405+0000","last_peered":"2024-09-13T22:50:19.739405+0000","last_clean":"2024-09-13T22:50:19.739405+0000","last_became_active":"2024-09-13T22:49:19.907655+0000","last_became_peered":"2024-09-13T22:49:19.907655+0000","last_unstale":"2024-09-13T22:50:19.739405+0000","last_undegraded":"2024-09-13T22:50:19.739405+0000","last_fullsized":"2024-09-13T22:50:19.739405+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4136,"log_dups_size":0,"ondisk_log_size":4136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:45.994437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271056896,"num_objects":4136,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4136,"num_write_kb":264704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.947965+0000","last_change":"2024-09-13T22:48:36.940636+0000","last_active":"2024-09-13T22:50:17.947965+0000","last_peered":"2024-09-13T22:50:17.947965+0000","last_clean":"2024-09-13T22:50:17.947965+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:50:17.947965+0000","last_undegraded":"2024-09-13T22:50:17.947965+0000","last_fullsized":"2024-09-13T22:50:17.947965+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:36.940280+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:36.940280+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"228'44","reported_seq":582,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945747+0000","last_change":"2024-09-13T22:50:16.286704+0000","last_active":"2024-09-13T22:50:17.945747+0000","last_peered":"2024-09-13T22:50:17.945747+0000","last_clean":"2024-09-13T22:50:17.945747+0000","last_became_active":"2024-09-13T22:50:03.985629+0000","last_became_peered":"2024-09-13T22:50:03.985629+0000","last_unstale":"2024-09-13T22:50:17.945747+0000","last_undegraded":"2024-09-13T22:50:17.945747+0000","last_fullsized":"2024-09-13T22:50:17.945747+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:50:16.286389+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:50:16.286389+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:44.161280+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":458,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958497+0000","last_change":"2024-09-13T22:50:04.002780+0000","last_active":"2024-09-13T22:50:17.958497+0000","last_peered":"2024-09-13T22:50:17.958497+0000","last_clean":"2024-09-13T22:50:17.958497+0000","last_became_active":"2024-09-13T22:50:04.002375+0000","last_became_peered":"2024-09-13T22:50:04.002375+0000","last_unstale":"2024-09-13T22:50:17.958497+0000","last_undegraded":"2024-09-13T22:50:17.958497+0000","last_fullsized":"2024-09-13T22:50:17.958497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:26.959784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.959784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:54.645210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":459,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.964332+0000","last_change":"2024-09-13T22:50:03.978683+0000","last_active":"2024-09-13T22:50:17.964332+0000","last_peered":"2024-09-13T22:50:17.964332+0000","last_clean":"2024-09-13T22:50:17.964332+0000","last_became_active":"2024-09-13T22:50:03.976801+0000","last_became_peered":"2024-09-13T22:50:03.976801+0000","last_unstale":"2024-09-13T22:50:17.964332+0000","last_undegraded":"2024-09-13T22:50:17.964332+0000","last_fullsized":"2024-09-13T22:50:17.964332+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:26.457236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.457236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:49.435363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":363,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.960540+0000","last_change":"2024-09-13T22:50:03.998970+0000","last_active":"2024-09-13T22:50:17.960540+0000","last_peered":"2024-09-13T22:50:17.960540+0000","last_clean":"2024-09-13T22:50:17.960540+0000","last_became_active":"2024-09-13T22:50:03.998504+0000","last_became_peered":"2024-09-13T22:50:03.998504+0000","last_unstale":"2024-09-13T22:50:17.960540+0000","last_undegraded":"2024-09-13T22:50:17.960540+0000","last_fullsized":"2024-09-13T22:50:17.960540+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:20.263925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.263925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:20.903797+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"271'4157","reported_seq":4207,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:20.443953+0000","last_change":"2024-09-13T22:49:19.909371+0000","last_active":"2024-09-13T22:50:20.443953+0000","last_peered":"2024-09-13T22:50:20.443953+0000","last_clean":"2024-09-13T22:50:20.443953+0000","last_became_active":"2024-09-13T22:49:19.908686+0000","last_became_peered":"2024-09-13T22:49:19.908686+0000","last_unstale":"2024-09-13T22:50:20.443953+0000","last_undegraded":"2024-09-13T22:50:20.443953+0000","last_fullsized":"2024-09-13T22:50:20.443953+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:31.803424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":266048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":725,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.961292+0000","last_change":"2024-09-13T22:49:20.975665+0000","last_active":"2024-09-13T22:50:17.961292+0000","last_peered":"2024-09-13T22:50:17.961292+0000","last_clean":"2024-09-13T22:50:17.961292+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:50:17.961292+0000","last_undegraded":"2024-09-13T22:50:17.961292+0000","last_fullsized":"2024-09-13T22:50:17.961292+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:20.975463+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.975463+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:30.138844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"228'19","reported_seq":504,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958859+0000","last_change":"2024-09-13T22:50:04.032174+0000","last_active":"2024-09-13T22:50:17.958859+0000","last_peered":"2024-09-13T22:50:17.958859+0000","last_clean":"2024-09-13T22:50:17.958859+0000","last_became_active":"2024-09-13T22:50:03.999716+0000","last_became_peered":"2024-09-13T22:50:03.999716+0000","last_unstale":"2024-09-13T22:50:17.958859+0000","last_undegraded":"2024-09-13T22:50:17.958859+0000","last_fullsized":"2024-09-13T22:50:17.958859+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:13.725631+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.725631+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963964+0000","last_change":"2024-09-13T22:49:25.446162+0000","last_active":"2024-09-13T22:50:17.963964+0000","last_peered":"2024-09-13T22:50:17.963964+0000","last_clean":"2024-09-13T22:50:17.963964+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:50:17.963964+0000","last_undegraded":"2024-09-13T22:50:17.963964+0000","last_fullsized":"2024-09-13T22:50:17.963964+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:25.445965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:25.445965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:50.875335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":72,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958572+0000","last_change":"2024-09-13T22:49:45.642573+0000","last_active":"2024-09-13T22:50:17.958572+0000","last_peered":"2024-09-13T22:50:17.958572+0000","last_clean":"2024-09-13T22:50:17.958572+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T22:50:17.958572+0000","last_undegraded":"2024-09-13T22:50:17.958572+0000","last_fullsized":"2024-09-13T22:50:17.958572+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:28.189507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.189507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969720+0000","last_change":"2024-09-13T22:50:04.000374+0000","last_active":"2024-09-13T22:50:17.969720+0000","last_peered":"2024-09-13T22:50:17.969720+0000","last_clean":"2024-09-13T22:50:17.969720+0000","last_became_active":"2024-09-13T22:50:04.000025+0000","last_became_peered":"2024-09-13T22:50:04.000025+0000","last_unstale":"2024-09-13T22:50:17.969720+0000","last_undegraded":"2024-09-13T22:50:17.969720+0000","last_fullsized":"2024-09-13T22:50:17.969720+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.090972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.090972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"271'3997","reported_seq":4047,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:20.429060+0000","last_change":"2024-09-13T22:49:19.826930+0000","last_active":"2024-09-13T22:50:20.429060+0000","last_peered":"2024-09-13T22:50:20.429060+0000","last_clean":"2024-09-13T22:50:20.429060+0000","last_became_active":"2024-09-13T22:49:19.826580+0000","last_became_peered":"2024-09-13T22:49:19.826580+0000","last_unstale":"2024-09-13T22:50:20.429060+0000","last_undegraded":"2024-09-13T22:50:20.429060+0000","last_fullsized":"2024-09-13T22:50:20.429060+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:37.064153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261947392,"num_objects":3997,"num_object_clones":0,"num_object_copies":11991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3997,"num_write_kb":255808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.948440+0000","last_change":"2024-09-13T22:48:24.860899+0000","last_active":"2024-09-13T22:50:17.948440+0000","last_peered":"2024-09-13T22:50:17.948440+0000","last_clean":"2024-09-13T22:50:17.948440+0000","last_became_active":"2024-09-13T22:45:58.880701+0000","last_became_peered":"2024-09-13T22:45:58.880701+0000","last_unstale":"2024-09-13T22:50:17.948440+0000","last_undegraded":"2024-09-13T22:50:17.948440+0000","last_fullsized":"2024-09-13T22:50:17.948440+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:24.860776+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.860776+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"228'27","reported_seq":486,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.960863+0000","last_change":"2024-09-13T22:48:23.929452+0000","last_active":"2024-09-13T22:50:17.960863+0000","last_peered":"2024-09-13T22:50:17.960863+0000","last_clean":"2024-09-13T22:50:17.960863+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:50:17.960863+0000","last_undegraded":"2024-09-13T22:50:17.960863+0000","last_fullsized":"2024-09-13T22:50:17.960863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:48:23.929342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.929342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.956329+0000","last_change":"2024-09-13T22:48:37.212544+0000","last_active":"2024-09-13T22:50:17.956329+0000","last_peered":"2024-09-13T22:50:17.956329+0000","last_clean":"2024-09-13T22:50:17.956329+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:50:17.956329+0000","last_undegraded":"2024-09-13T22:50:17.956329+0000","last_fullsized":"2024-09-13T22:50:17.956329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.201926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.201926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"266'14","reported_seq":32,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969806+0000","last_change":"2024-09-13T22:50:04.013350+0000","last_active":"2024-09-13T22:50:17.969806+0000","last_peered":"2024-09-13T22:50:17.969806+0000","last_clean":"2024-09-13T22:50:17.969806+0000","last_became_active":"2024-09-13T22:50:03.990705+0000","last_became_peered":"2024-09-13T22:50:03.990705+0000","last_unstale":"2024-09-13T22:50:17.969806+0000","last_undegraded":"2024-09-13T22:50:17.969806+0000","last_fullsized":"2024-09-13T22:50:17.969806+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:08.989696+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.989696+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":434,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950403+0000","last_change":"2024-09-13T22:48:39.747979+0000","last_active":"2024-09-13T22:50:17.950403+0000","last_peered":"2024-09-13T22:50:17.950403+0000","last_clean":"2024-09-13T22:50:17.950403+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:50:17.950403+0000","last_undegraded":"2024-09-13T22:50:17.950403+0000","last_fullsized":"2024-09-13T22:50:17.950403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.747881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.747881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"271'3969","reported_seq":4018,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.791116+0000","last_change":"2024-09-13T22:49:19.839310+0000","last_active":"2024-09-13T22:50:18.791116+0000","last_peered":"2024-09-13T22:50:18.791116+0000","last_clean":"2024-09-13T22:50:18.791116+0000","last_became_active":"2024-09-13T22:49:19.838528+0000","last_became_peered":"2024-09-13T22:49:19.838528+0000","last_unstale":"2024-09-13T22:50:18.791116+0000","last_undegraded":"2024-09-13T22:50:18.791116+0000","last_fullsized":"2024-09-13T22:50:18.791116+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:22.177237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":11907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":254016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"228'18","reported_seq":37,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.968963+0000","last_change":"2024-09-13T22:50:04.022082+0000","last_active":"2024-09-13T22:50:17.968963+0000","last_peered":"2024-09-13T22:50:17.968963+0000","last_clean":"2024-09-13T22:50:17.968963+0000","last_became_active":"2024-09-13T22:50:03.994289+0000","last_became_peered":"2024-09-13T22:50:03.994289+0000","last_unstale":"2024-09-13T22:50:17.968963+0000","last_undegraded":"2024-09-13T22:50:17.968963+0000","last_fullsized":"2024-09-13T22:50:17.968963+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:20.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.191528+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.962209+0000","last_change":"2024-09-13T22:49:38.845359+0000","last_active":"2024-09-13T22:50:17.962209+0000","last_peered":"2024-09-13T22:50:17.962209+0000","last_clean":"2024-09-13T22:50:17.962209+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:50:17.962209+0000","last_undegraded":"2024-09-13T22:50:17.962209+0000","last_fullsized":"2024-09-13T22:50:17.962209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:38.845293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:38.845293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:40.087606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":255,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958894+0000","last_change":"2024-09-13T22:50:03.996275+0000","last_active":"2024-09-13T22:50:17.958894+0000","last_peered":"2024-09-13T22:50:17.958894+0000","last_clean":"2024-09-13T22:50:17.958894+0000","last_became_active":"2024-09-13T22:50:03.995778+0000","last_became_peered":"2024-09-13T22:50:03.995778+0000","last_unstale":"2024-09-13T22:50:17.958894+0000","last_undegraded":"2024-09-13T22:50:17.958894+0000","last_fullsized":"2024-09-13T22:50:17.958894+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.194219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.194219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":427,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963235+0000","last_change":"2024-09-13T22:45:03.478116+0000","last_active":"2024-09-13T22:50:17.963235+0000","last_peered":"2024-09-13T22:50:17.963235+0000","last_clean":"2024-09-13T22:50:17.963235+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:50:17.963235+0000","last_undegraded":"2024-09-13T22:50:17.963235+0000","last_fullsized":"2024-09-13T22:50:17.963235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.477932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.477932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"271'4213","reported_seq":5331,"reported_epoch":271,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:50:20.468407+0000","last_change":"2024-09-13T22:50:05.311733+0000","last_active":"2024-09-13T22:50:20.468407+0000","last_peered":"2024-09-13T22:50:20.468407+0000","last_clean":"2024-09-13T22:50:02.919985+0000","last_became_active":"2024-09-13T22:50:05.285332+0000","last_became_peered":"2024-09-13T22:50:05.285332+0000","last_unstale":"2024-09-13T22:50:20.468407+0000","last_undegraded":"2024-09-13T22:50:20.468407+0000","last_fullsized":"2024-09-13T22:50:04.949038+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276103168,"num_objects":4213,"num_object_clones":0,"num_object_copies":12639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2686,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4213,"num_write_kb":269632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":532,"num_bytes_recovered":34865152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,7],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":2686},{"shards":"3,7","objects":1527}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":492,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963138+0000","last_change":"2024-09-13T22:45:59.360761+0000","last_active":"2024-09-13T22:50:17.963138+0000","last_peered":"2024-09-13T22:50:17.963138+0000","last_clean":"2024-09-13T22:50:17.963138+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:50:17.963138+0000","last_undegraded":"2024-09-13T22:50:17.963138+0000","last_fullsized":"2024-09-13T22:50:17.963138+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.360631+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"228'24","reported_seq":527,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943472+0000","last_change":"2024-09-13T22:50:04.013222+0000","last_active":"2024-09-13T22:50:17.943472+0000","last_peered":"2024-09-13T22:50:17.943472+0000","last_clean":"2024-09-13T22:50:17.943472+0000","last_became_active":"2024-09-13T22:50:03.973976+0000","last_became_peered":"2024-09-13T22:50:03.973976+0000","last_unstale":"2024-09-13T22:50:17.943472+0000","last_undegraded":"2024-09-13T22:50:17.943472+0000","last_fullsized":"2024-09-13T22:50:17.943472+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:49:36.082957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:36.082957+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.700193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.946612+0000","last_change":"2024-09-13T22:48:19.112621+0000","last_active":"2024-09-13T22:50:17.946612+0000","last_peered":"2024-09-13T22:50:17.946612+0000","last_clean":"2024-09-13T22:50:17.946612+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:50:17.946612+0000","last_undegraded":"2024-09-13T22:50:17.946612+0000","last_fullsized":"2024-09-13T22:50:17.946612+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:19.112032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:19.112032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":29,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.968859+0000","last_change":"2024-09-13T22:50:04.005424+0000","last_active":"2024-09-13T22:50:17.968859+0000","last_peered":"2024-09-13T22:50:17.968859+0000","last_clean":"2024-09-13T22:50:17.968859+0000","last_became_active":"2024-09-13T22:50:03.983986+0000","last_became_peered":"2024-09-13T22:50:03.983986+0000","last_unstale":"2024-09-13T22:50:17.968859+0000","last_undegraded":"2024-09-13T22:50:17.968859+0000","last_fullsized":"2024-09-13T22:50:17.968859+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:48:34.822707+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.822707+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"266'13","reported_seq":615,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957636+0000","last_change":"2024-09-13T22:50:04.034140+0000","last_active":"2024-09-13T22:50:17.957636+0000","last_peered":"2024-09-13T22:50:17.957636+0000","last_clean":"2024-09-13T22:50:17.957636+0000","last_became_active":"2024-09-13T22:50:03.999194+0000","last_became_peered":"2024-09-13T22:50:03.999194+0000","last_unstale":"2024-09-13T22:50:17.957636+0000","last_undegraded":"2024-09-13T22:50:17.957636+0000","last_fullsized":"2024-09-13T22:50:17.957636+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:38.494521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:38.494521+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957303+0000","last_change":"2024-09-13T22:49:30.942556+0000","last_active":"2024-09-13T22:50:17.957303+0000","last_peered":"2024-09-13T22:50:17.957303+0000","last_clean":"2024-09-13T22:50:17.957303+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:50:17.957303+0000","last_undegraded":"2024-09-13T22:50:17.957303+0000","last_fullsized":"2024-09-13T22:50:17.957303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:30.942259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:30.942259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:40.801299+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"271'3969","reported_seq":4019,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.771998+0000","last_change":"2024-09-13T22:49:19.844311+0000","last_active":"2024-09-13T22:50:18.771998+0000","last_peered":"2024-09-13T22:50:18.771998+0000","last_clean":"2024-09-13T22:50:18.771998+0000","last_became_active":"2024-09-13T22:49:19.843295+0000","last_became_peered":"2024-09-13T22:49:19.843295+0000","last_unstale":"2024-09-13T22:50:18.771998+0000","last_undegraded":"2024-09-13T22:50:18.771998+0000","last_fullsized":"2024-09-13T22:50:18.771998+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.082316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":11907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":254016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":468,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969527+0000","last_change":"2024-09-13T22:50:06.962705+0000","last_active":"2024-09-13T22:50:17.969527+0000","last_peered":"2024-09-13T22:50:17.969527+0000","last_clean":"2024-09-13T22:50:17.969527+0000","last_became_active":"2024-09-13T22:50:06.962282+0000","last_became_peered":"2024-09-13T22:50:06.962282+0000","last_unstale":"2024-09-13T22:50:17.969527+0000","last_undegraded":"2024-09-13T22:50:17.969527+0000","last_fullsized":"2024-09-13T22:50:17.969527+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:26.230025+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.230025+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:39.657626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"228'22","reported_seq":481,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.960802+0000","last_change":"2024-09-13T22:49:23.376019+0000","last_active":"2024-09-13T22:50:17.960802+0000","last_peered":"2024-09-13T22:50:17.960802+0000","last_clean":"2024-09-13T22:50:17.960802+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:50:17.960802+0000","last_undegraded":"2024-09-13T22:50:17.960802+0000","last_fullsized":"2024-09-13T22:50:17.960802+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:49:23.375848+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:49:23.375848+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:25.230518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":477,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.964328+0000","last_change":"2024-09-13T22:50:04.017241+0000","last_active":"2024-09-13T22:50:17.964328+0000","last_peered":"2024-09-13T22:50:17.964328+0000","last_clean":"2024-09-13T22:50:17.964328+0000","last_became_active":"2024-09-13T22:50:03.979523+0000","last_became_peered":"2024-09-13T22:50:03.979523+0000","last_unstale":"2024-09-13T22:50:17.964328+0000","last_undegraded":"2024-09-13T22:50:17.964328+0000","last_fullsized":"2024-09-13T22:50:17.964328+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:49:31.481514+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:49:31.481514+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:38.405524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":1182,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.772131+0000","last_change":"2024-09-13T22:45:54.489191+0000","last_active":"2024-09-13T22:50:18.772131+0000","last_peered":"2024-09-13T22:50:18.772131+0000","last_clean":"2024-09-13T22:50:18.772131+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:50:18.772131+0000","last_undegraded":"2024-09-13T22:50:18.772131+0000","last_fullsized":"2024-09-13T22:50:18.772131+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:45:54.489086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.489086+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":736,"num_read_kb":731,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.968874+0000","last_change":"2024-09-13T22:50:03.995424+0000","last_active":"2024-09-13T22:50:17.968874+0000","last_peered":"2024-09-13T22:50:17.968874+0000","last_clean":"2024-09-13T22:50:17.968874+0000","last_became_active":"2024-09-13T22:50:03.993502+0000","last_became_peered":"2024-09-13T22:50:03.993502+0000","last_unstale":"2024-09-13T22:50:17.968874+0000","last_undegraded":"2024-09-13T22:50:17.968874+0000","last_fullsized":"2024-09-13T22:50:17.968874+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:51.936005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.936005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969352+0000","last_change":"2024-09-13T22:50:03.984972+0000","last_active":"2024-09-13T22:50:17.969352+0000","last_peered":"2024-09-13T22:50:17.969352+0000","last_clean":"2024-09-13T22:50:17.969352+0000","last_became_active":"2024-09-13T22:50:03.982511+0000","last_became_peered":"2024-09-13T22:50:03.982511+0000","last_unstale":"2024-09-13T22:50:17.969352+0000","last_undegraded":"2024-09-13T22:50:17.969352+0000","last_fullsized":"2024-09-13T22:50:17.969352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:37.838733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:45:37.838733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"271'3867","reported_seq":5539,"reported_epoch":271,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:50:18.792723+0000","last_change":"2024-09-13T22:50:05.315402+0000","last_active":"2024-09-13T22:50:18.792723+0000","last_peered":"2024-09-13T22:50:18.792723+0000","last_clean":"2024-09-13T22:50:02.923925+0000","last_became_active":"2024-09-13T22:50:05.279429+0000","last_became_peered":"2024-09-13T22:50:05.279429+0000","last_unstale":"2024-09-13T22:50:18.792723+0000","last_undegraded":"2024-09-13T22:50:18.792723+0000","last_fullsized":"2024-09-13T22:50:04.947825+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253427712,"num_objects":3867,"num_object_clones":0,"num_object_copies":11601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2225,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3867,"num_write_kb":247488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":53018624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,1],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":2225},{"shards":"1,7","objects":1642}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":513,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.246875+0000","last_change":"2024-09-13T22:50:18.246875+0000","last_active":"2024-09-13T22:50:18.246875+0000","last_peered":"2024-09-13T22:50:18.246875+0000","last_clean":"2024-09-13T22:50:18.246875+0000","last_became_active":"2024-09-13T22:45:58.866066+0000","last_became_peered":"2024-09-13T22:45:58.866066+0000","last_unstale":"2024-09-13T22:50:18.246875+0000","last_undegraded":"2024-09-13T22:50:18.246875+0000","last_fullsized":"2024-09-13T22:50:18.246875+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:50:18.246752+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:50:18.246752+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:23.953992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"255'52","reported_seq":682,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.965264+0000","last_change":"2024-09-13T22:49:40.725923+0000","last_active":"2024-09-13T22:50:17.965264+0000","last_peered":"2024-09-13T22:50:17.965264+0000","last_clean":"2024-09-13T22:50:17.965264+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:50:17.965264+0000","last_undegraded":"2024-09-13T22:50:17.965264+0000","last_fullsized":"2024-09-13T22:50:17.965264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'52","last_scrub_stamp":"2024-09-13T22:49:40.725760+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:49:40.725760+0000","last_clean_scrub_stamp":"2024-09-13T22:49:40.725760+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:52.490962+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.959165+0000","last_change":"2024-09-13T22:50:03.997450+0000","last_active":"2024-09-13T22:50:17.959165+0000","last_peered":"2024-09-13T22:50:17.959165+0000","last_clean":"2024-09-13T22:50:17.959165+0000","last_became_active":"2024-09-13T22:50:03.996855+0000","last_became_peered":"2024-09-13T22:50:03.996855+0000","last_unstale":"2024-09-13T22:50:17.959165+0000","last_undegraded":"2024-09-13T22:50:17.959165+0000","last_fullsized":"2024-09-13T22:50:17.959165+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.960501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.960501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:40.386485+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":463,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945502+0000","last_change":"2024-09-13T22:49:29.115748+0000","last_active":"2024-09-13T22:50:17.945502+0000","last_peered":"2024-09-13T22:50:17.945502+0000","last_clean":"2024-09-13T22:50:17.945502+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:50:17.945502+0000","last_undegraded":"2024-09-13T22:50:17.945502+0000","last_fullsized":"2024-09-13T22:50:17.945502+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:49:29.115656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.845058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":436,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.956025+0000","last_change":"2024-09-13T22:48:11.214965+0000","last_active":"2024-09-13T22:50:17.956025+0000","last_peered":"2024-09-13T22:50:17.956025+0000","last_clean":"2024-09-13T22:50:17.956025+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:50:17.956025+0000","last_undegraded":"2024-09-13T22:50:17.956025+0000","last_fullsized":"2024-09-13T22:50:17.956025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.214834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.214834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"271'4026","reported_seq":4076,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.819235+0000","last_change":"2024-09-13T22:49:19.836085+0000","last_active":"2024-09-13T22:50:18.819235+0000","last_peered":"2024-09-13T22:50:18.819235+0000","last_clean":"2024-09-13T22:50:18.819235+0000","last_became_active":"2024-09-13T22:49:19.834656+0000","last_became_peered":"2024-09-13T22:49:19.834656+0000","last_unstale":"2024-09-13T22:50:18.819235+0000","last_undegraded":"2024-09-13T22:50:18.819235+0000","last_fullsized":"2024-09-13T22:50:18.819235+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:31.862109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4026,"num_write_kb":257664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":513,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945792+0000","last_change":"2024-09-13T22:49:31.103122+0000","last_active":"2024-09-13T22:50:17.945792+0000","last_peered":"2024-09-13T22:50:17.945792+0000","last_clean":"2024-09-13T22:50:17.945792+0000","last_became_active":"2024-09-13T22:45:58.862759+0000","last_became_peered":"2024-09-13T22:45:58.862759+0000","last_unstale":"2024-09-13T22:50:17.945792+0000","last_undegraded":"2024-09-13T22:50:17.945792+0000","last_fullsized":"2024-09-13T22:50:17.945792+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:31.103011+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:31.103011+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:31.125605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"228'36","reported_seq":651,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950907+0000","last_change":"2024-09-13T22:49:40.203401+0000","last_active":"2024-09-13T22:50:17.950907+0000","last_peered":"2024-09-13T22:50:17.950907+0000","last_clean":"2024-09-13T22:50:17.950907+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:50:17.950907+0000","last_undegraded":"2024-09-13T22:50:17.950907+0000","last_fullsized":"2024-09-13T22:50:17.950907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'36","last_scrub_stamp":"2024-09-13T22:49:40.203232+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:49:40.203232+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:07.644837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.965315+0000","last_change":"2024-09-13T22:48:35.268201+0000","last_active":"2024-09-13T22:50:17.965315+0000","last_peered":"2024-09-13T22:50:17.965315+0000","last_clean":"2024-09-13T22:50:17.965315+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:50:17.965315+0000","last_undegraded":"2024-09-13T22:50:17.965315+0000","last_fullsized":"2024-09-13T22:50:17.965315+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:48:35.268073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.268073+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":481,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963776+0000","last_change":"2024-09-13T22:50:04.020638+0000","last_active":"2024-09-13T22:50:17.963776+0000","last_peered":"2024-09-13T22:50:17.963776+0000","last_clean":"2024-09-13T22:50:17.963776+0000","last_became_active":"2024-09-13T22:50:03.977296+0000","last_became_peered":"2024-09-13T22:50:03.977296+0000","last_unstale":"2024-09-13T22:50:17.963776+0000","last_undegraded":"2024-09-13T22:50:17.963776+0000","last_fullsized":"2024-09-13T22:50:17.963776+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:49:14.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.400987+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:35.907603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":611,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942872+0000","last_change":"2024-09-13T22:48:28.132907+0000","last_active":"2024-09-13T22:50:17.942872+0000","last_peered":"2024-09-13T22:50:17.942872+0000","last_clean":"2024-09-13T22:50:17.942872+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:50:17.942872+0000","last_undegraded":"2024-09-13T22:50:17.942872+0000","last_fullsized":"2024-09-13T22:50:17.942872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:48:28.132780+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.132780+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":444,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942354+0000","last_change":"2024-09-13T22:49:39.109496+0000","last_active":"2024-09-13T22:50:17.942354+0000","last_peered":"2024-09-13T22:50:17.942354+0000","last_clean":"2024-09-13T22:50:17.942354+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:50:17.942354+0000","last_undegraded":"2024-09-13T22:50:17.942354+0000","last_fullsized":"2024-09-13T22:50:17.942354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:39.109375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:39.109375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:06.200469+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"271'4093","reported_seq":4143,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:20.329002+0000","last_change":"2024-09-13T22:49:19.836077+0000","last_active":"2024-09-13T22:50:20.329002+0000","last_peered":"2024-09-13T22:50:20.329002+0000","last_clean":"2024-09-13T22:50:20.329002+0000","last_became_active":"2024-09-13T22:49:19.835836+0000","last_became_peered":"2024-09-13T22:49:19.835836+0000","last_unstale":"2024-09-13T22:50:20.329002+0000","last_undegraded":"2024-09-13T22:50:20.329002+0000","last_fullsized":"2024-09-13T22:50:20.329002+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.716731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268238848,"num_objects":4093,"num_object_clones":0,"num_object_copies":12279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":261952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.948109+0000","last_change":"2024-09-13T22:48:26.832776+0000","last_active":"2024-09-13T22:50:17.948109+0000","last_peered":"2024-09-13T22:50:17.948109+0000","last_clean":"2024-09-13T22:50:17.948109+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:50:17.948109+0000","last_undegraded":"2024-09-13T22:50:17.948109+0000","last_fullsized":"2024-09-13T22:50:17.948109+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.832692+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"228'29","reported_seq":43,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969184+0000","last_change":"2024-09-13T22:50:04.027622+0000","last_active":"2024-09-13T22:50:17.969184+0000","last_peered":"2024-09-13T22:50:17.969184+0000","last_clean":"2024-09-13T22:50:17.969184+0000","last_became_active":"2024-09-13T22:50:03.997974+0000","last_became_peered":"2024-09-13T22:50:03.997974+0000","last_unstale":"2024-09-13T22:50:17.969184+0000","last_undegraded":"2024-09-13T22:50:17.969184+0000","last_fullsized":"2024-09-13T22:50:17.969184+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'29","last_scrub_stamp":"2024-09-13T22:49:25.093976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:25.093976+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:37.516006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.948949+0000","last_change":"2024-09-13T22:48:14.833285+0000","last_active":"2024-09-13T22:50:17.948949+0000","last_peered":"2024-09-13T22:50:17.948949+0000","last_clean":"2024-09-13T22:50:17.948949+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:50:17.948949+0000","last_undegraded":"2024-09-13T22:50:17.948949+0000","last_fullsized":"2024-09-13T22:50:17.948949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943247+0000","last_change":"2024-09-13T22:50:03.973607+0000","last_active":"2024-09-13T22:50:17.943247+0000","last_peered":"2024-09-13T22:50:17.943247+0000","last_clean":"2024-09-13T22:50:17.943247+0000","last_became_active":"2024-09-13T22:50:03.972954+0000","last_became_peered":"2024-09-13T22:50:03.972954+0000","last_unstale":"2024-09-13T22:50:17.943247+0000","last_undegraded":"2024-09-13T22:50:17.943247+0000","last_fullsized":"2024-09-13T22:50:17.943247+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:19.165932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:49:19.165932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:32.871652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957919+0000","last_change":"2024-09-13T22:50:03.999169+0000","last_active":"2024-09-13T22:50:17.957919+0000","last_peered":"2024-09-13T22:50:17.957919+0000","last_clean":"2024-09-13T22:50:17.957919+0000","last_became_active":"2024-09-13T22:50:03.998579+0000","last_became_peered":"2024-09-13T22:50:03.998579+0000","last_unstale":"2024-09-13T22:50:17.957919+0000","last_undegraded":"2024-09-13T22:50:17.957919+0000","last_fullsized":"2024-09-13T22:50:17.957919+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.535993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.535993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"271'4034","reported_seq":4084,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.845821+0000","last_change":"2024-09-13T22:49:19.844519+0000","last_active":"2024-09-13T22:50:18.845821+0000","last_peered":"2024-09-13T22:50:18.845821+0000","last_clean":"2024-09-13T22:50:18.845821+0000","last_became_active":"2024-09-13T22:49:19.840996+0000","last_became_peered":"2024-09-13T22:49:19.840996+0000","last_unstale":"2024-09-13T22:50:18.845821+0000","last_undegraded":"2024-09-13T22:50:18.845821+0000","last_fullsized":"2024-09-13T22:50:18.845821+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4034,"log_dups_size":0,"ondisk_log_size":4034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:25.003453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264372224,"num_objects":4034,"num_object_clones":0,"num_object_copies":12102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":258176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":743,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958637+0000","last_change":"2024-09-13T22:50:06.969941+0000","last_active":"2024-09-13T22:50:17.958637+0000","last_peered":"2024-09-13T22:50:17.958637+0000","last_clean":"2024-09-13T22:50:17.958637+0000","last_became_active":"2024-09-13T22:50:06.969615+0000","last_became_peered":"2024-09-13T22:50:06.969615+0000","last_unstale":"2024-09-13T22:50:17.958637+0000","last_undegraded":"2024-09-13T22:50:17.958637+0000","last_fullsized":"2024-09-13T22:50:17.958637+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:38.173263+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.173263+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"228'38","reported_seq":510,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.956340+0000","last_change":"2024-09-13T22:49:36.839833+0000","last_active":"2024-09-13T22:50:17.956340+0000","last_peered":"2024-09-13T22:50:17.956340+0000","last_clean":"2024-09-13T22:50:17.956340+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:50:17.956340+0000","last_undegraded":"2024-09-13T22:50:17.956340+0000","last_fullsized":"2024-09-13T22:50:17.956340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:49:36.839475+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:49:36.839475+0000","last_clean_scrub_stamp":"2024-09-13T22:49:36.839475+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:02.417612+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":472,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.952070+0000","last_change":"2024-09-13T22:49:24.271420+0000","last_active":"2024-09-13T22:50:17.952070+0000","last_peered":"2024-09-13T22:50:17.952070+0000","last_clean":"2024-09-13T22:50:17.952070+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:50:17.952070+0000","last_undegraded":"2024-09-13T22:50:17.952070+0000","last_fullsized":"2024-09-13T22:50:17.952070+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:49:24.271342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:24.271342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:38.148279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.953877+0000","last_change":"2024-09-13T22:48:22.142431+0000","last_active":"2024-09-13T22:50:17.953877+0000","last_peered":"2024-09-13T22:50:17.953877+0000","last_clean":"2024-09-13T22:50:17.953877+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:50:17.953877+0000","last_undegraded":"2024-09-13T22:50:17.953877+0000","last_fullsized":"2024-09-13T22:50:17.953877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:22.142327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:48:22.142327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.959417+0000","last_change":"2024-09-13T22:48:17.369617+0000","last_active":"2024-09-13T22:50:17.959417+0000","last_peered":"2024-09-13T22:50:17.959417+0000","last_clean":"2024-09-13T22:50:17.959417+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:50:17.959417+0000","last_undegraded":"2024-09-13T22:50:17.959417+0000","last_fullsized":"2024-09-13T22:50:17.959417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.369549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.369549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"271'4036","reported_seq":4086,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:20.343770+0000","last_change":"2024-09-13T22:49:19.908928+0000","last_active":"2024-09-13T22:50:20.343770+0000","last_peered":"2024-09-13T22:50:20.343770+0000","last_clean":"2024-09-13T22:50:20.343770+0000","last_became_active":"2024-09-13T22:49:19.908470+0000","last_became_peered":"2024-09-13T22:49:19.908470+0000","last_unstale":"2024-09-13T22:50:20.343770+0000","last_undegraded":"2024-09-13T22:50:20.343770+0000","last_fullsized":"2024-09-13T22:50:20.343770+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:22.290041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264503296,"num_objects":4036,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":258304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":518,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963299+0000","last_change":"2024-09-13T22:50:06.968858+0000","last_active":"2024-09-13T22:50:17.963299+0000","last_peered":"2024-09-13T22:50:17.963299+0000","last_clean":"2024-09-13T22:50:17.963299+0000","last_became_active":"2024-09-13T22:50:06.968526+0000","last_became_peered":"2024-09-13T22:50:06.968526+0000","last_unstale":"2024-09-13T22:50:17.963299+0000","last_undegraded":"2024-09-13T22:50:17.963299+0000","last_fullsized":"2024-09-13T22:50:17.963299+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:31.173761+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.802229+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.173761+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"228'28","reported_seq":538,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942527+0000","last_change":"2024-09-13T22:50:04.011454+0000","last_active":"2024-09-13T22:50:17.942527+0000","last_peered":"2024-09-13T22:50:17.942527+0000","last_clean":"2024-09-13T22:50:17.942527+0000","last_became_active":"2024-09-13T22:50:03.973390+0000","last_became_peered":"2024-09-13T22:50:03.973390+0000","last_unstale":"2024-09-13T22:50:17.942527+0000","last_undegraded":"2024-09-13T22:50:17.942527+0000","last_fullsized":"2024-09-13T22:50:17.942527+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'28","last_scrub_stamp":"2024-09-13T22:49:11.188389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.188389+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.667200+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.961921+0000","last_change":"2024-09-13T22:46:54.746202+0000","last_active":"2024-09-13T22:50:17.961921+0000","last_peered":"2024-09-13T22:50:17.961921+0000","last_clean":"2024-09-13T22:50:17.961921+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:50:17.961921+0000","last_undegraded":"2024-09-13T22:50:17.961921+0000","last_fullsized":"2024-09-13T22:50:17.961921+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:54.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:46:54.746095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.952867+0000","last_change":"2024-09-13T22:49:21.447561+0000","last_active":"2024-09-13T22:50:17.952867+0000","last_peered":"2024-09-13T22:50:17.952867+0000","last_clean":"2024-09-13T22:50:17.952867+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:50:17.952867+0000","last_undegraded":"2024-09-13T22:50:17.952867+0000","last_fullsized":"2024-09-13T22:50:17.952867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:21.447387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.447387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:27.672741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":444,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.964283+0000","last_change":"2024-09-13T22:49:28.469536+0000","last_active":"2024-09-13T22:50:17.964283+0000","last_peered":"2024-09-13T22:50:17.964283+0000","last_clean":"2024-09-13T22:50:17.964283+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:50:17.964283+0000","last_undegraded":"2024-09-13T22:50:17.964283+0000","last_fullsized":"2024-09-13T22:50:17.964283+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:28.469350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:49:28.469350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:42.153355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"271'4155","reported_seq":4205,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:20.441741+0000","last_change":"2024-09-13T22:49:19.832503+0000","last_active":"2024-09-13T22:50:20.441741+0000","last_peered":"2024-09-13T22:50:20.441741+0000","last_clean":"2024-09-13T22:50:20.441741+0000","last_became_active":"2024-09-13T22:49:19.831731+0000","last_became_peered":"2024-09-13T22:49:19.831731+0000","last_unstale":"2024-09-13T22:50:20.441741+0000","last_undegraded":"2024-09-13T22:50:20.441741+0000","last_fullsized":"2024-09-13T22:50:20.441741+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4155,"log_dups_size":0,"ondisk_log_size":4155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:32.889608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272302080,"num_objects":4155,"num_object_clones":0,"num_object_copies":12465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4155,"num_write_kb":265920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"228'20","reported_seq":474,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958324+0000","last_change":"2024-09-13T22:49:35.374424+0000","last_active":"2024-09-13T22:50:17.958324+0000","last_peered":"2024-09-13T22:50:17.958324+0000","last_clean":"2024-09-13T22:50:17.958324+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:50:17.958324+0000","last_undegraded":"2024-09-13T22:50:17.958324+0000","last_fullsized":"2024-09-13T22:50:17.958324+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:49:35.374321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:35.374321+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:46.636572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950260+0000","last_change":"2024-09-13T22:49:22.420187+0000","last_active":"2024-09-13T22:50:17.950260+0000","last_peered":"2024-09-13T22:50:17.950260+0000","last_clean":"2024-09-13T22:50:17.950260+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:50:17.950260+0000","last_undegraded":"2024-09-13T22:50:17.950260+0000","last_fullsized":"2024-09-13T22:50:17.950260+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:35.501613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":717,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.962457+0000","last_change":"2024-09-13T22:48:32.199596+0000","last_active":"2024-09-13T22:50:17.962457+0000","last_peered":"2024-09-13T22:50:17.962457+0000","last_clean":"2024-09-13T22:50:17.962457+0000","last_became_active":"2024-09-13T22:45:53.877322+0000","last_became_peered":"2024-09-13T22:45:53.877322+0000","last_unstale":"2024-09-13T22:50:17.962457+0000","last_undegraded":"2024-09-13T22:50:17.962457+0000","last_fullsized":"2024-09-13T22:50:17.962457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:32.199490+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.199490+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957478+0000","last_change":"2024-09-13T22:48:16.361464+0000","last_active":"2024-09-13T22:50:17.957478+0000","last_peered":"2024-09-13T22:50:17.957478+0000","last_clean":"2024-09-13T22:50:17.957478+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:50:17.957478+0000","last_undegraded":"2024-09-13T22:50:17.957478+0000","last_fullsized":"2024-09-13T22:50:17.957478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:16.361322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:48:16.361322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":454,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.952820+0000","last_change":"2024-09-13T22:50:03.991394+0000","last_active":"2024-09-13T22:50:17.952820+0000","last_peered":"2024-09-13T22:50:17.952820+0000","last_clean":"2024-09-13T22:50:17.952820+0000","last_became_active":"2024-09-13T22:50:03.990614+0000","last_became_peered":"2024-09-13T22:50:03.990614+0000","last_unstale":"2024-09-13T22:50:17.952820+0000","last_undegraded":"2024-09-13T22:50:17.952820+0000","last_fullsized":"2024-09-13T22:50:17.952820+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.075948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.075948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"271'4077","reported_seq":4125,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.868607+0000","last_change":"2024-09-13T22:49:19.844752+0000","last_active":"2024-09-13T22:50:18.868607+0000","last_peered":"2024-09-13T22:50:18.868607+0000","last_clean":"2024-09-13T22:50:18.868607+0000","last_became_active":"2024-09-13T22:49:19.841262+0000","last_became_peered":"2024-09-13T22:49:19.841262+0000","last_unstale":"2024-09-13T22:50:18.868607+0000","last_undegraded":"2024-09-13T22:50:18.868607+0000","last_fullsized":"2024-09-13T22:50:18.868607+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:25.629314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267190272,"num_objects":4077,"num_object_clones":0,"num_object_copies":12231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":260928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"228'31","reported_seq":491,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958929+0000","last_change":"2024-09-13T22:49:13.909885+0000","last_active":"2024-09-13T22:50:17.958929+0000","last_peered":"2024-09-13T22:50:17.958929+0000","last_clean":"2024-09-13T22:50:17.958929+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:50:17.958929+0000","last_undegraded":"2024-09-13T22:50:17.958929+0000","last_fullsized":"2024-09-13T22:50:17.958929+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'31","last_scrub_stamp":"2024-09-13T22:49:13.909765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:13.909765+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:32.029113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":471,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.657008+0000","last_change":"2024-09-13T22:50:18.657042+0000","last_active":"2024-09-13T22:50:18.657008+0000","last_peered":"2024-09-13T22:50:18.657008+0000","last_clean":"2024-09-13T22:50:18.657008+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:50:18.657008+0000","last_undegraded":"2024-09-13T22:50:18.657008+0000","last_fullsized":"2024-09-13T22:50:18.657008+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:18.656932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:18.656932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:43.795095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":447,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.952134+0000","last_change":"2024-09-13T22:48:35.269277+0000","last_active":"2024-09-13T22:50:17.952134+0000","last_peered":"2024-09-13T22:50:17.952134+0000","last_clean":"2024-09-13T22:50:17.952134+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:50:17.952134+0000","last_undegraded":"2024-09-13T22:50:17.952134+0000","last_fullsized":"2024-09-13T22:50:17.952134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:35.269156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.269156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":434,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.962295+0000","last_change":"2024-09-13T22:49:22.976336+0000","last_active":"2024-09-13T22:50:17.962295+0000","last_peered":"2024-09-13T22:50:17.962295+0000","last_clean":"2024-09-13T22:50:17.962295+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:50:17.962295+0000","last_undegraded":"2024-09-13T22:50:17.962295+0000","last_fullsized":"2024-09-13T22:50:17.962295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.976109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.976109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:48.810377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"271'4044","reported_seq":4094,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:18.874934+0000","last_change":"2024-09-13T22:49:19.838975+0000","last_active":"2024-09-13T22:50:18.874934+0000","last_peered":"2024-09-13T22:50:18.874934+0000","last_clean":"2024-09-13T22:50:18.874934+0000","last_became_active":"2024-09-13T22:49:19.835650+0000","last_became_peered":"2024-09-13T22:49:19.835650+0000","last_unstale":"2024-09-13T22:50:18.874934+0000","last_undegraded":"2024-09-13T22:50:18.874934+0000","last_fullsized":"2024-09-13T22:50:18.874934+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:44.481789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":258816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"228'20","reported_seq":480,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950349+0000","last_change":"2024-09-13T22:49:14.431757+0000","last_active":"2024-09-13T22:50:17.950349+0000","last_peered":"2024-09-13T22:50:17.950349+0000","last_clean":"2024-09-13T22:50:17.950349+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:50:17.950349+0000","last_undegraded":"2024-09-13T22:50:17.950349+0000","last_fullsized":"2024-09-13T22:50:17.950349+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:49:14.431540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.431540+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.128652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":467,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963877+0000","last_change":"2024-09-13T22:50:03.980136+0000","last_active":"2024-09-13T22:50:17.963877+0000","last_peered":"2024-09-13T22:50:17.963877+0000","last_clean":"2024-09-13T22:50:17.963877+0000","last_became_active":"2024-09-13T22:50:03.977100+0000","last_became_peered":"2024-09-13T22:50:03.977100+0000","last_unstale":"2024-09-13T22:50:17.963877+0000","last_undegraded":"2024-09-13T22:50:17.963877+0000","last_fullsized":"2024-09-13T22:50:17.963877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:23.112116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.112116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969188+0000","last_change":"2024-09-13T22:50:03.984674+0000","last_active":"2024-09-13T22:50:17.969188+0000","last_peered":"2024-09-13T22:50:17.969188+0000","last_clean":"2024-09-13T22:50:17.969188+0000","last_became_active":"2024-09-13T22:50:03.982295+0000","last_became_peered":"2024-09-13T22:50:03.982295+0000","last_unstale":"2024-09-13T22:50:17.969188+0000","last_undegraded":"2024-09-13T22:50:17.969188+0000","last_fullsized":"2024-09-13T22:50:17.969188+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.282342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.282342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:40.635174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942433+0000","last_change":"2024-09-13T22:48:11.570730+0000","last_active":"2024-09-13T22:50:17.942433+0000","last_peered":"2024-09-13T22:50:17.942433+0000","last_clean":"2024-09-13T22:50:17.942433+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:50:17.942433+0000","last_undegraded":"2024-09-13T22:50:17.942433+0000","last_fullsized":"2024-09-13T22:50:17.942433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.570502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.570502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"271'4121","reported_seq":4171,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:20.316095+0000","last_change":"2024-09-13T22:49:19.838725+0000","last_active":"2024-09-13T22:50:20.316095+0000","last_peered":"2024-09-13T22:50:20.316095+0000","last_clean":"2024-09-13T22:50:20.316095+0000","last_became_active":"2024-09-13T22:49:19.838415+0000","last_became_peered":"2024-09-13T22:49:19.838415+0000","last_unstale":"2024-09-13T22:50:20.316095+0000","last_undegraded":"2024-09-13T22:50:20.316095+0000","last_fullsized":"2024-09-13T22:50:20.316095+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:28.533179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270073856,"num_objects":4121,"num_object_clones":0,"num_object_copies":12363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4121,"num_write_kb":263744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"228'34","reported_seq":536,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.959051+0000","last_change":"2024-09-13T22:50:04.032442+0000","last_active":"2024-09-13T22:50:17.959051+0000","last_peered":"2024-09-13T22:50:17.959051+0000","last_clean":"2024-09-13T22:50:17.959051+0000","last_became_active":"2024-09-13T22:50:03.995906+0000","last_became_peered":"2024-09-13T22:50:03.995906+0000","last_unstale":"2024-09-13T22:50:17.959051+0000","last_undegraded":"2024-09-13T22:50:17.959051+0000","last_fullsized":"2024-09-13T22:50:17.959051+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:49:11.973550+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.973550+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":38,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958840+0000","last_change":"2024-09-13T22:49:45.646088+0000","last_active":"2024-09-13T22:50:17.958840+0000","last_peered":"2024-09-13T22:50:17.958840+0000","last_clean":"2024-09-13T22:50:17.958840+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T22:50:17.958840+0000","last_undegraded":"2024-09-13T22:50:17.958840+0000","last_fullsized":"2024-09-13T22:50:17.958840+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.191239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.191239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"266'13","reported_seq":629,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.960989+0000","last_change":"2024-09-13T22:50:04.041189+0000","last_active":"2024-09-13T22:50:17.960989+0000","last_peered":"2024-09-13T22:50:17.960989+0000","last_clean":"2024-09-13T22:50:17.960989+0000","last_became_active":"2024-09-13T22:50:04.001753+0000","last_became_peered":"2024-09-13T22:50:04.001753+0000","last_unstale":"2024-09-13T22:50:17.960989+0000","last_undegraded":"2024-09-13T22:50:17.960989+0000","last_fullsized":"2024-09-13T22:50:17.960989+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'12","last_scrub_stamp":"2024-09-13T22:49:29.894307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:29.894307+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:51.125275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":442,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.948537+0000","last_change":"2024-09-13T22:49:33.362540+0000","last_active":"2024-09-13T22:50:17.948537+0000","last_peered":"2024-09-13T22:50:17.948537+0000","last_clean":"2024-09-13T22:50:17.948537+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:50:17.948537+0000","last_undegraded":"2024-09-13T22:50:17.948537+0000","last_fullsized":"2024-09-13T22:50:17.948537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:33.362411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:33.362411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:50.062673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"271'4104","reported_seq":5150,"reported_epoch":271,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:50:19.764360+0000","last_change":"2024-09-13T22:50:05.321435+0000","last_active":"2024-09-13T22:50:19.764360+0000","last_peered":"2024-09-13T22:50:19.764360+0000","last_clean":"2024-09-13T22:50:02.938290+0000","last_became_active":"2024-09-13T22:50:05.288363+0000","last_became_peered":"2024-09-13T22:50:05.288363+0000","last_unstale":"2024-09-13T22:50:19.764360+0000","last_undegraded":"2024-09-13T22:50:19.764360+0000","last_fullsized":"2024-09-13T22:50:04.952548+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4104,"log_dups_size":0,"ondisk_log_size":4104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268959744,"num_objects":4104,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2696,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4104,"num_write_kb":262656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":32505856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,3],"avail_no_missing":["1","0","3"],"object_location_counts":[{"shards":"0,1,3","objects":2696},{"shards":"1,3","objects":1408}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"228'19","reported_seq":517,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.968139+0000","last_change":"2024-09-13T22:50:04.038898+0000","last_active":"2024-09-13T22:50:17.968139+0000","last_peered":"2024-09-13T22:50:17.968139+0000","last_clean":"2024-09-13T22:50:17.968139+0000","last_became_active":"2024-09-13T22:50:03.994867+0000","last_became_peered":"2024-09-13T22:50:03.994867+0000","last_unstale":"2024-09-13T22:50:17.968139+0000","last_undegraded":"2024-09-13T22:50:17.968139+0000","last_fullsized":"2024-09-13T22:50:17.968139+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:49:31.961809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:31.961809+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:36.428509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":31,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.970028+0000","last_change":"2024-09-13T22:50:04.010479+0000","last_active":"2024-09-13T22:50:17.970028+0000","last_peered":"2024-09-13T22:50:17.970028+0000","last_clean":"2024-09-13T22:50:17.970028+0000","last_became_active":"2024-09-13T22:50:03.990408+0000","last_became_peered":"2024-09-13T22:50:03.990408+0000","last_unstale":"2024-09-13T22:50:17.970028+0000","last_undegraded":"2024-09-13T22:50:17.970028+0000","last_fullsized":"2024-09-13T22:50:17.970028+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:49:32.135936+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:49:32.135936+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:38.776874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942581+0000","last_change":"2024-09-13T22:49:45.649979+0000","last_active":"2024-09-13T22:50:17.942581+0000","last_peered":"2024-09-13T22:50:17.942581+0000","last_clean":"2024-09-13T22:50:17.942581+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T22:50:17.942581+0000","last_undegraded":"2024-09-13T22:50:17.942581+0000","last_fullsized":"2024-09-13T22:50:17.942581+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:40.290665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:48:40.290665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958322+0000","last_change":"2024-09-13T22:48:17.839741+0000","last_active":"2024-09-13T22:50:17.958322+0000","last_peered":"2024-09-13T22:50:17.958322+0000","last_clean":"2024-09-13T22:50:17.958322+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:50:17.958322+0000","last_undegraded":"2024-09-13T22:50:17.958322+0000","last_fullsized":"2024-09-13T22:50:17.958322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.839516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.839516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"228'24","reported_seq":514,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.961150+0000","last_change":"2024-09-13T22:50:04.030185+0000","last_active":"2024-09-13T22:50:17.961150+0000","last_peered":"2024-09-13T22:50:17.961150+0000","last_clean":"2024-09-13T22:50:17.961150+0000","last_became_active":"2024-09-13T22:50:03.995225+0000","last_became_peered":"2024-09-13T22:50:03.995225+0000","last_unstale":"2024-09-13T22:50:17.961150+0000","last_undegraded":"2024-09-13T22:50:17.961150+0000","last_fullsized":"2024-09-13T22:50:17.961150+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.168396+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":463,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.952324+0000","last_change":"2024-09-13T22:49:19.256364+0000","last_active":"2024-09-13T22:50:17.952324+0000","last_peered":"2024-09-13T22:50:17.952324+0000","last_clean":"2024-09-13T22:50:17.952324+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:50:17.952324+0000","last_undegraded":"2024-09-13T22:50:17.952324+0000","last_fullsized":"2024-09-13T22:50:17.952324+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:19.256282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:49:19.256282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:43.438849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.960719+0000","last_change":"2024-09-13T22:48:15.782055+0000","last_active":"2024-09-13T22:50:17.960719+0000","last_peered":"2024-09-13T22:50:17.960719+0000","last_clean":"2024-09-13T22:50:17.960719+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:50:17.960719+0000","last_undegraded":"2024-09-13T22:50:17.960719+0000","last_fullsized":"2024-09-13T22:50:17.960719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.778860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.778860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":471,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.952342+0000","last_change":"2024-09-13T22:50:16.597759+0000","last_active":"2024-09-13T22:50:17.952342+0000","last_peered":"2024-09-13T22:50:17.952342+0000","last_clean":"2024-09-13T22:50:17.952342+0000","last_became_active":"2024-09-13T22:50:03.968467+0000","last_became_peered":"2024-09-13T22:50:03.968467+0000","last_unstale":"2024-09-13T22:50:17.952342+0000","last_undegraded":"2024-09-13T22:50:17.952342+0000","last_fullsized":"2024-09-13T22:50:17.952342+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:16.597647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:50:16.597647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:37.510497+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"235'10","reported_seq":472,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945487+0000","last_change":"2024-09-13T22:48:31.992271+0000","last_active":"2024-09-13T22:50:17.945487+0000","last_peered":"2024-09-13T22:50:17.945487+0000","last_clean":"2024-09-13T22:50:17.945487+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:50:17.945487+0000","last_undegraded":"2024-09-13T22:50:17.945487+0000","last_fullsized":"2024-09-13T22:50:17.945487+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'10","last_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.992124+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":474,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945927+0000","last_change":"2024-09-13T22:50:03.988879+0000","last_active":"2024-09-13T22:50:17.945927+0000","last_peered":"2024-09-13T22:50:17.945927+0000","last_clean":"2024-09-13T22:50:17.945927+0000","last_became_active":"2024-09-13T22:50:03.987950+0000","last_became_peered":"2024-09-13T22:50:03.987950+0000","last_unstale":"2024-09-13T22:50:17.945927+0000","last_undegraded":"2024-09-13T22:50:17.945927+0000","last_fullsized":"2024-09-13T22:50:17.945927+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.082437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.082437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.951894+0000","last_change":"2024-09-13T22:49:32.393122+0000","last_active":"2024-09-13T22:50:17.951894+0000","last_peered":"2024-09-13T22:50:17.951894+0000","last_clean":"2024-09-13T22:50:17.951894+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:50:17.951894+0000","last_undegraded":"2024-09-13T22:50:17.951894+0000","last_fullsized":"2024-09-13T22:50:17.951894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:32.392914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:32.392914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:47.161912+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957315+0000","last_change":"2024-09-13T22:50:03.997746+0000","last_active":"2024-09-13T22:50:17.957315+0000","last_peered":"2024-09-13T22:50:17.957315+0000","last_clean":"2024-09-13T22:50:17.957315+0000","last_became_active":"2024-09-13T22:50:03.997081+0000","last_became_peered":"2024-09-13T22:50:03.997081+0000","last_unstale":"2024-09-13T22:50:17.957315+0000","last_undegraded":"2024-09-13T22:50:17.957315+0000","last_fullsized":"2024-09-13T22:50:17.957315+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:27.968957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:27.968957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:29.783547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"228'19","reported_seq":509,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957557+0000","last_change":"2024-09-13T22:50:04.045917+0000","last_active":"2024-09-13T22:50:17.957557+0000","last_peered":"2024-09-13T22:50:17.957557+0000","last_clean":"2024-09-13T22:50:17.957557+0000","last_became_active":"2024-09-13T22:50:04.002950+0000","last_became_peered":"2024-09-13T22:50:04.002950+0000","last_unstale":"2024-09-13T22:50:17.957557+0000","last_undegraded":"2024-09-13T22:50:17.957557+0000","last_fullsized":"2024-09-13T22:50:17.957557+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:18.321070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:18.321070+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":454,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.948878+0000","last_change":"2024-09-13T22:48:15.804011+0000","last_active":"2024-09-13T22:50:17.948878+0000","last_peered":"2024-09-13T22:50:17.948878+0000","last_clean":"2024-09-13T22:50:17.948878+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:50:17.948878+0000","last_undegraded":"2024-09-13T22:50:17.948878+0000","last_fullsized":"2024-09-13T22:50:17.948878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.803899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.803899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":611,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.950550+0000","last_change":"2024-09-13T22:48:04.929694+0000","last_active":"2024-09-13T22:50:17.950550+0000","last_peered":"2024-09-13T22:50:17.950550+0000","last_clean":"2024-09-13T22:50:17.950550+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:50:17.950550+0000","last_undegraded":"2024-09-13T22:50:17.950550+0000","last_fullsized":"2024-09-13T22:50:17.950550+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:04.929561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:04.929561+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":443,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943236+0000","last_change":"2024-09-13T22:49:42.161884+0000","last_active":"2024-09-13T22:50:17.943236+0000","last_peered":"2024-09-13T22:50:17.943236+0000","last_clean":"2024-09-13T22:50:17.943236+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:50:17.943236+0000","last_undegraded":"2024-09-13T22:50:17.943236+0000","last_fullsized":"2024-09-13T22:50:17.943236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:42.161805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:42.161805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:01.895853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"228'15","reported_seq":476,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942605+0000","last_change":"2024-09-13T22:49:38.143966+0000","last_active":"2024-09-13T22:50:17.942605+0000","last_peered":"2024-09-13T22:50:17.942605+0000","last_clean":"2024-09-13T22:50:17.942605+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:50:17.942605+0000","last_undegraded":"2024-09-13T22:50:17.942605+0000","last_fullsized":"2024-09-13T22:50:17.942605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:49:38.143760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:38.143760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:46.823665+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":467,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943344+0000","last_change":"2024-09-13T22:50:03.974441+0000","last_active":"2024-09-13T22:50:17.943344+0000","last_peered":"2024-09-13T22:50:17.943344+0000","last_clean":"2024-09-13T22:50:17.943344+0000","last_became_active":"2024-09-13T22:50:03.973419+0000","last_became_peered":"2024-09-13T22:50:03.973419+0000","last_unstale":"2024-09-13T22:50:17.943344+0000","last_undegraded":"2024-09-13T22:50:17.943344+0000","last_fullsized":"2024-09-13T22:50:17.943344+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:04.615328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:04.615328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611299+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":601,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963376+0000","last_change":"2024-09-13T22:45:48.537059+0000","last_active":"2024-09-13T22:50:17.963376+0000","last_peered":"2024-09-13T22:50:17.963376+0000","last_clean":"2024-09-13T22:50:17.963376+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:50:17.963376+0000","last_undegraded":"2024-09-13T22:50:17.963376+0000","last_fullsized":"2024-09-13T22:50:17.963376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:48.536952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:48.536952+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969898+0000","last_change":"2024-09-13T22:50:03.980831+0000","last_active":"2024-09-13T22:50:17.969898+0000","last_peered":"2024-09-13T22:50:17.969898+0000","last_clean":"2024-09-13T22:50:17.969898+0000","last_became_active":"2024-09-13T22:50:03.980403+0000","last_became_peered":"2024-09-13T22:50:03.980403+0000","last_unstale":"2024-09-13T22:50:17.969898+0000","last_undegraded":"2024-09-13T22:50:17.969898+0000","last_fullsized":"2024-09-13T22:50:17.969898+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:08.144897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.144897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"228'15","reported_seq":499,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958841+0000","last_change":"2024-09-13T22:50:04.041895+0000","last_active":"2024-09-13T22:50:17.958841+0000","last_peered":"2024-09-13T22:50:17.958841+0000","last_clean":"2024-09-13T22:50:17.958841+0000","last_became_active":"2024-09-13T22:50:03.997804+0000","last_became_peered":"2024-09-13T22:50:03.997804+0000","last_unstale":"2024-09-13T22:50:17.958841+0000","last_undegraded":"2024-09-13T22:50:17.958841+0000","last_fullsized":"2024-09-13T22:50:17.958841+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:21.238464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:21.238464+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945566+0000","last_change":"2024-09-13T22:48:36.942296+0000","last_active":"2024-09-13T22:50:17.945566+0000","last_peered":"2024-09-13T22:50:17.945566+0000","last_clean":"2024-09-13T22:50:17.945566+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:50:17.945566+0000","last_undegraded":"2024-09-13T22:50:17.945566+0000","last_fullsized":"2024-09-13T22:50:17.945566+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:36.942121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:36.942121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":439,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958418+0000","last_change":"2024-09-13T22:49:17.277840+0000","last_active":"2024-09-13T22:50:17.958418+0000","last_peered":"2024-09-13T22:50:17.958418+0000","last_clean":"2024-09-13T22:50:17.958418+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:50:17.958418+0000","last_undegraded":"2024-09-13T22:50:17.958418+0000","last_fullsized":"2024-09-13T22:50:17.958418+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.277656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.277656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.471913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.942959+0000","last_change":"2024-09-13T22:48:33.071864+0000","last_active":"2024-09-13T22:50:17.942959+0000","last_peered":"2024-09-13T22:50:17.942959+0000","last_clean":"2024-09-13T22:50:17.942959+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:50:17.942959+0000","last_undegraded":"2024-09-13T22:50:17.942959+0000","last_fullsized":"2024-09-13T22:50:17.942959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:33.071751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:33.071751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611322+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"228'20","reported_seq":37,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969659+0000","last_change":"2024-09-13T22:50:04.019803+0000","last_active":"2024-09-13T22:50:17.969659+0000","last_peered":"2024-09-13T22:50:17.969659+0000","last_clean":"2024-09-13T22:50:17.969659+0000","last_became_active":"2024-09-13T22:50:03.992294+0000","last_became_peered":"2024-09-13T22:50:03.992294+0000","last_unstale":"2024-09-13T22:50:17.969659+0000","last_undegraded":"2024-09-13T22:50:17.969659+0000","last_fullsized":"2024-09-13T22:50:17.969659+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:48:20.275130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.275130+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963775+0000","last_change":"2024-09-13T22:49:16.390793+0000","last_active":"2024-09-13T22:50:17.963775+0000","last_peered":"2024-09-13T22:50:17.963775+0000","last_clean":"2024-09-13T22:50:17.963775+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:50:17.963775+0000","last_undegraded":"2024-09-13T22:50:17.963775+0000","last_fullsized":"2024-09-13T22:50:17.963775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.389928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:03.849281+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.389928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.083913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":447,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963608+0000","last_change":"2024-09-13T22:48:38.188618+0000","last_active":"2024-09-13T22:50:17.963608+0000","last_peered":"2024-09-13T22:50:17.963608+0000","last_clean":"2024-09-13T22:50:17.963608+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:50:17.963608+0000","last_undegraded":"2024-09-13T22:50:17.963608+0000","last_fullsized":"2024-09-13T22:50:17.963608+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:38.188506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.188506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957809+0000","last_change":"2024-09-13T22:49:20.405583+0000","last_active":"2024-09-13T22:50:17.957809+0000","last_peered":"2024-09-13T22:50:17.957809+0000","last_clean":"2024-09-13T22:50:17.957809+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:50:17.957809+0000","last_undegraded":"2024-09-13T22:50:17.957809+0000","last_fullsized":"2024-09-13T22:50:17.957809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:20.405502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.405502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:41.635537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"228'12","reported_seq":469,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.944122+0000","last_change":"2024-09-13T22:48:32.051004+0000","last_active":"2024-09-13T22:50:17.944122+0000","last_peered":"2024-09-13T22:50:17.944122+0000","last_clean":"2024-09-13T22:50:17.944122+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:50:17.944122+0000","last_undegraded":"2024-09-13T22:50:17.944122+0000","last_fullsized":"2024-09-13T22:50:17.944122+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12","last_scrub_stamp":"2024-09-13T22:48:32.050878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.050878+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":454,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.964414+0000","last_change":"2024-09-13T22:49:28.371324+0000","last_active":"2024-09-13T22:50:17.964414+0000","last_peered":"2024-09-13T22:50:17.964414+0000","last_clean":"2024-09-13T22:50:17.964414+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:50:17.964414+0000","last_undegraded":"2024-09-13T22:50:17.964414+0000","last_fullsized":"2024-09-13T22:50:17.964414+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:28.371115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:49:28.371115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:36.022385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945662+0000","last_change":"2024-09-13T22:50:03.988511+0000","last_active":"2024-09-13T22:50:17.945662+0000","last_peered":"2024-09-13T22:50:17.945662+0000","last_clean":"2024-09-13T22:50:17.945662+0000","last_became_active":"2024-09-13T22:50:03.987270+0000","last_became_peered":"2024-09-13T22:50:03.987270+0000","last_unstale":"2024-09-13T22:50:17.945662+0000","last_undegraded":"2024-09-13T22:50:17.945662+0000","last_fullsized":"2024-09-13T22:50:17.945662+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:14.149187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.149187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957661+0000","last_change":"2024-09-13T22:50:03.996248+0000","last_active":"2024-09-13T22:50:17.957661+0000","last_peered":"2024-09-13T22:50:17.957661+0000","last_clean":"2024-09-13T22:50:17.957661+0000","last_became_active":"2024-09-13T22:50:03.995547+0000","last_became_peered":"2024-09-13T22:50:03.995547+0000","last_unstale":"2024-09-13T22:50:17.957661+0000","last_undegraded":"2024-09-13T22:50:17.957661+0000","last_fullsized":"2024-09-13T22:50:17.957661+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:15.907800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:15.907800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"228'15","reported_seq":475,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.951977+0000","last_change":"2024-09-13T22:49:24.436104+0000","last_active":"2024-09-13T22:50:17.951977+0000","last_peered":"2024-09-13T22:50:17.951977+0000","last_clean":"2024-09-13T22:50:17.951977+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:50:17.951977+0000","last_undegraded":"2024-09-13T22:50:17.951977+0000","last_fullsized":"2024-09-13T22:50:17.951977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:49:24.436026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:24.436026+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:33.631379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.946517+0000","last_change":"2024-09-13T22:48:13.617965+0000","last_active":"2024-09-13T22:50:17.946517+0000","last_peered":"2024-09-13T22:50:17.946517+0000","last_clean":"2024-09-13T22:50:17.946517+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:50:17.946517+0000","last_undegraded":"2024-09-13T22:50:17.946517+0000","last_fullsized":"2024-09-13T22:50:17.946517+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:13.617680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.617680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":450,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.961938+0000","last_change":"2024-09-13T22:50:03.995218+0000","last_active":"2024-09-13T22:50:17.961938+0000","last_peered":"2024-09-13T22:50:17.961938+0000","last_clean":"2024-09-13T22:50:17.961938+0000","last_became_active":"2024-09-13T22:50:03.994661+0000","last_became_peered":"2024-09-13T22:50:03.994661+0000","last_unstale":"2024-09-13T22:50:17.961938+0000","last_undegraded":"2024-09-13T22:50:17.961938+0000","last_fullsized":"2024-09-13T22:50:17.961938+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.934047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.934047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:19.169477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":435,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.958760+0000","last_change":"2024-09-13T22:45:40.486348+0000","last_active":"2024-09-13T22:50:17.958760+0000","last_peered":"2024-09-13T22:50:17.958760+0000","last_clean":"2024-09-13T22:50:17.958760+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:50:17.958760+0000","last_undegraded":"2024-09-13T22:50:17.958760+0000","last_fullsized":"2024-09-13T22:50:17.958760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:40.486247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:45:40.486247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":436,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957466+0000","last_change":"2024-09-13T22:48:29.166658+0000","last_active":"2024-09-13T22:50:17.957466+0000","last_peered":"2024-09-13T22:50:17.957466+0000","last_clean":"2024-09-13T22:50:17.957466+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:50:17.957466+0000","last_undegraded":"2024-09-13T22:50:17.957466+0000","last_fullsized":"2024-09-13T22:50:17.957466+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:48:29.166588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:29.166588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"228'34","reported_seq":545,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.956175+0000","last_change":"2024-09-13T22:50:04.026569+0000","last_active":"2024-09-13T22:50:17.956175+0000","last_peered":"2024-09-13T22:50:17.956175+0000","last_clean":"2024-09-13T22:50:17.956175+0000","last_became_active":"2024-09-13T22:50:03.978438+0000","last_became_peered":"2024-09-13T22:50:03.978438+0000","last_unstale":"2024-09-13T22:50:17.956175+0000","last_undegraded":"2024-09-13T22:50:17.956175+0000","last_fullsized":"2024-09-13T22:50:17.956175+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:28.140653+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.140653+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.947960+0000","last_change":"2024-09-13T22:50:03.976681+0000","last_active":"2024-09-13T22:50:17.947960+0000","last_peered":"2024-09-13T22:50:17.947960+0000","last_clean":"2024-09-13T22:50:17.947960+0000","last_became_active":"2024-09-13T22:50:03.975395+0000","last_became_peered":"2024-09-13T22:50:03.975395+0000","last_unstale":"2024-09-13T22:50:17.947960+0000","last_undegraded":"2024-09-13T22:50:17.947960+0000","last_fullsized":"2024-09-13T22:50:17.947960+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:12.849206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:48:12.849206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"266'14","reported_seq":642,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.946400+0000","last_change":"2024-09-13T22:50:04.017420+0000","last_active":"2024-09-13T22:50:17.946400+0000","last_peered":"2024-09-13T22:50:17.946400+0000","last_clean":"2024-09-13T22:50:17.946400+0000","last_became_active":"2024-09-13T22:50:03.985520+0000","last_became_peered":"2024-09-13T22:50:03.985520+0000","last_unstale":"2024-09-13T22:50:17.946400+0000","last_undegraded":"2024-09-13T22:50:17.946400+0000","last_fullsized":"2024-09-13T22:50:17.946400+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:34.950872+0000","last_deep_scrub":"172'12","last_deep_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.950872+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963473+0000","last_change":"2024-09-13T22:50:03.980316+0000","last_active":"2024-09-13T22:50:17.963473+0000","last_peered":"2024-09-13T22:50:17.963473+0000","last_clean":"2024-09-13T22:50:17.963473+0000","last_became_active":"2024-09-13T22:50:03.979831+0000","last_became_peered":"2024-09-13T22:50:03.979831+0000","last_unstale":"2024-09-13T22:50:17.963473+0000","last_undegraded":"2024-09-13T22:50:17.963473+0000","last_fullsized":"2024-09-13T22:50:17.963473+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.144743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.144743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"228'15","reported_seq":468,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.959292+0000","last_change":"2024-09-13T22:48:24.914421+0000","last_active":"2024-09-13T22:50:17.959292+0000","last_peered":"2024-09-13T22:50:17.959292+0000","last_clean":"2024-09-13T22:50:17.959292+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:50:17.959292+0000","last_undegraded":"2024-09-13T22:50:17.959292+0000","last_fullsized":"2024-09-13T22:50:17.959292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:24.913947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.913947+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":440,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.957933+0000","last_change":"2024-09-13T22:44:58.543023+0000","last_active":"2024-09-13T22:50:17.957933+0000","last_peered":"2024-09-13T22:50:17.957933+0000","last_clean":"2024-09-13T22:50:17.957933+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:50:17.957933+0000","last_undegraded":"2024-09-13T22:50:17.957933+0000","last_fullsized":"2024-09-13T22:50:17.957933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:58.542781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.542781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":448,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943329+0000","last_change":"2024-09-13T22:48:30.083820+0000","last_active":"2024-09-13T22:50:17.943329+0000","last_peered":"2024-09-13T22:50:17.943329+0000","last_clean":"2024-09-13T22:50:17.943329+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:50:17.943329+0000","last_undegraded":"2024-09-13T22:50:17.943329+0000","last_fullsized":"2024-09-13T22:50:17.943329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.083568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969900+0000","last_change":"2024-09-13T22:50:03.985020+0000","last_active":"2024-09-13T22:50:17.969900+0000","last_peered":"2024-09-13T22:50:17.969900+0000","last_clean":"2024-09-13T22:50:17.969900+0000","last_became_active":"2024-09-13T22:50:03.984625+0000","last_became_peered":"2024-09-13T22:50:03.984625+0000","last_unstale":"2024-09-13T22:50:17.969900+0000","last_undegraded":"2024-09-13T22:50:17.969900+0000","last_fullsized":"2024-09-13T22:50:17.969900+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:38.014231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:38.014231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:39.626531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":474,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.945999+0000","last_change":"2024-09-13T22:50:03.986647+0000","last_active":"2024-09-13T22:50:17.945999+0000","last_peered":"2024-09-13T22:50:17.945999+0000","last_clean":"2024-09-13T22:50:17.945999+0000","last_became_active":"2024-09-13T22:50:03.985970+0000","last_became_peered":"2024-09-13T22:50:03.985970+0000","last_unstale":"2024-09-13T22:50:17.945999+0000","last_undegraded":"2024-09-13T22:50:17.945999+0000","last_fullsized":"2024-09-13T22:50:17.945999+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.917904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.917904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"228'9","reported_seq":490,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.943181+0000","last_change":"2024-09-13T22:50:04.012414+0000","last_active":"2024-09-13T22:50:17.943181+0000","last_peered":"2024-09-13T22:50:17.943181+0000","last_clean":"2024-09-13T22:50:17.943181+0000","last_became_active":"2024-09-13T22:50:03.983134+0000","last_became_peered":"2024-09-13T22:50:03.983134+0000","last_unstale":"2024-09-13T22:50:17.943181+0000","last_undegraded":"2024-09-13T22:50:17.943181+0000","last_fullsized":"2024-09-13T22:50:17.943181+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-09-13T22:46:58.546268+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:46:58.546268+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:18.611213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":459,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.963721+0000","last_change":"2024-09-13T22:50:03.978676+0000","last_active":"2024-09-13T22:50:17.963721+0000","last_peered":"2024-09-13T22:50:17.963721+0000","last_clean":"2024-09-13T22:50:17.963721+0000","last_became_active":"2024-09-13T22:50:03.976594+0000","last_became_peered":"2024-09-13T22:50:03.976594+0000","last_unstale":"2024-09-13T22:50:17.963721+0000","last_undegraded":"2024-09-13T22:50:17.963721+0000","last_fullsized":"2024-09-13T22:50:17.963721+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:41.110195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:48:41.110195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":437,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.960692+0000","last_change":"2024-09-13T22:45:51.542656+0000","last_active":"2024-09-13T22:50:17.960692+0000","last_peered":"2024-09-13T22:50:17.960692+0000","last_clean":"2024-09-13T22:50:17.960692+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:50:17.960692+0000","last_undegraded":"2024-09-13T22:50:17.960692+0000","last_fullsized":"2024-09-13T22:50:17.960692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:45:51.542373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.542373+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":447,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.959024+0000","last_change":"2024-09-13T22:48:28.089740+0000","last_active":"2024-09-13T22:50:17.959024+0000","last_peered":"2024-09-13T22:50:17.959024+0000","last_clean":"2024-09-13T22:50:17.959024+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:50:17.959024+0000","last_undegraded":"2024-09-13T22:50:17.959024+0000","last_fullsized":"2024-09-13T22:50:17.959024+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:28.089548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.089548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"228'19","reported_seq":35,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969093+0000","last_change":"2024-09-13T22:50:04.012611+0000","last_active":"2024-09-13T22:50:17.969093+0000","last_peered":"2024-09-13T22:50:17.969093+0000","last_clean":"2024-09-13T22:50:17.969093+0000","last_became_active":"2024-09-13T22:50:03.971811+0000","last_became_peered":"2024-09-13T22:50:03.971811+0000","last_unstale":"2024-09-13T22:50:17.969093+0000","last_undegraded":"2024-09-13T22:50:17.969093+0000","last_fullsized":"2024-09-13T22:50:17.969093+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:49:21.457510+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.457510+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:50:25.846226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T22:50:17.969574+0000","last_change":"2024-09-13T22:50:03.980217+0000","last_active":"2024-09-13T22:50:17.969574+0000","last_peered":"2024-09-13T22:50:17.969574+0000","last_clean":"2024-09-13T22:50:17.969574+0000","last_became_active":"2024-09-13T22:50:03.979868+0000","last_became_peered":"2024-09-13T22:50:03.979868+0000","last_unstale":"2024-09-13T22:50:17.969574+0000","last_undegraded":"2024-09-13T22:50:17.969574+0000","last_fullsized":"2024-09-13T22:50:17.969574+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:50.912196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:45:50.912196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4259708928,"num_objects":64998,"num_object_clones":0,"num_object_copies":194994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7607,"num_objects_unfound":0,"num_objects_dirty":64998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64998,"num_write_kb":4159872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1837,"num_bytes_recovered":120389632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12850167808,"data_stored":12850167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64998,"ondisk_log_size":64998,"up":48,"acting":45,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":736,"num_read_kb":731,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":3960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":769,"ondisk_log_size":769,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762808992,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2594572,"kb_used_data":2278780,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91150580,"statfs":{"total":95995035648,"available":93338193920,"internally_reserved":0,"allocated":2333470720,"data_stored":2332966074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136043,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1051072,"kb_used_data":767212,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92694080,"statfs":{"total":95995035648,"available":94918737920,"internally_reserved":0,"allocated":785625088,"data_stored":785141468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528502,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2599496,"kb_used_data":2293428,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91145656,"statfs":{"total":95995035648,"available":93333151744,"internally_reserved":0,"allocated":2348470272,"data_stored":2347979406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888256,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461408,"kb_used_data":267792,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93283744,"statfs":{"total":95995035648,"available":95522553856,"internally_reserved":0,"allocated":274219008,"data_stored":273699456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084490,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2530564,"kb_used_data":2116212,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91214588,"statfs":{"total":95995035648,"available":93403738112,"internally_reserved":0,"allocated":2167001088,"data_stored":2166510643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378838,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052164,"kb_used_data":768308,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92692988,"statfs":{"total":95995035648,"available":94917619712,"internally_reserved":0,"allocated":786747392,"data_stored":786257040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2659408,"kb_used_data":2331132,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91085744,"statfs":{"total":95995035648,"available":93271801856,"internally_reserved":0,"allocated":2387079168,"data_stored":2386580475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738603,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1973396,"kb_used_data":1645120,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91771756,"statfs":{"total":95995035648,"available":93974278144,"internally_reserved":0,"allocated":1684602880,"data_stored":1684100851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1693843456,"data_stored":1693843456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2388066304,"data_stored":2388066304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":790626304,"data_stored":790626304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2165964800,"data_stored":2165964800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":280625152,"data_stored":280625152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2374631424,"data_stored":2374631424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":791412736,"data_stored":791412736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2364997632,"data_stored":2364997632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:50:24.383 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T22:50:24.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_2 pg_num 26 2024-09-13T22:50:24.403 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:24.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:24.564 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:24.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:24.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:24.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:24.835 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:24.853 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:24.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:24.980 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:24.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:25.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:25.130 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:25.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:25.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:25 smithi192 bash[24608]: audit 2024-09-13T22:50:24.476321+0000 mon.b (mon.1) 195 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:50:25.260 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:25.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 53 16 70367 70351 82.9445 54.875 0.020596 0.012051 2024-09-13T22:50:25.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:25.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:25 smithi178 bash[29889]: audit 2024-09-13T22:50:24.476321+0000 mon.b (mon.1) 195 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:50:25.374 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:25 smithi178 bash[22840]: audit 2024-09-13T22:50:24.476321+0000 mon.b (mon.1) 195 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:50:25.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:25.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:25.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:50:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:50:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:25.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:25.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:25.664 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:25.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:25.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:25.898 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:50:25.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:25.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:25.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:26.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:26.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:26.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:26.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:26.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:26.332 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 54 15 71380 71365 82.5821 63.375 0.0128379 0.0121032 2024-09-13T22:50:26.337 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:26.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:26 smithi192 bash[24608]: cluster 2024-09-13T22:50:24.636151+0000 osd.2 (osd.2) 205 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:50:26.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:26 smithi192 bash[24608]: cluster 2024-09-13T22:50:24.643546+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:50:26.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:26 smithi192 bash[24608]: cluster 2024-09-13T22:50:24.715566+0000 mgr.y (mgr.14613) 475 : cluster [DBG] pgmap v535: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1022 B/s rd, 61 MiB/s wr, 976 op/s; 7248/201921 objects misplaced (3.590%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T22:50:26.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:26 smithi192 bash[24608]: audit 2024-09-13T22:50:25.435961+0000 mgr.y (mgr.14613) 476 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[22840]: cluster 2024-09-13T22:50:24.636151+0000 osd.2 (osd.2) 205 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:50:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[22840]: cluster 2024-09-13T22:50:24.643546+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:50:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[22840]: cluster 2024-09-13T22:50:24.715566+0000 mgr.y (mgr.14613) 475 : cluster [DBG] pgmap v535: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1022 B/s rd, 61 MiB/s wr, 976 op/s; 7248/201921 objects misplaced (3.590%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T22:50:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[22840]: audit 2024-09-13T22:50:25.435961+0000 mgr.y (mgr.14613) 476 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:26.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[29889]: cluster 2024-09-13T22:50:24.636151+0000 osd.2 (osd.2) 205 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:50:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[29889]: cluster 2024-09-13T22:50:24.643546+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:50:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[29889]: cluster 2024-09-13T22:50:24.715566+0000 mgr.y (mgr.14613) 475 : cluster [DBG] pgmap v535: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1022 B/s rd, 61 MiB/s wr, 976 op/s; 7248/201921 objects misplaced (3.590%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T22:50:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:26 smithi178 bash[29889]: audit 2024-09-13T22:50:25.435961+0000 mgr.y (mgr.14613) 476 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:26.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:26.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:26.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:26.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:26.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:26.730 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:26.834 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:26.858 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:26.877 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:26.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:26.989 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:26.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:27.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:27.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:27.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:27.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:27 smithi192 bash[24608]: cluster 2024-09-13T22:50:26.145058+0000 osd.5 (osd.5) 281 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:50:27.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:27 smithi192 bash[24608]: cluster 2024-09-13T22:50:26.150239+0000 osd.5 (osd.5) 282 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:50:27.267 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:27.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 55 16 72322 72306 82.1498 58.8125 0.0535821 0.0121639 2024-09-13T22:50:27.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:27.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:27.510 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:27.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:27.580 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:27 smithi178 bash[22840]: cluster 2024-09-13T22:50:26.145058+0000 osd.5 (osd.5) 281 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:50:27.580 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:27 smithi178 bash[22840]: cluster 2024-09-13T22:50:26.150239+0000 osd.5 (osd.5) 282 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:50:27.580 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:27 smithi178 bash[29889]: cluster 2024-09-13T22:50:26.145058+0000 osd.5 (osd.5) 281 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:50:27.580 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:27 smithi178 bash[29889]: cluster 2024-09-13T22:50:26.150239+0000 osd.5 (osd.5) 282 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:50:27.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:27.657 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:27.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:27.788 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:27.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:27.920 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:27.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:28.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:28.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:28.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:28.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[29889]: cluster 2024-09-13T22:50:26.278133+0000 osd.6 (osd.6) 207 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:50:28.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[29889]: cluster 2024-09-13T22:50:26.282464+0000 osd.6 (osd.6) 208 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:50:28.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[29889]: cluster 2024-09-13T22:50:26.716824+0000 mgr.y (mgr.14613) 477 : cluster [DBG] pgmap v536: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1022 B/s rd, 59 MiB/s wr, 952 op/s; 6869/209973 objects misplaced (3.271%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T22:50:28.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[29889]: cluster 2024-09-13T22:50:27.107442+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:50:28.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[29889]: cluster 2024-09-13T22:50:27.114373+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:50:28.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[22840]: cluster 2024-09-13T22:50:26.278133+0000 osd.6 (osd.6) 207 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:50:28.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[22840]: cluster 2024-09-13T22:50:26.282464+0000 osd.6 (osd.6) 208 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:50:28.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[22840]: cluster 2024-09-13T22:50:26.716824+0000 mgr.y (mgr.14613) 477 : cluster [DBG] pgmap v536: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1022 B/s rd, 59 MiB/s wr, 952 op/s; 6869/209973 objects misplaced (3.271%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T22:50:28.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[22840]: cluster 2024-09-13T22:50:27.107442+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:50:28.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:28 smithi178 bash[22840]: cluster 2024-09-13T22:50:27.114373+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:50:28.194 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:28.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:28.322 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:28.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 56 16 73108 73092 81.5601 49.125 0.00257989 0.0122277 2024-09-13T22:50:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:28 smithi192 bash[24608]: cluster 2024-09-13T22:50:26.278133+0000 osd.6 (osd.6) 207 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:50:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:28 smithi192 bash[24608]: cluster 2024-09-13T22:50:26.282464+0000 osd.6 (osd.6) 208 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:50:28.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:28 smithi192 bash[24608]: cluster 2024-09-13T22:50:26.716824+0000 mgr.y (mgr.14613) 477 : cluster [DBG] pgmap v536: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1022 B/s rd, 59 MiB/s wr, 952 op/s; 6869/209973 objects misplaced (3.271%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T22:50:28.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:28 smithi192 bash[24608]: cluster 2024-09-13T22:50:27.107442+0000 osd.5 (osd.5) 283 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:50:28.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:28 smithi192 bash[24608]: cluster 2024-09-13T22:50:27.114373+0000 osd.5 (osd.5) 284 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:50:28.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:28.451 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:28.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:28.582 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:28.685 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:28.719 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:28.825 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:28.860 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:28.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:28.914 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:28.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:29.009 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:29.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:29.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:29.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:29.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 57 16 73731 73715 80.8123 38.9375 0.0226207 0.0123692 2024-09-13T22:50:29.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:29.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:29 smithi192 bash[24608]: cluster 2024-09-13T22:50:27.752964+0000 osd.0 (osd.0) 251 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:50:29.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:29 smithi192 bash[24608]: cluster 2024-09-13T22:50:27.757216+0000 osd.0 (osd.0) 252 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:50:29.411 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:29 smithi178 bash[29889]: cluster 2024-09-13T22:50:27.752964+0000 osd.0 (osd.0) 251 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:50:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:29 smithi178 bash[29889]: cluster 2024-09-13T22:50:27.757216+0000 osd.0 (osd.0) 252 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:50:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:29 smithi178 bash[22840]: cluster 2024-09-13T22:50:27.752964+0000 osd.0 (osd.0) 251 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:50:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:29 smithi178 bash[22840]: cluster 2024-09-13T22:50:27.757216+0000 osd.0 (osd.0) 252 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:50:29.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:29.557 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:29.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:29.678 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:29.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:29.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:29.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:29.921 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:29.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:30.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:30.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:30.151 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:50:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:50:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:50:30.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:30.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:28.468431+0000 osd.6 (osd.6) 209 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:50:30.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:28.469981+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:50:30.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:28.717492+0000 mgr.y (mgr.14613) 478 : cluster [DBG] pgmap v537: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 950 B/s rd, 55 MiB/s wr, 885 op/s; 6869/209973 objects misplaced (3.271%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:28.735334+0000 osd.0 (osd.0) 253 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:28.737658+0000 osd.0 (osd.0) 254 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:29.387944+0000 osd.7 (osd.7) 269 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: cluster 2024-09-13T22:50:29.388003+0000 osd.7 (osd.7) 270 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: audit 2024-09-13T22:50:29.832432+0000 mon.a (mon.0) 1331 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: audit 2024-09-13T22:50:29.838043+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: audit 2024-09-13T22:50:30.059438+0000 mon.a (mon.0) 1333 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[29889]: audit 2024-09-13T22:50:30.072649+0000 mon.a (mon.0) 1334 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:28.468431+0000 osd.6 (osd.6) 209 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:28.469981+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:28.717492+0000 mgr.y (mgr.14613) 478 : cluster [DBG] pgmap v537: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 950 B/s rd, 55 MiB/s wr, 885 op/s; 6869/209973 objects misplaced (3.271%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:28.735334+0000 osd.0 (osd.0) 253 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:28.737658+0000 osd.0 (osd.0) 254 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:29.387944+0000 osd.7 (osd.7) 269 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: cluster 2024-09-13T22:50:29.388003+0000 osd.7 (osd.7) 270 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: audit 2024-09-13T22:50:29.832432+0000 mon.a (mon.0) 1331 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: audit 2024-09-13T22:50:29.838043+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: audit 2024-09-13T22:50:30.059438+0000 mon.a (mon.0) 1333 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:30 smithi178 bash[22840]: audit 2024-09-13T22:50:30.072649+0000 mon.a (mon.0) 1334 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:30.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:30.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 58 16 74300 74284 80.0319 35.5625 0.0484311 0.012489 2024-09-13T22:50:30.334 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:30.439 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:30.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:30.469 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:28.468431+0000 osd.6 (osd.6) 209 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:50:30.469 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:28.469981+0000 osd.6 (osd.6) 210 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:50:30.469 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:28.717492+0000 mgr.y (mgr.14613) 478 : cluster [DBG] pgmap v537: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 950 B/s rd, 55 MiB/s wr, 885 op/s; 6869/209973 objects misplaced (3.271%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:28.735334+0000 osd.0 (osd.0) 253 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:28.737658+0000 osd.0 (osd.0) 254 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:29.387944+0000 osd.7 (osd.7) 269 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: cluster 2024-09-13T22:50:29.388003+0000 osd.7 (osd.7) 270 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: audit 2024-09-13T22:50:29.832432+0000 mon.a (mon.0) 1331 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: audit 2024-09-13T22:50:29.838043+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: audit 2024-09-13T22:50:30.059438+0000 mon.a (mon.0) 1333 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.470 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:30 smithi192 bash[24608]: audit 2024-09-13T22:50:30.072649+0000 mon.a (mon.0) 1334 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:30.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:30.609 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:30.689 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:50:30.690 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:50:30.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:30.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:30.806 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:50:30.807 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:50:30.844 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:50:30.845 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:50:30.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:30.863 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:50:30.864 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:50:30.879 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:30.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:30.918 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:50:30.918 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:50:30.949 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:50:30.949 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:50:30.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:30.989 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:50:30.989 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:50:31.008 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:31.027 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:50:31.028 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:50:31.078 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:50:31.078 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:50:31.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:31.142 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:31.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:31.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:31.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 59 15 75111 75096 79.5356 50.75 0.00813998 0.0125675 2024-09-13T22:50:31.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: cluster 2024-09-13T22:50:29.959157+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: cluster 2024-09-13T22:50:29.967358+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: cluster 2024-09-13T22:50:30.173890+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: cluster 2024-09-13T22:50:30.190339+0000 osd.5 (osd.5) 285 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: cluster 2024-09-13T22:50:30.194168+0000 osd.5 (osd.5) 286 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: audit 2024-09-13T22:50:30.775704+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.178:0/1165406756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: audit 2024-09-13T22:50:30.948904+0000 mon.b (mon.1) 196 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: audit 2024-09-13T22:50:30.950120+0000 mon.b (mon.1) 197 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:50:31.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:31 smithi192 bash[24608]: audit 2024-09-13T22:50:30.960224+0000 mon.a (mon.0) 1336 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:31.412 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:31.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: cluster 2024-09-13T22:50:29.959157+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:50:31.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: cluster 2024-09-13T22:50:29.967358+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: cluster 2024-09-13T22:50:30.173890+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: cluster 2024-09-13T22:50:30.190339+0000 osd.5 (osd.5) 285 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: cluster 2024-09-13T22:50:30.194168+0000 osd.5 (osd.5) 286 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: audit 2024-09-13T22:50:30.775704+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.178:0/1165406756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: audit 2024-09-13T22:50:30.948904+0000 mon.b (mon.1) 196 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: audit 2024-09-13T22:50:30.950120+0000 mon.b (mon.1) 197 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[29889]: audit 2024-09-13T22:50:30.960224+0000 mon.a (mon.0) 1336 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:31.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: cluster 2024-09-13T22:50:29.959157+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: cluster 2024-09-13T22:50:29.967358+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: cluster 2024-09-13T22:50:30.173890+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: cluster 2024-09-13T22:50:30.190339+0000 osd.5 (osd.5) 285 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: cluster 2024-09-13T22:50:30.194168+0000 osd.5 (osd.5) 286 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: audit 2024-09-13T22:50:30.775704+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.178:0/1165406756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: audit 2024-09-13T22:50:30.948904+0000 mon.b (mon.1) 196 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: audit 2024-09-13T22:50:30.950120+0000 mon.b (mon.1) 197 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:50:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:31 smithi178 bash[22840]: audit 2024-09-13T22:50:30.960224+0000 mon.a (mon.0) 1336 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:31.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:31.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:31.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:31.665 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:31.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:31.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:31.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T22:50:31.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:31.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:32.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:32.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:32.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:32 smithi192 bash[24608]: cluster 2024-09-13T22:50:30.718520+0000 mgr.y (mgr.14613) 479 : cluster [DBG] pgmap v539: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 6151/222384 objects misplaced (2.766%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:50:32.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:32.271 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:32.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:50:32.333060+0000 min lat: 0.00136126 max lat: 2.39712 avg lat: 0.0126487 2024-09-13T22:50:32.331 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:50:32.331 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 60 16 75897 75881 79.0277 49.0625 0.0129572 0.0126487 2024-09-13T22:50:32.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:32.403 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:32.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:32 smithi178 bash[29889]: cluster 2024-09-13T22:50:30.718520+0000 mgr.y (mgr.14613) 479 : cluster [DBG] pgmap v539: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 6151/222384 objects misplaced (2.766%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:50:32.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:32 smithi178 bash[22840]: cluster 2024-09-13T22:50:30.718520+0000 mgr.y (mgr.14613) 479 : cluster [DBG] pgmap v539: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 6151/222384 objects misplaced (2.766%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:50:32.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:32.540 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:32.645 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:32.669 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:32.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:32.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:32.904 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:32.941 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:32.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:33.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:33.084 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:33.190 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:33.218 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:33.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:33.330 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 61 15 76882 76867 78.7423 61.625 0.0146614 0.0126941 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[29889]: cluster 2024-09-13T22:50:32.237113+0000 osd.5 (osd.5) 287 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[29889]: cluster 2024-09-13T22:50:32.240521+0000 osd.5 (osd.5) 288 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[29889]: cluster 2024-09-13T22:50:32.754177+0000 osd.0 (osd.0) 257 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[29889]: cluster 2024-09-13T22:50:32.755465+0000 osd.0 (osd.0) 258 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[29889]: cluster 2024-09-13T22:50:32.942669+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[22840]: cluster 2024-09-13T22:50:32.237113+0000 osd.5 (osd.5) 287 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:50:33.337 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[22840]: cluster 2024-09-13T22:50:32.240521+0000 osd.5 (osd.5) 288 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:50:33.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[22840]: cluster 2024-09-13T22:50:32.754177+0000 osd.0 (osd.0) 257 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:50:33.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[22840]: cluster 2024-09-13T22:50:32.755465+0000 osd.0 (osd.0) 258 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:50:33.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:33 smithi178 bash[22840]: cluster 2024-09-13T22:50:32.942669+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-13T22:50:33.356 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:33.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:33.498 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:33.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:33.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:33.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:33 smithi192 bash[24608]: cluster 2024-09-13T22:50:32.237113+0000 osd.5 (osd.5) 287 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:50:33.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:33 smithi192 bash[24608]: cluster 2024-09-13T22:50:32.240521+0000 osd.5 (osd.5) 288 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:50:33.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:33 smithi192 bash[24608]: cluster 2024-09-13T22:50:32.754177+0000 osd.0 (osd.0) 257 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:50:33.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:33 smithi192 bash[24608]: cluster 2024-09-13T22:50:32.755465+0000 osd.0 (osd.0) 258 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:50:33.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:33 smithi192 bash[24608]: cluster 2024-09-13T22:50:32.942669+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-13T22:50:33.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:33.758 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:33.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:33.892 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:33.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:33.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:34.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:34.132 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:34.166 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:34.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:34.291 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:34.330 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[29889]: cluster 2024-09-13T22:50:32.719108+0000 mgr.y (mgr.14613) 480 : cluster [DBG] pgmap v540: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 6151/222384 objects misplaced (2.766%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:50:34.330 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[29889]: audit 2024-09-13T22:50:33.656291+0000 mon.b (mon.1) 198 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:34.330 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[29889]: cluster 2024-09-13T22:50:33.718667+0000 osd.0 (osd.0) 259 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:50:34.330 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[29889]: cluster 2024-09-13T22:50:33.724003+0000 osd.0 (osd.0) 260 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:50:34.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[22840]: cluster 2024-09-13T22:50:32.719108+0000 mgr.y (mgr.14613) 480 : cluster [DBG] pgmap v540: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 6151/222384 objects misplaced (2.766%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:50:34.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[22840]: audit 2024-09-13T22:50:33.656291+0000 mon.b (mon.1) 198 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:34.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[22840]: cluster 2024-09-13T22:50:33.718667+0000 osd.0 (osd.0) 259 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:50:34.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:34 smithi178 bash[22840]: cluster 2024-09-13T22:50:33.724003+0000 osd.0 (osd.0) 260 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:50:34.333 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 62 16 77741 77725 78.3339 53.625 0.00837488 0.0127465 2024-09-13T22:50:34.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:34.434 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:34.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:34 smithi192 bash[24608]: cluster 2024-09-13T22:50:32.719108+0000 mgr.y (mgr.14613) 480 : cluster [DBG] pgmap v540: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s; 6151/222384 objects misplaced (2.766%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:50:34.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:34 smithi192 bash[24608]: audit 2024-09-13T22:50:33.656291+0000 mon.b (mon.1) 198 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:34.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:34 smithi192 bash[24608]: cluster 2024-09-13T22:50:33.718667+0000 osd.0 (osd.0) 259 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:50:34.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:34 smithi192 bash[24608]: cluster 2024-09-13T22:50:33.724003+0000 osd.0 (osd.0) 260 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:50:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:34.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:34.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:34.673 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:34.704 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:34.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:34.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:34.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:34.963 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:34.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:35.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:35.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:35.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:35.230 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:35.333 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 63 16 78676 78660 78.018 58.4375 0.00826414 0.0128112 2024-09-13T22:50:35.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:35.344 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:35 smithi192 bash[24608]: cluster 2024-09-13T22:50:34.239737+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:50:35.344 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:35 smithi192 bash[24608]: cluster 2024-09-13T22:50:34.245980+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:50:35.367 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:35.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:35 smithi178 bash[29889]: cluster 2024-09-13T22:50:34.239737+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:50:35.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:35 smithi178 bash[29889]: cluster 2024-09-13T22:50:34.245980+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:50:35.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:35 smithi178 bash[22840]: cluster 2024-09-13T22:50:34.239737+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:50:35.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:35 smithi178 bash[22840]: cluster 2024-09-13T22:50:34.245980+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:50:35.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:35.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:35.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:35.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:50:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:50:35.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:35.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:35.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:35.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:35.901 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:35.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:36.005 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:36.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:36.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:36.169 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:36.202 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:50:36.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:36.283 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:36 smithi178 bash[29889]: cluster 2024-09-13T22:50:34.719893+0000 mgr.y (mgr.14613) 481 : cluster [DBG] pgmap v542: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 5727/227535 objects misplaced (2.517%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T22:50:36.283 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:36 smithi178 bash[29889]: audit 2024-09-13T22:50:35.447941+0000 mgr.y (mgr.14613) 482 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:36.283 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:36 smithi178 bash[29889]: audit 2024-09-13T22:50:36.094058+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? v1:172.21.15.178:0/637987545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:50:36.284 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:36 smithi178 bash[22840]: cluster 2024-09-13T22:50:34.719893+0000 mgr.y (mgr.14613) 481 : cluster [DBG] pgmap v542: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 5727/227535 objects misplaced (2.517%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T22:50:36.284 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:36 smithi178 bash[22840]: audit 2024-09-13T22:50:35.447941+0000 mgr.y (mgr.14613) 482 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:36.284 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:36 smithi178 bash[22840]: audit 2024-09-13T22:50:36.094058+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? v1:172.21.15.178:0/637987545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:50:36.308 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:36.333 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 64 16 79511 79495 77.6144 52.1875 0.00790166 0.0128789 2024-09-13T22:50:36.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:36.446 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:36.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:36 smithi192 bash[24608]: cluster 2024-09-13T22:50:34.719893+0000 mgr.y (mgr.14613) 481 : cluster [DBG] pgmap v542: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 5727/227535 objects misplaced (2.517%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T22:50:36.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:36 smithi192 bash[24608]: audit 2024-09-13T22:50:35.447941+0000 mgr.y (mgr.14613) 482 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:36.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:36 smithi192 bash[24608]: audit 2024-09-13T22:50:36.094058+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? v1:172.21.15.178:0/637987545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:50:36.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:36.584 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:36.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:36.720 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:36.755 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:50:36.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:36.857 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:36.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:36.986 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:36.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:37.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:37.115 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:37.213 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 9 pg_num to 26 2024-09-13T22:50:37.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:37.245 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:35.325606+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:50:37.245 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:35.328936+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:50:37.245 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.186735+0000 mon.a (mon.0) 1339 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:50:37.245 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.186757+0000 mon.a (mon.0) 1340 : cluster [INF] Cluster is now healthy 2024-09-13T22:50:37.245 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: audit 2024-09-13T22:50:36.200743+0000 mon.a (mon.0) 1341 : audit [INF] from='client.? v1:172.21.15.178:0/637987545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:50:37.245 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.213590+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.308846+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.321539+0000 osd.5 (osd.5) 292 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: audit 2024-09-13T22:50:36.326824+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.178:0/3455143371' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[29889]: audit 2024-09-13T22:50:36.328184+0000 mon.a (mon.0) 1343 : audit [INF] from='client.25367 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:35.325606+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:35.328936+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.186735+0000 mon.a (mon.0) 1339 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.186757+0000 mon.a (mon.0) 1340 : cluster [INF] Cluster is now healthy 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: audit 2024-09-13T22:50:36.200743+0000 mon.a (mon.0) 1341 : audit [INF] from='client.? v1:172.21.15.178:0/637987545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.213590+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-13T22:50:37.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.308846+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:50:37.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.321539+0000 osd.5 (osd.5) 292 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:50:37.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: audit 2024-09-13T22:50:36.326824+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.178:0/3455143371' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-09-13T22:50:37.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:37 smithi178 bash[22840]: audit 2024-09-13T22:50:36.328184+0000 mon.a (mon.0) 1343 : audit [INF] from='client.25367 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-09-13T22:50:37.250 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:37.333 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 65 16 80497 80481 77.3684 61.625 0.0103004 0.0129197 2024-09-13T22:50:37.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:37.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:37.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:37.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:37.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:50:37.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:37.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:37.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:35.325606+0000 osd.6 (osd.6) 211 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:35.328936+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.186735+0000 mon.a (mon.0) 1339 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.186757+0000 mon.a (mon.0) 1340 : cluster [INF] Cluster is now healthy 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: audit 2024-09-13T22:50:36.200743+0000 mon.a (mon.0) 1341 : audit [INF] from='client.? v1:172.21.15.178:0/637987545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.213590+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.308846+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.321539+0000 osd.5 (osd.5) 292 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: audit 2024-09-13T22:50:36.326824+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.178:0/3455143371' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-09-13T22:50:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:37 smithi192 bash[24608]: audit 2024-09-13T22:50:36.328184+0000 mon.a (mon.0) 1343 : audit [INF] from='client.25367 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-09-13T22:50:37.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:37.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:37.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:37.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:37.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:38.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:38.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:38.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:38.192 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:38.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:38.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:38.317 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:38.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 66 16 81172 81156 76.8353 42.1875 0.00570442 0.0130036 2024-09-13T22:50:38.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.333538+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.335127+0000 osd.6 (osd.6) 214 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: cluster 2024-09-13T22:50:36.720554+0000 mgr.y (mgr.14613) 483 : cluster [DBG] pgmap v544: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 56 MiB/s wr, 897 op/s; 5378/234858 objects misplaced (2.290%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: audit 2024-09-13T22:50:37.210633+0000 mon.a (mon.0) 1344 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: cluster 2024-09-13T22:50:37.214738+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: cluster 2024-09-13T22:50:37.290697+0000 osd.5 (osd.5) 293 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[29889]: cluster 2024-09-13T22:50:37.305804+0000 osd.5 (osd.5) 294 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.333538+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:50:38.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.335127+0000 osd.6 (osd.6) 214 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:50:38.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: cluster 2024-09-13T22:50:36.720554+0000 mgr.y (mgr.14613) 483 : cluster [DBG] pgmap v544: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 56 MiB/s wr, 897 op/s; 5378/234858 objects misplaced (2.290%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T22:50:38.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: audit 2024-09-13T22:50:37.210633+0000 mon.a (mon.0) 1344 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-13T22:50:38.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: cluster 2024-09-13T22:50:37.214738+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-13T22:50:38.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: cluster 2024-09-13T22:50:37.290697+0000 osd.5 (osd.5) 293 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:50:38.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:38 smithi178 bash[22840]: cluster 2024-09-13T22:50:37.305804+0000 osd.5 (osd.5) 294 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.333538+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.335127+0000 osd.6 (osd.6) 214 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: cluster 2024-09-13T22:50:36.720554+0000 mgr.y (mgr.14613) 483 : cluster [DBG] pgmap v544: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 56 MiB/s wr, 897 op/s; 5378/234858 objects misplaced (2.290%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: audit 2024-09-13T22:50:37.210633+0000 mon.a (mon.0) 1344 : audit [INF] from='client.25367 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: cluster 2024-09-13T22:50:37.214738+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: cluster 2024-09-13T22:50:37.290697+0000 osd.5 (osd.5) 293 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:50:38.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:38 smithi192 bash[24608]: cluster 2024-09-13T22:50:37.305804+0000 osd.5 (osd.5) 294 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:50:38.448 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:38.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:38.575 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:38.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:38.703 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:38.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:38.827 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:38.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:38.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:38.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:39.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:39.092 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:39.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:39.227 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:39.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:39.360 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 67 16 81943 81927 76.4077 48.1875 0.106288 0.0130655 2024-09-13T22:50:39.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:39 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.266100+0000 osd.5 (osd.5) 295 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:50:39.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:39 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.278048+0000 osd.5 (osd.5) 296 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:50:39.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:39 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.714261+0000 osd.0 (osd.0) 261 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:50:39.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:39 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.718321+0000 osd.0 (osd.0) 262 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:50:39.372 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:39 smithi192 bash[24608]: audit 2024-09-13T22:50:38.723701+0000 mon.b (mon.1) 199 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:39.372 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:39 smithi192 bash[24608]: audit 2024-09-13T22:50:38.725415+0000 mon.a (mon.0) 1346 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:39.401 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:39.503 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.266100+0000 osd.5 (osd.5) 295 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.278048+0000 osd.5 (osd.5) 296 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.714261+0000 osd.0 (osd.0) 261 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.718321+0000 osd.0 (osd.0) 262 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[29889]: audit 2024-09-13T22:50:38.723701+0000 mon.b (mon.1) 199 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[29889]: audit 2024-09-13T22:50:38.725415+0000 mon.a (mon.0) 1346 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.266100+0000 osd.5 (osd.5) 295 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.278048+0000 osd.5 (osd.5) 296 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:50:39.511 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.714261+0000 osd.0 (osd.0) 261 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:50:39.512 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.718321+0000 osd.0 (osd.0) 262 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:50:39.512 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[22840]: audit 2024-09-13T22:50:38.723701+0000 mon.b (mon.1) 199 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:39.512 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[22840]: audit 2024-09-13T22:50:38.725415+0000 mon.a (mon.0) 1346 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:39.521 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:39.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:39.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:39.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:39.770 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:39.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:39.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:39.897 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:39.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:40.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:40.021 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:40.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:40.135 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:50:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:50:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:50:40.150 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:40.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.420162+0000 osd.6 (osd.6) 215 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.438623+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.721004+0000 mgr.y (mgr.14613) 484 : cluster [DBG] pgmap v546: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 43 MiB/s wr, 693 op/s; 5378/234858 objects misplaced (2.290%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.814512+0000 osd.0 (osd.0) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:38.814720+0000 osd.0 (osd.0) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: audit 2024-09-13T22:50:39.228849+0000 mon.a (mon.0) 1347 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:39.232917+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-13T22:50:40.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:39.638748+0000 osd.2 (osd.2) 207 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:50:40.266 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:40 smithi192 bash[24608]: cluster 2024-09-13T22:50:39.640881+0000 osd.2 (osd.2) 208 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:50:40.286 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:40.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 68 16 82000 81984 75.3366 3.5625 0.00356332 0.0132285 2024-09-13T22:50:40.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.420162+0000 osd.6 (osd.6) 215 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:50:40.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.438623+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:50:40.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.721004+0000 mgr.y (mgr.14613) 484 : cluster [DBG] pgmap v546: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 43 MiB/s wr, 693 op/s; 5378/234858 objects misplaced (2.290%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T22:50:40.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.814512+0000 osd.0 (osd.0) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:40.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:38.814720+0000 osd.0 (osd.0) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:40.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: audit 2024-09-13T22:50:39.228849+0000 mon.a (mon.0) 1347 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:39.232917+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:39.638748+0000 osd.2 (osd.2) 207 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[29889]: cluster 2024-09-13T22:50:39.640881+0000 osd.2 (osd.2) 208 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.420162+0000 osd.6 (osd.6) 215 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.438623+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.721004+0000 mgr.y (mgr.14613) 484 : cluster [DBG] pgmap v546: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 43 MiB/s wr, 693 op/s; 5378/234858 objects misplaced (2.290%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.814512+0000 osd.0 (osd.0) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:38.814720+0000 osd.0 (osd.0) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: audit 2024-09-13T22:50:39.228849+0000 mon.a (mon.0) 1347 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:39.232917+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-13T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:39.638748+0000 osd.2 (osd.2) 207 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:50:40.394 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:40 smithi178 bash[22840]: cluster 2024-09-13T22:50:39.640881+0000 osd.2 (osd.2) 208 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:50:40.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:40.418 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:40.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:40.544 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:40.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:40.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:40.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:40.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:40.792 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:40.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:40.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:40.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:41.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:41.041 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:41.078 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:50:41.078 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:50:41.079 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-09-13T22:50:41.079 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:50:41.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:41.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:50:41.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:41.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:41.285 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:41.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 69 16 82319 82303 74.5338 19.9375 0.549266 0.0134059 2024-09-13T22:50:41.391 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:41 smithi192 bash[24608]: cluster 2024-09-13T22:50:40.234315+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-13T22:50:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:41 smithi192 bash[24608]: cluster 2024-09-13T22:50:40.705755+0000 osd.0 (osd.0) 265 : cluster [DBG] 9.9 scrub starts 2024-09-13T22:50:41.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:41 smithi192 bash[24608]: audit 2024-09-13T22:50:40.722992+0000 mon.a (mon.0) 1350 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:41.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:41 smithi192 bash[24608]: audit 2024-09-13T22:50:40.724362+0000 mon.b (mon.1) 200 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:41.412 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:41.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:41.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[29889]: cluster 2024-09-13T22:50:40.234315+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-13T22:50:41.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[29889]: cluster 2024-09-13T22:50:40.705755+0000 osd.0 (osd.0) 265 : cluster [DBG] 9.9 scrub starts 2024-09-13T22:50:41.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[29889]: audit 2024-09-13T22:50:40.722992+0000 mon.a (mon.0) 1350 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:41.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[29889]: audit 2024-09-13T22:50:40.724362+0000 mon.b (mon.1) 200 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:41.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[22840]: cluster 2024-09-13T22:50:40.234315+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-13T22:50:41.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[22840]: cluster 2024-09-13T22:50:40.705755+0000 osd.0 (osd.0) 265 : cluster [DBG] 9.9 scrub starts 2024-09-13T22:50:41.524 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[22840]: audit 2024-09-13T22:50:40.722992+0000 mon.a (mon.0) 1350 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:41.524 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:41 smithi178 bash[22840]: audit 2024-09-13T22:50:40.724362+0000 mon.b (mon.1) 200 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-13T22:50:41.534 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:41.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:41.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:41.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:41.784 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:41.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:41.913 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:41.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:42.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:42.036 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:42.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:42.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:42.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T22:50:42.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:42.287 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:40.391179+0000 osd.6 (osd.6) 217 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:40.392236+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:40.721691+0000 mgr.y (mgr.14613) 485 : cluster [DBG] pgmap v549: 168 pgs: 1 peering, 1 remapped+peering, 5 unknown, 2 active+recovering+undersized+remapped, 159 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 838 op/s; 2363/246207 objects misplaced (0.960%); 14 MiB/s, 229 objects/s recovering 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.028781+0000 osd.0 (osd.0) 266 : cluster [DBG] 9.9 scrub ok 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: audit 2024-09-13T22:50:41.235117+0000 mon.a (mon.0) 1351 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.241755+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.293488+0000 osd.5 (osd.5) 297 : cluster [DBG] 9.7 scrub starts 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.684620+0000 osd.5 (osd.5) 298 : cluster [DBG] 9.7 scrub ok 2024-09-13T22:50:42.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.699743+0000 osd.0 (osd.0) 267 : cluster [DBG] 9.5 deep-scrub starts 2024-09-13T22:50:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:40.391179+0000 osd.6 (osd.6) 217 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:40.392236+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:40.721691+0000 mgr.y (mgr.14613) 485 : cluster [DBG] pgmap v549: 168 pgs: 1 peering, 1 remapped+peering, 5 unknown, 2 active+recovering+undersized+remapped, 159 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 838 op/s; 2363/246207 objects misplaced (0.960%); 14 MiB/s, 229 objects/s recovering 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.028781+0000 osd.0 (osd.0) 266 : cluster [DBG] 9.9 scrub ok 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: audit 2024-09-13T22:50:41.235117+0000 mon.a (mon.0) 1351 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.241755+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.293488+0000 osd.5 (osd.5) 297 : cluster [DBG] 9.7 scrub starts 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.684620+0000 osd.5 (osd.5) 298 : cluster [DBG] 9.7 scrub ok 2024-09-13T22:50:42.291 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:42 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.699743+0000 osd.0 (osd.0) 267 : cluster [DBG] 9.5 deep-scrub starts 2024-09-13T22:50:42.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 70 16 82348 82332 73.4951 1.8125 0.00633655 0.0134517 2024-09-13T22:50:42.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:40.391179+0000 osd.6 (osd.6) 217 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:40.392236+0000 osd.6 (osd.6) 218 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:40.721691+0000 mgr.y (mgr.14613) 485 : cluster [DBG] pgmap v549: 168 pgs: 1 peering, 1 remapped+peering, 5 unknown, 2 active+recovering+undersized+remapped, 159 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 838 op/s; 2363/246207 objects misplaced (0.960%); 14 MiB/s, 229 objects/s recovering 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.028781+0000 osd.0 (osd.0) 266 : cluster [DBG] 9.9 scrub ok 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: audit 2024-09-13T22:50:41.235117+0000 mon.a (mon.0) 1351 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.241755+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.293488+0000 osd.5 (osd.5) 297 : cluster [DBG] 9.7 scrub starts 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.684620+0000 osd.5 (osd.5) 298 : cluster [DBG] 9.7 scrub ok 2024-09-13T22:50:42.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:42 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.699743+0000 osd.0 (osd.0) 267 : cluster [DBG] 9.5 deep-scrub starts 2024-09-13T22:50:42.419 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:42.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:42.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:42.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:42.672 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:42.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:42.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:42.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:42.921 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:42.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:43.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:43.045 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:43.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:43.170 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:43.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.422442+0000 osd.6 (osd.6) 219 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:41.423513+0000 osd.6 (osd.6) 220 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:42.239759+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:42.281790+0000 osd.5 (osd.5) 299 : cluster [DBG] 9.6 scrub starts 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:42.637260+0000 osd.5 (osd.5) 300 : cluster [DBG] 9.6 scrub ok 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:42.691982+0000 osd.0 (osd.0) 268 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:42.692960+0000 osd.0 (osd.0) 269 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:50:43.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:43 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.240798+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-13T22:50:43.329 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:43.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 71 16 82369 82353 72.4785 1.3125 0.0152618 0.0135601 2024-09-13T22:50:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:43.411 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:43.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.422442+0000 osd.6 (osd.6) 219 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:41.423513+0000 osd.6 (osd.6) 220 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:42.239759+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:42.281790+0000 osd.5 (osd.5) 299 : cluster [DBG] 9.6 scrub starts 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:42.637260+0000 osd.5 (osd.5) 300 : cluster [DBG] 9.6 scrub ok 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:42.691982+0000 osd.0 (osd.0) 268 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:42.692960+0000 osd.0 (osd.0) 269 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.240798+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-13T22:50:43.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.422442+0000 osd.6 (osd.6) 219 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:41.423513+0000 osd.6 (osd.6) 220 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:42.239759+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:42.281790+0000 osd.5 (osd.5) 299 : cluster [DBG] 9.6 scrub starts 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:42.637260+0000 osd.5 (osd.5) 300 : cluster [DBG] 9.6 scrub ok 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:42.691982+0000 osd.0 (osd.0) 268 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:42.692960+0000 osd.0 (osd.0) 269 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:50:43.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:43 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.240798+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-13T22:50:43.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:43.465 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:43.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:43.603 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:43.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:43.735 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:43.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:43.873 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:43.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:44.005 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:44.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:44.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:44.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.276 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:44.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 72 16 83255 83239 72.241 55.375 0.0180427 0.0138376 2024-09-13T22:50:44.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:44.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:42.722217+0000 mgr.y (mgr.14613) 486 : cluster [DBG] pgmap v552: 168 pgs: 1 peering, 1 remapped+peering, 5 unknown, 2 active+recovering+undersized+remapped, 159 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 946 op/s; 2363/246207 objects misplaced (0.960%); 16 MiB/s, 259 objects/s recovering 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.239084+0000 osd.0 (osd.0) 270 : cluster [DBG] 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5186 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.239644+0000 osd.0 (osd.0) 271 : cluster [DBG] 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 277'5136 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.242181+0000 osd.1 (osd.1) 255 : cluster [DBG] 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 277'5040 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.242758+0000 osd.1 (osd.1) 256 : cluster [DBG] 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 277'5210 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.242847+0000 osd.7 (osd.7) 271 : cluster [DBG] 9.15 starting backfill to osd.1 from (0'0,0'0] MAX to 277'5132 2024-09-13T22:50:44.523 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.243355+0000 osd.7 (osd.7) 272 : cluster [DBG] 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 277'5113 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.245662+0000 osd.0 (osd.0) 272 : cluster [DBG] 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 277'5186 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.245943+0000 osd.7 (osd.7) 273 : cluster [DBG] 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 277'5178 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.253038+0000 osd.7 (osd.7) 274 : cluster [DBG] 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 277'5132 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.253400+0000 osd.7 (osd.7) 275 : cluster [DBG] 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 277'5178 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.253671+0000 osd.7 (osd.7) 276 : cluster [DBG] 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5113 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.259906+0000 osd.0 (osd.0) 273 : cluster [DBG] 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5136 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.262287+0000 osd.1 (osd.1) 257 : cluster [DBG] 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 277'5040 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.273630+0000 osd.5 (osd.5) 301 : cluster [DBG] 1.a scrub starts 2024-09-13T22:50:44.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:44 smithi192 bash[24608]: cluster 2024-09-13T22:50:43.282591+0000 osd.5 (osd.5) 302 : cluster [DBG] 1.a scrub ok 2024-09-13T22:50:44.538 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:44.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.674 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:44.679 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:42.722217+0000 mgr.y (mgr.14613) 486 : cluster [DBG] pgmap v552: 168 pgs: 1 peering, 1 remapped+peering, 5 unknown, 2 active+recovering+undersized+remapped, 159 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 946 op/s; 2363/246207 objects misplaced (0.960%); 16 MiB/s, 259 objects/s recovering 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.239084+0000 osd.0 (osd.0) 270 : cluster [DBG] 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5186 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.239644+0000 osd.0 (osd.0) 271 : cluster [DBG] 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 277'5136 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.242181+0000 osd.1 (osd.1) 255 : cluster [DBG] 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 277'5040 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.242758+0000 osd.1 (osd.1) 256 : cluster [DBG] 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 277'5210 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.242847+0000 osd.7 (osd.7) 271 : cluster [DBG] 9.15 starting backfill to osd.1 from (0'0,0'0] MAX to 277'5132 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.243355+0000 osd.7 (osd.7) 272 : cluster [DBG] 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 277'5113 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.245662+0000 osd.0 (osd.0) 272 : cluster [DBG] 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 277'5186 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.245943+0000 osd.7 (osd.7) 273 : cluster [DBG] 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 277'5178 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.253038+0000 osd.7 (osd.7) 274 : cluster [DBG] 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 277'5132 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.253400+0000 osd.7 (osd.7) 275 : cluster [DBG] 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 277'5178 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.253671+0000 osd.7 (osd.7) 276 : cluster [DBG] 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5113 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.259906+0000 osd.0 (osd.0) 273 : cluster [DBG] 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5136 2024-09-13T22:50:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.262287+0000 osd.1 (osd.1) 257 : cluster [DBG] 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 277'5040 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.273630+0000 osd.5 (osd.5) 301 : cluster [DBG] 1.a scrub starts 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[29889]: cluster 2024-09-13T22:50:43.282591+0000 osd.5 (osd.5) 302 : cluster [DBG] 1.a scrub ok 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:42.722217+0000 mgr.y (mgr.14613) 486 : cluster [DBG] pgmap v552: 168 pgs: 1 peering, 1 remapped+peering, 5 unknown, 2 active+recovering+undersized+remapped, 159 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 946 op/s; 2363/246207 objects misplaced (0.960%); 16 MiB/s, 259 objects/s recovering 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.239084+0000 osd.0 (osd.0) 270 : cluster [DBG] 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5186 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.239644+0000 osd.0 (osd.0) 271 : cluster [DBG] 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 277'5136 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.242181+0000 osd.1 (osd.1) 255 : cluster [DBG] 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 277'5040 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.242758+0000 osd.1 (osd.1) 256 : cluster [DBG] 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 277'5210 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.242847+0000 osd.7 (osd.7) 271 : cluster [DBG] 9.15 starting backfill to osd.1 from (0'0,0'0] MAX to 277'5132 2024-09-13T22:50:44.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.243355+0000 osd.7 (osd.7) 272 : cluster [DBG] 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 277'5113 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.245662+0000 osd.0 (osd.0) 272 : cluster [DBG] 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 277'5186 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.245943+0000 osd.7 (osd.7) 273 : cluster [DBG] 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 277'5178 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.253038+0000 osd.7 (osd.7) 274 : cluster [DBG] 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 277'5132 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.253400+0000 osd.7 (osd.7) 275 : cluster [DBG] 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 277'5178 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.253671+0000 osd.7 (osd.7) 276 : cluster [DBG] 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5113 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.259906+0000 osd.0 (osd.0) 273 : cluster [DBG] 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 277'5136 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.262287+0000 osd.1 (osd.1) 257 : cluster [DBG] 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 277'5040 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.273630+0000 osd.5 (osd.5) 301 : cluster [DBG] 1.a scrub starts 2024-09-13T22:50:44.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:44 smithi178 bash[22840]: cluster 2024-09-13T22:50:43.282591+0000 osd.5 (osd.5) 302 : cluster [DBG] 1.a scrub ok 2024-09-13T22:50:44.777 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:44.801 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:44.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:44.937 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:44.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:45.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:45.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:45.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:45.204 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:45.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:45.330 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:45.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:45.335 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 73 16 84167 84151 72.0321 57 0.0170838 0.0138767 2024-09-13T22:50:45.395 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:50:45.395 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:50:45.395 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T22:50:45.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T22:50:45.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:45.453 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:45.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:45 smithi192 bash[24608]: cluster 2024-09-13T22:50:45.265124+0000 mon.a (mon.0) 1355 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:50:45.480 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:45.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:45.594 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:50:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:50:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:45.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:45 smithi178 bash[22840]: cluster 2024-09-13T22:50:45.265124+0000 mon.a (mon.0) 1355 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:50:45.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:45 smithi178 bash[29889]: cluster 2024-09-13T22:50:45.265124+0000 mon.a (mon.0) 1355 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:50:45.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:45.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:45.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:45.881 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:45.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:45.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:46.004 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:46.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:46.132 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:46.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:46.276 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:46.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 74 16 84918 84902 71.693 46.9375 0.0157159 0.013943 2024-09-13T22:50:46.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:46.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:46.556 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:46.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:46 smithi192 bash[24608]: cluster 2024-09-13T22:50:44.723092+0000 mgr.y (mgr.14613) 487 : cluster [DBG] pgmap v554: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 peering, 1 remapped+peering, 3 unknown, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 159 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17327/240042 objects misplaced (7.218%); 11 MiB/s, 176 objects/s recovering 2024-09-13T22:50:46.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:46 smithi192 bash[24608]: audit 2024-09-13T22:50:45.458552+0000 mgr.y (mgr.14613) 488 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:46.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:46 smithi192 bash[24608]: cluster 2024-09-13T22:50:45.629737+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:50:46.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:46 smithi192 bash[24608]: cluster 2024-09-13T22:50:45.633086+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:50:46.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[29889]: cluster 2024-09-13T22:50:44.723092+0000 mgr.y (mgr.14613) 487 : cluster [DBG] pgmap v554: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 peering, 1 remapped+peering, 3 unknown, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 159 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17327/240042 objects misplaced (7.218%); 11 MiB/s, 176 objects/s recovering 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[29889]: audit 2024-09-13T22:50:45.458552+0000 mgr.y (mgr.14613) 488 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[29889]: cluster 2024-09-13T22:50:45.629737+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[29889]: cluster 2024-09-13T22:50:45.633086+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[22840]: cluster 2024-09-13T22:50:44.723092+0000 mgr.y (mgr.14613) 487 : cluster [DBG] pgmap v554: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 peering, 1 remapped+peering, 3 unknown, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 159 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17327/240042 objects misplaced (7.218%); 11 MiB/s, 176 objects/s recovering 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[22840]: audit 2024-09-13T22:50:45.458552+0000 mgr.y (mgr.14613) 488 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[22840]: cluster 2024-09-13T22:50:45.629737+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:50:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:46 smithi178 bash[22840]: cluster 2024-09-13T22:50:45.633086+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:50:46.685 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:46.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:46.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:46.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:46.961 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:46.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:47.056 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:50:47.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:47.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:47.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:47.229 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:47.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:47.334 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 75 16 85736 85720 71.4187 51.125 0.018372 0.0139944 2024-09-13T22:50:47.375 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:47.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.511 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:47.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:47.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:47.648 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:47.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:47.787 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:47.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:47.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:47.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:48.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:48.053 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:48.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:48.205 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:48.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:48.337 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 76 16 86576 86560 71.168 52.5 0.0314229 0.0140446 2024-09-13T22:50:48.339 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:48.443 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:48.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:48.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:48.605 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:48 smithi192 bash[24608]: cluster 2024-09-13T22:50:46.724156+0000 mgr.y (mgr.14613) 489 : cluster [DBG] pgmap v555: 168 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 1 active+recovering, 2 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 42895/251058 objects misplaced (17.086%); 31 MiB/s, 492 objects/s recovering 2024-09-13T22:50:48.605 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:48 smithi192 bash[24608]: cluster 2024-09-13T22:50:47.947347+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-13T22:50:48.624 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:48 smithi178 bash[29889]: cluster 2024-09-13T22:50:46.724156+0000 mgr.y (mgr.14613) 489 : cluster [DBG] pgmap v555: 168 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 1 active+recovering, 2 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 42895/251058 objects misplaced (17.086%); 31 MiB/s, 492 objects/s recovering 2024-09-13T22:50:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:48 smithi178 bash[29889]: cluster 2024-09-13T22:50:47.947347+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-13T22:50:48.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:48 smithi178 bash[22840]: cluster 2024-09-13T22:50:46.724156+0000 mgr.y (mgr.14613) 489 : cluster [DBG] pgmap v555: 168 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 1 active+recovering, 2 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 42895/251058 objects misplaced (17.086%); 31 MiB/s, 492 objects/s recovering 2024-09-13T22:50:48.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:48 smithi178 bash[22840]: cluster 2024-09-13T22:50:47.947347+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-13T22:50:48.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:48.755 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:48.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:48.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:48.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:48.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:49.018 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:49.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:49.153 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:49.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:49.285 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:49.336 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 77 16 87472 87456 70.971 56 0.0168591 0.0140823 2024-09-13T22:50:49.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:49.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:49.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:49.534 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:50:49.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:49.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:49.690 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:49.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:49.828 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:49.929 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:50:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:50:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:50:49.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:49 smithi178 bash[29889]: audit 2024-09-13T22:50:48.671637+0000 mon.a (mon.0) 1357 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:49.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:49 smithi178 bash[29889]: audit 2024-09-13T22:50:48.677487+0000 mon.b (mon.1) 201 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:49.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:49 smithi178 bash[22840]: audit 2024-09-13T22:50:48.671637+0000 mon.a (mon.0) 1357 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:49.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:49 smithi178 bash[22840]: audit 2024-09-13T22:50:48.677487+0000 mon.b (mon.1) 201 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:49.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:49.943 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:49 smithi192 bash[24608]: audit 2024-09-13T22:50:48.671637+0000 mon.a (mon.0) 1357 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:50:49.943 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:49 smithi192 bash[24608]: audit 2024-09-13T22:50:48.677487+0000 mon.b (mon.1) 201 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:50:49.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:49.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:50.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:50.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:50.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:50.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:50.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:50.337 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 78 16 88165 88149 70.6155 43.3125 0.00841385 0.014154 2024-09-13T22:50:50.355 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:50.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:50.485 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:50.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:50.615 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:50.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:50.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:50.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:50.882 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:50.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:50.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:50 smithi178 bash[29889]: cluster 2024-09-13T22:50:48.724759+0000 mgr.y (mgr.14613) 490 : cluster [DBG] pgmap v557: 168 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 1 active+recovering, 2 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 789 B/s rd, 0 op/s; 42895/251058 objects misplaced (17.086%); 28 MiB/s, 455 objects/s recovering 2024-09-13T22:50:50.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:50 smithi178 bash[22840]: cluster 2024-09-13T22:50:48.724759+0000 mgr.y (mgr.14613) 490 : cluster [DBG] pgmap v557: 168 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 1 active+recovering, 2 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 789 B/s rd, 0 op/s; 42895/251058 objects misplaced (17.086%); 28 MiB/s, 455 objects/s recovering 2024-09-13T22:50:50.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:50.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:50 smithi192 bash[24608]: cluster 2024-09-13T22:50:48.724759+0000 mgr.y (mgr.14613) 490 : cluster [DBG] pgmap v557: 168 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 1 active+recovering, 2 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 789 B/s rd, 0 op/s; 42895/251058 objects misplaced (17.086%); 28 MiB/s, 455 objects/s recovering 2024-09-13T22:50:51.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:51.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:51.155 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:51.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:51.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:51.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 79 16 88954 88938 70.3459 49.3125 0.0144078 0.0142106 2024-09-13T22:50:51.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:51.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:51.557 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:51.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:51.697 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:51.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:51.830 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:51.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:51.961 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:51.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:52.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:52.110 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:52.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:52.242 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:52.274 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:50:52.274 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:50:52.274 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:50:52.274 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:50:52.275 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:50:52.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:50:52.340241+0000 min lat: 0.00136126 max lat: 2.39712 avg lat: 0.0142328 2024-09-13T22:50:52.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:50:52.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 80 16 89936 89920 70.2337 61.375 0.0150704 0.0142328 2024-09-13T22:50:52.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:52.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:52.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:52.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:52.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:52.633 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:52.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:52.747 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:52 smithi178 bash[22840]: cluster 2024-09-13T22:50:50.725608+0000 mgr.y (mgr.14613) 491 : cluster [DBG] pgmap v558: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39785/263475 objects misplaced (15.100%); 45 MiB/s, 712 objects/s recovering 2024-09-13T22:50:52.747 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:52 smithi178 bash[22840]: cluster 2024-09-13T22:50:51.032828+0000 osd.0 (osd.0) 274 : cluster [DBG] 9.5 deep-scrub ok 2024-09-13T22:50:52.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:52 smithi178 bash[29889]: cluster 2024-09-13T22:50:50.725608+0000 mgr.y (mgr.14613) 491 : cluster [DBG] pgmap v558: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39785/263475 objects misplaced (15.100%); 45 MiB/s, 712 objects/s recovering 2024-09-13T22:50:52.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:52 smithi178 bash[29889]: cluster 2024-09-13T22:50:51.032828+0000 osd.0 (osd.0) 274 : cluster [DBG] 9.5 deep-scrub ok 2024-09-13T22:50:52.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:52.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:52.899 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:52.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:53.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:53.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:53.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:53.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:52 smithi192 bash[24608]: cluster 2024-09-13T22:50:50.725608+0000 mgr.y (mgr.14613) 491 : cluster [DBG] pgmap v558: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39785/263475 objects misplaced (15.100%); 45 MiB/s, 712 objects/s recovering 2024-09-13T22:50:53.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:52 smithi192 bash[24608]: cluster 2024-09-13T22:50:51.032828+0000 osd.0 (osd.0) 274 : cluster [DBG] 9.5 deep-scrub ok 2024-09-13T22:50:53.157 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:53.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.295 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:53.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:53.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 81 16 90687 90671 69.946 46.9375 0.0125285 0.0142908 2024-09-13T22:50:53.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:53.428 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:53.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:53.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:53.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:53.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:53.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T22:50:53.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:53.851 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:53.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:53 smithi178 bash[29889]: cluster 2024-09-13T22:50:52.942026+0000 mon.a (mon.0) 1358 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:50:53.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:53 smithi178 bash[22840]: cluster 2024-09-13T22:50:52.942026+0000 mon.a (mon.0) 1358 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:50:53.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:53.965 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:53 smithi192 bash[24608]: cluster 2024-09-13T22:50:52.942026+0000 mon.a (mon.0) 1358 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:50:53.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:53.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:54.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:54.139 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:54.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:54.271 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:54.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 82 15 91703 91688 69.8681 63.5625 0.0170414 0.0143078 2024-09-13T22:50:54.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:54.405 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:54.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:54.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:54.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:54.676 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:54.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:54.821 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:54.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:54 smithi192 bash[24608]: cluster 2024-09-13T22:50:52.726086+0000 mgr.y (mgr.14613) 492 : cluster [DBG] pgmap v559: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 39785/263475 objects misplaced (15.100%); 38 MiB/s, 601 objects/s recovering 2024-09-13T22:50:54.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:54.959 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:54.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:54.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:55.003 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:54 smithi178 bash[22840]: cluster 2024-09-13T22:50:52.726086+0000 mgr.y (mgr.14613) 492 : cluster [DBG] pgmap v559: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 39785/263475 objects misplaced (15.100%); 38 MiB/s, 601 objects/s recovering 2024-09-13T22:50:55.003 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:54 smithi178 bash[29889]: cluster 2024-09-13T22:50:52.726086+0000 mgr.y (mgr.14613) 492 : cluster [DBG] pgmap v559: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 39785/263475 objects misplaced (15.100%); 38 MiB/s, 601 objects/s recovering 2024-09-13T22:50:55.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:55.091 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:55.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:55.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:55.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 83 16 92700 92684 69.7763 62.25 0.00729655 0.014326 2024-09-13T22:50:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:55.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:55.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:55.483 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:55.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:55.621 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:55.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:55.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:50:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:50:55.756 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:55.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:55.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:55.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:55.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:55.990 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:56.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:56.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:56.140 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:56.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:56.273 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:56.338 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 84 16 93662 93646 69.6613 60.125 0.0079412 0.0143493 2024-09-13T22:50:56.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:56.412 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:56.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:56.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:56.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:56.672 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:56.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:56 smithi192 bash[24608]: cluster 2024-09-13T22:50:54.726958+0000 mgr.y (mgr.14613) 493 : cluster [DBG] pgmap v560: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 719 op/s; 38611/269379 objects misplaced (14.333%); 38 MiB/s, 610 objects/s recovering 2024-09-13T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:56 smithi192 bash[24608]: audit 2024-09-13T22:50:55.467913+0000 mgr.y (mgr.14613) 494 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:56.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:56.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:56.931 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:56.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:57.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:57.046 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:56 smithi178 bash[22840]: cluster 2024-09-13T22:50:54.726958+0000 mgr.y (mgr.14613) 493 : cluster [DBG] pgmap v560: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 719 op/s; 38611/269379 objects misplaced (14.333%); 38 MiB/s, 610 objects/s recovering 2024-09-13T22:50:57.046 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:56 smithi178 bash[22840]: audit 2024-09-13T22:50:55.467913+0000 mgr.y (mgr.14613) 494 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:57.046 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:56 smithi178 bash[29889]: cluster 2024-09-13T22:50:54.726958+0000 mgr.y (mgr.14613) 493 : cluster [DBG] pgmap v560: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 158 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 719 op/s; 38611/269379 objects misplaced (14.333%); 38 MiB/s, 610 objects/s recovering 2024-09-13T22:50:57.046 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:56 smithi178 bash[29889]: audit 2024-09-13T22:50:55.467913+0000 mgr.y (mgr.14613) 494 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:50:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:57.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:57.180 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:57.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:57.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:57.340 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 85 16 94653 94637 69.5686 61.9375 0.0171513 0.0143677 2024-09-13T22:50:57.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:57.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:57.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:57.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:57.618 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:57.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:57.745 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:57.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:57.885 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:57.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:57.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:57.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:58.005 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:58.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:58.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:58.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:58.275 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:58.340 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 86 16 95553 95537 69.4137 56.25 0.0122561 0.0144011 2024-09-13T22:50:58.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:58.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:50:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:50:58.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:58.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:58.637 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:50:58.643 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:58.676 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:50:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:50:58.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:58.792 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:50:58 smithi178 bash[22840]: cluster 2024-09-13T22:50:56.728268+0000 mgr.y (mgr.14613) 495 : cluster [DBG] pgmap v561: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 37 MiB/s wr, 589 op/s; 35769/276609 objects misplaced (12.931%); 37 MiB/s, 592 objects/s recovering 2024-09-13T22:50:58.792 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:50:58 smithi178 bash[29889]: cluster 2024-09-13T22:50:56.728268+0000 mgr.y (mgr.14613) 495 : cluster [DBG] pgmap v561: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 37 MiB/s wr, 589 op/s; 35769/276609 objects misplaced (12.931%); 37 MiB/s, 592 objects/s recovering 2024-09-13T22:50:58.812 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:58.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:58.939 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:50:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:58.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:50:59.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:59.052 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:50:58 smithi192 bash[24608]: cluster 2024-09-13T22:50:56.728268+0000 mgr.y (mgr.14613) 495 : cluster [DBG] pgmap v561: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 37 MiB/s wr, 589 op/s; 35769/276609 objects misplaced (12.931%); 37 MiB/s, 592 objects/s recovering 2024-09-13T22:50:59.077 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:59.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:59.225 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:50:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:50:59.330 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:59.340 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 87 16 96500 96484 69.2961 59.1875 0.0111056 0.0144249 2024-09-13T22:50:59.366 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:50:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:50:59.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:59.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:59.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:59.616 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:50:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:50:59.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:59.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:50:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:50:59.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:59.876 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:50:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:50:59.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:50:59.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:00.004 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:00.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:00.135 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:00.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:50:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:50:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:51:00.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:00.277 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:00.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 88 16 97377 97361 69.1315 54.8125 0.00855437 0.01446 2024-09-13T22:51:00.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:00.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:00.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:00.531 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:00.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:00.667 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:00.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:00.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:00.900 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:00.910 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:00 smithi178 bash[29889]: cluster 2024-09-13T22:50:58.728796+0000 mgr.y (mgr.14613) 496 : cluster [DBG] pgmap v562: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 949 B/s rd, 34 MiB/s wr, 547 op/s; 35769/276609 objects misplaced (12.931%); 34 MiB/s, 549 objects/s recovering 2024-09-13T22:51:00.910 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:00 smithi178 bash[22840]: cluster 2024-09-13T22:50:58.728796+0000 mgr.y (mgr.14613) 496 : cluster [DBG] pgmap v562: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 949 B/s rd, 34 MiB/s wr, 547 op/s; 35769/276609 objects misplaced (12.931%); 34 MiB/s, 549 objects/s recovering 2024-09-13T22:51:00.931 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:00.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:01.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:01.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:00 smithi192 bash[24608]: cluster 2024-09-13T22:50:58.728796+0000 mgr.y (mgr.14613) 496 : cluster [DBG] pgmap v562: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 949 B/s rd, 34 MiB/s wr, 547 op/s; 35769/276609 objects misplaced (12.931%); 34 MiB/s, 549 objects/s recovering 2024-09-13T22:51:01.060 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:01.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:01.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:01.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:01.328 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:01.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 89 16 98259 98243 68.9741 55.125 0.00749774 0.0144928 2024-09-13T22:51:01.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:01.475 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:01.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:01.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:01.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:01.757 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:01.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:01.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:01.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:01.995 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:01 smithi178 bash[29889]: cluster 2024-09-13T22:51:00.729856+0000 mgr.y (mgr.14613) 497 : cluster [DBG] pgmap v563: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 887 op/s; 32100/290850 objects misplaced (11.037%); 47 MiB/s, 745 objects/s recovering 2024-09-13T22:51:01.995 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:01 smithi178 bash[22840]: cluster 2024-09-13T22:51:00.729856+0000 mgr.y (mgr.14613) 497 : cluster [DBG] pgmap v563: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 887 op/s; 32100/290850 objects misplaced (11.037%); 47 MiB/s, 745 objects/s recovering 2024-09-13T22:51:02.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:02.036 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:02.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:02.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:02.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:01 smithi192 bash[24608]: cluster 2024-09-13T22:51:00.729856+0000 mgr.y (mgr.14613) 497 : cluster [DBG] pgmap v563: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 887 op/s; 32100/290850 objects misplaced (11.037%); 47 MiB/s, 745 objects/s recovering 2024-09-13T22:51:02.174 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:02.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:02.236 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:51:02.237 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:51:02.277 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:51:02.277 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:51:02.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:02.313 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:51:02.313 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:51:02.314 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:02.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:02.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total time run: 90.0176 2024-09-13T22:51:02.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total writes made: 99159 2024-09-13T22:51:02.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Write size: 65536 2024-09-13T22:51:02.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object size: 65536 2024-09-13T22:51:02.341 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Bandwidth (MB/sec): 68.8469 2024-09-13T22:51:02.342 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Bandwidth: 33.3062 2024-09-13T22:51:02.342 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max bandwidth (MB/sec): 178.125 2024-09-13T22:51:02.342 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min bandwidth (MB/sec): 0 2024-09-13T22:51:02.342 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average IOPS: 1101 2024-09-13T22:51:02.342 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev IOPS: 532.898 2024-09-13T22:51:02.343 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max IOPS: 2850 2024-09-13T22:51:02.343 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min IOPS: 0 2024-09-13T22:51:02.343 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average Latency(s): 0.0145204 2024-09-13T22:51:02.343 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Latency(s): 0.0461842 2024-09-13T22:51:02.343 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max latency(s): 2.39712 2024-09-13T22:51:02.343 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min latency(s): 0.00136126 2024-09-13T22:51:02.345 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T22:51:02.359 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:51:02.360 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:51:02.418 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:02.421 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:51:02.421 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:51:02.461 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:02.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:02.468 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:51:02.468 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:51:02.514 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:51:02.514 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:51:02.548 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:51:02.548 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:51:02.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:02.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:02.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:02.704 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:02.806 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:02.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:02.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:02.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:02 smithi178 bash[29889]: audit 2024-09-13T22:51:02.196841+0000 mon.a (mon.0) 1359 : audit [DBG] from='client.? v1:172.21.15.178:0/4077851050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:02.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:02 smithi178 bash[22840]: audit 2024-09-13T22:51:02.196841+0000 mon.a (mon.0) 1359 : audit [DBG] from='client.? v1:172.21.15.178:0/4077851050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:02.930 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:51:02.931 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 281, 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "fsid": "03ae43e2-721f-11ef-bceb-c7b262605968", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "created": "2024-09-13T22:27:31.500176+0000", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "modified": "2024-09-13T22:50:47.940916+0000", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "last_up_change": "2024-09-13T22:37:07.372146+0000", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "last_in_change": "2024-09-13T22:50:02.929279+0000", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_num": 5799936, 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_set": [ 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "pglog_hardlimit", 2024-09-13T22:51:02.932 INFO:teuthology.orchestra.run.smithi178.stdout: "purged_snapdirs", 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_deletes", 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "sortbitwise" 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_version": 18, 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_snapshot": "", 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_max": 9, 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "max_osd": 8, 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "require_min_compat_client": "luminous", 2024-09-13T22:51:02.933 INFO:teuthology.orchestra.run.smithi178.stdout: "min_compat_client": "luminous", 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "require_osd_release": "squid", 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "allow_crimson": false, 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "pools": [ 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 1, 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".mgr", 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:32:40.780155+0000", 2024-09-13T22:51:02.934 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.935 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 11, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 11, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 11, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 11, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 11, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.936 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "189", 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "187", 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.937 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.938 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_max": 32, 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_min": 1 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "mgr": {} 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.940 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.1800000667572021, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.1800000667572021, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 2, 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": ".rgw.root", 2024-09-13T22:51:02.941 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:39.227580+0000", 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.942 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.943 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "76", 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:51:02.944 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.945 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.946 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:51:02.947 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 3, 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.log", 2024-09-13T22:51:02.948 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.229251+0000", 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.949 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.950 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "78", 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.951 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.952 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.953 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 4, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "datapool", 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:41.624329+0000", 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 8193, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.956 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 3, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 3, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 3, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 3, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 3, 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "85", 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.958 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 2, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 85, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.959 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.960 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "rbd": {} 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.961 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.6500000953674316, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.6500000953674316, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 5, 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.control", 2024-09-13T22:51:02.962 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:43.199974+0000", 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.964 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "80", 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:51:02.965 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.966 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.967 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 1.5, 2024-09-13T22:51:02.968 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 1.5, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 1.5, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 1.5, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 6, 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "default.rgw.meta", 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:37:45.423695+0000", 2024-09-13T22:51:02.969 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.970 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 32, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 32, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 32, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 32, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 32, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.971 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "82", 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.973 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.974 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "options": { 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_bias": 4 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "rgw": {} 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2, 2024-09-13T22:51:02.975 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2, 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "pool": 9, 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_name": "unique_pool_2", 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "create_time": "2024-09-13T22:49:18.761265+0000", 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "flags": 1, 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "flags_names": "hashpspool", 2024-09-13T22:51:02.976 INFO:teuthology.orchestra.run.smithi178.stdout: "type": 1, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 3, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "min_size": 2, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_rule": 0, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "is_stretch_pool": false, 2024-09-13T22:51:02.977 INFO:teuthology.orchestra.run.smithi178.stdout: "object_hash": 2, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num": 26, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num": 26, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_placement_num_target": 26, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_target": 26, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_num_pending": 26, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "last_pg_merge_meta": { 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "source_pgid": "0.0", 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "ready_epoch": 0, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_started": 0, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "last_epoch_clean": 0, 2024-09-13T22:51:02.978 INFO:teuthology.orchestra.run.smithi178.stdout: "source_version": "0'0", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "target_version": "0'0" 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "last_change": "278", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend": "0", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "last_force_op_resend_preluminous": "276", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "auid": 0, 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_seq": 0, 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "snap_epoch": 0, 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "pool_snaps": [], 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps": "[]", 2024-09-13T22:51:02.979 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_bytes": 0, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "quota_max_objects": 0, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "tiers": [], 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_of": -1, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "read_tier": -1, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "write_tier": -1, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_mode": "none", 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_bytes": 0, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "target_max_objects": 0, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-13T22:51:02.980 INFO:teuthology.orchestra.run.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profile": "", 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_params": { 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "none" 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_period": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_count": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "use_gmt_hitset": true, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "grade_table": [], 2024-09-13T22:51:02.981 INFO:teuthology.orchestra.run.smithi178.stdout: "stripe_width": 0, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "expected_num_objects": 0, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "fast_read": false, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "options": {}, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "application_metadata": { 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "rados": {} 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "read_balance": { 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "score_type": "Fair distribution", 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "score_acting": 2.1500000953674316, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "score_stable": 2.1500000953674316, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "optimal_score": 1, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_acting": 2.1500000953674316, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "raw_score_stable": 2.1500000953674316, 2024-09-13T22:51:02.982 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity_weighted": 0.9270782470703125, 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "394f2664-bba1-4b15-ace8-78f70f970cac", 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:02.983 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 8, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 279, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6801", 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:51:02.984 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6802", 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6804", 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.986 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6803", 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1489719354 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6801/1489719354", 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6802/1489719354", 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6804/1489719354", 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6803/1489719354", 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:02.987 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "8e65c08b-6431-4849-a6af-2da9cb1efdfa", 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 13, 2024-09-13T22:51:02.988 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 279, 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6805", 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.989 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6806", 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6808", 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.990 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6807", 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 4187084287 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6805/4187084287", 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6806/4187084287", 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6808/4187084287", 2024-09-13T22:51:02.991 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6807/4187084287", 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "40a79075-5102-4f6c-b41a-255102064624", 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 0.4166259765625, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 19, 2024-09-13T22:51:02.992 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 265, 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6809", 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:02.993 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6810", 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6812", 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:51:02.994 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6811", 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 2303802730 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6809/2303802730", 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6810/2303802730", 2024-09-13T22:51:02.995 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6812/2303802730", 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6811/2303802730", 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "3962c796-6f97-4cc6-a186-4b6ef58d67ad", 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:02.996 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 28, 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 278, 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6813", 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.997 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6814", 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6816", 2024-09-13T22:51:02.998 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.178:6815", 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 320675660 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.178:6813/320675660", 2024-09-13T22:51:02.999 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.178:6814/320675660", 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.178:6816/320675660", 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.178:6815/320675660", 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "1b89db2f-9e81-40b1-89bd-8de1dbff133b", 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:03.000 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 34, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 278, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6800", 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:51:03.001 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6801", 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.002 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6803", 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6802", 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3593670527 2024-09-13T22:51:03.003 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6800/3593670527", 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6801/3593670527", 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6803/3593670527", 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6802/3593670527", 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.004 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "efa849a2-d789-4fdd-acc7-04f2c1bebfa8", 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 45, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 278, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:03.005 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6804", 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6805", 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:51:03.006 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6807", 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:03.007 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6806", 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 3271425338 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.008 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6804/3271425338", 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6805/3271425338", 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6807/3271425338", 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6806/3271425338", 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "54a26ae9-bf31-49cf-a924-901eed4a0c98", 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:03.009 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 57, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 278, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6808", 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6809", 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.011 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6811", 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.012 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6810", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 684205162 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6808/684205162", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6809/684205162", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6811/684205162", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6810/684205162", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "uuid": "13e146db-61ac-4fe6-8f26-cbf8d1623a52", 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "up": 1, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "in": 1, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "weight": 1, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_affinity": 1, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_begin": 0, 2024-09-13T22:51:03.014 INFO:teuthology.orchestra.run.smithi178.stdout: "last_clean_end": 0, 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "up_from": 67, 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "up_thru": 279, 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "down_at": 0, 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "lost_at": 0, 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addrs": { 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6812", 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.015 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addrs": { 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6813", 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.016 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6815", 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "addrvec": [ 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "type": "v1", 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "addr": "172.21.15.192:6814", 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: "nonce": 1303971738 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.017 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "public_addr": "172.21.15.192:6812/1303971738", 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "cluster_addr": "172.21.15.192:6813/1303971738", 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1303971738", 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "heartbeat_front_addr": "172.21.15.192:6814/1303971738", 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "state": [ 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "exists", 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "up" 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_xinfo": [ 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 0, 2024-09-13T22:51:03.018 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:50:38.814725+0000", 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.019 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 1, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:50:17.887682+0000", 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 2, 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.020 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:50:12.108430+0000", 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 3, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.021 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:49:58.726439+0000", 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 4, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:49:53.884082+0000", 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 5, 2024-09-13T22:51:03.022 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:49:53.502214+0000", 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 6, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.023 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:50:11.276559+0000", 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "osd": 7, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "down_stamp": "0.000000", 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_probability": 0, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "laggy_interval": 0, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "features": 4540701547738038271, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "old_weight": 0, 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-13T22:50:29.388005+0000", 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: "dead_epoch": 0 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.024 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap": [], 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_items": [ 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "2.1f", 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 7, 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 2 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "5.f", 2024-09-13T22:51:03.025 INFO:teuthology.orchestra.run.smithi178.stdout: "mappings": [ 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: "from": 5, 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: "to": 7 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_upmap_primaries": [], 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: "pg_temp": [ 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.c", 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: 3, 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: 7 2024-09-13T22:51:03.026 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.10", 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: 1, 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: 0 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.12", 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: 1, 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: 2, 2024-09-13T22:51:03.027 INFO:teuthology.orchestra.run.smithi178.stdout: 3 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.13", 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: 7, 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: 1, 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: 3 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.15", 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.028 INFO:teuthology.orchestra.run.smithi178.stdout: 7, 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: 0, 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: 5 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.16", 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: 1, 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: 2, 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: 5 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.029 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.17", 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: 7, 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: 3, 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: 5 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.18", 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: 0, 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: 5, 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: 3 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.030 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: "pgid": "9.19", 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: "osds": [ 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: 0, 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: 5, 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: 7 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: "primary_temp": [], 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: "blocklist": { 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/268210503": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:51:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2039160385": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/669877409": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/129655740": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3632788039": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/388635774": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3265105147": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2959920382": "2024-09-14T22:27:58.436396+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2494442689": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:6800/2676020773": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2675812306": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/1834442338": "2024-09-14T22:38:32.396576+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/2835351236": "2024-09-14T22:28:30.168687+0000", 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "172.21.15.178:0/3836928220": "2024-09-14T22:38:32.396576+0000" 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.032 INFO:teuthology.orchestra.run.smithi178.stdout: "range_blocklist": {}, 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "erasure_code_profiles": { 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "default": { 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "crush-failure-domain": "osd", 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "k": "2", 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "m": "1", 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "plugin": "jerasure", 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "technique": "reed_sol_van" 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "removed_snaps_queue": [], 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "new_removed_snaps": [], 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "new_purged_snaps": [], 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "crush_node_flags": {}, 2024-09-13T22:51:03.033 INFO:teuthology.orchestra.run.smithi178.stdout: "device_class_flags": {}, 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode": { 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_enabled": false, 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_bucket_count": 0, 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: "degraded_stretch_mode": 0, 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: "recovering_stretch_mode": 0, 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: "stretch_mode_bucket": 0 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:51:03.034 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:51:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:03.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:03.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.147 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:02 smithi192 bash[24608]: audit 2024-09-13T22:51:02.196841+0000 mon.a (mon.0) 1359 : audit [DBG] from='client.? v1:172.21.15.178:0/4077851050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:03.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:03.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:03.289 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:03.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:03.421 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:03.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:03.559 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:03.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:03.748 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:51:03.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.790 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:03 smithi192 bash[24608]: cluster 2024-09-13T22:51:02.730416+0000 mgr.y (mgr.14613) 498 : cluster [DBG] pgmap v564: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 34 MiB/s wr, 542 op/s; 32100/290850 objects misplaced (11.037%); 32 MiB/s, 516 objects/s recovering 2024-09-13T22:51:03.791 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:03 smithi192 bash[24608]: audit 2024-09-13T22:51:02.930359+0000 mon.a (mon.0) 1360 : audit [DBG] from='client.? v1:172.21.15.178:0/3580495301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:51:03.791 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:03 smithi192 bash[24608]: audit 2024-09-13T22:51:02.952021+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.178:0/129398572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:51:03.791 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:03 smithi192 bash[24608]: audit 2024-09-13T22:51:02.952962+0000 mon.a (mon.0) 1361 : audit [INF] from='client.25382 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:51:03.791 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:03 smithi192 bash[24608]: audit 2024-09-13T22:51:03.692007+0000 mon.b (mon.1) 202 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.824 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:03.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.950 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:03.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:03.970 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:51:03.970 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:51:04.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[29889]: cluster 2024-09-13T22:51:02.730416+0000 mgr.y (mgr.14613) 498 : cluster [DBG] pgmap v564: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 34 MiB/s wr, 542 op/s; 32100/290850 objects misplaced (11.037%); 32 MiB/s, 516 objects/s recovering 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[29889]: audit 2024-09-13T22:51:02.930359+0000 mon.a (mon.0) 1360 : audit [DBG] from='client.? v1:172.21.15.178:0/3580495301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[29889]: audit 2024-09-13T22:51:02.952021+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.178:0/129398572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[29889]: audit 2024-09-13T22:51:02.952962+0000 mon.a (mon.0) 1361 : audit [INF] from='client.25382 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[29889]: audit 2024-09-13T22:51:03.692007+0000 mon.b (mon.1) 202 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[22840]: cluster 2024-09-13T22:51:02.730416+0000 mgr.y (mgr.14613) 498 : cluster [DBG] pgmap v564: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 34 MiB/s wr, 542 op/s; 32100/290850 objects misplaced (11.037%); 32 MiB/s, 516 objects/s recovering 2024-09-13T22:51:04.068 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[22840]: audit 2024-09-13T22:51:02.930359+0000 mon.a (mon.0) 1360 : audit [DBG] from='client.? v1:172.21.15.178:0/3580495301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-13T22:51:04.069 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[22840]: audit 2024-09-13T22:51:02.952021+0000 mon.c (mon.2) 56 : audit [INF] from='client.? v1:172.21.15.178:0/129398572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:51:04.069 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[22840]: audit 2024-09-13T22:51:02.952962+0000 mon.a (mon.0) 1361 : audit [INF] from='client.25382 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:51:04.069 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:03 smithi178 bash[22840]: audit 2024-09-13T22:51:03.692007+0000 mon.b (mon.1) 202 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:04.090 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:04.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:04.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:04.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:04.354 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:04.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:04.492 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:04.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:04.629 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:04.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:04.758 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:04.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:04.883 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:04.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:04.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:04.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:04 smithi192 bash[24608]: cluster 2024-09-13T22:51:03.724856+0000 mon.a (mon.0) 1362 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:04.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:04 smithi192 bash[24608]: audit 2024-09-13T22:51:03.732066+0000 mon.a (mon.0) 1363 : audit [INF] from='client.25382 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:51:04.994 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:04 smithi192 bash[24608]: cluster 2024-09-13T22:51:03.737814+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-13T22:51:05.006 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:05.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:05.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:05.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:04 smithi178 bash[29889]: cluster 2024-09-13T22:51:03.724856+0000 mon.a (mon.0) 1362 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:05.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:04 smithi178 bash[29889]: audit 2024-09-13T22:51:03.732066+0000 mon.a (mon.0) 1363 : audit [INF] from='client.25382 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:51:05.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:04 smithi178 bash[29889]: cluster 2024-09-13T22:51:03.737814+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-13T22:51:05.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:04 smithi178 bash[22840]: cluster 2024-09-13T22:51:03.724856+0000 mon.a (mon.0) 1362 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:05.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:04 smithi178 bash[22840]: audit 2024-09-13T22:51:03.732066+0000 mon.a (mon.0) 1363 : audit [INF] from='client.25382 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:51:05.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:04 smithi178 bash[22840]: cluster 2024-09-13T22:51:03.737814+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-13T22:51:05.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:05.259 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:05.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:05.396 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:05.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:05.522 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:05.625 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:05.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:05.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:51:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:51:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:05.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T22:51:05.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:05.815 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:05.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:05.927 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:05 smithi178 bash[29889]: cluster 2024-09-13T22:51:04.731125+0000 mgr.y (mgr.14613) 499 : cluster [DBG] pgmap v566: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 654 op/s; 30356/293484 objects misplaced (10.343%); 39 MiB/s, 624 objects/s recovering 2024-09-13T22:51:05.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:05 smithi178 bash[29889]: audit 2024-09-13T22:51:05.479967+0000 mgr.y (mgr.14613) 500 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:05 smithi178 bash[22840]: cluster 2024-09-13T22:51:04.731125+0000 mgr.y (mgr.14613) 499 : cluster [DBG] pgmap v566: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 654 op/s; 30356/293484 objects misplaced (10.343%); 39 MiB/s, 624 objects/s recovering 2024-09-13T22:51:05.928 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:05 smithi178 bash[22840]: audit 2024-09-13T22:51:05.479967+0000 mgr.y (mgr.14613) 500 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:05.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:05.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:06.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:06.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:06.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:05 smithi192 bash[24608]: cluster 2024-09-13T22:51:04.731125+0000 mgr.y (mgr.14613) 499 : cluster [DBG] pgmap v566: 168 pgs: 2 active+recovering+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 654 op/s; 30356/293484 objects misplaced (10.343%); 39 MiB/s, 624 objects/s recovering 2024-09-13T22:51:06.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:05 smithi192 bash[24608]: audit 2024-09-13T22:51:05.479967+0000 mgr.y (mgr.14613) 500 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:06.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:06.201 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:06.304 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:06.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:06.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:06.454 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:06.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:06.585 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:06.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:06.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:06.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:06.826 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:06 smithi178 bash[29889]: cluster 2024-09-13T22:51:05.816900+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-13T22:51:06.827 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:06 smithi178 bash[22840]: cluster 2024-09-13T22:51:05.816900+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-13T22:51:06.846 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:06.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:06.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:07.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:07.088 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:06 smithi192 bash[24608]: cluster 2024-09-13T22:51:05.816900+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-13T22:51:07.105 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:07.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:07.231 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:07.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:07.366 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:07.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:07.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:07.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:07.629 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:07.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:07.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:07.883 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:07.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:07 smithi192 bash[24608]: cluster 2024-09-13T22:51:06.731608+0000 mgr.y (mgr.14613) 501 : cluster [DBG] pgmap v568: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 1.13k op/s; 26598/292653 objects misplaced (9.089%); 50 MiB/s, 792 objects/s recovering 2024-09-13T22:51:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:07 smithi192 bash[24608]: cluster 2024-09-13T22:51:06.796042+0000 mon.a (mon.0) 1366 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T22:51:07.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:07 smithi192 bash[24608]: cluster 2024-09-13T22:51:06.805004+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-13T22:51:07.987 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:08.018 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:08.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:08.148 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:08.179 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:07 smithi178 bash[29889]: cluster 2024-09-13T22:51:06.731608+0000 mgr.y (mgr.14613) 501 : cluster [DBG] pgmap v568: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 1.13k op/s; 26598/292653 objects misplaced (9.089%); 50 MiB/s, 792 objects/s recovering 2024-09-13T22:51:08.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:07 smithi178 bash[29889]: cluster 2024-09-13T22:51:06.796042+0000 mon.a (mon.0) 1366 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T22:51:08.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:07 smithi178 bash[29889]: cluster 2024-09-13T22:51:06.805004+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-13T22:51:08.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:07 smithi178 bash[22840]: cluster 2024-09-13T22:51:06.731608+0000 mgr.y (mgr.14613) 501 : cluster [DBG] pgmap v568: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 1.13k op/s; 26598/292653 objects misplaced (9.089%); 50 MiB/s, 792 objects/s recovering 2024-09-13T22:51:08.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:07 smithi178 bash[22840]: cluster 2024-09-13T22:51:06.796042+0000 mon.a (mon.0) 1366 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T22:51:08.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:07 smithi178 bash[22840]: cluster 2024-09-13T22:51:06.805004+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-13T22:51:08.251 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:08.277 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:08.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:08.405 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:08.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:08.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:08.660 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:08.763 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:08.785 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:08.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:08.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:08.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:09.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:09.028 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:08 smithi178 bash[29889]: cluster 2024-09-13T22:51:07.887901+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-13T22:51:09.028 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:08 smithi178 bash[22840]: cluster 2024-09-13T22:51:07.887901+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-13T22:51:09.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:09.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:08 smithi192 bash[24608]: cluster 2024-09-13T22:51:07.887901+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-13T22:51:09.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:09.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:09.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:09.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:09.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.563 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:09.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.690 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:09.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:09.814 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:09.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:09.949 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:09.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:10.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:10.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:09 smithi192 bash[24608]: cluster 2024-09-13T22:51:08.732185+0000 mgr.y (mgr.14613) 502 : cluster [DBG] pgmap v571: 168 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 870 op/s; 25878/292461 objects misplaced (8.848%); 50 MiB/s, 805 objects/s recovering 2024-09-13T22:51:10.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:09 smithi192 bash[24608]: cluster 2024-09-13T22:51:08.894035+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-13T22:51:10.091 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:10.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:51:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:51:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:51:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:09 smithi178 bash[29889]: cluster 2024-09-13T22:51:08.732185+0000 mgr.y (mgr.14613) 502 : cluster [DBG] pgmap v571: 168 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 870 op/s; 25878/292461 objects misplaced (8.848%); 50 MiB/s, 805 objects/s recovering 2024-09-13T22:51:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:09 smithi178 bash[29889]: cluster 2024-09-13T22:51:08.894035+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-13T22:51:10.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:09 smithi178 bash[22840]: cluster 2024-09-13T22:51:08.732185+0000 mgr.y (mgr.14613) 502 : cluster [DBG] pgmap v571: 168 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 159 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 870 op/s; 25878/292461 objects misplaced (8.848%); 50 MiB/s, 805 objects/s recovering 2024-09-13T22:51:10.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:09 smithi178 bash[22840]: cluster 2024-09-13T22:51:08.894035+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-13T22:51:10.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:10.218 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:10.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:10.343 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:10.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:10.467 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:10.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:10.590 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:10.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:10.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:10.837 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:10.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:10.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:10.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:11.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:11.106 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:11.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:10 smithi192 bash[24608]: cluster 2024-09-13T22:51:09.895266+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-13T22:51:11.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:10 smithi178 bash[29889]: cluster 2024-09-13T22:51:09.895266+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-13T22:51:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:10 smithi178 bash[22840]: cluster 2024-09-13T22:51:09.895266+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-13T22:51:11.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:11.236 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:11.339 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:11.358 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:11.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:11.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:11.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:11.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:11.733 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:11.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:11.860 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:11.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:11.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:11.974 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:11 smithi192 bash[24608]: cluster 2024-09-13T22:51:10.732734+0000 mgr.y (mgr.14613) 503 : cluster [DBG] pgmap v574: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 743 op/s; 18919/283746 objects misplaced (6.668%); 60 MiB/s, 955 objects/s recovering 2024-09-13T22:51:11.974 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:11 smithi192 bash[24608]: cluster 2024-09-13T22:51:10.901697+0000 mon.a (mon.0) 1371 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T22:51:11.974 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:11 smithi192 bash[24608]: cluster 2024-09-13T22:51:10.914633+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-13T22:51:11.996 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:12.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:12.127 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:11 smithi178 bash[29889]: cluster 2024-09-13T22:51:10.732734+0000 mgr.y (mgr.14613) 503 : cluster [DBG] pgmap v574: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 743 op/s; 18919/283746 objects misplaced (6.668%); 60 MiB/s, 955 objects/s recovering 2024-09-13T22:51:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:11 smithi178 bash[29889]: cluster 2024-09-13T22:51:10.901697+0000 mon.a (mon.0) 1371 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T22:51:12.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:11 smithi178 bash[29889]: cluster 2024-09-13T22:51:10.914633+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-13T22:51:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:11 smithi178 bash[22840]: cluster 2024-09-13T22:51:10.732734+0000 mgr.y (mgr.14613) 503 : cluster [DBG] pgmap v574: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 743 op/s; 18919/283746 objects misplaced (6.668%); 60 MiB/s, 955 objects/s recovering 2024-09-13T22:51:12.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:11 smithi178 bash[22840]: cluster 2024-09-13T22:51:10.901697+0000 mon.a (mon.0) 1371 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T22:51:12.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:11 smithi178 bash[22840]: cluster 2024-09-13T22:51:10.914633+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-13T22:51:12.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:12.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:12.333 INFO:tasks.thrashosds.thrasher:j is {'epoch': 281, 'fsid': '03ae43e2-721f-11ef-bceb-c7b262605968', 'created': '2024-09-13T22:27:31.500176+0000', 'modified': '2024-09-13T22:50:47.940916+0000', 'last_up_change': '2024-09-13T22:37:07.372146+0000', 'last_in_change': '2024-09-13T22:50:02.929279+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T22:32:40.780155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '187', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T22:37:39.227580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T22:37:41.229251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-09-13T22:37:41.624329+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 85, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T22:37:43.199974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T22:37:45.423695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-13T22:49:18.761265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1500000953674316, 'score_stable': 2.1500000953674316, 'optimal_score': 1, 'raw_score_acting': 2.1500000953674316, 'raw_score_stable': 2.1500000953674316, 'primary_affinity_weighted': 0.9270782470703125, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}], 'osds': [{'osd': 0, 'uuid': '394f2664-bba1-4b15-ace8-78f70f970cac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 279, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6801', 'nonce': 1489719354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6802', 'nonce': 1489719354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6804', 'nonce': 1489719354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6803', 'nonce': 1489719354}]}, 'public_addr': '172.21.15.178:6801/1489719354', 'cluster_addr': '172.21.15.178:6802/1489719354', 'heartbeat_back_addr': '172.21.15.178:6804/1489719354', 'heartbeat_front_addr': '172.21.15.178:6803/1489719354', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e65c08b-6431-4849-a6af-2da9cb1efdfa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 279, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6805', 'nonce': 4187084287}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6806', 'nonce': 4187084287}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6808', 'nonce': 4187084287}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6807', 'nonce': 4187084287}]}, 'public_addr': '172.21.15.178:6805/4187084287', 'cluster_addr': '172.21.15.178:6806/4187084287', 'heartbeat_back_addr': '172.21.15.178:6808/4187084287', 'heartbeat_front_addr': '172.21.15.178:6807/4187084287', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '40a79075-5102-4f6c-b41a-255102064624', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4166259765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 265, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6809', 'nonce': 2303802730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6810', 'nonce': 2303802730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6812', 'nonce': 2303802730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6811', 'nonce': 2303802730}]}, 'public_addr': '172.21.15.178:6809/2303802730', 'cluster_addr': '172.21.15.178:6810/2303802730', 'heartbeat_back_addr': '172.21.15.178:6812/2303802730', 'heartbeat_front_addr': '172.21.15.178:6811/2303802730', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3962c796-6f97-4cc6-a186-4b6ef58d67ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6813', 'nonce': 320675660}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6814', 'nonce': 320675660}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6816', 'nonce': 320675660}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.178:6815', 'nonce': 320675660}]}, 'public_addr': '172.21.15.178:6813/320675660', 'cluster_addr': '172.21.15.178:6814/320675660', 'heartbeat_back_addr': '172.21.15.178:6816/320675660', 'heartbeat_front_addr': '172.21.15.178:6815/320675660', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1b89db2f-9e81-40b1-89bd-8de1dbff133b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 34, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 3593670527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 3593670527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 3593670527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 3593670527}]}, 'public_addr': '172.21.15.192:6800/3593670527', 'cluster_addr': '172.21.15.192:6801/3593670527', 'heartbeat_back_addr': '172.21.15.192:6803/3593670527', 'heartbeat_front_addr': '172.21.15.192:6802/3593670527', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'efa849a2-d789-4fdd-acc7-04f2c1bebfa8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 3271425338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 3271425338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 3271425338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 3271425338}]}, 'public_addr': '172.21.15.192:6804/3271425338', 'cluster_addr': '172.21.15.192:6805/3271425338', 'heartbeat_back_addr': '172.21.15.192:6807/3271425338', 'heartbeat_front_addr': '172.21.15.192:6806/3271425338', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '54a26ae9-bf31-49cf-a924-901eed4a0c98', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 684205162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 684205162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 684205162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 684205162}]}, 'public_addr': '172.21.15.192:6808/684205162', 'cluster_addr': '172.21.15.192:6809/684205162', 'heartbeat_back_addr': '172.21.15.192:6811/684205162', 'heartbeat_front_addr': '172.21.15.192:6810/684205162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '13e146db-61ac-4fe6-8f26-cbf8d1623a52', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 279, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 1303971738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1303971738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1303971738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 1303971738}]}, 'public_addr': '172.21.15.192:6812/1303971738', 'cluster_addr': '172.21.15.192:6813/1303971738', 'heartbeat_back_addr': '172.21.15.192:6815/1303971738', 'heartbeat_front_addr': '172.21.15.192:6814/1303971738', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:50:38.814725+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:50:17.887682+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:50:12.108430+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:49:58.726439+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:49:53.884082+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:49:53.502214+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:50:11.276559+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T22:50:29.388005+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1f', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '5.f', 'mappings': [{'from': 5, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.c', 'osds': [3, 7]}, {'pgid': '9.10', 'osds': [1, 0]}, {'pgid': '9.12', 'osds': [1, 2, 3]}, {'pgid': '9.13', 'osds': [7, 1, 3]}, {'pgid': '9.15', 'osds': [7, 0, 5]}, {'pgid': '9.16', 'osds': [1, 2, 5]}, {'pgid': '9.17', 'osds': [7, 3, 5]}, {'pgid': '9.18', 'osds': [0, 5, 3]}, {'pgid': '9.19', 'osds': [0, 5, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.178:0/268210503': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:6800/2039160385': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/669877409': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/129655740': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/3632788039': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:6800/388635774': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3265105147': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2959920382': '2024-09-14T22:27:58.436396+0000', '172.21.15.178:0/2494442689': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:6800/2676020773': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2675812306': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/1834442338': '2024-09-14T22:38:32.396576+0000', '172.21.15.178:0/2835351236': '2024-09-14T22:28:30.168687+0000', '172.21.15.178:0/3836928220': '2024-09-14T22:38:32.396576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T22:51:12.333 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:51:12.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:12.411 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:12.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:12.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:12.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:12.680 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:12.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:12.810 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:12.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:12.937 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:12.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:13.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:13.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:12 smithi192 bash[24608]: cluster 2024-09-13T22:51:11.923484+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-13T22:51:13.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:13.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:13.183 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:12 smithi178 bash[29889]: cluster 2024-09-13T22:51:11.923484+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-13T22:51:13.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:12 smithi178 bash[22840]: cluster 2024-09-13T22:51:11.923484+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-13T22:51:13.208 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:13.313 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:13.341 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:13.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:13.468 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:13.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:13.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:13.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:13.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:13.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:13.879 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:13.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:13.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:13.993 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:13 smithi178 bash[29889]: cluster 2024-09-13T22:51:12.733259+0000 mgr.y (mgr.14613) 504 : cluster [DBG] pgmap v577: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 727 op/s; 18919/283746 objects misplaced (6.668%); 49 MiB/s, 779 objects/s recovering 2024-09-13T22:51:13.993 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:13 smithi178 bash[22840]: cluster 2024-09-13T22:51:12.733259+0000 mgr.y (mgr.14613) 504 : cluster [DBG] pgmap v577: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 727 op/s; 18919/283746 objects misplaced (6.668%); 49 MiB/s, 779 objects/s recovering 2024-09-13T22:51:14.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:14.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:14.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:13 smithi192 bash[24608]: cluster 2024-09-13T22:51:12.733259+0000 mgr.y (mgr.14613) 504 : cluster [DBG] pgmap v577: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 727 op/s; 18919/283746 objects misplaced (6.668%); 49 MiB/s, 779 objects/s recovering 2024-09-13T22:51:14.140 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:14.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:14.275 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:14.321 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:51:14.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:51:14.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:14.402 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:14.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:14.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:14.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:14.679 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:14.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:14.818 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:14.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:14.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:14.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:15.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:15.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:15.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:15.207 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:15.213 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:15.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:15.347 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:15.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:15.473 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:15.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:15.606 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:15.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:15.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:15.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:15.890 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:15.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:15.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:51:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:51:15.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:16.025 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:16.132 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[29889]: cluster 2024-09-13T22:51:14.734163+0000 mgr.y (mgr.14613) 505 : cluster [DBG] pgmap v578: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 700 op/s; 17843/277401 objects misplaced (6.432%); 19 MiB/s, 307 objects/s recovering 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[29889]: cluster 2024-09-13T22:51:15.213987+0000 osd.4 (osd.4) 181 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[29889]: cluster 2024-09-13T22:51:15.214055+0000 osd.4 (osd.4) 182 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[29889]: audit 2024-09-13T22:51:15.487881+0000 mgr.y (mgr.14613) 506 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[22840]: cluster 2024-09-13T22:51:14.734163+0000 mgr.y (mgr.14613) 505 : cluster [DBG] pgmap v578: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 700 op/s; 17843/277401 objects misplaced (6.432%); 19 MiB/s, 307 objects/s recovering 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[22840]: cluster 2024-09-13T22:51:15.213987+0000 osd.4 (osd.4) 181 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:16.143 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[22840]: cluster 2024-09-13T22:51:15.214055+0000 osd.4 (osd.4) 182 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:16.144 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:15 smithi178 bash[22840]: audit 2024-09-13T22:51:15.487881+0000 mgr.y (mgr.14613) 506 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:16.160 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:16.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:16.276 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:15 smithi192 bash[24608]: cluster 2024-09-13T22:51:14.734163+0000 mgr.y (mgr.14613) 505 : cluster [DBG] pgmap v578: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 700 op/s; 17843/277401 objects misplaced (6.432%); 19 MiB/s, 307 objects/s recovering 2024-09-13T22:51:16.276 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:15 smithi192 bash[24608]: cluster 2024-09-13T22:51:15.213987+0000 osd.4 (osd.4) 181 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:16.276 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:15 smithi192 bash[24608]: cluster 2024-09-13T22:51:15.214055+0000 osd.4 (osd.4) 182 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:16.276 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:15 smithi192 bash[24608]: audit 2024-09-13T22:51:15.487881+0000 mgr.y (mgr.14613) 506 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:16.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:16.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:16.420 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:16.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:16.555 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:16.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:16.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:16.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:16.832 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:16.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:16.969 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:16.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:17.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.103 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:17.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:17.229 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:17.230 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:16 smithi192 bash[24608]: cluster 2024-09-13T22:51:15.958267+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-13T22:51:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:17.256 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:17.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.362 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:17.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:16 smithi178 bash[22840]: cluster 2024-09-13T22:51:15.958267+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-13T22:51:17.391 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:16 smithi178 bash[29889]: cluster 2024-09-13T22:51:15.958267+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-13T22:51:17.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.492 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:17.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:17.623 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:17.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:17.747 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:17.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.871 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:17.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:17.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:18.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:18.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:18.116 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[29889]: cluster 2024-09-13T22:51:16.734666+0000 mgr.y (mgr.14613) 507 : cluster [DBG] pgmap v580: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 754 op/s; 16429/270180 objects misplaced (6.081%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[29889]: cluster 2024-09-13T22:51:16.963507+0000 mon.a (mon.0) 1375 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[29889]: cluster 2024-09-13T22:51:17.499035+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[29889]: cluster 2024-09-13T22:51:17.499114+0000 osd.3 (osd.3) 242 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[29889]: cluster 2024-09-13T22:51:17.953844+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[22840]: cluster 2024-09-13T22:51:16.734666+0000 mgr.y (mgr.14613) 507 : cluster [DBG] pgmap v580: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 754 op/s; 16429/270180 objects misplaced (6.081%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[22840]: cluster 2024-09-13T22:51:16.963507+0000 mon.a (mon.0) 1375 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[22840]: cluster 2024-09-13T22:51:17.499035+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[22840]: cluster 2024-09-13T22:51:17.499114+0000 osd.3 (osd.3) 242 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:18.117 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:17 smithi178 bash[22840]: cluster 2024-09-13T22:51:17.953844+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-13T22:51:18.135 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:18.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:18.246 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:17 smithi192 bash[24608]: cluster 2024-09-13T22:51:16.734666+0000 mgr.y (mgr.14613) 507 : cluster [DBG] pgmap v580: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 754 op/s; 16429/270180 objects misplaced (6.081%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T22:51:18.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:17 smithi192 bash[24608]: cluster 2024-09-13T22:51:16.963507+0000 mon.a (mon.0) 1375 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:51:18.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:17 smithi192 bash[24608]: cluster 2024-09-13T22:51:17.499035+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:18.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:17 smithi192 bash[24608]: cluster 2024-09-13T22:51:17.499114+0000 osd.3 (osd.3) 242 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:18.247 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:17 smithi192 bash[24608]: cluster 2024-09-13T22:51:17.953844+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-13T22:51:18.258 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:18.268 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:18.269 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:18.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:18.384 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:18.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:18.515 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:18.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:18.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:18.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:18.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:18.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:18.904 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:18.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:19.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[29889]: cluster 2024-09-13T22:51:17.475457+0000 osd.5 (osd.5) 303 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[29889]: cluster 2024-09-13T22:51:17.475518+0000 osd.5 (osd.5) 304 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[29889]: audit 2024-09-13T22:51:18.713628+0000 mon.a (mon.0) 1377 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[29889]: audit 2024-09-13T22:51:18.716807+0000 mon.b (mon.1) 203 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[22840]: cluster 2024-09-13T22:51:17.475457+0000 osd.5 (osd.5) 303 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[22840]: cluster 2024-09-13T22:51:17.475518+0000 osd.5 (osd.5) 304 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:19.020 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[22840]: audit 2024-09-13T22:51:18.713628+0000 mon.a (mon.0) 1377 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:19.021 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:18 smithi178 bash[22840]: audit 2024-09-13T22:51:18.716807+0000 mon.b (mon.1) 203 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:19.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:19.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:19.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:18 smithi192 bash[24608]: cluster 2024-09-13T22:51:17.475457+0000 osd.5 (osd.5) 303 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:18 smithi192 bash[24608]: cluster 2024-09-13T22:51:17.475518+0000 osd.5 (osd.5) 304 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:18 smithi192 bash[24608]: audit 2024-09-13T22:51:18.713628+0000 mon.a (mon.0) 1377 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:19.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:18 smithi192 bash[24608]: audit 2024-09-13T22:51:18.716807+0000 mon.b (mon.1) 203 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:19.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:19.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:19.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:19.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:19.302 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:19.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:19.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:20.111 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:51:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:51:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:51:20.129 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:20.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:20.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:20.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:20.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:20.371 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:20.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:20.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:20.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:21.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:20 smithi192 bash[24608]: cluster 2024-09-13T22:51:18.735449+0000 mgr.y (mgr.14613) 508 : cluster [DBG] pgmap v582: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 690 op/s; 16429/269652 objects misplaced (6.093%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T22:51:21.330 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:20 smithi178 bash[29889]: cluster 2024-09-13T22:51:18.735449+0000 mgr.y (mgr.14613) 508 : cluster [DBG] pgmap v582: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 690 op/s; 16429/269652 objects misplaced (6.093%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T22:51:21.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:20 smithi178 bash[22840]: cluster 2024-09-13T22:51:18.735449+0000 mgr.y (mgr.14613) 508 : cluster [DBG] pgmap v582: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 690 op/s; 16429/269652 objects misplaced (6.093%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T22:51:21.431 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:21.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:21.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:21.584 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:21.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:21.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:21.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:21.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:21.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:21.872 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:21.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:21.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:21.983 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:21 smithi178 bash[29889]: cluster 2024-09-13T22:51:20.736474+0000 mgr.y (mgr.14613) 509 : cluster [DBG] pgmap v583: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 14094/253230 objects misplaced (5.566%); 12 MiB/s, 198 objects/s recovering 2024-09-13T22:51:21.984 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:21 smithi178 bash[22840]: cluster 2024-09-13T22:51:20.736474+0000 mgr.y (mgr.14613) 509 : cluster [DBG] pgmap v583: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 14094/253230 objects misplaced (5.566%); 12 MiB/s, 198 objects/s recovering 2024-09-13T22:51:21.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:21.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:21.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:22.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:22.107 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:21 smithi192 bash[24608]: cluster 2024-09-13T22:51:20.736474+0000 mgr.y (mgr.14613) 509 : cluster [DBG] pgmap v583: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 14094/253230 objects misplaced (5.566%); 12 MiB/s, 198 objects/s recovering 2024-09-13T22:51:22.119 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:22.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:22.222 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:22.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:22.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:22.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:22.732 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:22.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:22.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:24.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:23 smithi192 bash[24608]: cluster 2024-09-13T22:51:22.736963+0000 mgr.y (mgr.14613) 510 : cluster [DBG] pgmap v584: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 14094/253230 objects misplaced (5.566%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:51:24.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:23 smithi192 bash[24608]: cluster 2024-09-13T22:51:23.204942+0000 osd.1 (osd.1) 258 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:24.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:23 smithi192 bash[24608]: cluster 2024-09-13T22:51:23.205005+0000 osd.1 (osd.1) 259 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:24.188 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:23 smithi178 bash[29889]: cluster 2024-09-13T22:51:22.736963+0000 mgr.y (mgr.14613) 510 : cluster [DBG] pgmap v584: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 14094/253230 objects misplaced (5.566%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:51:24.188 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:23 smithi178 bash[29889]: cluster 2024-09-13T22:51:23.204942+0000 osd.1 (osd.1) 258 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:24.188 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:23 smithi178 bash[29889]: cluster 2024-09-13T22:51:23.205005+0000 osd.1 (osd.1) 259 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:24.189 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:23 smithi178 bash[22840]: cluster 2024-09-13T22:51:22.736963+0000 mgr.y (mgr.14613) 510 : cluster [DBG] pgmap v584: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 14094/253230 objects misplaced (5.566%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T22:51:24.189 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:23 smithi178 bash[22840]: cluster 2024-09-13T22:51:23.204942+0000 osd.1 (osd.1) 258 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:24.189 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:23 smithi178 bash[22840]: cluster 2024-09-13T22:51:23.205005+0000 osd.1 (osd.1) 259 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:24.189 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:24.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:24.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:24.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:24.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:24.412 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:25.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:24 smithi192 bash[24608]: cluster 2024-09-13T22:51:23.806463+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-13T22:51:25.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:24 smithi178 bash[29889]: cluster 2024-09-13T22:51:23.806463+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-13T22:51:25.167 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:24 smithi178 bash[22840]: cluster 2024-09-13T22:51:23.806463+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-13T22:51:25.184 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:25.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:25.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:25.347 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:25.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:25.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:25.486 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:25.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:25.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:25.716 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:25.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:25.808 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:51:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:51:25.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:25.844 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:25.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:25.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:25 smithi178 bash[22840]: cluster 2024-09-13T22:51:24.738332+0000 mgr.y (mgr.14613) 511 : cluster [DBG] pgmap v586: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 13760/244647 objects misplaced (5.624%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T22:51:25.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:25 smithi178 bash[22840]: audit 2024-09-13T22:51:25.497820+0000 mgr.y (mgr.14613) 512 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:25.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:25 smithi178 bash[29889]: cluster 2024-09-13T22:51:24.738332+0000 mgr.y (mgr.14613) 511 : cluster [DBG] pgmap v586: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 13760/244647 objects misplaced (5.624%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T22:51:25.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:25 smithi178 bash[29889]: audit 2024-09-13T22:51:25.497820+0000 mgr.y (mgr.14613) 512 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:25.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:25.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:26.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:26.112 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:26.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:25 smithi192 bash[24608]: cluster 2024-09-13T22:51:24.738332+0000 mgr.y (mgr.14613) 511 : cluster [DBG] pgmap v586: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 13760/244647 objects misplaced (5.624%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T22:51:26.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:25 smithi192 bash[24608]: audit 2024-09-13T22:51:25.497820+0000 mgr.y (mgr.14613) 512 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:26.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:26.258 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:26.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:26.389 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:26.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:26.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:26.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:26.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:26.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:26.780 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:26.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:26.963 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:26.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:27.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:27.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:27.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:27.226 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:27.330 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:27.354 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:27.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:27.480 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:27.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:27.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:27.721 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:27.764 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:27.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:27.894 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:27.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:27.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:28.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:28.022 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:27 smithi178 bash[29889]: cluster 2024-09-13T22:51:26.739161+0000 mgr.y (mgr.14613) 513 : cluster [DBG] pgmap v587: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 11414/237462 objects misplaced (4.807%); 13 MiB/s, 200 objects/s recovering 2024-09-13T22:51:28.022 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:27 smithi178 bash[22840]: cluster 2024-09-13T22:51:26.739161+0000 mgr.y (mgr.14613) 513 : cluster [DBG] pgmap v587: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 11414/237462 objects misplaced (4.807%); 13 MiB/s, 200 objects/s recovering 2024-09-13T22:51:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:28.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:28.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:27 smithi192 bash[24608]: cluster 2024-09-13T22:51:26.739161+0000 mgr.y (mgr.14613) 513 : cluster [DBG] pgmap v587: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 11414/237462 objects misplaced (4.807%); 13 MiB/s, 200 objects/s recovering 2024-09-13T22:51:28.146 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:28.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:28.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:28.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:28.403 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:28.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:28.528 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:28.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:28.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:28.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:28.773 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:28.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:28.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:28.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:29.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:29.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:29.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:28 smithi192 bash[24608]: cluster 2024-09-13T22:51:27.955208+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-13T22:51:29.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:29.184 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:29.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:28 smithi178 bash[22840]: cluster 2024-09-13T22:51:27.955208+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-13T22:51:29.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:28 smithi178 bash[29889]: cluster 2024-09-13T22:51:27.955208+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-13T22:51:29.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:29.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:29.418 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:29.440 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:29.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:29.567 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:29.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:29.699 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:29.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:29.841 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:29.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:29.971 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:29.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:30.072 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:51:30.072 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:51:30.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:30.085 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:51:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:51:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:51:30.085 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:29 smithi178 bash[29889]: cluster 2024-09-13T22:51:28.739736+0000 mgr.y (mgr.14613) 514 : cluster [DBG] pgmap v589: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 11414/237081 objects misplaced (4.814%); 8.0 MiB/s, 127 objects/s recovering 2024-09-13T22:51:30.085 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:29 smithi178 bash[22840]: cluster 2024-09-13T22:51:28.739736+0000 mgr.y (mgr.14613) 514 : cluster [DBG] pgmap v589: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 11414/237081 objects misplaced (4.814%); 8.0 MiB/s, 127 objects/s recovering 2024-09-13T22:51:30.110 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:30.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:29 smithi192 bash[24608]: cluster 2024-09-13T22:51:28.739736+0000 mgr.y (mgr.14613) 514 : cluster [DBG] pgmap v589: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 11414/237081 objects misplaced (4.814%); 8.0 MiB/s, 127 objects/s recovering 2024-09-13T22:51:30.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:30.248 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:30.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:30.385 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:30.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:30.516 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:30.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:30.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:30.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:30.781 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:30.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:30.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:30.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:31.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:31.041 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:31.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.189 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:31.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.326 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:31.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:31.458 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:31.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:31.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.739 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:31.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.874 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:31.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:31.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:31.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:32.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:32.098 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:51:32.098 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:51:32.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:32.139 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:51:32.139 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:51:32.173 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:51:32.173 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:51:32.206 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:51:32.206 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:51:32.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:32.247 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:51:32.247 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:51:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:32.280 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:51:32.280 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:51:32.299 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:51:32.299 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:51:32.316 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:51:32.317 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:51:32.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: cluster 2024-09-13T22:51:30.740610+0000 mgr.y (mgr.14613) 515 : cluster [DBG] pgmap v590: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 9177/224814 objects misplaced (4.082%); 15 MiB/s, 237 objects/s recovering 2024-09-13T22:51:32.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: audit 2024-09-13T22:51:31.086351+0000 mon.b (mon.1) 204 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:51:32.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: cluster 2024-09-13T22:51:31.098578+0000 osd.2 (osd.2) 211 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:32.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: cluster 2024-09-13T22:51:31.098629+0000 osd.2 (osd.2) 212 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:32.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: cluster 2024-09-13T22:51:30.740610+0000 mgr.y (mgr.14613) 515 : cluster [DBG] pgmap v590: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 9177/224814 objects misplaced (4.082%); 15 MiB/s, 237 objects/s recovering 2024-09-13T22:51:32.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: audit 2024-09-13T22:51:31.086351+0000 mon.b (mon.1) 204 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:51:32.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: cluster 2024-09-13T22:51:31.098578+0000 osd.2 (osd.2) 211 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:32.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: cluster 2024-09-13T22:51:31.098629+0000 osd.2 (osd.2) 212 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:32.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:31 smithi192 bash[24608]: cluster 2024-09-13T22:51:30.740610+0000 mgr.y (mgr.14613) 515 : cluster [DBG] pgmap v590: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 9177/224814 objects misplaced (4.082%); 15 MiB/s, 237 objects/s recovering 2024-09-13T22:51:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:31 smithi192 bash[24608]: audit 2024-09-13T22:51:31.086351+0000 mon.b (mon.1) 204 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:51:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:31 smithi192 bash[24608]: cluster 2024-09-13T22:51:31.098578+0000 osd.2 (osd.2) 211 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:31 smithi192 bash[24608]: cluster 2024-09-13T22:51:31.098629+0000 osd.2 (osd.2) 212 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:32.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:32.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:32.517 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:32.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:32.653 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:32.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:32.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:32.893 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:32.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:32.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:33.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:33.032 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:32 smithi192 bash[24608]: cluster 2024-09-13T22:51:31.968558+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-13T22:51:33.032 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:32 smithi192 bash[24608]: audit 2024-09-13T22:51:32.066129+0000 mon.a (mon.0) 1381 : audit [DBG] from='client.? v1:172.21.15.178:0/1039327553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:33.032 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:32 smithi192 bash[24608]: audit 2024-09-13T22:51:32.595446+0000 mon.b (mon.1) 205 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:51:33.032 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:32 smithi192 bash[24608]: audit 2024-09-13T22:51:32.596460+0000 mon.b (mon.1) 206 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:51:33.032 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:32 smithi192 bash[24608]: audit 2024-09-13T22:51:32.601785+0000 mon.a (mon.0) 1382 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:33.043 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:33.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: cluster 2024-09-13T22:51:31.968558+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: audit 2024-09-13T22:51:32.066129+0000 mon.a (mon.0) 1381 : audit [DBG] from='client.? v1:172.21.15.178:0/1039327553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: audit 2024-09-13T22:51:32.595446+0000 mon.b (mon.1) 205 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: audit 2024-09-13T22:51:32.596460+0000 mon.b (mon.1) 206 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[22840]: audit 2024-09-13T22:51:32.601785+0000 mon.a (mon.0) 1382 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: cluster 2024-09-13T22:51:31.968558+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: audit 2024-09-13T22:51:32.066129+0000 mon.a (mon.0) 1381 : audit [DBG] from='client.? v1:172.21.15.178:0/1039327553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:33.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: audit 2024-09-13T22:51:32.595446+0000 mon.b (mon.1) 205 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:51:33.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: audit 2024-09-13T22:51:32.596460+0000 mon.b (mon.1) 206 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:51:33.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:32 smithi178 bash[29889]: audit 2024-09-13T22:51:32.601785+0000 mon.a (mon.0) 1382 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:33.178 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:33.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:33.305 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:33.409 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:33.432 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:33.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:33.554 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:33.657 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:33.807 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:33.909 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:33.932 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:33.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:34.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:34.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[29889]: cluster 2024-09-13T22:51:32.741003+0000 mgr.y (mgr.14613) 516 : cluster [DBG] pgmap v592: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 827 op/s; 9177/224814 objects misplaced (4.082%); 15 MiB/s, 237 objects/s recovering 2024-09-13T22:51:34.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[29889]: cluster 2024-09-13T22:51:32.969582+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-13T22:51:34.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[29889]: audit 2024-09-13T22:51:33.728003+0000 mon.b (mon.1) 207 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:34.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[29889]: cluster 2024-09-13T22:51:33.979724+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-13T22:51:34.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[22840]: cluster 2024-09-13T22:51:32.741003+0000 mgr.y (mgr.14613) 516 : cluster [DBG] pgmap v592: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 827 op/s; 9177/224814 objects misplaced (4.082%); 15 MiB/s, 237 objects/s recovering 2024-09-13T22:51:34.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[22840]: cluster 2024-09-13T22:51:32.969582+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-13T22:51:34.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[22840]: audit 2024-09-13T22:51:33.728003+0000 mon.b (mon.1) 207 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:34.169 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:33 smithi178 bash[22840]: cluster 2024-09-13T22:51:33.979724+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-13T22:51:34.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:34.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:34.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:33 smithi192 bash[24608]: cluster 2024-09-13T22:51:32.741003+0000 mgr.y (mgr.14613) 516 : cluster [DBG] pgmap v592: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 827 op/s; 9177/224814 objects misplaced (4.082%); 15 MiB/s, 237 objects/s recovering 2024-09-13T22:51:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:33 smithi192 bash[24608]: cluster 2024-09-13T22:51:32.969582+0000 mon.a (mon.0) 1383 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-13T22:51:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:33 smithi192 bash[24608]: audit 2024-09-13T22:51:33.728003+0000 mon.b (mon.1) 207 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:33 smithi192 bash[24608]: cluster 2024-09-13T22:51:33.979724+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-13T22:51:34.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:34.437 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:34.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:34.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:34.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:34.697 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:34.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:34.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:34.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:34.968 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:34.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:35.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:35.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:35.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:35.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:35.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:35.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:35.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T22:51:35.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:35.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:35.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:35.511 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:35.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:35.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:35.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:35.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:35.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:35.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:51:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:51:35.872 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:35.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:35.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:36.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:36.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.112 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:35 smithi192 bash[24608]: cluster 2024-09-13T22:51:34.741660+0000 mgr.y (mgr.14613) 517 : cluster [DBG] pgmap v595: 168 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 8603/217332 objects misplaced (3.958%); 11 MiB/s, 175 objects/s recovering 2024-09-13T22:51:36.112 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:35 smithi192 bash[24608]: audit 2024-09-13T22:51:35.507830+0000 mgr.y (mgr.14613) 518 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:36.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:36.154 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:51:36.156 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:51:36.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:36.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:35 smithi178 bash[29889]: cluster 2024-09-13T22:51:34.741660+0000 mgr.y (mgr.14613) 517 : cluster [DBG] pgmap v595: 168 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 8603/217332 objects misplaced (3.958%); 11 MiB/s, 175 objects/s recovering 2024-09-13T22:51:36.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:35 smithi178 bash[29889]: audit 2024-09-13T22:51:35.507830+0000 mgr.y (mgr.14613) 518 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:36.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:35 smithi178 bash[22840]: cluster 2024-09-13T22:51:34.741660+0000 mgr.y (mgr.14613) 517 : cluster [DBG] pgmap v595: 168 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 8603/217332 objects misplaced (3.958%); 11 MiB/s, 175 objects/s recovering 2024-09-13T22:51:36.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:35 smithi178 bash[22840]: audit 2024-09-13T22:51:35.507830+0000 mgr.y (mgr.14613) 518 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:36.268 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:36.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:36.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:36.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:36.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:36.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.662 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:36.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:36.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:36.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:37.002 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:51:37.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:37.061 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:37.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:37 smithi192 bash[24608]: audit 2024-09-13T22:51:36.131914+0000 mgr.y (mgr.14613) 519 : audit [DBG] from='client.25397 v1:172.21.15.178:0/1124012902' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:51:37.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:37 smithi192 bash[24608]: audit 2024-09-13T22:51:36.147896+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.178:0/1950165451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:51:37.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:37 smithi192 bash[24608]: audit 2024-09-13T22:51:36.148061+0000 mon.a (mon.0) 1385 : audit [INF] from='client.25312 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:51:37.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:37.183 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:37.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:37.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:37.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:37 smithi178 bash[29889]: audit 2024-09-13T22:51:36.131914+0000 mgr.y (mgr.14613) 519 : audit [DBG] from='client.25397 v1:172.21.15.178:0/1124012902' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:51:37.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:37 smithi178 bash[29889]: audit 2024-09-13T22:51:36.147896+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.178:0/1950165451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:51:37.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:37 smithi178 bash[29889]: audit 2024-09-13T22:51:36.148061+0000 mon.a (mon.0) 1385 : audit [INF] from='client.25312 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:51:37.375 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:37 smithi178 bash[22840]: audit 2024-09-13T22:51:36.131914+0000 mgr.y (mgr.14613) 519 : audit [DBG] from='client.25397 v1:172.21.15.178:0/1124012902' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:51:37.375 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:37 smithi178 bash[22840]: audit 2024-09-13T22:51:36.147896+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.178:0/1950165451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:51:37.375 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:37 smithi178 bash[22840]: audit 2024-09-13T22:51:36.148061+0000 mon.a (mon.0) 1385 : audit [INF] from='client.25312 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:51:37.415 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:37.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:37.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:37.580 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:37.683 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:37.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:37.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:37.845 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:37.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:37.982 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:37.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:38.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:38.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:38 smithi192 bash[24608]: cluster 2024-09-13T22:51:36.743316+0000 mgr.y (mgr.14613) 520 : cluster [DBG] pgmap v596: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_toofull, 165 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 717 op/s; 7968/211917 objects misplaced (3.760%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T22:51:38.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:38 smithi192 bash[24608]: cluster 2024-09-13T22:51:36.993116+0000 mon.a (mon.0) 1386 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:38.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:38 smithi192 bash[24608]: audit 2024-09-13T22:51:36.995921+0000 mon.a (mon.0) 1387 : audit [INF] from='client.25312 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:51:38.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:38 smithi192 bash[24608]: cluster 2024-09-13T22:51:37.007997+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-13T22:51:38.103 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:38.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:38.216 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[29889]: cluster 2024-09-13T22:51:36.743316+0000 mgr.y (mgr.14613) 520 : cluster [DBG] pgmap v596: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_toofull, 165 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 717 op/s; 7968/211917 objects misplaced (3.760%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[29889]: cluster 2024-09-13T22:51:36.993116+0000 mon.a (mon.0) 1386 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[29889]: audit 2024-09-13T22:51:36.995921+0000 mon.a (mon.0) 1387 : audit [INF] from='client.25312 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[29889]: cluster 2024-09-13T22:51:37.007997+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[22840]: cluster 2024-09-13T22:51:36.743316+0000 mgr.y (mgr.14613) 520 : cluster [DBG] pgmap v596: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_toofull, 165 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 717 op/s; 7968/211917 objects misplaced (3.760%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[22840]: cluster 2024-09-13T22:51:36.993116+0000 mon.a (mon.0) 1386 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[22840]: audit 2024-09-13T22:51:36.995921+0000 mon.a (mon.0) 1387 : audit [INF] from='client.25312 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:51:38.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:38 smithi178 bash[22840]: cluster 2024-09-13T22:51:37.007997+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-13T22:51:38.233 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:38.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:38.363 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:38.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:38.496 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:38.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:38.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:38.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:38.759 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:38.864 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:38.952 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:38.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:39.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:39.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:39.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:39.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:39.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:39.335 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:39.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:51:39.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:39.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:39.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:39.588 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:39.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:39.719 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:39.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:39.849 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:39.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:39.989 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:39.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:40.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:40.100 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:51:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:51:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:51:40.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:40 smithi178 bash[29889]: cluster 2024-09-13T22:51:38.743808+0000 mgr.y (mgr.14613) 521 : cluster [DBG] pgmap v598: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_toofull, 165 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 742 op/s; 7968/211464 objects misplaced (3.768%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T22:51:40.101 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:40 smithi178 bash[22840]: cluster 2024-09-13T22:51:38.743808+0000 mgr.y (mgr.14613) 521 : cluster [DBG] pgmap v598: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_toofull, 165 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 742 op/s; 7968/211464 objects misplaced (3.768%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T22:51:40.114 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:40.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:40 smithi192 bash[24608]: cluster 2024-09-13T22:51:38.743808+0000 mgr.y (mgr.14613) 521 : cluster [DBG] pgmap v598: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_toofull, 165 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 742 op/s; 7968/211464 objects misplaced (3.768%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T22:51:40.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:40.242 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:40.262 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-09-13T22:51:34.741273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4735048243,"num_objects":72444,"num_object_clones":0,"num_object_copies":217332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8603,"num_objects_unfound":0,"num_objects_dirty":72444,"num_whiteouts":0,"num_read":1968,"num_read_kb":1720,"num_write":116429,"num_write_kb":5680430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28022,"num_bytes_recovered":1822953650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151064,"ondisk_log_size":151064,"up":504,"acting":503,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":518,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18425932,"kb_used_data":15263648,"kb_used_omap":12,"kb_used_meta":3161971,"kb_avail":731535284,"statfs":{"total":767960285184,"available":749092130816,"internally_reserved":0,"allocated":15629975552,"data_stored":15625004435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3237858864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-431423488,"num_objects":-6583,"num_object_clones":0,"num_object_copies":-19749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2811,"num_objects_unfound":0,"num_objects_dirty":-6583,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6583,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":69009408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001940"},"pg_stats":[{"pgid":"9.14","version":"293'6518","reported_seq":10957,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:29.916130+0000","last_change":"2024-09-13T22:50:43.959317+0000","last_active":"2024-09-13T22:51:29.916130+0000","last_peered":"2024-09-13T22:51:29.916130+0000","last_clean":"2024-09-13T22:51:29.916130+0000","last_became_active":"2024-09-13T22:50:40.321393+0000","last_became_peered":"2024-09-13T22:50:40.321393+0000","last_unstale":"2024-09-13T22:51:29.916130+0000","last_undegraded":"2024-09-13T22:51:29.916130+0000","last_fullsized":"2024-09-13T22:51:29.916130+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6518,"log_dups_size":0,"ondisk_log_size":6518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:33.029312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156631040,"num_objects":2390,"num_object_clones":0,"num_object_copies":7170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":201216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1559,"num_bytes_recovered":102137856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.964981+0000","last_change":"2024-09-13T22:49:37.429809+0000","last_active":"2024-09-13T22:51:27.964981+0000","last_peered":"2024-09-13T22:51:27.964981+0000","last_clean":"2024-09-13T22:51:27.964981+0000","last_became_active":"2024-09-13T22:37:47.691682+0000","last_became_peered":"2024-09-13T22:37:47.691682+0000","last_unstale":"2024-09-13T22:51:27.964981+0000","last_undegraded":"2024-09-13T22:51:27.964981+0000","last_fullsized":"2024-09-13T22:51:27.964981+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:37.429474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:49:37.429474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":518,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975371+0000","last_change":"2024-09-13T22:50:03.973780+0000","last_active":"2024-09-13T22:51:27.975371+0000","last_peered":"2024-09-13T22:51:27.975371+0000","last_clean":"2024-09-13T22:51:27.975371+0000","last_became_active":"2024-09-13T22:50:03.973150+0000","last_became_peered":"2024-09-13T22:50:03.973150+0000","last_unstale":"2024-09-13T22:51:27.975371+0000","last_undegraded":"2024-09-13T22:51:27.975371+0000","last_fullsized":"2024-09-13T22:51:27.975371+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:21.195029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:46.613466+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.195029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"228'25","reported_seq":523,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.959907+0000","last_change":"2024-09-13T22:47:03.372631+0000","last_active":"2024-09-13T22:51:27.959907+0000","last_peered":"2024-09-13T22:51:27.959907+0000","last_clean":"2024-09-13T22:51:27.959907+0000","last_became_active":"2024-09-13T22:37:43.299916+0000","last_became_peered":"2024-09-13T22:37:43.299916+0000","last_unstale":"2024-09-13T22:51:27.959907+0000","last_undegraded":"2024-09-13T22:51:27.959907+0000","last_fullsized":"2024-09-13T22:51:27.959907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'15","last_scrub_stamp":"2024-09-13T22:47:03.372362+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:47:03.372362+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611723+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962788+0000","last_change":"2024-09-13T22:50:03.983646+0000","last_active":"2024-09-13T22:51:32.962788+0000","last_peered":"2024-09-13T22:51:32.962788+0000","last_clean":"2024-09-13T22:51:32.962788+0000","last_became_active":"2024-09-13T22:50:03.982074+0000","last_became_peered":"2024-09-13T22:50:03.982074+0000","last_unstale":"2024-09-13T22:51:32.962788+0000","last_undegraded":"2024-09-13T22:51:32.962788+0000","last_fullsized":"2024-09-13T22:51:32.962788+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:14.210087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:48:14.210087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.15","version":"295'6556","reported_seq":15474,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.390585+0000","last_change":"2024-09-13T22:51:11.948789+0000","last_active":"2024-09-13T22:51:33.390585+0000","last_peered":"2024-09-13T22:51:33.390585+0000","last_clean":"2024-09-13T22:51:33.390585+0000","last_became_active":"2024-09-13T22:51:11.941823+0000","last_became_peered":"2024-09-13T22:51:11.941823+0000","last_unstale":"2024-09-13T22:51:33.390585+0000","last_undegraded":"2024-09-13T22:51:33.390585+0000","last_fullsized":"2024-09-13T22:51:33.390585+0000","mapping_epoch":288,"log_start":"255'388","ondisk_log_start":"255'388","created":276,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6168,"log_dups_size":0,"ondisk_log_size":6168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:34.163047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144343040,"num_objects":2203,"num_object_clones":0,"num_object_copies":6609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":198304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2932,"num_bytes_recovered":192151552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":492,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960007+0000","last_change":"2024-09-13T22:49:16.307170+0000","last_active":"2024-09-13T22:51:27.960007+0000","last_peered":"2024-09-13T22:51:27.960007+0000","last_clean":"2024-09-13T22:51:27.960007+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:51:27.960007+0000","last_undegraded":"2024-09-13T22:51:27.960007+0000","last_fullsized":"2024-09-13T22:51:27.960007+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.306966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.306966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"228'24","reported_seq":532,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975187+0000","last_change":"2024-09-13T22:49:22.149867+0000","last_active":"2024-09-13T22:51:27.975187+0000","last_peered":"2024-09-13T22:51:27.975187+0000","last_clean":"2024-09-13T22:51:27.975187+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:51:27.975187+0000","last_undegraded":"2024-09-13T22:51:27.975187+0000","last_fullsized":"2024-09-13T22:51:27.975187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:49:22.149762+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.149762+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979147+0000","last_change":"2024-09-13T22:48:30.135471+0000","last_active":"2024-09-13T22:51:27.979147+0000","last_peered":"2024-09-13T22:51:27.979147+0000","last_clean":"2024-09-13T22:51:27.979147+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:51:27.979147+0000","last_undegraded":"2024-09-13T22:51:27.979147+0000","last_fullsized":"2024-09-13T22:51:27.979147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:30.135258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:48:30.135258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963360+0000","last_change":"2024-09-13T22:50:04.001279+0000","last_active":"2024-09-13T22:51:32.963360+0000","last_peered":"2024-09-13T22:51:32.963360+0000","last_clean":"2024-09-13T22:51:32.963360+0000","last_became_active":"2024-09-13T22:50:04.000951+0000","last_became_peered":"2024-09-13T22:50:04.000951+0000","last_unstale":"2024-09-13T22:51:32.963360+0000","last_undegraded":"2024-09-13T22:51:32.963360+0000","last_fullsized":"2024-09-13T22:51:32.963360+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.803924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.803924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"295'6634","reported_seq":12715,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.525269+0000","last_change":"2024-09-13T22:51:08.905509+0000","last_active":"2024-09-13T22:51:33.525269+0000","last_peered":"2024-09-13T22:51:33.525269+0000","last_clean":"2024-09-13T22:51:33.525269+0000","last_became_active":"2024-09-13T22:51:08.902124+0000","last_became_peered":"2024-09-13T22:51:08.902124+0000","last_unstale":"2024-09-13T22:51:33.525269+0000","last_undegraded":"2024-09-13T22:51:33.525269+0000","last_fullsized":"2024-09-13T22:51:33.525269+0000","mapping_epoch":285,"log_start":"255'500","ondisk_log_start":"255'500","created":276,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:35.405071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151879680,"num_objects":2318,"num_object_clones":0,"num_object_copies":6954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":203232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2981,"num_bytes_recovered":195362816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":494,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972585+0000","last_change":"2024-09-13T22:48:11.350608+0000","last_active":"2024-09-13T22:51:32.972585+0000","last_peered":"2024-09-13T22:51:32.972585+0000","last_clean":"2024-09-13T22:51:32.972585+0000","last_became_active":"2024-09-13T22:37:40.757419+0000","last_became_peered":"2024-09-13T22:37:40.757419+0000","last_unstale":"2024-09-13T22:51:32.972585+0000","last_undegraded":"2024-09-13T22:51:32.972585+0000","last_fullsized":"2024-09-13T22:51:32.972585+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.350541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.350541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"228'34","reported_seq":600,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973368+0000","last_change":"2024-09-13T22:50:38.278190+0000","last_active":"2024-09-13T22:51:32.973368+0000","last_peered":"2024-09-13T22:51:32.973368+0000","last_clean":"2024-09-13T22:51:32.973368+0000","last_became_active":"2024-09-13T22:50:03.990784+0000","last_became_peered":"2024-09-13T22:50:03.990784+0000","last_unstale":"2024-09-13T22:51:32.973368+0000","last_undegraded":"2024-09-13T22:51:32.973368+0000","last_fullsized":"2024-09-13T22:51:32.973368+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:50:38.278054+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:50:38.278054+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:58.589142+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":490,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971570+0000","last_change":"2024-09-13T22:50:03.995918+0000","last_active":"2024-09-13T22:51:32.971570+0000","last_peered":"2024-09-13T22:51:32.971570+0000","last_clean":"2024-09-13T22:51:32.971570+0000","last_became_active":"2024-09-13T22:50:03.995043+0000","last_became_peered":"2024-09-13T22:50:03.995043+0000","last_unstale":"2024-09-13T22:51:32.971570+0000","last_undegraded":"2024-09-13T22:51:32.971570+0000","last_fullsized":"2024-09-13T22:51:32.971570+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:10.365482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:48:10.365482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":490,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.974024+0000","last_change":"2024-09-13T22:50:27.114545+0000","last_active":"2024-09-13T22:51:32.974024+0000","last_peered":"2024-09-13T22:51:32.974024+0000","last_clean":"2024-09-13T22:51:32.974024+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:51:32.974024+0000","last_undegraded":"2024-09-13T22:51:32.974024+0000","last_fullsized":"2024-09-13T22:51:32.974024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:27.114380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:50:27.114380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"295'6621","reported_seq":15453,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.537394+0000","last_change":"2024-09-13T22:51:09.915992+0000","last_active":"2024-09-13T22:51:33.537394+0000","last_peered":"2024-09-13T22:51:33.537394+0000","last_clean":"2024-09-13T22:51:33.537394+0000","last_became_active":"2024-09-13T22:51:09.907668+0000","last_became_peered":"2024-09-13T22:51:09.907668+0000","last_unstale":"2024-09-13T22:51:33.537394+0000","last_undegraded":"2024-09-13T22:51:33.537394+0000","last_fullsized":"2024-09-13T22:51:33.537394+0000","mapping_epoch":286,"log_start":"255'500","ondisk_log_start":"255'500","created":276,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:37.171661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143556608,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":199264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2904,"num_bytes_recovered":190316544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":502,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972537+0000","last_change":"2024-09-13T22:49:39.881881+0000","last_active":"2024-09-13T22:51:32.972537+0000","last_peered":"2024-09-13T22:51:32.972537+0000","last_clean":"2024-09-13T22:51:32.972537+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:51:32.972537+0000","last_undegraded":"2024-09-13T22:51:32.972537+0000","last_fullsized":"2024-09-13T22:51:32.972537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:39.881753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:39.881753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"228'18","reported_seq":574,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975575+0000","last_change":"2024-09-13T22:50:19.094582+0000","last_active":"2024-09-13T22:51:32.975575+0000","last_peered":"2024-09-13T22:51:32.975575+0000","last_clean":"2024-09-13T22:51:32.975575+0000","last_became_active":"2024-09-13T22:50:03.977915+0000","last_became_peered":"2024-09-13T22:50:03.977915+0000","last_unstale":"2024-09-13T22:51:32.975575+0000","last_undegraded":"2024-09-13T22:51:32.975575+0000","last_fullsized":"2024-09-13T22:51:32.975575+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:50:19.094498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:50:19.094498+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":494,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.976155+0000","last_change":"2024-09-13T22:49:42.177553+0000","last_active":"2024-09-13T22:51:32.976155+0000","last_peered":"2024-09-13T22:51:32.976155+0000","last_clean":"2024-09-13T22:51:32.976155+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:51:32.976155+0000","last_undegraded":"2024-09-13T22:51:32.976155+0000","last_fullsized":"2024-09-13T22:51:32.976155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T22:49:42.177430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":487,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.984508+0000","last_change":"2024-09-13T22:49:34.085378+0000","last_active":"2024-09-13T22:51:27.984508+0000","last_peered":"2024-09-13T22:51:27.984508+0000","last_clean":"2024-09-13T22:51:27.984508+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:51:27.984508+0000","last_undegraded":"2024-09-13T22:51:27.984508+0000","last_fullsized":"2024-09-13T22:51:27.984508+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:34.085275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:34.085275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"293'6456","reported_seq":6560,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:29.872409+0000","last_change":"2024-09-13T22:50:40.252078+0000","last_active":"2024-09-13T22:51:29.872409+0000","last_peered":"2024-09-13T22:51:29.872409+0000","last_clean":"2024-09-13T22:51:29.872409+0000","last_became_active":"2024-09-13T22:50:40.249239+0000","last_became_peered":"2024-09-13T22:50:40.249239+0000","last_unstale":"2024-09-13T22:51:29.872409+0000","last_undegraded":"2024-09-13T22:51:29.872409+0000","last_fullsized":"2024-09-13T22:51:29.872409+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:30.050603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155877376,"num_objects":2378,"num_object_clones":0,"num_object_copies":7134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3878,"num_write_kb":200224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":556,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973178+0000","last_change":"2024-09-13T22:50:37.308204+0000","last_active":"2024-09-13T22:51:32.973178+0000","last_peered":"2024-09-13T22:51:32.973178+0000","last_clean":"2024-09-13T22:51:32.973178+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:51:32.973178+0000","last_undegraded":"2024-09-13T22:51:32.973178+0000","last_fullsized":"2024-09-13T22:51:32.973178+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:50:37.305809+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:50:37.305809+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:43.300274+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"228'44","reported_seq":631,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.974976+0000","last_change":"2024-09-13T22:50:16.286704+0000","last_active":"2024-09-13T22:51:32.974976+0000","last_peered":"2024-09-13T22:51:32.974976+0000","last_clean":"2024-09-13T22:51:32.974976+0000","last_became_active":"2024-09-13T22:50:03.985629+0000","last_became_peered":"2024-09-13T22:50:03.985629+0000","last_unstale":"2024-09-13T22:51:32.974976+0000","last_undegraded":"2024-09-13T22:51:32.974976+0000","last_fullsized":"2024-09-13T22:51:32.974976+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:50:16.286389+0000","last_deep_scrub":"114'28","last_deep_scrub_stamp":"2024-09-13T22:40:44.161919+0000","last_clean_scrub_stamp":"2024-09-13T22:50:16.286389+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:44.161280+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":507,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971526+0000","last_change":"2024-09-13T22:50:04.002780+0000","last_active":"2024-09-13T22:51:32.971526+0000","last_peered":"2024-09-13T22:51:32.971526+0000","last_clean":"2024-09-13T22:51:32.971526+0000","last_became_active":"2024-09-13T22:50:04.002375+0000","last_became_peered":"2024-09-13T22:50:04.002375+0000","last_unstale":"2024-09-13T22:51:32.971526+0000","last_undegraded":"2024-09-13T22:51:32.971526+0000","last_fullsized":"2024-09-13T22:51:32.971526+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:26.959784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.959784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":504,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979312+0000","last_change":"2024-09-13T22:50:03.978683+0000","last_active":"2024-09-13T22:51:27.979312+0000","last_peered":"2024-09-13T22:51:27.979312+0000","last_clean":"2024-09-13T22:51:27.979312+0000","last_became_active":"2024-09-13T22:50:03.976801+0000","last_became_peered":"2024-09-13T22:50:03.976801+0000","last_unstale":"2024-09-13T22:51:27.979312+0000","last_undegraded":"2024-09-13T22:51:27.979312+0000","last_fullsized":"2024-09-13T22:51:27.979312+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:26.457236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.457236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":407,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960498+0000","last_change":"2024-09-13T22:50:03.998970+0000","last_active":"2024-09-13T22:51:27.960498+0000","last_peered":"2024-09-13T22:51:27.960498+0000","last_clean":"2024-09-13T22:51:27.960498+0000","last_became_active":"2024-09-13T22:50:03.998504+0000","last_became_peered":"2024-09-13T22:50:03.998504+0000","last_unstale":"2024-09-13T22:51:27.960498+0000","last_undegraded":"2024-09-13T22:51:27.960498+0000","last_fullsized":"2024-09-13T22:51:27.960498+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:20.263925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.263925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"293'6484","reported_seq":6588,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.599124+0000","last_change":"2024-09-13T22:50:40.269907+0000","last_active":"2024-09-13T22:51:30.599124+0000","last_peered":"2024-09-13T22:51:30.599124+0000","last_clean":"2024-09-13T22:51:30.599124+0000","last_became_active":"2024-09-13T22:50:40.267554+0000","last_became_peered":"2024-09-13T22:50:40.267554+0000","last_unstale":"2024-09-13T22:51:30.599124+0000","last_undegraded":"2024-09-13T22:51:30.599124+0000","last_fullsized":"2024-09-13T22:51:30.599124+0000","mapping_epoch":251,"log_start":"255'300","ondisk_log_start":"255'300","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6184,"log_dups_size":0,"ondisk_log_size":6184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:34.666714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153812992,"num_objects":2347,"num_object_clones":0,"num_object_copies":7041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":200256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":773,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.970914+0000","last_change":"2024-09-13T22:49:20.975665+0000","last_active":"2024-09-13T22:51:32.970914+0000","last_peered":"2024-09-13T22:51:32.970914+0000","last_clean":"2024-09-13T22:51:32.970914+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:51:32.970914+0000","last_undegraded":"2024-09-13T22:51:32.970914+0000","last_fullsized":"2024-09-13T22:51:32.970914+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:20.975463+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.975463+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"228'19","reported_seq":549,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.965404+0000","last_change":"2024-09-13T22:50:04.032174+0000","last_active":"2024-09-13T22:51:27.965404+0000","last_peered":"2024-09-13T22:51:27.965404+0000","last_clean":"2024-09-13T22:51:27.965404+0000","last_became_active":"2024-09-13T22:50:03.999716+0000","last_became_peered":"2024-09-13T22:50:03.999716+0000","last_unstale":"2024-09-13T22:51:27.965404+0000","last_undegraded":"2024-09-13T22:51:27.965404+0000","last_fullsized":"2024-09-13T22:51:27.965404+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:13.725631+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:48:13.725631+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611767+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.992072+0000","last_change":"2024-09-13T22:49:25.446162+0000","last_active":"2024-09-13T22:51:27.992072+0000","last_peered":"2024-09-13T22:51:27.992072+0000","last_clean":"2024-09-13T22:51:27.992072+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:51:27.992072+0000","last_undegraded":"2024-09-13T22:51:27.992072+0000","last_fullsized":"2024-09-13T22:51:27.992072+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:25.445965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:25.445965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972377+0000","last_change":"2024-09-13T22:49:45.642573+0000","last_active":"2024-09-13T22:51:32.972377+0000","last_peered":"2024-09-13T22:51:32.972377+0000","last_clean":"2024-09-13T22:51:32.972377+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T22:51:32.972377+0000","last_undegraded":"2024-09-13T22:51:32.972377+0000","last_fullsized":"2024-09-13T22:51:32.972377+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:48:28.189507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.189507+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964282+0000","last_change":"2024-09-13T22:50:04.000374+0000","last_active":"2024-09-13T22:51:32.964282+0000","last_peered":"2024-09-13T22:51:32.964282+0000","last_clean":"2024-09-13T22:51:32.964282+0000","last_became_active":"2024-09-13T22:50:04.000025+0000","last_became_peered":"2024-09-13T22:50:04.000025+0000","last_unstale":"2024-09-13T22:51:32.964282+0000","last_undegraded":"2024-09-13T22:51:32.964282+0000","last_fullsized":"2024-09-13T22:51:32.964282+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:26.090972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.090972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"293'6309","reported_seq":6413,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.545815+0000","last_change":"2024-09-13T22:50:40.261469+0000","last_active":"2024-09-13T22:51:30.545815+0000","last_peered":"2024-09-13T22:51:30.545815+0000","last_clean":"2024-09-13T22:51:30.545815+0000","last_became_active":"2024-09-13T22:50:40.258686+0000","last_became_peered":"2024-09-13T22:50:40.258686+0000","last_unstale":"2024-09-13T22:51:30.545815+0000","last_undegraded":"2024-09-13T22:51:30.545815+0000","last_fullsized":"2024-09-13T22:51:30.545815+0000","mapping_epoch":251,"log_start":"255'200","ondisk_log_start":"255'200","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6109,"log_dups_size":0,"ondisk_log_size":6109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:34.666774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149815296,"num_objects":2286,"num_object_clones":0,"num_object_copies":6858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3794,"num_write_kb":194560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":556,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975332+0000","last_change":"2024-09-13T22:50:43.282723+0000","last_active":"2024-09-13T22:51:32.975332+0000","last_peered":"2024-09-13T22:51:32.975332+0000","last_clean":"2024-09-13T22:51:32.975332+0000","last_became_active":"2024-09-13T22:45:58.880701+0000","last_became_peered":"2024-09-13T22:45:58.880701+0000","last_unstale":"2024-09-13T22:51:32.975332+0000","last_undegraded":"2024-09-13T22:51:32.975332+0000","last_fullsized":"2024-09-13T22:51:32.975332+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:50:43.282598+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:50:43.282598+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:48.519519+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"228'27","reported_seq":531,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960831+0000","last_change":"2024-09-13T22:48:23.929452+0000","last_active":"2024-09-13T22:51:27.960831+0000","last_peered":"2024-09-13T22:51:27.960831+0000","last_clean":"2024-09-13T22:51:27.960831+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:51:27.960831+0000","last_undegraded":"2024-09-13T22:51:27.960831+0000","last_fullsized":"2024-09-13T22:51:27.960831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:48:23.929342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.929342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612007+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979302+0000","last_change":"2024-09-13T22:48:37.212544+0000","last_active":"2024-09-13T22:51:27.979302+0000","last_peered":"2024-09-13T22:51:27.979302+0000","last_clean":"2024-09-13T22:51:27.979302+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:51:27.979302+0000","last_undegraded":"2024-09-13T22:51:27.979302+0000","last_fullsized":"2024-09-13T22:51:27.979302+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:37.201926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:37.201926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"266'14","reported_seq":95,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964362+0000","last_change":"2024-09-13T22:50:04.013350+0000","last_active":"2024-09-13T22:51:32.964362+0000","last_peered":"2024-09-13T22:51:32.964362+0000","last_clean":"2024-09-13T22:51:32.964362+0000","last_became_active":"2024-09-13T22:50:03.990705+0000","last_became_peered":"2024-09-13T22:50:03.990705+0000","last_unstale":"2024-09-13T22:51:32.964362+0000","last_undegraded":"2024-09-13T22:51:32.964362+0000","last_fullsized":"2024-09-13T22:51:32.964362+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-09-13T22:48:08.989696+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.989696+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":483,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973695+0000","last_change":"2024-09-13T22:48:39.747979+0000","last_active":"2024-09-13T22:51:32.973695+0000","last_peered":"2024-09-13T22:51:32.973695+0000","last_clean":"2024-09-13T22:51:32.973695+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:51:32.973695+0000","last_undegraded":"2024-09-13T22:51:32.973695+0000","last_fullsized":"2024-09-13T22:51:32.973695+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.747881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.747881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"295'6510","reported_seq":6617,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.495954+0000","last_change":"2024-09-13T22:50:40.266156+0000","last_active":"2024-09-13T22:51:33.495954+0000","last_peered":"2024-09-13T22:51:33.495954+0000","last_clean":"2024-09-13T22:51:33.495954+0000","last_became_active":"2024-09-13T22:50:40.263855+0000","last_became_peered":"2024-09-13T22:50:40.263855+0000","last_unstale":"2024-09-13T22:51:33.495954+0000","last_undegraded":"2024-09-13T22:51:33.495954+0000","last_fullsized":"2024-09-13T22:51:33.495954+0000","mapping_epoch":251,"log_start":"255'900","ondisk_log_start":"255'900","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:34.163020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143261696,"num_objects":2186,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3960,"num_write_kb":196672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"228'18","reported_seq":85,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962893+0000","last_change":"2024-09-13T22:50:04.022082+0000","last_active":"2024-09-13T22:51:32.962893+0000","last_peered":"2024-09-13T22:51:32.962893+0000","last_clean":"2024-09-13T22:51:32.962893+0000","last_became_active":"2024-09-13T22:50:03.994289+0000","last_became_peered":"2024-09-13T22:50:03.994289+0000","last_unstale":"2024-09-13T22:51:32.962893+0000","last_undegraded":"2024-09-13T22:51:32.962893+0000","last_fullsized":"2024-09-13T22:51:32.962893+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:48:20.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.191528+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":495,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.970947+0000","last_change":"2024-09-13T22:49:38.845359+0000","last_active":"2024-09-13T22:51:32.970947+0000","last_peered":"2024-09-13T22:51:32.970947+0000","last_clean":"2024-09-13T22:51:32.970947+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:51:32.970947+0000","last_undegraded":"2024-09-13T22:51:32.970947+0000","last_fullsized":"2024-09-13T22:51:32.970947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:38.845293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:38.845293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":304,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972224+0000","last_change":"2024-09-13T22:50:03.996275+0000","last_active":"2024-09-13T22:51:32.972224+0000","last_peered":"2024-09-13T22:51:32.972224+0000","last_clean":"2024-09-13T22:51:32.972224+0000","last_became_active":"2024-09-13T22:50:03.995778+0000","last_became_peered":"2024-09-13T22:50:03.995778+0000","last_unstale":"2024-09-13T22:51:32.972224+0000","last_undegraded":"2024-09-13T22:51:32.972224+0000","last_fullsized":"2024-09-13T22:51:32.972224+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:27.194219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:48:27.194219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":472,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.961304+0000","last_change":"2024-09-13T22:45:03.478116+0000","last_active":"2024-09-13T22:51:27.961304+0000","last_peered":"2024-09-13T22:51:27.961304+0000","last_clean":"2024-09-13T22:51:27.961304+0000","last_became_active":"2024-09-13T22:37:47.691581+0000","last_became_peered":"2024-09-13T22:37:47.691581+0000","last_unstale":"2024-09-13T22:51:27.961304+0000","last_undegraded":"2024-09-13T22:51:27.961304+0000","last_fullsized":"2024-09-13T22:51:27.961304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:03.477932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:03.477932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"293'7886","reported_seq":14430,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.605703+0000","last_change":"2024-09-13T22:51:06.814044+0000","last_active":"2024-09-13T22:51:30.605703+0000","last_peered":"2024-09-13T22:51:30.605703+0000","last_clean":"2024-09-13T22:51:30.605703+0000","last_became_active":"2024-09-13T22:51:06.808985+0000","last_became_peered":"2024-09-13T22:51:06.808985+0000","last_unstale":"2024-09-13T22:51:30.605703+0000","last_undegraded":"2024-09-13T22:51:30.605703+0000","last_fullsized":"2024-09-13T22:51:30.605703+0000","mapping_epoch":283,"log_start":"267'3496","ondisk_log_start":"267'3496","created":251,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":400576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3218,"num_bytes_recovered":210894848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":536,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.961220+0000","last_change":"2024-09-13T22:45:59.360761+0000","last_active":"2024-09-13T22:51:27.961220+0000","last_peered":"2024-09-13T22:51:27.961220+0000","last_clean":"2024-09-13T22:51:27.961220+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:51:27.961220+0000","last_undegraded":"2024-09-13T22:51:27.961220+0000","last_fullsized":"2024-09-13T22:51:27.961220+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.360631+0000","last_clean_scrub_stamp":"2024-09-13T22:45:59.360631+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"228'24","reported_seq":572,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975281+0000","last_change":"2024-09-13T22:50:04.013222+0000","last_active":"2024-09-13T22:51:27.975281+0000","last_peered":"2024-09-13T22:51:27.975281+0000","last_clean":"2024-09-13T22:51:27.975281+0000","last_became_active":"2024-09-13T22:50:03.973976+0000","last_became_peered":"2024-09-13T22:50:03.973976+0000","last_unstale":"2024-09-13T22:51:27.975281+0000","last_undegraded":"2024-09-13T22:51:27.975281+0000","last_fullsized":"2024-09-13T22:51:27.975281+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:49:36.082957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:36.082957+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.976940+0000","last_change":"2024-09-13T22:50:26.282546+0000","last_active":"2024-09-13T22:51:32.976940+0000","last_peered":"2024-09-13T22:51:32.976940+0000","last_clean":"2024-09-13T22:51:32.976940+0000","last_became_active":"2024-09-13T22:37:40.742041+0000","last_became_peered":"2024-09-13T22:37:40.742041+0000","last_unstale":"2024-09-13T22:51:32.976940+0000","last_undegraded":"2024-09-13T22:51:32.976940+0000","last_fullsized":"2024-09-13T22:51:32.976940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:26.282468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:26.282468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:50.299890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":77,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962798+0000","last_change":"2024-09-13T22:50:04.005424+0000","last_active":"2024-09-13T22:51:32.962798+0000","last_peered":"2024-09-13T22:51:32.962798+0000","last_clean":"2024-09-13T22:51:32.962798+0000","last_became_active":"2024-09-13T22:50:03.983986+0000","last_became_peered":"2024-09-13T22:50:03.983986+0000","last_unstale":"2024-09-13T22:51:32.962798+0000","last_undegraded":"2024-09-13T22:51:32.962798+0000","last_fullsized":"2024-09-13T22:51:32.962798+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:48:34.822707+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:48:34.822707+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"266'13","reported_seq":673,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960163+0000","last_change":"2024-09-13T22:50:04.034140+0000","last_active":"2024-09-13T22:51:27.960163+0000","last_peered":"2024-09-13T22:51:27.960163+0000","last_clean":"2024-09-13T22:51:27.960163+0000","last_became_active":"2024-09-13T22:50:03.999194+0000","last_became_peered":"2024-09-13T22:50:03.999194+0000","last_unstale":"2024-09-13T22:51:27.960163+0000","last_undegraded":"2024-09-13T22:51:27.960163+0000","last_fullsized":"2024-09-13T22:51:27.960163+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:38.494521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:38.494521+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611506+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":483,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971265+0000","last_change":"2024-09-13T22:49:30.942556+0000","last_active":"2024-09-13T22:51:32.971265+0000","last_peered":"2024-09-13T22:51:32.971265+0000","last_clean":"2024-09-13T22:51:32.971265+0000","last_became_active":"2024-09-13T22:37:47.688654+0000","last_became_peered":"2024-09-13T22:37:47.688654+0000","last_unstale":"2024-09-13T22:51:32.971265+0000","last_undegraded":"2024-09-13T22:51:32.971265+0000","last_fullsized":"2024-09-13T22:51:32.971265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:30.942259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:30.942259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"295'7842","reported_seq":7944,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.519859+0000","last_change":"2024-09-13T22:49:19.844311+0000","last_active":"2024-09-13T22:51:33.519859+0000","last_peered":"2024-09-13T22:51:33.519859+0000","last_clean":"2024-09-13T22:51:33.519859+0000","last_became_active":"2024-09-13T22:49:19.843295+0000","last_became_peered":"2024-09-13T22:49:19.843295+0000","last_unstale":"2024-09-13T22:51:33.519859+0000","last_undegraded":"2024-09-13T22:51:33.519859+0000","last_fullsized":"2024-09-13T22:51:33.519859+0000","mapping_epoch":251,"log_start":"271'4002","ondisk_log_start":"271'4002","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":3840,"log_dups_size":0,"ondisk_log_size":3840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287113240,"num_objects":4382,"num_object_clones":0,"num_object_copies":13146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7842,"num_write_kb":391105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":516,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964376+0000","last_change":"2024-09-13T22:50:06.962705+0000","last_active":"2024-09-13T22:51:32.964376+0000","last_peered":"2024-09-13T22:51:32.964376+0000","last_clean":"2024-09-13T22:51:32.964376+0000","last_became_active":"2024-09-13T22:50:06.962282+0000","last_became_peered":"2024-09-13T22:50:06.962282+0000","last_unstale":"2024-09-13T22:51:32.964376+0000","last_undegraded":"2024-09-13T22:51:32.964376+0000","last_fullsized":"2024-09-13T22:51:32.964376+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:49:26.230025+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:49:26.230025+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"228'22","reported_seq":526,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960895+0000","last_change":"2024-09-13T22:49:23.376019+0000","last_active":"2024-09-13T22:51:27.960895+0000","last_peered":"2024-09-13T22:51:27.960895+0000","last_clean":"2024-09-13T22:51:27.960895+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:51:27.960895+0000","last_undegraded":"2024-09-13T22:51:27.960895+0000","last_fullsized":"2024-09-13T22:51:27.960895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:49:23.375848+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:49:23.375848+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":522,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.989963+0000","last_change":"2024-09-13T22:50:04.017241+0000","last_active":"2024-09-13T22:51:27.989963+0000","last_peered":"2024-09-13T22:51:27.989963+0000","last_clean":"2024-09-13T22:51:27.989963+0000","last_became_active":"2024-09-13T22:50:03.979523+0000","last_became_peered":"2024-09-13T22:50:03.979523+0000","last_unstale":"2024-09-13T22:51:27.989963+0000","last_undegraded":"2024-09-13T22:51:27.989963+0000","last_fullsized":"2024-09-13T22:51:27.989963+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:49:31.481514+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:49:31.481514+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'5","reported_seq":1296,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:29.193215+0000","last_change":"2024-09-13T22:45:54.489191+0000","last_active":"2024-09-13T22:51:29.193215+0000","last_peered":"2024-09-13T22:51:29.193215+0000","last_clean":"2024-09-13T22:51:29.193215+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:51:29.193215+0000","last_undegraded":"2024-09-13T22:51:29.193215+0000","last_fullsized":"2024-09-13T22:51:29.193215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-13T22:45:54.489086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:45:54.489086+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611636+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":806,"num_read_kb":801,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962818+0000","last_change":"2024-09-13T22:50:03.995424+0000","last_active":"2024-09-13T22:51:32.962818+0000","last_peered":"2024-09-13T22:51:32.962818+0000","last_clean":"2024-09-13T22:51:32.962818+0000","last_became_active":"2024-09-13T22:50:03.993502+0000","last_became_peered":"2024-09-13T22:50:03.993502+0000","last_unstale":"2024-09-13T22:51:32.962818+0000","last_undegraded":"2024-09-13T22:51:32.962818+0000","last_fullsized":"2024-09-13T22:51:32.962818+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:51.936005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.936005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963459+0000","last_change":"2024-09-13T22:50:03.984972+0000","last_active":"2024-09-13T22:51:32.963459+0000","last_peered":"2024-09-13T22:51:32.963459+0000","last_clean":"2024-09-13T22:51:32.963459+0000","last_became_active":"2024-09-13T22:50:03.982511+0000","last_became_peered":"2024-09-13T22:50:03.982511+0000","last_unstale":"2024-09-13T22:51:32.963459+0000","last_undegraded":"2024-09-13T22:51:32.963459+0000","last_fullsized":"2024-09-13T22:51:32.963459+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:37.838733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:45:37.838733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"295'7680","reported_seq":13859,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.424917+0000","last_change":"2024-09-13T22:50:42.258365+0000","last_active":"2024-09-13T22:51:33.424917+0000","last_peered":"2024-09-13T22:51:33.424917+0000","last_clean":"2024-09-13T22:51:33.424917+0000","last_became_active":"2024-09-13T22:50:42.254724+0000","last_became_peered":"2024-09-13T22:50:42.254724+0000","last_unstale":"2024-09-13T22:51:33.424917+0000","last_undegraded":"2024-09-13T22:51:33.424917+0000","last_fullsized":"2024-09-13T22:51:33.424917+0000","mapping_epoch":278,"log_start":"271'3843","ondisk_log_start":"271'3843","created":251,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283508736,"num_objects":4326,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7680,"num_write_kb":384192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3034,"num_bytes_recovered":198836224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":561,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.976329+0000","last_change":"2024-09-13T22:50:18.246875+0000","last_active":"2024-09-13T22:51:32.976329+0000","last_peered":"2024-09-13T22:51:32.976329+0000","last_clean":"2024-09-13T22:51:32.976329+0000","last_became_active":"2024-09-13T22:45:58.866066+0000","last_became_peered":"2024-09-13T22:45:58.866066+0000","last_unstale":"2024-09-13T22:51:32.976329+0000","last_undegraded":"2024-09-13T22:51:32.976329+0000","last_fullsized":"2024-09-13T22:51:32.976329+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:50:18.246752+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:50:18.246752+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"255'52","reported_seq":740,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.989485+0000","last_change":"2024-09-13T22:49:40.725923+0000","last_active":"2024-09-13T22:51:27.989485+0000","last_peered":"2024-09-13T22:51:27.989485+0000","last_clean":"2024-09-13T22:51:27.989485+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:51:27.989485+0000","last_undegraded":"2024-09-13T22:51:27.989485+0000","last_fullsized":"2024-09-13T22:51:27.989485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'52","last_scrub_stamp":"2024-09-13T22:49:40.725760+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:49:40.725760+0000","last_clean_scrub_stamp":"2024-09-13T22:49:40.725760+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":507,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972665+0000","last_change":"2024-09-13T22:50:03.997450+0000","last_active":"2024-09-13T22:51:32.972665+0000","last_peered":"2024-09-13T22:51:32.972665+0000","last_clean":"2024-09-13T22:51:32.972665+0000","last_became_active":"2024-09-13T22:50:03.996855+0000","last_became_peered":"2024-09-13T22:50:03.996855+0000","last_unstale":"2024-09-13T22:51:32.972665+0000","last_undegraded":"2024-09-13T22:51:32.972665+0000","last_fullsized":"2024-09-13T22:51:32.972665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.960501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.960501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.974842+0000","last_change":"2024-09-13T22:49:29.115748+0000","last_active":"2024-09-13T22:51:32.974842+0000","last_peered":"2024-09-13T22:51:32.974842+0000","last_clean":"2024-09-13T22:51:32.974842+0000","last_became_active":"2024-09-13T22:37:44.878445+0000","last_became_peered":"2024-09-13T22:37:44.878445+0000","last_unstale":"2024-09-13T22:51:32.974842+0000","last_undegraded":"2024-09-13T22:51:32.974842+0000","last_fullsized":"2024-09-13T22:51:32.974842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:49:29.115656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":480,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979027+0000","last_change":"2024-09-13T22:48:11.214965+0000","last_active":"2024-09-13T22:51:27.979027+0000","last_peered":"2024-09-13T22:51:27.979027+0000","last_clean":"2024-09-13T22:51:27.979027+0000","last_became_active":"2024-09-13T22:37:47.691745+0000","last_became_peered":"2024-09-13T22:37:47.691745+0000","last_unstale":"2024-09-13T22:51:27.979027+0000","last_undegraded":"2024-09-13T22:51:27.979027+0000","last_fullsized":"2024-09-13T22:51:27.979027+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:11.214834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:11.214834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"295'7914","reported_seq":8016,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.478048+0000","last_change":"2024-09-13T22:49:19.836085+0000","last_active":"2024-09-13T22:51:33.478048+0000","last_peered":"2024-09-13T22:51:33.478048+0000","last_clean":"2024-09-13T22:51:33.478048+0000","last_became_active":"2024-09-13T22:49:19.834656+0000","last_became_peered":"2024-09-13T22:49:19.834656+0000","last_unstale":"2024-09-13T22:51:33.478048+0000","last_undegraded":"2024-09-13T22:51:33.478048+0000","last_fullsized":"2024-09-13T22:51:33.478048+0000","mapping_epoch":251,"log_start":"267'3548","ondisk_log_start":"267'3548","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285999104,"num_objects":4364,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7914,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":569,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.976601+0000","last_change":"2024-09-13T22:50:41.423595+0000","last_active":"2024-09-13T22:51:32.976601+0000","last_peered":"2024-09-13T22:51:32.976601+0000","last_clean":"2024-09-13T22:51:32.976601+0000","last_became_active":"2024-09-13T22:45:58.862759+0000","last_became_peered":"2024-09-13T22:45:58.862759+0000","last_unstale":"2024-09-13T22:51:32.976601+0000","last_undegraded":"2024-09-13T22:51:32.976601+0000","last_fullsized":"2024-09-13T22:51:32.976601+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:50:41.423517+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:50:41.423517+0000","last_clean_scrub_stamp":"2024-09-13T22:50:41.423517+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:52:04.264158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"228'36","reported_seq":714,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975732+0000","last_change":"2024-09-13T22:49:40.203401+0000","last_active":"2024-09-13T22:51:32.975732+0000","last_peered":"2024-09-13T22:51:32.975732+0000","last_clean":"2024-09-13T22:51:32.975732+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:51:32.975732+0000","last_undegraded":"2024-09-13T22:51:32.975732+0000","last_fullsized":"2024-09-13T22:51:32.975732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'36","last_scrub_stamp":"2024-09-13T22:49:40.203232+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:49:40.203232+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.989542+0000","last_change":"2024-09-13T22:48:35.268201+0000","last_active":"2024-09-13T22:51:27.989542+0000","last_peered":"2024-09-13T22:51:27.989542+0000","last_clean":"2024-09-13T22:51:27.989542+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:51:27.989542+0000","last_undegraded":"2024-09-13T22:51:27.989542+0000","last_fullsized":"2024-09-13T22:51:27.989542+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:48:35.268073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:35.268073+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'2","reported_seq":525,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.991989+0000","last_change":"2024-09-13T22:50:04.020638+0000","last_active":"2024-09-13T22:51:27.991989+0000","last_peered":"2024-09-13T22:51:27.991989+0000","last_clean":"2024-09-13T22:51:27.991989+0000","last_became_active":"2024-09-13T22:50:03.977296+0000","last_became_peered":"2024-09-13T22:50:03.977296+0000","last_unstale":"2024-09-13T22:51:27.991989+0000","last_undegraded":"2024-09-13T22:51:27.991989+0000","last_fullsized":"2024-09-13T22:51:27.991989+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2","last_scrub_stamp":"2024-09-13T22:49:14.400987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.400987+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'12","reported_seq":677,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975617+0000","last_change":"2024-09-13T22:50:38.718417+0000","last_active":"2024-09-13T22:51:27.975617+0000","last_peered":"2024-09-13T22:51:27.975617+0000","last_clean":"2024-09-13T22:51:27.975617+0000","last_became_active":"2024-09-13T22:37:44.873478+0000","last_became_peered":"2024-09-13T22:37:44.873478+0000","last_unstale":"2024-09-13T22:51:27.975617+0000","last_undegraded":"2024-09-13T22:51:27.975617+0000","last_fullsized":"2024-09-13T22:51:27.975617+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:50:38.718325+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:50:38.718325+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:55.090544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":488,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.984142+0000","last_change":"2024-09-13T22:49:39.109496+0000","last_active":"2024-09-13T22:51:27.984142+0000","last_peered":"2024-09-13T22:51:27.984142+0000","last_clean":"2024-09-13T22:51:27.984142+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:51:27.984142+0000","last_undegraded":"2024-09-13T22:51:27.984142+0000","last_fullsized":"2024-09-13T22:51:27.984142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:39.109375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:39.109375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"293'6482","reported_seq":6619,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.623893+0000","last_change":"2024-09-13T22:50:41.028972+0000","last_active":"2024-09-13T22:51:30.623893+0000","last_peered":"2024-09-13T22:51:30.623893+0000","last_clean":"2024-09-13T22:51:30.623893+0000","last_became_active":"2024-09-13T22:50:40.254411+0000","last_became_peered":"2024-09-13T22:50:40.254411+0000","last_unstale":"2024-09-13T22:51:30.623893+0000","last_undegraded":"2024-09-13T22:51:30.623893+0000","last_fullsized":"2024-09-13T22:51:30.623893+0000","mapping_epoch":251,"log_start":"255'100","ondisk_log_start":"255'100","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'5128","last_scrub_stamp":"2024-09-13T22:50:41.028787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:50:41.028787+0000","objects_scrubbed":2594,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:52:03.585536+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152567808,"num_objects":2328,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1354,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":548,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973792+0000","last_change":"2024-09-13T22:48:26.832776+0000","last_active":"2024-09-13T22:51:32.973792+0000","last_peered":"2024-09-13T22:51:32.973792+0000","last_clean":"2024-09-13T22:51:32.973792+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:51:32.973792+0000","last_undegraded":"2024-09-13T22:51:32.973792+0000","last_fullsized":"2024-09-13T22:51:32.973792+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:48:26.832692+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"228'29","reported_seq":91,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963783+0000","last_change":"2024-09-13T22:50:04.027622+0000","last_active":"2024-09-13T22:51:32.963783+0000","last_peered":"2024-09-13T22:51:32.963783+0000","last_clean":"2024-09-13T22:51:32.963783+0000","last_became_active":"2024-09-13T22:50:03.997974+0000","last_became_peered":"2024-09-13T22:50:03.997974+0000","last_unstale":"2024-09-13T22:51:32.963783+0000","last_undegraded":"2024-09-13T22:51:32.963783+0000","last_fullsized":"2024-09-13T22:51:32.963783+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'29","last_scrub_stamp":"2024-09-13T22:49:25.093976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:25.093976+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.982451+0000","last_change":"2024-09-13T22:50:32.240630+0000","last_active":"2024-09-13T22:51:32.982451+0000","last_peered":"2024-09-13T22:51:32.982451+0000","last_clean":"2024-09-13T22:51:32.982451+0000","last_became_active":"2024-09-13T22:37:40.744235+0000","last_became_peered":"2024-09-13T22:37:40.744235+0000","last_unstale":"2024-09-13T22:51:32.982451+0000","last_undegraded":"2024-09-13T22:51:32.982451+0000","last_fullsized":"2024-09-13T22:51:32.982451+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:32.240527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:50:32.240527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:43.470181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":504,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979876+0000","last_change":"2024-09-13T22:50:03.973607+0000","last_active":"2024-09-13T22:51:27.979876+0000","last_peered":"2024-09-13T22:51:27.979876+0000","last_clean":"2024-09-13T22:51:27.979876+0000","last_became_active":"2024-09-13T22:50:03.972954+0000","last_became_peered":"2024-09-13T22:50:03.972954+0000","last_unstale":"2024-09-13T22:51:27.979876+0000","last_undegraded":"2024-09-13T22:51:27.979876+0000","last_fullsized":"2024-09-13T22:51:27.979876+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:19.165932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:49:19.165932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":484,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.959673+0000","last_change":"2024-09-13T22:50:03.999169+0000","last_active":"2024-09-13T22:51:27.959673+0000","last_peered":"2024-09-13T22:51:27.959673+0000","last_clean":"2024-09-13T22:51:27.959673+0000","last_became_active":"2024-09-13T22:50:03.998579+0000","last_became_peered":"2024-09-13T22:50:03.998579+0000","last_unstale":"2024-09-13T22:51:27.959673+0000","last_undegraded":"2024-09-13T22:51:27.959673+0000","last_fullsized":"2024-09-13T22:51:27.959673+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:42.535993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:42.535993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"295'8079","reported_seq":8181,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.466314+0000","last_change":"2024-09-13T22:49:19.844519+0000","last_active":"2024-09-13T22:51:33.466314+0000","last_peered":"2024-09-13T22:51:33.466314+0000","last_clean":"2024-09-13T22:51:33.466314+0000","last_became_active":"2024-09-13T22:49:19.840996+0000","last_became_peered":"2024-09-13T22:49:19.840996+0000","last_unstale":"2024-09-13T22:51:33.466314+0000","last_undegraded":"2024-09-13T22:51:33.466314+0000","last_fullsized":"2024-09-13T22:51:33.466314+0000","mapping_epoch":251,"log_start":"268'3694","ondisk_log_start":"268'3694","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287375360,"num_objects":4385,"num_object_clones":0,"num_object_copies":13155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":398848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":791,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971769+0000","last_change":"2024-09-13T22:50:06.969941+0000","last_active":"2024-09-13T22:51:32.971769+0000","last_peered":"2024-09-13T22:51:32.971769+0000","last_clean":"2024-09-13T22:51:32.971769+0000","last_became_active":"2024-09-13T22:50:06.969615+0000","last_became_peered":"2024-09-13T22:50:06.969615+0000","last_unstale":"2024-09-13T22:51:32.971769+0000","last_undegraded":"2024-09-13T22:51:32.971769+0000","last_fullsized":"2024-09-13T22:51:32.971769+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:38.173263+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.173263+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"228'38","reported_seq":554,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.978769+0000","last_change":"2024-09-13T22:49:36.839833+0000","last_active":"2024-09-13T22:51:27.978769+0000","last_peered":"2024-09-13T22:51:27.978769+0000","last_clean":"2024-09-13T22:51:27.978769+0000","last_became_active":"2024-09-13T22:37:43.300055+0000","last_became_peered":"2024-09-13T22:37:43.300055+0000","last_unstale":"2024-09-13T22:51:27.978769+0000","last_undegraded":"2024-09-13T22:51:27.978769+0000","last_fullsized":"2024-09-13T22:51:27.978769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:49:36.839475+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:49:36.839475+0000","last_clean_scrub_stamp":"2024-09-13T22:49:36.839475+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":528,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962246+0000","last_change":"2024-09-13T22:50:39.641050+0000","last_active":"2024-09-13T22:51:32.962246+0000","last_peered":"2024-09-13T22:51:32.962246+0000","last_clean":"2024-09-13T22:51:32.962246+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:51:32.962246+0000","last_undegraded":"2024-09-13T22:51:32.962246+0000","last_fullsized":"2024-09-13T22:51:32.962246+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:50:39.640890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:39.640890+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:51.539482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":512,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963910+0000","last_change":"2024-09-13T22:50:24.643629+0000","last_active":"2024-09-13T22:51:32.963910+0000","last_peered":"2024-09-13T22:51:32.963910+0000","last_clean":"2024-09-13T22:51:32.963910+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:51:32.963910+0000","last_undegraded":"2024-09-13T22:51:32.963910+0000","last_fullsized":"2024-09-13T22:51:32.963910+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:24.643550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:50:24.643550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:33.788259+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":483,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972353+0000","last_change":"2024-09-13T22:48:17.369617+0000","last_active":"2024-09-13T22:51:32.972353+0000","last_peered":"2024-09-13T22:51:32.972353+0000","last_clean":"2024-09-13T22:51:32.972353+0000","last_became_active":"2024-09-13T22:37:47.686302+0000","last_became_peered":"2024-09-13T22:37:47.686302+0000","last_unstale":"2024-09-13T22:51:32.972353+0000","last_undegraded":"2024-09-13T22:51:32.972353+0000","last_fullsized":"2024-09-13T22:51:32.972353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.369549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.369549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"293'7705","reported_seq":7802,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.603973+0000","last_change":"2024-09-13T22:49:19.908928+0000","last_active":"2024-09-13T22:51:30.603973+0000","last_peered":"2024-09-13T22:51:30.603973+0000","last_clean":"2024-09-13T22:51:30.603973+0000","last_became_active":"2024-09-13T22:49:19.908470+0000","last_became_peered":"2024-09-13T22:49:19.908470+0000","last_unstale":"2024-09-13T22:51:30.603973+0000","last_undegraded":"2024-09-13T22:51:30.603973+0000","last_fullsized":"2024-09-13T22:51:30.603973+0000","mapping_epoch":251,"log_start":"266'3223","ondisk_log_start":"266'3223","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":395712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":562,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.978861+0000","last_change":"2024-09-13T22:50:06.968858+0000","last_active":"2024-09-13T22:51:27.978861+0000","last_peered":"2024-09-13T22:51:27.978861+0000","last_clean":"2024-09-13T22:51:27.978861+0000","last_became_active":"2024-09-13T22:50:06.968526+0000","last_became_peered":"2024-09-13T22:50:06.968526+0000","last_unstale":"2024-09-13T22:51:27.978861+0000","last_undegraded":"2024-09-13T22:51:27.978861+0000","last_fullsized":"2024-09-13T22:51:27.978861+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:31.173761+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:45:59.802229+0000","last_clean_scrub_stamp":"2024-09-13T22:48:31.173761+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"228'28","reported_seq":582,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.974801+0000","last_change":"2024-09-13T22:50:04.011454+0000","last_active":"2024-09-13T22:51:27.974801+0000","last_peered":"2024-09-13T22:51:27.974801+0000","last_clean":"2024-09-13T22:51:27.974801+0000","last_became_active":"2024-09-13T22:50:03.973390+0000","last_became_peered":"2024-09-13T22:50:03.973390+0000","last_unstale":"2024-09-13T22:51:27.974801+0000","last_undegraded":"2024-09-13T22:51:27.974801+0000","last_fullsized":"2024-09-13T22:51:27.974801+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'28","last_scrub_stamp":"2024-09-13T22:49:11.188389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.188389+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":270,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.970585+0000","last_change":"2024-09-13T22:46:54.746202+0000","last_active":"2024-09-13T22:51:32.970585+0000","last_peered":"2024-09-13T22:51:32.970585+0000","last_clean":"2024-09-13T22:51:32.970585+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:51:32.970585+0000","last_undegraded":"2024-09-13T22:51:32.970585+0000","last_fullsized":"2024-09-13T22:51:32.970585+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:46:54.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:46:54.746095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":495,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975423+0000","last_change":"2024-09-13T22:49:21.447561+0000","last_active":"2024-09-13T22:51:32.975423+0000","last_peered":"2024-09-13T22:51:32.975423+0000","last_clean":"2024-09-13T22:51:32.975423+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:51:32.975423+0000","last_undegraded":"2024-09-13T22:51:32.975423+0000","last_fullsized":"2024-09-13T22:51:32.975423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:21.447387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.447387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":488,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979214+0000","last_change":"2024-09-13T22:49:28.469536+0000","last_active":"2024-09-13T22:51:27.979214+0000","last_peered":"2024-09-13T22:51:27.979214+0000","last_clean":"2024-09-13T22:51:27.979214+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:51:27.979214+0000","last_undegraded":"2024-09-13T22:51:27.979214+0000","last_fullsized":"2024-09-13T22:51:27.979214+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:28.469350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:11.075324+0000","last_clean_scrub_stamp":"2024-09-13T22:49:28.469350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"293'6488","reported_seq":6592,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.614787+0000","last_change":"2024-09-13T22:50:40.259290+0000","last_active":"2024-09-13T22:51:30.614787+0000","last_peered":"2024-09-13T22:51:30.614787+0000","last_clean":"2024-09-13T22:51:30.614787+0000","last_became_active":"2024-09-13T22:50:40.256019+0000","last_became_peered":"2024-09-13T22:50:40.256019+0000","last_unstale":"2024-09-13T22:51:30.614787+0000","last_undegraded":"2024-09-13T22:51:30.614787+0000","last_fullsized":"2024-09-13T22:51:30.614787+0000","mapping_epoch":251,"log_start":"255'400","ondisk_log_start":"255'400","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6088,"log_dups_size":0,"ondisk_log_size":6088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:34.666813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150962176,"num_objects":2303,"num_object_clones":0,"num_object_copies":6909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":198560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"228'20","reported_seq":518,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960173+0000","last_change":"2024-09-13T22:49:35.374424+0000","last_active":"2024-09-13T22:51:27.960173+0000","last_peered":"2024-09-13T22:51:27.960173+0000","last_clean":"2024-09-13T22:51:27.960173+0000","last_became_active":"2024-09-13T22:37:42.765423+0000","last_became_peered":"2024-09-13T22:37:42.765423+0000","last_unstale":"2024-09-13T22:51:27.960173+0000","last_undegraded":"2024-09-13T22:51:27.960173+0000","last_fullsized":"2024-09-13T22:51:27.960173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:49:35.374321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:35.374321+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612291+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":503,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975564+0000","last_change":"2024-09-13T22:49:22.420187+0000","last_active":"2024-09-13T22:51:32.975564+0000","last_peered":"2024-09-13T22:51:32.975564+0000","last_clean":"2024-09-13T22:51:32.975564+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:51:32.975564+0000","last_undegraded":"2024-09-13T22:51:32.975564+0000","last_fullsized":"2024-09-13T22:51:32.975564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":766,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971128+0000","last_change":"2024-09-13T22:48:32.199596+0000","last_active":"2024-09-13T22:51:32.971128+0000","last_peered":"2024-09-13T22:51:32.971128+0000","last_clean":"2024-09-13T22:51:32.971128+0000","last_became_active":"2024-09-13T22:45:53.877322+0000","last_became_peered":"2024-09-13T22:45:53.877322+0000","last_unstale":"2024-09-13T22:51:32.971128+0000","last_undegraded":"2024-09-13T22:51:32.971128+0000","last_fullsized":"2024-09-13T22:51:32.971128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:48:32.199490+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:48:32.199490+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":487,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972271+0000","last_change":"2024-09-13T22:48:16.361464+0000","last_active":"2024-09-13T22:51:32.972271+0000","last_peered":"2024-09-13T22:51:32.972271+0000","last_clean":"2024-09-13T22:51:32.972271+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:51:32.972271+0000","last_undegraded":"2024-09-13T22:51:32.972271+0000","last_fullsized":"2024-09-13T22:51:32.972271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:16.361322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:48:16.361322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.974137+0000","last_change":"2024-09-13T22:50:36.321611+0000","last_active":"2024-09-13T22:51:32.974137+0000","last_peered":"2024-09-13T22:51:32.974137+0000","last_clean":"2024-09-13T22:51:32.974137+0000","last_became_active":"2024-09-13T22:50:03.990614+0000","last_became_peered":"2024-09-13T22:50:03.990614+0000","last_unstale":"2024-09-13T22:51:32.974137+0000","last_undegraded":"2024-09-13T22:51:32.974137+0000","last_fullsized":"2024-09-13T22:51:32.974137+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:36.321543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:50:36.321543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:50.552608+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"295'6555","reported_seq":6699,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.499179+0000","last_change":"2024-09-13T22:50:41.684809+0000","last_active":"2024-09-13T22:51:33.499179+0000","last_peered":"2024-09-13T22:51:33.499179+0000","last_clean":"2024-09-13T22:51:33.499179+0000","last_became_active":"2024-09-13T22:50:40.281644+0000","last_became_peered":"2024-09-13T22:50:40.281644+0000","last_unstale":"2024-09-13T22:51:33.499179+0000","last_undegraded":"2024-09-13T22:51:33.499179+0000","last_fullsized":"2024-09-13T22:51:33.499179+0000","mapping_epoch":251,"log_start":"255'500","ondisk_log_start":"255'500","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'5184","last_scrub_stamp":"2024-09-13T22:50:41.684624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:50:41.684624+0000","objects_scrubbed":2594,"log_size":6055,"log_dups_size":0,"ondisk_log_size":6055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:43.582047+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":6585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1371,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"228'31","reported_seq":540,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972489+0000","last_change":"2024-09-13T22:49:13.909885+0000","last_active":"2024-09-13T22:51:32.972489+0000","last_peered":"2024-09-13T22:51:32.972489+0000","last_clean":"2024-09-13T22:51:32.972489+0000","last_became_active":"2024-09-13T22:37:42.768191+0000","last_became_peered":"2024-09-13T22:37:42.768191+0000","last_unstale":"2024-09-13T22:51:32.972489+0000","last_undegraded":"2024-09-13T22:51:32.972489+0000","last_fullsized":"2024-09-13T22:51:32.972489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'31","last_scrub_stamp":"2024-09-13T22:49:13.909765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:13.909765+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":520,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963170+0000","last_change":"2024-09-13T22:50:18.657042+0000","last_active":"2024-09-13T22:51:32.963170+0000","last_peered":"2024-09-13T22:51:32.963170+0000","last_clean":"2024-09-13T22:51:32.963170+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:51:32.963170+0000","last_undegraded":"2024-09-13T22:51:32.963170+0000","last_fullsized":"2024-09-13T22:51:32.963170+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:18.656932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:18.656932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:43.795095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":504,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962581+0000","last_change":"2024-09-13T22:50:21.662731+0000","last_active":"2024-09-13T22:51:32.962581+0000","last_peered":"2024-09-13T22:51:32.962581+0000","last_clean":"2024-09-13T22:51:32.962581+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:51:32.962581+0000","last_undegraded":"2024-09-13T22:51:32.962581+0000","last_fullsized":"2024-09-13T22:51:32.962581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:21.662613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:50:21.662613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:37.775839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":483,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971030+0000","last_change":"2024-09-13T22:49:22.976336+0000","last_active":"2024-09-13T22:51:32.971030+0000","last_peered":"2024-09-13T22:51:32.971030+0000","last_clean":"2024-09-13T22:51:32.971030+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:51:32.971030+0000","last_undegraded":"2024-09-13T22:51:32.971030+0000","last_fullsized":"2024-09-13T22:51:32.971030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.976109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.976109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"295'6663","reported_seq":6805,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.493594+0000","last_change":"2024-09-13T22:50:42.637384+0000","last_active":"2024-09-13T22:51:33.493594+0000","last_peered":"2024-09-13T22:51:33.493594+0000","last_clean":"2024-09-13T22:51:33.493594+0000","last_became_active":"2024-09-13T22:50:40.273733+0000","last_became_peered":"2024-09-13T22:50:40.273733+0000","last_unstale":"2024-09-13T22:51:33.493594+0000","last_undegraded":"2024-09-13T22:51:33.493594+0000","last_fullsized":"2024-09-13T22:51:33.493594+0000","mapping_epoch":251,"log_start":"255'500","ondisk_log_start":"255'500","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'5210","last_scrub_stamp":"2024-09-13T22:50:42.637264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:50:42.637264+0000","objects_scrubbed":2621,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:46.350340+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1453,"num_write_kb":34944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"228'20","reported_seq":529,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973502+0000","last_change":"2024-09-13T22:49:14.431757+0000","last_active":"2024-09-13T22:51:32.973502+0000","last_peered":"2024-09-13T22:51:32.973502+0000","last_clean":"2024-09-13T22:51:32.973502+0000","last_became_active":"2024-09-13T22:37:42.758412+0000","last_became_peered":"2024-09-13T22:37:42.758412+0000","last_unstale":"2024-09-13T22:51:32.973502+0000","last_undegraded":"2024-09-13T22:51:32.973502+0000","last_fullsized":"2024-09-13T22:51:32.973502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:49:14.431540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:14.431540+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":512,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.993137+0000","last_change":"2024-09-13T22:50:03.980136+0000","last_active":"2024-09-13T22:51:27.993137+0000","last_peered":"2024-09-13T22:51:27.993137+0000","last_clean":"2024-09-13T22:51:27.993137+0000","last_became_active":"2024-09-13T22:50:03.977100+0000","last_became_peered":"2024-09-13T22:50:03.977100+0000","last_unstale":"2024-09-13T22:51:27.993137+0000","last_undegraded":"2024-09-13T22:51:27.993137+0000","last_fullsized":"2024-09-13T22:51:27.993137+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:23.112116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:23.112116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963167+0000","last_change":"2024-09-13T22:50:03.984674+0000","last_active":"2024-09-13T22:51:32.963167+0000","last_peered":"2024-09-13T22:51:32.963167+0000","last_clean":"2024-09-13T22:51:32.963167+0000","last_became_active":"2024-09-13T22:50:03.982295+0000","last_became_peered":"2024-09-13T22:50:03.982295+0000","last_unstale":"2024-09-13T22:51:32.963167+0000","last_undegraded":"2024-09-13T22:51:32.963167+0000","last_fullsized":"2024-09-13T22:51:32.963167+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:22.282342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:22.282342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.974695+0000","last_change":"2024-09-13T22:50:29.967472+0000","last_active":"2024-09-13T22:51:27.974695+0000","last_peered":"2024-09-13T22:51:27.974695+0000","last_clean":"2024-09-13T22:51:27.974695+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:51:27.974695+0000","last_undegraded":"2024-09-13T22:51:27.974695+0000","last_fullsized":"2024-09-13T22:51:27.974695+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:29.967365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:50:29.967365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:33.320480+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"293'8318","reported_seq":8534,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.604585+0000","last_change":"2024-09-13T22:50:51.033811+0000","last_active":"2024-09-13T22:51:30.604585+0000","last_peered":"2024-09-13T22:51:30.604585+0000","last_clean":"2024-09-13T22:51:30.604585+0000","last_became_active":"2024-09-13T22:50:40.270471+0000","last_became_peered":"2024-09-13T22:50:40.270471+0000","last_unstale":"2024-09-13T22:51:30.604585+0000","last_undegraded":"2024-09-13T22:51:30.604585+0000","last_fullsized":"2024-09-13T22:51:30.604585+0000","mapping_epoch":251,"log_start":"259'2580","ondisk_log_start":"259'2580","created":251,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7218","last_scrub_stamp":"2024-09-13T22:50:51.032837+0000","last_deep_scrub":"281'7218","last_deep_scrub_stamp":"2024-09-13T22:50:51.032837+0000","last_clean_scrub_stamp":"2024-09-13T22:50:51.032837+0000","objects_scrubbed":2625,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:52:14.308036+0000","scrub_duration":9331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147390464,"num_objects":2249,"num_object_clones":0,"num_object_copies":6747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"228'34","reported_seq":585,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973548+0000","last_change":"2024-09-13T22:50:04.032442+0000","last_active":"2024-09-13T22:51:32.973548+0000","last_peered":"2024-09-13T22:51:32.973548+0000","last_clean":"2024-09-13T22:51:32.973548+0000","last_became_active":"2024-09-13T22:50:03.995906+0000","last_became_peered":"2024-09-13T22:50:03.995906+0000","last_unstale":"2024-09-13T22:51:32.973548+0000","last_undegraded":"2024-09-13T22:51:32.973548+0000","last_fullsized":"2024-09-13T22:51:32.973548+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:49:11.973550+0000","last_deep_scrub":"107'22","last_deep_scrub_stamp":"2024-09-13T22:39:46.684514+0000","last_clean_scrub_stamp":"2024-09-13T22:49:11.973550+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":82,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960109+0000","last_change":"2024-09-13T22:49:45.646088+0000","last_active":"2024-09-13T22:51:27.960109+0000","last_peered":"2024-09-13T22:51:27.960109+0000","last_clean":"2024-09-13T22:51:27.960109+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T22:51:27.960109+0000","last_undegraded":"2024-09-13T22:51:27.960109+0000","last_fullsized":"2024-09-13T22:51:27.960109+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.191239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:38.779536+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.191239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"266'13","reported_seq":693,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971222+0000","last_change":"2024-09-13T22:50:04.041189+0000","last_active":"2024-09-13T22:51:32.971222+0000","last_peered":"2024-09-13T22:51:32.971222+0000","last_clean":"2024-09-13T22:51:32.971222+0000","last_became_active":"2024-09-13T22:50:04.001753+0000","last_became_peered":"2024-09-13T22:50:04.001753+0000","last_unstale":"2024-09-13T22:51:32.971222+0000","last_undegraded":"2024-09-13T22:51:32.971222+0000","last_fullsized":"2024-09-13T22:51:32.971222+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'12","last_scrub_stamp":"2024-09-13T22:49:29.894307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:29.894307+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":491,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975491+0000","last_change":"2024-09-13T22:49:33.362540+0000","last_active":"2024-09-13T22:51:32.975491+0000","last_peered":"2024-09-13T22:51:32.975491+0000","last_clean":"2024-09-13T22:51:32.975491+0000","last_became_active":"2024-09-13T22:37:47.686240+0000","last_became_peered":"2024-09-13T22:37:47.686240+0000","last_unstale":"2024-09-13T22:51:32.975491+0000","last_undegraded":"2024-09-13T22:51:32.975491+0000","last_fullsized":"2024-09-13T22:51:32.975491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:33.362411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:33.362411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"293'6609","reported_seq":11201,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:29.929202+0000","last_change":"2024-09-13T22:50:48.283597+0000","last_active":"2024-09-13T22:51:29.929202+0000","last_peered":"2024-09-13T22:51:29.929202+0000","last_clean":"2024-09-13T22:51:29.929202+0000","last_became_active":"2024-09-13T22:50:41.270747+0000","last_became_peered":"2024-09-13T22:50:41.270747+0000","last_unstale":"2024-09-13T22:51:29.929202+0000","last_undegraded":"2024-09-13T22:51:29.929202+0000","last_fullsized":"2024-09-13T22:51:29.929202+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6609,"log_dups_size":0,"ondisk_log_size":6609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:30.050621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152633344,"num_objects":2329,"num_object_clones":0,"num_object_copies":6987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":202176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":107184128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"228'19","reported_seq":566,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972981+0000","last_change":"2024-09-13T22:50:04.038898+0000","last_active":"2024-09-13T22:51:32.972981+0000","last_peered":"2024-09-13T22:51:32.972981+0000","last_clean":"2024-09-13T22:51:32.972981+0000","last_became_active":"2024-09-13T22:50:03.994867+0000","last_became_peered":"2024-09-13T22:50:03.994867+0000","last_unstale":"2024-09-13T22:51:32.972981+0000","last_undegraded":"2024-09-13T22:51:32.972981+0000","last_fullsized":"2024-09-13T22:51:32.972981+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:49:31.961809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:31.961809+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":79,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964826+0000","last_change":"2024-09-13T22:50:04.010479+0000","last_active":"2024-09-13T22:51:32.964826+0000","last_peered":"2024-09-13T22:51:32.964826+0000","last_clean":"2024-09-13T22:51:32.964826+0000","last_became_active":"2024-09-13T22:50:03.990408+0000","last_became_peered":"2024-09-13T22:50:03.990408+0000","last_unstale":"2024-09-13T22:51:32.964826+0000","last_undegraded":"2024-09-13T22:51:32.964826+0000","last_fullsized":"2024-09-13T22:51:32.964826+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:49:32.135936+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:49:32.135936+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":82,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.984599+0000","last_change":"2024-09-13T22:49:45.649979+0000","last_active":"2024-09-13T22:51:27.984599+0000","last_peered":"2024-09-13T22:51:27.984599+0000","last_clean":"2024-09-13T22:51:27.984599+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T22:51:27.984599+0000","last_undegraded":"2024-09-13T22:51:27.984599+0000","last_fullsized":"2024-09-13T22:51:27.984599+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:40.290665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:48:40.290665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":480,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.965146+0000","last_change":"2024-09-13T22:48:17.839741+0000","last_active":"2024-09-13T22:51:27.965146+0000","last_peered":"2024-09-13T22:51:27.965146+0000","last_clean":"2024-09-13T22:51:27.965146+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:51:27.965146+0000","last_undegraded":"2024-09-13T22:51:27.965146+0000","last_fullsized":"2024-09-13T22:51:27.965146+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:17.839516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:48:17.839516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611805+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"228'24","reported_seq":562,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971659+0000","last_change":"2024-09-13T22:50:04.030185+0000","last_active":"2024-09-13T22:51:32.971659+0000","last_peered":"2024-09-13T22:51:32.971659+0000","last_clean":"2024-09-13T22:51:32.971659+0000","last_became_active":"2024-09-13T22:50:03.995225+0000","last_became_peered":"2024-09-13T22:50:03.995225+0000","last_unstale":"2024-09-13T22:51:32.971659+0000","last_undegraded":"2024-09-13T22:51:32.971659+0000","last_fullsized":"2024-09-13T22:51:32.971659+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:48:24.168396+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.168396+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963472+0000","last_change":"2024-09-13T22:50:45.633223+0000","last_active":"2024-09-13T22:51:32.963472+0000","last_peered":"2024-09-13T22:51:32.963472+0000","last_clean":"2024-09-13T22:51:32.963472+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:51:32.963472+0000","last_undegraded":"2024-09-13T22:51:32.963472+0000","last_fullsized":"2024-09-13T22:51:32.963472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:45.633092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:50:45.633092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:52.705950+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960797+0000","last_change":"2024-09-13T22:48:15.782055+0000","last_active":"2024-09-13T22:51:27.960797+0000","last_peered":"2024-09-13T22:51:27.960797+0000","last_clean":"2024-09-13T22:51:27.960797+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:51:27.960797+0000","last_undegraded":"2024-09-13T22:51:27.960797+0000","last_fullsized":"2024-09-13T22:51:27.960797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:15.778860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:48:15.778860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611883+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.962783+0000","last_change":"2024-09-13T22:50:16.597759+0000","last_active":"2024-09-13T22:51:32.962783+0000","last_peered":"2024-09-13T22:51:32.962783+0000","last_clean":"2024-09-13T22:51:32.962783+0000","last_became_active":"2024-09-13T22:50:03.968467+0000","last_became_peered":"2024-09-13T22:50:03.968467+0000","last_unstale":"2024-09-13T22:51:32.962783+0000","last_undegraded":"2024-09-13T22:51:32.962783+0000","last_fullsized":"2024-09-13T22:51:32.962783+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:16.597647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:50:16.597647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:37.510497+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"235'10","reported_seq":528,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.977592+0000","last_change":"2024-09-13T22:50:21.280965+0000","last_active":"2024-09-13T22:51:32.977592+0000","last_peered":"2024-09-13T22:51:32.977592+0000","last_clean":"2024-09-13T22:51:32.977592+0000","last_became_active":"2024-09-13T22:37:42.756061+0000","last_became_peered":"2024-09-13T22:37:42.756061+0000","last_unstale":"2024-09-13T22:51:32.977592+0000","last_undegraded":"2024-09-13T22:51:32.977592+0000","last_fullsized":"2024-09-13T22:51:32.977592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'10","last_scrub_stamp":"2024-09-13T22:50:21.280830+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:50:21.280830+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:41.462143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":530,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975909+0000","last_change":"2024-09-13T22:50:28.470138+0000","last_active":"2024-09-13T22:51:32.975909+0000","last_peered":"2024-09-13T22:51:32.975909+0000","last_clean":"2024-09-13T22:51:32.975909+0000","last_became_active":"2024-09-13T22:50:03.987950+0000","last_became_peered":"2024-09-13T22:50:03.987950+0000","last_unstale":"2024-09-13T22:51:32.975909+0000","last_undegraded":"2024-09-13T22:51:32.975909+0000","last_fullsized":"2024-09-13T22:51:32.975909+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:28.469986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:04.034995+0000","last_clean_scrub_stamp":"2024-09-13T22:50:28.469986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:36.701399+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":494,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973503+0000","last_change":"2024-09-13T22:49:32.393122+0000","last_active":"2024-09-13T22:51:32.973503+0000","last_peered":"2024-09-13T22:51:32.973503+0000","last_clean":"2024-09-13T22:51:32.973503+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:51:32.973503+0000","last_undegraded":"2024-09-13T22:51:32.973503+0000","last_fullsized":"2024-09-13T22:51:32.973503+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:32.392914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:32.392914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":494,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972288+0000","last_change":"2024-09-13T22:50:03.997746+0000","last_active":"2024-09-13T22:51:32.972288+0000","last_peered":"2024-09-13T22:51:32.972288+0000","last_clean":"2024-09-13T22:51:32.972288+0000","last_became_active":"2024-09-13T22:50:03.997081+0000","last_became_peered":"2024-09-13T22:50:03.997081+0000","last_unstale":"2024-09-13T22:51:32.972288+0000","last_undegraded":"2024-09-13T22:51:32.972288+0000","last_fullsized":"2024-09-13T22:51:32.972288+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:27.968957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:27.968957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.19","version":"293'6472","reported_seq":15147,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:30.416198+0000","last_change":"2024-09-13T22:51:06.817530+0000","last_active":"2024-09-13T22:51:30.416198+0000","last_peered":"2024-09-13T22:51:30.416198+0000","last_clean":"2024-09-13T22:51:30.416198+0000","last_became_active":"2024-09-13T22:51:06.812284+0000","last_became_peered":"2024-09-13T22:51:06.812284+0000","last_unstale":"2024-09-13T22:51:30.416198+0000","last_undegraded":"2024-09-13T22:51:30.416198+0000","last_fullsized":"2024-09-13T22:51:30.416198+0000","mapping_epoch":283,"log_start":"255'100","ondisk_log_start":"255'100","created":276,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:33.671410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155680768,"num_objects":2376,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3912,"num_write_kb":201184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2853,"num_bytes_recovered":186974208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"228'19","reported_seq":557,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971376+0000","last_change":"2024-09-13T22:50:04.045917+0000","last_active":"2024-09-13T22:51:32.971376+0000","last_peered":"2024-09-13T22:51:32.971376+0000","last_clean":"2024-09-13T22:51:32.971376+0000","last_became_active":"2024-09-13T22:50:04.002950+0000","last_became_peered":"2024-09-13T22:50:04.002950+0000","last_unstale":"2024-09-13T22:51:32.971376+0000","last_undegraded":"2024-09-13T22:51:32.971376+0000","last_fullsized":"2024-09-13T22:51:32.971376+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:48:18.321070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:18.321070+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":510,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.984583+0000","last_change":"2024-09-13T22:50:34.246091+0000","last_active":"2024-09-13T22:51:32.984583+0000","last_peered":"2024-09-13T22:51:32.984583+0000","last_clean":"2024-09-13T22:51:32.984583+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:51:32.984583+0000","last_undegraded":"2024-09-13T22:51:32.984583+0000","last_fullsized":"2024-09-13T22:51:32.984583+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:34.245985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:50:34.245985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:39.547914+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"82'11","reported_seq":682,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.974022+0000","last_change":"2024-09-13T22:50:26.150354+0000","last_active":"2024-09-13T22:51:32.974022+0000","last_peered":"2024-09-13T22:51:32.974022+0000","last_clean":"2024-09-13T22:51:32.974022+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:51:32.974022+0000","last_undegraded":"2024-09-13T22:51:32.974022+0000","last_fullsized":"2024-09-13T22:51:32.974022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:50:26.150246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:50:26.150246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:43.147899+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":487,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.974997+0000","last_change":"2024-09-13T22:49:42.161884+0000","last_active":"2024-09-13T22:51:27.974997+0000","last_peered":"2024-09-13T22:51:27.974997+0000","last_clean":"2024-09-13T22:51:27.974997+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:51:27.974997+0000","last_undegraded":"2024-09-13T22:51:27.974997+0000","last_fullsized":"2024-09-13T22:51:27.974997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:42.161805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:42.161805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"294'6596","reported_seq":14397,"reported_epoch":295,"state":"peering","last_fresh":"2024-09-13T22:51:32.974810+0000","last_change":"2024-09-13T22:51:32.974810+0000","last_active":"2024-09-13T22:51:32.974810+0000","last_peered":"2024-09-13T22:51:32.879071+0000","last_clean":"2024-09-13T22:51:32.879071+0000","last_became_active":"2024-09-13T22:50:43.401356+0000","last_became_peered":"2024-09-13T22:50:43.401356+0000","last_unstale":"2024-09-13T22:51:32.974810+0000","last_undegraded":"2024-09-13T22:51:32.974810+0000","last_fullsized":"2024-09-13T22:51:32.974810+0000","mapping_epoch":295,"log_start":"255'400","ondisk_log_start":"255'400","created":276,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6196,"log_dups_size":0,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147816448,"num_objects":2256,"num_object_clones":0,"num_object_copies":6768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":200480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":167837696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"228'15","reported_seq":520,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975112+0000","last_change":"2024-09-13T22:49:38.143966+0000","last_active":"2024-09-13T22:51:27.975112+0000","last_peered":"2024-09-13T22:51:27.975112+0000","last_clean":"2024-09-13T22:51:27.975112+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:51:27.975112+0000","last_undegraded":"2024-09-13T22:51:27.975112+0000","last_fullsized":"2024-09-13T22:51:27.975112+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:49:38.143760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:38.143760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":519,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975986+0000","last_change":"2024-09-13T22:50:32.755563+0000","last_active":"2024-09-13T22:51:27.975986+0000","last_peered":"2024-09-13T22:51:27.975986+0000","last_clean":"2024-09-13T22:51:27.975986+0000","last_became_active":"2024-09-13T22:50:03.973419+0000","last_became_peered":"2024-09-13T22:50:03.973419+0000","last_unstale":"2024-09-13T22:51:27.975986+0000","last_undegraded":"2024-09-13T22:51:27.975986+0000","last_fullsized":"2024-09-13T22:51:27.975986+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:32.755470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:32.755470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:41.978846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":659,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.961434+0000","last_change":"2024-09-13T22:45:48.537059+0000","last_active":"2024-09-13T22:51:27.961434+0000","last_peered":"2024-09-13T22:51:27.961434+0000","last_clean":"2024-09-13T22:51:27.961434+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:51:27.961434+0000","last_undegraded":"2024-09-13T22:51:27.961434+0000","last_fullsized":"2024-09-13T22:51:27.961434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:45:48.536952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:45:48.536952+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964701+0000","last_change":"2024-09-13T22:50:03.980831+0000","last_active":"2024-09-13T22:51:32.964701+0000","last_peered":"2024-09-13T22:51:32.964701+0000","last_clean":"2024-09-13T22:51:32.964701+0000","last_became_active":"2024-09-13T22:50:03.980403+0000","last_became_peered":"2024-09-13T22:50:03.980403+0000","last_unstale":"2024-09-13T22:51:32.964701+0000","last_undegraded":"2024-09-13T22:51:32.964701+0000","last_fullsized":"2024-09-13T22:51:32.964701+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:08.144897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:08.144897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"228'15","reported_seq":547,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972448+0000","last_change":"2024-09-13T22:50:04.041895+0000","last_active":"2024-09-13T22:51:32.972448+0000","last_peered":"2024-09-13T22:51:32.972448+0000","last_clean":"2024-09-13T22:51:32.972448+0000","last_became_active":"2024-09-13T22:50:03.997804+0000","last_became_peered":"2024-09-13T22:50:03.997804+0000","last_unstale":"2024-09-13T22:51:32.972448+0000","last_undegraded":"2024-09-13T22:51:32.972448+0000","last_fullsized":"2024-09-13T22:51:32.972448+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:21.238464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:21.238464+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":518,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975954+0000","last_change":"2024-09-13T22:50:38.438761+0000","last_active":"2024-09-13T22:51:32.975954+0000","last_peered":"2024-09-13T22:51:32.975954+0000","last_clean":"2024-09-13T22:51:32.975954+0000","last_became_active":"2024-09-13T22:37:40.750456+0000","last_became_peered":"2024-09-13T22:37:40.750456+0000","last_unstale":"2024-09-13T22:51:32.975954+0000","last_undegraded":"2024-09-13T22:51:32.975954+0000","last_fullsized":"2024-09-13T22:51:32.975954+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:38.438629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:38.438629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:52.514646+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.965232+0000","last_change":"2024-09-13T22:49:17.277840+0000","last_active":"2024-09-13T22:51:27.965232+0000","last_peered":"2024-09-13T22:51:27.965232+0000","last_clean":"2024-09-13T22:51:27.965232+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:51:27.965232+0000","last_undegraded":"2024-09-13T22:51:27.965232+0000","last_fullsized":"2024-09-13T22:51:27.965232+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:17.277656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:49:17.277656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":487,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.984674+0000","last_change":"2024-09-13T22:50:33.724150+0000","last_active":"2024-09-13T22:51:27.984674+0000","last_peered":"2024-09-13T22:51:27.984674+0000","last_clean":"2024-09-13T22:51:27.984674+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:51:27.984674+0000","last_undegraded":"2024-09-13T22:51:27.984674+0000","last_fullsized":"2024-09-13T22:51:27.984674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:33.724012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:50:33.724012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:34.168221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"228'20","reported_seq":85,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964155+0000","last_change":"2024-09-13T22:50:04.019803+0000","last_active":"2024-09-13T22:51:32.964155+0000","last_peered":"2024-09-13T22:51:32.964155+0000","last_clean":"2024-09-13T22:51:32.964155+0000","last_became_active":"2024-09-13T22:50:03.992294+0000","last_became_peered":"2024-09-13T22:50:03.992294+0000","last_unstale":"2024-09-13T22:51:32.964155+0000","last_undegraded":"2024-09-13T22:51:32.964155+0000","last_fullsized":"2024-09-13T22:51:32.964155+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:48:20.275130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:20.275130+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.993000+0000","last_change":"2024-09-13T22:49:16.390793+0000","last_active":"2024-09-13T22:51:27.993000+0000","last_peered":"2024-09-13T22:51:27.993000+0000","last_clean":"2024-09-13T22:51:27.993000+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:51:27.993000+0000","last_undegraded":"2024-09-13T22:51:27.993000+0000","last_fullsized":"2024-09-13T22:51:27.993000+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.389928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:03.849281+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.389928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":491,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.989528+0000","last_change":"2024-09-13T22:48:38.188618+0000","last_active":"2024-09-13T22:51:27.989528+0000","last_peered":"2024-09-13T22:51:27.989528+0000","last_clean":"2024-09-13T22:51:27.989528+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:51:27.989528+0000","last_undegraded":"2024-09-13T22:51:27.989528+0000","last_fullsized":"2024-09-13T22:51:27.989528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:38.188506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:48:38.188506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.961922+0000","last_change":"2024-09-13T22:49:20.405583+0000","last_active":"2024-09-13T22:51:27.961922+0000","last_peered":"2024-09-13T22:51:27.961922+0000","last_clean":"2024-09-13T22:51:27.961922+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:51:27.961922+0000","last_undegraded":"2024-09-13T22:51:27.961922+0000","last_fullsized":"2024-09-13T22:51:27.961922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:20.405502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:49:20.405502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.612247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"228'12","reported_seq":521,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975278+0000","last_change":"2024-09-13T22:50:28.737756+0000","last_active":"2024-09-13T22:51:27.975278+0000","last_peered":"2024-09-13T22:51:27.975278+0000","last_clean":"2024-09-13T22:51:27.975278+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:51:27.975278+0000","last_undegraded":"2024-09-13T22:51:27.975278+0000","last_fullsized":"2024-09-13T22:51:27.975278+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12","last_scrub_stamp":"2024-09-13T22:50:28.737662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:50:28.737662+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:58.058382+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":502,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973799+0000","last_change":"2024-09-13T22:49:28.371324+0000","last_active":"2024-09-13T22:51:32.973799+0000","last_peered":"2024-09-13T22:51:32.973799+0000","last_clean":"2024-09-13T22:51:32.973799+0000","last_became_active":"2024-09-13T22:37:40.745262+0000","last_became_peered":"2024-09-13T22:37:40.745262+0000","last_unstale":"2024-09-13T22:51:32.973799+0000","last_undegraded":"2024-09-13T22:51:32.973799+0000","last_fullsized":"2024-09-13T22:51:32.973799+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:28.371115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:49:28.371115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":522,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975133+0000","last_change":"2024-09-13T22:50:40.392350+0000","last_active":"2024-09-13T22:51:32.975133+0000","last_peered":"2024-09-13T22:51:32.975133+0000","last_clean":"2024-09-13T22:51:32.975133+0000","last_became_active":"2024-09-13T22:50:03.987270+0000","last_became_peered":"2024-09-13T22:50:03.987270+0000","last_unstale":"2024-09-13T22:51:32.975133+0000","last_undegraded":"2024-09-13T22:51:32.975133+0000","last_fullsized":"2024-09-13T22:51:32.975133+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:40.392242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:50:40.392242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:52:07.359658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":494,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.971485+0000","last_change":"2024-09-13T22:50:03.996248+0000","last_active":"2024-09-13T22:51:32.971485+0000","last_peered":"2024-09-13T22:51:32.971485+0000","last_clean":"2024-09-13T22:51:32.971485+0000","last_became_active":"2024-09-13T22:50:03.995547+0000","last_became_peered":"2024-09-13T22:50:03.995547+0000","last_unstale":"2024-09-13T22:51:32.971485+0000","last_undegraded":"2024-09-13T22:51:32.971485+0000","last_fullsized":"2024-09-13T22:51:32.971485+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:15.907800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:15.907800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"228'15","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973533+0000","last_change":"2024-09-13T22:49:24.436104+0000","last_active":"2024-09-13T22:51:32.973533+0000","last_peered":"2024-09-13T22:51:32.973533+0000","last_clean":"2024-09-13T22:51:32.973533+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:51:32.973533+0000","last_undegraded":"2024-09-13T22:51:32.973533+0000","last_fullsized":"2024-09-13T22:51:32.973533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:49:24.436026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:49:24.436026+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":518,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.976863+0000","last_change":"2024-09-13T22:50:20.262300+0000","last_active":"2024-09-13T22:51:32.976863+0000","last_peered":"2024-09-13T22:51:32.976863+0000","last_clean":"2024-09-13T22:51:32.976863+0000","last_became_active":"2024-09-13T22:37:40.743775+0000","last_became_peered":"2024-09-13T22:37:40.743775+0000","last_unstale":"2024-09-13T22:51:32.976863+0000","last_undegraded":"2024-09-13T22:51:32.976863+0000","last_fullsized":"2024-09-13T22:51:32.976863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:20.262205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:50:20.262205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":498,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.970640+0000","last_change":"2024-09-13T22:50:03.995218+0000","last_active":"2024-09-13T22:51:32.970640+0000","last_peered":"2024-09-13T22:51:32.970640+0000","last_clean":"2024-09-13T22:51:32.970640+0000","last_became_active":"2024-09-13T22:50:03.994661+0000","last_became_peered":"2024-09-13T22:50:03.994661+0000","last_unstale":"2024-09-13T22:51:32.970640+0000","last_undegraded":"2024-09-13T22:51:32.970640+0000","last_fullsized":"2024-09-13T22:51:32.970640+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:16.934047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:49:16.934047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.965315+0000","last_change":"2024-09-13T22:45:40.486348+0000","last_active":"2024-09-13T22:51:27.965315+0000","last_peered":"2024-09-13T22:51:27.965315+0000","last_clean":"2024-09-13T22:51:27.965315+0000","last_became_active":"2024-09-13T22:37:47.689773+0000","last_became_peered":"2024-09-13T22:37:47.689773+0000","last_unstale":"2024-09-13T22:51:27.965315+0000","last_undegraded":"2024-09-13T22:51:27.965315+0000","last_fullsized":"2024-09-13T22:51:27.965315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:40.486247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:45:40.486247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"295'6475","reported_seq":6592,"reported_epoch":295,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T22:51:33.439438+0000","last_change":"2024-09-13T22:51:13.315471+0000","last_active":"2024-09-13T22:51:33.439438+0000","last_peered":"2024-09-13T22:51:33.439438+0000","last_clean":"2024-09-13T22:50:40.759943+0000","last_became_active":"2024-09-13T22:50:43.305514+0000","last_became_peered":"2024-09-13T22:50:43.305514+0000","last_unstale":"2024-09-13T22:51:33.439438+0000","last_undegraded":"2024-09-13T22:51:33.439438+0000","last_fullsized":"2024-09-13T22:51:33.439438+0000","mapping_epoch":279,"log_start":"255'900","ondisk_log_start":"255'900","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145293312,"num_objects":2217,"num_object_clones":0,"num_object_copies":6651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4434,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":196544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,1,3],"avail_no_missing":["7","1","3"],"object_location_counts":[{"shards":"1,3,7","objects":2217}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"82'1","reported_seq":484,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.972158+0000","last_change":"2024-09-13T22:48:29.166658+0000","last_active":"2024-09-13T22:51:32.972158+0000","last_peered":"2024-09-13T22:51:32.972158+0000","last_clean":"2024-09-13T22:51:32.972158+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:51:32.972158+0000","last_undegraded":"2024-09-13T22:51:32.972158+0000","last_fullsized":"2024-09-13T22:51:32.972158+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:48:29.166588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:29.166588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"228'34","reported_seq":589,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.979278+0000","last_change":"2024-09-13T22:50:04.026569+0000","last_active":"2024-09-13T22:51:27.979278+0000","last_peered":"2024-09-13T22:51:27.979278+0000","last_clean":"2024-09-13T22:51:27.979278+0000","last_became_active":"2024-09-13T22:50:03.978438+0000","last_became_peered":"2024-09-13T22:50:03.978438+0000","last_unstale":"2024-09-13T22:51:27.979278+0000","last_undegraded":"2024-09-13T22:51:27.979278+0000","last_fullsized":"2024-09-13T22:51:27.979278+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:48:28.140653+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.140653+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":522,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.973415+0000","last_change":"2024-09-13T22:50:30.194280+0000","last_active":"2024-09-13T22:51:32.973415+0000","last_peered":"2024-09-13T22:51:32.973415+0000","last_clean":"2024-09-13T22:51:32.973415+0000","last_became_active":"2024-09-13T22:50:03.975395+0000","last_became_peered":"2024-09-13T22:50:03.975395+0000","last_unstale":"2024-09-13T22:51:32.973415+0000","last_undegraded":"2024-09-13T22:51:32.973415+0000","last_fullsized":"2024-09-13T22:51:32.973415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:30.194174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:50:30.194174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:51.286896+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"266'14","reported_seq":713,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.976765+0000","last_change":"2024-09-13T22:50:36.335246+0000","last_active":"2024-09-13T22:51:32.976765+0000","last_peered":"2024-09-13T22:51:32.976765+0000","last_clean":"2024-09-13T22:51:32.976765+0000","last_became_active":"2024-09-13T22:50:03.985520+0000","last_became_peered":"2024-09-13T22:50:03.985520+0000","last_unstale":"2024-09-13T22:51:32.976765+0000","last_undegraded":"2024-09-13T22:51:32.976765+0000","last_fullsized":"2024-09-13T22:51:32.976765+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'14","last_scrub_stamp":"2024-09-13T22:50:36.335134+0000","last_deep_scrub":"172'12","last_deep_scrub_stamp":"2024-09-13T22:44:59.017618+0000","last_clean_scrub_stamp":"2024-09-13T22:50:36.335134+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:42.779229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"295'6463","reported_seq":15240,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.404639+0000","last_change":"2024-09-13T22:51:10.932023+0000","last_active":"2024-09-13T22:51:33.404639+0000","last_peered":"2024-09-13T22:51:33.404639+0000","last_clean":"2024-09-13T22:51:33.404639+0000","last_became_active":"2024-09-13T22:51:10.924860+0000","last_became_peered":"2024-09-13T22:51:10.924860+0000","last_unstale":"2024-09-13T22:51:33.404639+0000","last_undegraded":"2024-09-13T22:51:33.404639+0000","last_fullsized":"2024-09-13T22:51:33.404639+0000","mapping_epoch":287,"log_start":"255'200","ondisk_log_start":"255'200","created":276,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6263,"log_dups_size":0,"ondisk_log_size":6263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144441344,"num_objects":2204,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3948,"num_write_kb":196864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2886,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.992644+0000","last_change":"2024-09-13T22:50:03.980316+0000","last_active":"2024-09-13T22:51:27.992644+0000","last_peered":"2024-09-13T22:51:27.992644+0000","last_clean":"2024-09-13T22:51:27.992644+0000","last_became_active":"2024-09-13T22:50:03.979831+0000","last_became_peered":"2024-09-13T22:50:03.979831+0000","last_unstale":"2024-09-13T22:51:27.992644+0000","last_undegraded":"2024-09-13T22:51:27.992644+0000","last_fullsized":"2024-09-13T22:51:27.992644+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:39.144743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:48:39.144743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"228'15","reported_seq":512,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960744+0000","last_change":"2024-09-13T22:48:24.914421+0000","last_active":"2024-09-13T22:51:27.960744+0000","last_peered":"2024-09-13T22:51:27.960744+0000","last_clean":"2024-09-13T22:51:27.960744+0000","last_became_active":"2024-09-13T22:37:42.762426+0000","last_became_peered":"2024-09-13T22:37:42.762426+0000","last_unstale":"2024-09-13T22:51:27.960744+0000","last_undegraded":"2024-09-13T22:51:27.960744+0000","last_fullsized":"2024-09-13T22:51:27.960744+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:48:24.913947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:48:24.913947+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611964+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":484,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.962112+0000","last_change":"2024-09-13T22:44:58.543023+0000","last_active":"2024-09-13T22:51:27.962112+0000","last_peered":"2024-09-13T22:51:27.962112+0000","last_clean":"2024-09-13T22:51:27.962112+0000","last_became_active":"2024-09-13T22:37:40.746424+0000","last_became_peered":"2024-09-13T22:37:40.746424+0000","last_unstale":"2024-09-13T22:51:27.962112+0000","last_undegraded":"2024-09-13T22:51:27.962112+0000","last_fullsized":"2024-09-13T22:51:27.962112+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:44:58.542781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:44:58.542781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975093+0000","last_change":"2024-09-13T22:50:42.693031+0000","last_active":"2024-09-13T22:51:27.975093+0000","last_peered":"2024-09-13T22:51:27.975093+0000","last_clean":"2024-09-13T22:51:27.975093+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:51:27.975093+0000","last_undegraded":"2024-09-13T22:51:27.975093+0000","last_fullsized":"2024-09-13T22:51:27.975093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:42.692964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:50:42.692964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:49.893461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.11","version":"295'6585","reported_seq":6697,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:33.410125+0000","last_change":"2024-09-13T22:50:42.254620+0000","last_active":"2024-09-13T22:51:33.410125+0000","last_peered":"2024-09-13T22:51:33.410125+0000","last_clean":"2024-09-13T22:51:33.410125+0000","last_became_active":"2024-09-13T22:50:42.250616+0000","last_became_peered":"2024-09-13T22:50:42.250616+0000","last_unstale":"2024-09-13T22:51:33.410125+0000","last_undegraded":"2024-09-13T22:51:33.410125+0000","last_fullsized":"2024-09-13T22:51:33.410125+0000","mapping_epoch":278,"log_start":"255'437","ondisk_log_start":"255'437","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:35.164047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":198656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964444+0000","last_change":"2024-09-13T22:50:03.985020+0000","last_active":"2024-09-13T22:51:32.964444+0000","last_peered":"2024-09-13T22:51:32.964444+0000","last_clean":"2024-09-13T22:51:32.964444+0000","last_became_active":"2024-09-13T22:50:03.984625+0000","last_became_peered":"2024-09-13T22:50:03.984625+0000","last_unstale":"2024-09-13T22:51:32.964444+0000","last_undegraded":"2024-09-13T22:51:32.964444+0000","last_fullsized":"2024-09-13T22:51:32.964444+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:38.014231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:49:38.014231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":530,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.975828+0000","last_change":"2024-09-13T22:50:35.329066+0000","last_active":"2024-09-13T22:51:32.975828+0000","last_peered":"2024-09-13T22:51:32.975828+0000","last_clean":"2024-09-13T22:51:32.975828+0000","last_became_active":"2024-09-13T22:50:03.985970+0000","last_became_peered":"2024-09-13T22:50:03.985970+0000","last_unstale":"2024-09-13T22:51:32.975828+0000","last_undegraded":"2024-09-13T22:51:32.975828+0000","last_fullsized":"2024-09-13T22:51:32.975828+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:50:35.328942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:50:35.328942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:50.931422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"228'9","reported_seq":542,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.975857+0000","last_change":"2024-09-13T22:50:27.757366+0000","last_active":"2024-09-13T22:51:27.975857+0000","last_peered":"2024-09-13T22:51:27.975857+0000","last_clean":"2024-09-13T22:51:27.975857+0000","last_became_active":"2024-09-13T22:50:03.983134+0000","last_became_peered":"2024-09-13T22:50:03.983134+0000","last_unstale":"2024-09-13T22:51:27.975857+0000","last_undegraded":"2024-09-13T22:51:27.975857+0000","last_fullsized":"2024-09-13T22:51:27.975857+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'9","last_scrub_stamp":"2024-09-13T22:50:27.757224+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:50:27.757224+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:28.796158+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":503,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.992913+0000","last_change":"2024-09-13T22:50:03.978676+0000","last_active":"2024-09-13T22:51:27.992913+0000","last_peered":"2024-09-13T22:51:27.992913+0000","last_clean":"2024-09-13T22:51:27.992913+0000","last_became_active":"2024-09-13T22:50:03.976594+0000","last_became_peered":"2024-09-13T22:50:03.976594+0000","last_unstale":"2024-09-13T22:51:27.992913+0000","last_undegraded":"2024-09-13T22:51:27.992913+0000","last_fullsized":"2024-09-13T22:51:27.992913+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:41.110195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:23.072590+0000","last_clean_scrub_stamp":"2024-09-13T22:48:41.110195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"293'6556","reported_seq":11675,"reported_epoch":293,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:51:29.927039+0000","last_change":"2024-09-13T22:50:43.993260+0000","last_active":"2024-09-13T22:51:29.927039+0000","last_peered":"2024-09-13T22:51:29.927039+0000","last_clean":"2024-09-13T22:50:40.569328+0000","last_became_active":"2024-09-13T22:50:43.420229+0000","last_became_peered":"2024-09-13T22:50:43.420229+0000","last_unstale":"2024-09-13T22:51:29.927039+0000","last_undegraded":"2024-09-13T22:51:29.927039+0000","last_fullsized":"2024-09-13T22:50:43.242142+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:18.811427+0000","last_clean_scrub_stamp":"2024-09-13T22:49:18.811427+0000","objects_scrubbed":0,"log_size":6556,"log_dups_size":0,"ondisk_log_size":6556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149585920,"num_objects":2283,"num_object_clones":0,"num_object_copies":6849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4169,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":200352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":82116608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[1,0],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"0,1","objects":893},{"shards":"1,2,4,6","objects":1},{"shards":"1,2,6","objects":1389}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":481,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.960706+0000","last_change":"2024-09-13T22:45:51.542656+0000","last_active":"2024-09-13T22:51:27.960706+0000","last_peered":"2024-09-13T22:51:27.960706+0000","last_clean":"2024-09-13T22:51:27.960706+0000","last_became_active":"2024-09-13T22:37:47.683784+0000","last_became_peered":"2024-09-13T22:37:47.683784+0000","last_unstale":"2024-09-13T22:51:27.960706+0000","last_undegraded":"2024-09-13T22:51:27.960706+0000","last_fullsized":"2024-09-13T22:51:27.960706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:45:51.542373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:45:51.542373+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":491,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T22:51:27.965549+0000","last_change":"2024-09-13T22:48:28.089740+0000","last_active":"2024-09-13T22:51:27.965549+0000","last_peered":"2024-09-13T22:51:27.965549+0000","last_clean":"2024-09-13T22:51:27.965549+0000","last_became_active":"2024-09-13T22:37:40.746483+0000","last_became_peered":"2024-09-13T22:37:40.746483+0000","last_unstale":"2024-09-13T22:51:27.965549+0000","last_undegraded":"2024-09-13T22:51:27.965549+0000","last_fullsized":"2024-09-13T22:51:27.965549+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:48:28.089548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:48:28.089548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:51:31.611921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"228'19","reported_seq":83,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.963058+0000","last_change":"2024-09-13T22:50:04.012611+0000","last_active":"2024-09-13T22:51:32.963058+0000","last_peered":"2024-09-13T22:51:32.963058+0000","last_clean":"2024-09-13T22:51:32.963058+0000","last_became_active":"2024-09-13T22:50:03.971811+0000","last_became_peered":"2024-09-13T22:50:03.971811+0000","last_unstale":"2024-09-13T22:51:32.963058+0000","last_undegraded":"2024-09-13T22:51:32.963058+0000","last_fullsized":"2024-09-13T22:51:32.963058+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:49:21.457510+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:49:21.457510+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T22:51:32.964065+0000","last_change":"2024-09-13T22:50:03.980217+0000","last_active":"2024-09-13T22:51:32.964065+0000","last_peered":"2024-09-13T22:51:32.964065+0000","last_clean":"2024-09-13T22:51:32.964065+0000","last_became_active":"2024-09-13T22:50:03.979868+0000","last_became_peered":"2024-09-13T22:50:03.979868+0000","last_unstale":"2024-09-13T22:51:32.964065+0000","last_undegraded":"2024-09-13T22:51:32.964065+0000","last_fullsized":"2024-09-13T22:51:32.964065+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:45:50.912196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:45:50.912196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":4734451736,"num_objects":72245,"num_object_clones":0,"num_object_copies":216735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8603,"num_objects_unfound":0,"num_objects_dirty":72245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115707,"num_write_kb":5680385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27816,"num_bytes_recovered":1822949376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15513432064,"data_stored":15513419848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148745,"ondisk_log_size":148745,"up":78,"acting":77,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":806,"num_read_kb":801,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":3960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":769,"ondisk_log_size":769,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809007,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3048900,"kb_used_data":2612332,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":90696252,"statfs":{"total":95995035648,"available":92872962048,"internally_reserved":0,"allocated":2675027968,"data_stored":2674388247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":447019466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136058,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1531104,"kb_used_data":1175052,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92214048,"statfs":{"total":95995035648,"available":94427185152,"internally_reserved":0,"allocated":1203253248,"data_stored":1202643949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528517,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2679664,"kb_used_data":2273624,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91065488,"statfs":{"total":95995035648,"available":93251059712,"internally_reserved":0,"allocated":2328190976,"data_stored":2327570359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888271,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1362760,"kb_used_data":1060848,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92382392,"statfs":{"total":95995035648,"available":94599569408,"internally_reserved":0,"allocated":1086308352,"data_stored":1085663121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084504,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2930380,"kb_used_data":2399292,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":90814772,"statfs":{"total":95995035648,"available":92994326528,"internally_reserved":0,"allocated":2456875008,"data_stored":2456269570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":543685063},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378853,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1211948,"kb_used_data":887828,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92533204,"statfs":{"total":95995035648,"available":94754000896,"internally_reserved":0,"allocated":909135872,"data_stored":908519841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3415260,"kb_used_data":3005064,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":90329892,"statfs":{"total":95995035648,"available":92497809408,"internally_reserved":0,"allocated":3077185536,"data_stored":3076567778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738617,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2245916,"kb_used_data":1849608,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91499236,"statfs":{"total":95995035648,"available":93695217664,"internally_reserved":0,"allocated":1893998592,"data_stored":1893381570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1876754432,"data_stored":1876754432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3040743424,"data_stored":3040739352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905576448,"data_stored":905576448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2416508928,"data_stored":2416508928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1085472768,"data_stored":1085472768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2321813504,"data_stored":2321809432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1200816128,"data_stored":1200816128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2665746432,"data_stored":2665742360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:51:40.269 INFO:tasks.thrashosds.thrasher:Setting 3.1d to [4, 2, 5, 7, 0, 1] 2024-09-13T22:51:40.269 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.1d', '4', '2', '5', '7', '0', '1'] 2024-09-13T22:51:40.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 3.1d 4 2 5 7 0 1 2024-09-13T22:51:40.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:40.395 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:40.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:40.527 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:40.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:40.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:40.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:40.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:40.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:40.923 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:40.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:41.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:41.056 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:41.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:41.189 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:41.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:41.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:41.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:41.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:41.452 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:41.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:41.579 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:41.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:41.714 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:41.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:41.857 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:41.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:41.988 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:41.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:42.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:42.101 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:42 smithi192 bash[24608]: cluster 2024-09-13T22:51:40.744730+0000 mgr.y (mgr.14613) 522 : cluster [DBG] pgmap v599: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 7080/196275 objects misplaced (3.607%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T22:51:42.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:42.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:42.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:42.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:51:42.280 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:42 smithi178 bash[29889]: cluster 2024-09-13T22:51:40.744730+0000 mgr.y (mgr.14613) 522 : cluster [DBG] pgmap v599: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 7080/196275 objects misplaced (3.607%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T22:51:42.281 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:42 smithi178 bash[22840]: cluster 2024-09-13T22:51:40.744730+0000 mgr.y (mgr.14613) 522 : cluster [DBG] pgmap v599: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 7080/196275 objects misplaced (3.607%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T22:51:42.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:42.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:42.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:42.523 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:42.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:42.656 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:42.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:42.798 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:42.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:42.937 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:42.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:43.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:43.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:43.229 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:43.333 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:43.342 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[29889]: cluster 2024-09-13T22:51:41.824308+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[29889]: cluster 2024-09-13T22:51:41.824380+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[29889]: cluster 2024-09-13T22:51:42.034119+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[29889]: cluster 2024-09-13T22:51:42.960760+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[22840]: cluster 2024-09-13T22:51:41.824308+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[22840]: cluster 2024-09-13T22:51:41.824380+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[22840]: cluster 2024-09-13T22:51:42.034119+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-13T22:51:43.343 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:43 smithi178 bash[22840]: cluster 2024-09-13T22:51:42.960760+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-13T22:51:43.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:43.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:43.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:43 smithi192 bash[24608]: cluster 2024-09-13T22:51:41.824308+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:43.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:43 smithi192 bash[24608]: cluster 2024-09-13T22:51:41.824380+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:43.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:43 smithi192 bash[24608]: cluster 2024-09-13T22:51:42.034119+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-13T22:51:43.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:43 smithi192 bash[24608]: cluster 2024-09-13T22:51:42.960760+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-13T22:51:43.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:43.482 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:43.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.625 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:43.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:43.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:43.751 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:43.768 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:43.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:43.887 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:43.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:43.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:44.024 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:44.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:44.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:44 smithi178 bash[29889]: cluster 2024-09-13T22:51:42.745110+0000 mgr.y (mgr.14613) 523 : cluster [DBG] pgmap v601: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 878 op/s; 7080/196275 objects misplaced (3.607%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T22:51:44.136 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:44 smithi178 bash[22840]: cluster 2024-09-13T22:51:42.745110+0000 mgr.y (mgr.14613) 523 : cluster [DBG] pgmap v601: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 878 op/s; 7080/196275 objects misplaced (3.607%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T22:51:44.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:44.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:44 smithi192 bash[24608]: cluster 2024-09-13T22:51:42.745110+0000 mgr.y (mgr.14613) 523 : cluster [DBG] pgmap v601: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 878 op/s; 7080/196275 objects misplaced (3.607%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T22:51:44.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:44.287 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:44.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:44.416 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:44.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:44.544 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:44.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:44.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:44.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:44.786 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:44.786 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:44.824 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:44.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:44.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:44.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:45.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:45.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:45.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:45.227 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:45.333 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:45.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:45.367 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:45 smithi178 bash[29889]: cluster 2024-09-13T22:51:45.043898+0000 mon.a (mon.0) 1391 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T22:51:45.367 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:45 smithi178 bash[29889]: cluster 2024-09-13T22:51:45.043937+0000 mon.a (mon.0) 1392 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:51:45.367 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:45 smithi178 bash[22840]: cluster 2024-09-13T22:51:45.043898+0000 mon.a (mon.0) 1391 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T22:51:45.368 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:45 smithi178 bash[22840]: cluster 2024-09-13T22:51:45.043937+0000 mon.a (mon.0) 1392 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:51:45.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:45 smithi192 bash[24608]: cluster 2024-09-13T22:51:45.043898+0000 mon.a (mon.0) 1391 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T22:51:45.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:45 smithi192 bash[24608]: cluster 2024-09-13T22:51:45.043937+0000 mon.a (mon.0) 1392 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:51:45.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:45.480 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:45.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:45.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:45.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:45.733 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:45.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:45.846 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:51:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:51:45.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:45.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:45.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:45.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:46.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:46.117 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:46.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:46.241 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:46.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:46.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:46 smithi178 bash[22840]: cluster 2024-09-13T22:51:44.745745+0000 mgr.y (mgr.14613) 524 : cluster [DBG] pgmap v603: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 1.10k op/s; 6692/186294 objects misplaced (3.592%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T22:51:46.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:46 smithi178 bash[22840]: audit 2024-09-13T22:51:45.517592+0000 mgr.y (mgr.14613) 525 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:46.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:46 smithi178 bash[29889]: cluster 2024-09-13T22:51:44.745745+0000 mgr.y (mgr.14613) 524 : cluster [DBG] pgmap v603: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 1.10k op/s; 6692/186294 objects misplaced (3.592%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T22:51:46.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:46 smithi178 bash[29889]: audit 2024-09-13T22:51:45.517592+0000 mgr.y (mgr.14613) 525 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:46.370 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:46.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:46 smithi192 bash[24608]: cluster 2024-09-13T22:51:44.745745+0000 mgr.y (mgr.14613) 524 : cluster [DBG] pgmap v603: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 1.10k op/s; 6692/186294 objects misplaced (3.592%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T22:51:46.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:46 smithi192 bash[24608]: audit 2024-09-13T22:51:45.517592+0000 mgr.y (mgr.14613) 525 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:46.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:46.502 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:46.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:46.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:46.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:46.768 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:46.819 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:51:46.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:46.899 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:46.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:47.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:47.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:47.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:47.171 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:47.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:47.294 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:47.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:47.429 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:47.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:47.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:47.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:47.686 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:47.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:47.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:47.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:47.946 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:47.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:48.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:48.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:48.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:48.211 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:48.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:48.328 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:48 smithi192 bash[24608]: cluster 2024-09-13T22:51:46.747458+0000 mgr.y (mgr.14613) 526 : cluster [DBG] pgmap v604: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 6025/178449 objects misplaced (3.376%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T22:51:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:48.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:48 smithi178 bash[22840]: cluster 2024-09-13T22:51:46.747458+0000 mgr.y (mgr.14613) 526 : cluster [DBG] pgmap v604: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 6025/178449 objects misplaced (3.376%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T22:51:48.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:48 smithi178 bash[29889]: cluster 2024-09-13T22:51:46.747458+0000 mgr.y (mgr.14613) 526 : cluster [DBG] pgmap v604: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 6025/178449 objects misplaced (3.376%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T22:51:48.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:48.477 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:48.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:48.607 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:48.713 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:48.742 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:48.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:48.867 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:48.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:48.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:48.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:49.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:49.130 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:49.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:49.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:49.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:49.403 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:49.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:49.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:49.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:49.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:49.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:49.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:49.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:49 smithi192 bash[24608]: cluster 2024-09-13T22:51:48.747904+0000 mgr.y (mgr.14613) 527 : cluster [DBG] pgmap v605: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 763 op/s; 6025/177957 objects misplaced (3.386%); 2.1 MiB/s, 32 objects/s recovering 2024-09-13T22:51:49.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:49 smithi192 bash[24608]: audit 2024-09-13T22:51:48.751429+0000 mon.a (mon.0) 1393 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:49.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:49 smithi192 bash[24608]: audit 2024-09-13T22:51:48.756288+0000 mon.b (mon.1) 209 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:49.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:49.953 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:49.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:49.995 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:51:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:51:49.995 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[29889]: cluster 2024-09-13T22:51:48.747904+0000 mgr.y (mgr.14613) 527 : cluster [DBG] pgmap v605: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 763 op/s; 6025/177957 objects misplaced (3.386%); 2.1 MiB/s, 32 objects/s recovering 2024-09-13T22:51:49.996 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[29889]: audit 2024-09-13T22:51:48.751429+0000 mon.a (mon.0) 1393 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:49.996 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[29889]: audit 2024-09-13T22:51:48.756288+0000 mon.b (mon.1) 209 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:49.996 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[22840]: cluster 2024-09-13T22:51:48.747904+0000 mgr.y (mgr.14613) 527 : cluster [DBG] pgmap v605: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 763 op/s; 6025/177957 objects misplaced (3.386%); 2.1 MiB/s, 32 objects/s recovering 2024-09-13T22:51:49.996 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[22840]: audit 2024-09-13T22:51:48.751429+0000 mon.a (mon.0) 1393 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:51:49.996 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:49 smithi178 bash[22840]: audit 2024-09-13T22:51:48.756288+0000 mon.b (mon.1) 209 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:51:50.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:50.081 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:50.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:50.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:50.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:50.345 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:50.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:50.489 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:50.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:50.616 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:50.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:50.749 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:50.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:50.884 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:50.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:50.987 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:51.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:51.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:51.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:51.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:51.276 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:51.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:51.405 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:51.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:51.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:51.639 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:51.672 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:51.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:51.800 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:51.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[29889]: cluster 2024-09-13T22:51:50.492534+0000 osd.7 (osd.7) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[29889]: cluster 2024-09-13T22:51:50.492597+0000 osd.7 (osd.7) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[29889]: cluster 2024-09-13T22:51:50.748968+0000 mgr.y (mgr.14613) 528 : cluster [DBG] pgmap v606: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.23k op/s; 3302/164115 objects misplaced (2.012%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:51:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[29889]: cluster 2024-09-13T22:51:50.758641+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-13T22:51:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[22840]: cluster 2024-09-13T22:51:50.492534+0000 osd.7 (osd.7) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[22840]: cluster 2024-09-13T22:51:50.492597+0000 osd.7 (osd.7) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[22840]: cluster 2024-09-13T22:51:50.748968+0000 mgr.y (mgr.14613) 528 : cluster [DBG] pgmap v606: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.23k op/s; 3302/164115 objects misplaced (2.012%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:51:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:51 smithi178 bash[22840]: cluster 2024-09-13T22:51:50.758641+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-13T22:51:51.940 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:51.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:52.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:52.076 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:51 smithi192 bash[24608]: cluster 2024-09-13T22:51:50.492534+0000 osd.7 (osd.7) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:51 smithi192 bash[24608]: cluster 2024-09-13T22:51:50.492597+0000 osd.7 (osd.7) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:51 smithi192 bash[24608]: cluster 2024-09-13T22:51:50.748968+0000 mgr.y (mgr.14613) 528 : cluster [DBG] pgmap v606: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.23k op/s; 3302/164115 objects misplaced (2.012%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:51:52.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:51 smithi192 bash[24608]: cluster 2024-09-13T22:51:50.758641+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-13T22:51:52.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:52.214 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:52.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:52.346 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:52.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.481 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:52.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:52.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:52.613 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:52.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:52.747 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:52.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:52.886 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:52.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:52.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:53.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:53.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:53.185 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:53.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:53.339 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:53.368 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:51:53.368 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:51:53.368 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:51:53.368 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:51:53.368 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:51:53.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:53.452 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:51:53.453 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:51:53.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:53.503 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:51:53.504 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:51:53.548 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:51:53.548 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:51:53.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:53.586 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:51:53.586 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:51:53.615 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:53.627 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:51:53.627 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:51:53.659 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:51:53.659 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:51:53.697 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:51:53.697 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:51:53.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:53.742 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:51:53.742 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:51:53.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:53.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:53.878 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:53.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:53.980 INFO:teuthology.orchestra.run.smithi178.stderr:set 3.1d pg_upmap_items mapping to [4->2,5->7,0->1] 2024-09-13T22:51:53.983 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:51:53.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:54.010 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:54.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[29889]: cluster 2024-09-13T22:51:52.749470+0000 mgr.y (mgr.14613) 529 : cluster [DBG] pgmap v608: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 3302/164115 objects misplaced (2.012%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T22:51:54.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[29889]: cluster 2024-09-13T22:51:52.970884+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-13T22:51:54.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[29889]: audit 2024-09-13T22:51:53.346577+0000 mon.a (mon.0) 1396 : audit [DBG] from='client.? v1:172.21.15.178:0/3186499175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:54.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[29889]: audit 2024-09-13T22:51:53.730064+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? v1:172.21.15.178:0/1781673826' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [4, 2, 5, 7, 0, 1]}]: dispatch 2024-09-13T22:51:54.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[29889]: audit 2024-09-13T22:51:53.769517+0000 mon.a (mon.0) 1398 : audit [INF] from='client.25321 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:51:54.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[29889]: audit 2024-09-13T22:51:53.771065+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.178:0/3469977296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:51:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:54.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[22840]: cluster 2024-09-13T22:51:52.749470+0000 mgr.y (mgr.14613) 529 : cluster [DBG] pgmap v608: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 3302/164115 objects misplaced (2.012%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T22:51:54.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[22840]: cluster 2024-09-13T22:51:52.970884+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-13T22:51:54.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[22840]: audit 2024-09-13T22:51:53.346577+0000 mon.a (mon.0) 1396 : audit [DBG] from='client.? v1:172.21.15.178:0/3186499175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:54.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[22840]: audit 2024-09-13T22:51:53.730064+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? v1:172.21.15.178:0/1781673826' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [4, 2, 5, 7, 0, 1]}]: dispatch 2024-09-13T22:51:54.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[22840]: audit 2024-09-13T22:51:53.769517+0000 mon.a (mon.0) 1398 : audit [INF] from='client.25321 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:51:54.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:53 smithi178 bash[22840]: audit 2024-09-13T22:51:53.771065+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.178:0/3469977296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:51:54.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:54.138 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:54.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:53 smithi192 bash[24608]: cluster 2024-09-13T22:51:52.749470+0000 mgr.y (mgr.14613) 529 : cluster [DBG] pgmap v608: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 3302/164115 objects misplaced (2.012%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T22:51:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:53 smithi192 bash[24608]: cluster 2024-09-13T22:51:52.970884+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-13T22:51:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:53 smithi192 bash[24608]: audit 2024-09-13T22:51:53.346577+0000 mon.a (mon.0) 1396 : audit [DBG] from='client.? v1:172.21.15.178:0/3186499175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:51:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:53 smithi192 bash[24608]: audit 2024-09-13T22:51:53.730064+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? v1:172.21.15.178:0/1781673826' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [4, 2, 5, 7, 0, 1]}]: dispatch 2024-09-13T22:51:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:53 smithi192 bash[24608]: audit 2024-09-13T22:51:53.769517+0000 mon.a (mon.0) 1398 : audit [INF] from='client.25321 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:51:54.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:53 smithi192 bash[24608]: audit 2024-09-13T22:51:53.771065+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.178:0/3469977296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:51:54.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:54.264 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:54.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:54.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:54.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:54.513 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:54.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:54.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:54.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:54.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:54.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:54.898 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:54.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:55.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:55.022 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:55.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:54 smithi192 bash[24608]: cluster 2024-09-13T22:51:53.975865+0000 mon.a (mon.0) 1399 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:54 smithi192 bash[24608]: audit 2024-09-13T22:51:53.978192+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? v1:172.21.15.178:0/1781673826' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [4, 2, 5, 7, 0, 1]}]': finished 2024-09-13T22:51:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:54 smithi192 bash[24608]: audit 2024-09-13T22:51:53.978405+0000 mon.a (mon.0) 1401 : audit [INF] from='client.25321 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:51:55.134 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:54 smithi192 bash[24608]: cluster 2024-09-13T22:51:53.987708+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-13T22:51:55.147 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:55.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[22840]: cluster 2024-09-13T22:51:53.975865+0000 mon.a (mon.0) 1399 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:55.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[22840]: audit 2024-09-13T22:51:53.978192+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? v1:172.21.15.178:0/1781673826' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [4, 2, 5, 7, 0, 1]}]': finished 2024-09-13T22:51:55.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[22840]: audit 2024-09-13T22:51:53.978405+0000 mon.a (mon.0) 1401 : audit [INF] from='client.25321 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:51:55.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[22840]: cluster 2024-09-13T22:51:53.987708+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-13T22:51:55.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[29889]: cluster 2024-09-13T22:51:53.975865+0000 mon.a (mon.0) 1399 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:51:55.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[29889]: audit 2024-09-13T22:51:53.978192+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? v1:172.21.15.178:0/1781673826' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [4, 2, 5, 7, 0, 1]}]': finished 2024-09-13T22:51:55.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[29889]: audit 2024-09-13T22:51:53.978405+0000 mon.a (mon.0) 1401 : audit [INF] from='client.25321 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:51:55.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:54 smithi178 bash[29889]: cluster 2024-09-13T22:51:53.987708+0000 mon.a (mon.0) 1402 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-13T22:51:55.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:55.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:55.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:55.409 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:55.513 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:55.540 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:55.651 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:55.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:55.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:55.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:51:55.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:55.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:55.952 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:55.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:56.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:54.338697+0000 osd.6 (osd.6) 223 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:54.341370+0000 osd.6 (osd.6) 224 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:54.348464+0000 osd.5 (osd.5) 305 : cluster [DBG] 6.d scrub starts 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:54.350071+0000 osd.5 (osd.5) 306 : cluster [DBG] 6.d scrub ok 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:54.750404+0000 mgr.y (mgr.14613) 530 : cluster [DBG] pgmap v611: 168 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 166 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s; 1596/155604 objects misplaced (1.026%); 20 MiB/s, 318 objects/s recovering 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:54.986347+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: audit 2024-09-13T22:51:55.527939+0000 mgr.y (mgr.14613) 531 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:56.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:55.898857+0000 osd.0 (osd.0) 275 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:56.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:55 smithi192 bash[24608]: cluster 2024-09-13T22:51:55.898938+0000 osd.0 (osd.0) 276 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:56.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:56.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:54.338697+0000 osd.6 (osd.6) 223 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:51:56.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:54.341370+0000 osd.6 (osd.6) 224 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:51:56.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:54.348464+0000 osd.5 (osd.5) 305 : cluster [DBG] 6.d scrub starts 2024-09-13T22:51:56.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:54.350071+0000 osd.5 (osd.5) 306 : cluster [DBG] 6.d scrub ok 2024-09-13T22:51:56.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:54.750404+0000 mgr.y (mgr.14613) 530 : cluster [DBG] pgmap v611: 168 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 166 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s; 1596/155604 objects misplaced (1.026%); 20 MiB/s, 318 objects/s recovering 2024-09-13T22:51:56.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:54.986347+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-13T22:51:56.226 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: audit 2024-09-13T22:51:55.527939+0000 mgr.y (mgr.14613) 531 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:55.898857+0000 osd.0 (osd.0) 275 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[29889]: cluster 2024-09-13T22:51:55.898938+0000 osd.0 (osd.0) 276 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:54.338697+0000 osd.6 (osd.6) 223 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:54.341370+0000 osd.6 (osd.6) 224 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:54.348464+0000 osd.5 (osd.5) 305 : cluster [DBG] 6.d scrub starts 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:54.350071+0000 osd.5 (osd.5) 306 : cluster [DBG] 6.d scrub ok 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:54.750404+0000 mgr.y (mgr.14613) 530 : cluster [DBG] pgmap v611: 168 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 166 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s; 1596/155604 objects misplaced (1.026%); 20 MiB/s, 318 objects/s recovering 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:54.986347+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: audit 2024-09-13T22:51:55.527939+0000 mgr.y (mgr.14613) 531 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:51:56.227 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:55.898857+0000 osd.0 (osd.0) 275 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:51:56.228 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:55 smithi178 bash[22840]: cluster 2024-09-13T22:51:55.898938+0000 osd.0 (osd.0) 276 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:51:56.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:56.262 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:56.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:56.386 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:56.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T22:51:56.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:56.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:56.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:56.638 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:56.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:56.771 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:56.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:56.903 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:56.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:57.009 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:57.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:57.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:57.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:57.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:57.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:57 smithi192 bash[24608]: cluster 2024-09-13T22:51:55.297732+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:51:57.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:57 smithi192 bash[24608]: cluster 2024-09-13T22:51:55.298814+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:51:57.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:57 smithi192 bash[24608]: cluster 2024-09-13T22:51:55.784206+0000 osd.3 (osd.3) 243 : cluster [DBG] 9.19 scrub starts 2024-09-13T22:51:57.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:57 smithi192 bash[24608]: cluster 2024-09-13T22:51:55.997422+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-13T22:51:57.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:57 smithi192 bash[24608]: cluster 2024-09-13T22:51:56.310096+0000 osd.3 (osd.3) 244 : cluster [DBG] 9.19 scrub ok 2024-09-13T22:51:57.299 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:57.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[29889]: cluster 2024-09-13T22:51:55.297732+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[29889]: cluster 2024-09-13T22:51:55.298814+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[29889]: cluster 2024-09-13T22:51:55.784206+0000 osd.3 (osd.3) 243 : cluster [DBG] 9.19 scrub starts 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[29889]: cluster 2024-09-13T22:51:55.997422+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[29889]: cluster 2024-09-13T22:51:56.310096+0000 osd.3 (osd.3) 244 : cluster [DBG] 9.19 scrub ok 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[22840]: cluster 2024-09-13T22:51:55.297732+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[22840]: cluster 2024-09-13T22:51:55.298814+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[22840]: cluster 2024-09-13T22:51:55.784206+0000 osd.3 (osd.3) 243 : cluster [DBG] 9.19 scrub starts 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[22840]: cluster 2024-09-13T22:51:55.997422+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-13T22:51:57.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:57 smithi178 bash[22840]: cluster 2024-09-13T22:51:56.310096+0000 osd.3 (osd.3) 244 : cluster [DBG] 9.19 scrub ok 2024-09-13T22:51:57.426 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:57.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:57.553 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:57.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:57.696 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:51:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:51:57.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:57.826 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:57.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:57.948 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:51:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:57.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:51:58.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:58.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:56.272196+0000 osd.6 (osd.6) 227 : cluster [DBG] 9.18 scrub starts 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:56.441612+0000 osd.6 (osd.6) 228 : cluster [DBG] 9.18 scrub ok 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:56.747910+0000 osd.5 (osd.5) 307 : cluster [DBG] 9.12 scrub starts 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:56.751025+0000 mgr.y (mgr.14613) 532 : cluster [DBG] pgmap v614: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 166 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 624/149856 objects misplaced (0.416%); 17 MiB/s, 272 objects/s recovering 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:56.753217+0000 osd.3 (osd.3) 245 : cluster [DBG] 9.1 scrub starts 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:57.332211+0000 osd.3 (osd.3) 246 : cluster [DBG] 9.1 scrub ok 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[29889]: cluster 2024-09-13T22:51:57.968934+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-13T22:51:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:56.272196+0000 osd.6 (osd.6) 227 : cluster [DBG] 9.18 scrub starts 2024-09-13T22:51:58.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:56.441612+0000 osd.6 (osd.6) 228 : cluster [DBG] 9.18 scrub ok 2024-09-13T22:51:58.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:56.747910+0000 osd.5 (osd.5) 307 : cluster [DBG] 9.12 scrub starts 2024-09-13T22:51:58.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:56.751025+0000 mgr.y (mgr.14613) 532 : cluster [DBG] pgmap v614: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 166 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 624/149856 objects misplaced (0.416%); 17 MiB/s, 272 objects/s recovering 2024-09-13T22:51:58.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:56.753217+0000 osd.3 (osd.3) 245 : cluster [DBG] 9.1 scrub starts 2024-09-13T22:51:58.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:57.332211+0000 osd.3 (osd.3) 246 : cluster [DBG] 9.1 scrub ok 2024-09-13T22:51:58.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:58 smithi178 bash[22840]: cluster 2024-09-13T22:51:57.968934+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-13T22:51:58.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:58.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:58.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:56.272196+0000 osd.6 (osd.6) 227 : cluster [DBG] 9.18 scrub starts 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:56.441612+0000 osd.6 (osd.6) 228 : cluster [DBG] 9.18 scrub ok 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:56.747910+0000 osd.5 (osd.5) 307 : cluster [DBG] 9.12 scrub starts 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:56.751025+0000 mgr.y (mgr.14613) 532 : cluster [DBG] pgmap v614: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 166 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 624/149856 objects misplaced (0.416%); 17 MiB/s, 272 objects/s recovering 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:56.753217+0000 osd.3 (osd.3) 245 : cluster [DBG] 9.1 scrub starts 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:57.332211+0000 osd.3 (osd.3) 246 : cluster [DBG] 9.1 scrub ok 2024-09-13T22:51:58.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:58 smithi192 bash[24608]: cluster 2024-09-13T22:51:57.968934+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-13T22:51:58.331 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:58.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:58.455 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:58.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:58.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:58.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:58.721 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:58.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:58.859 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:51:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:58.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:51:58.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:58.995 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:59.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:59.134 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:51:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:51:59.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:59.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[22840]: cluster 2024-09-13T22:51:57.202371+0000 osd.5 (osd.5) 308 : cluster [DBG] 9.12 scrub ok 2024-09-13T22:51:59.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[22840]: cluster 2024-09-13T22:51:57.571895+0000 osd.6 (osd.6) 229 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:51:59.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[22840]: cluster 2024-09-13T22:51:57.575575+0000 osd.6 (osd.6) 230 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:51:59.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[22840]: cluster 2024-09-13T22:51:57.733873+0000 osd.3 (osd.3) 247 : cluster [DBG] 9.2 scrub starts 2024-09-13T22:51:59.248 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[29889]: cluster 2024-09-13T22:51:57.202371+0000 osd.5 (osd.5) 308 : cluster [DBG] 9.12 scrub ok 2024-09-13T22:51:59.248 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[29889]: cluster 2024-09-13T22:51:57.571895+0000 osd.6 (osd.6) 229 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:51:59.248 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[29889]: cluster 2024-09-13T22:51:57.575575+0000 osd.6 (osd.6) 230 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:51:59.248 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[29889]: cluster 2024-09-13T22:51:57.733873+0000 osd.3 (osd.3) 247 : cluster [DBG] 9.2 scrub starts 2024-09-13T22:51:59.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:59.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:59.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:51:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:51:59.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:59 smithi192 bash[24608]: cluster 2024-09-13T22:51:57.202371+0000 osd.5 (osd.5) 308 : cluster [DBG] 9.12 scrub ok 2024-09-13T22:51:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:59 smithi192 bash[24608]: cluster 2024-09-13T22:51:57.571895+0000 osd.6 (osd.6) 229 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:51:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:59 smithi192 bash[24608]: cluster 2024-09-13T22:51:57.575575+0000 osd.6 (osd.6) 230 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:51:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:51:59 smithi192 bash[24608]: cluster 2024-09-13T22:51:57.733873+0000 osd.3 (osd.3) 247 : cluster [DBG] 9.2 scrub starts 2024-09-13T22:51:59.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:59.512 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:51:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:51:59.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:59.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:51:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:51:59.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:59.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:59.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:59.893 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:51:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:51:59.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:51:59.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:00.007 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:51:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:51:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:52:00.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:00.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:58.619895+0000 osd.3 (osd.3) 248 : cluster [DBG] 9.2 scrub ok 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:58.747299+0000 osd.3 (osd.3) 249 : cluster [DBG] 9.8 scrub starts 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:58.751767+0000 mgr.y (mgr.14613) 533 : cluster [DBG] pgmap v616: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 166 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 422 op/s; 624/149568 objects misplaced (0.417%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:58.856462+0000 osd.2 (osd.2) 213 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:58.859519+0000 osd.2 (osd.2) 214 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.017897+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.039630+0000 osd.3 (osd.3) 250 : cluster [DBG] 9.8 scrub ok 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.202193+0000 osd.4 (osd.4) 183 : cluster [DBG] 9.16 scrub starts 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.226581+0000 osd.7 (osd.7) 279 : cluster [DBG] 9.17 scrub starts 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.460061+0000 osd.4 (osd.4) 184 : cluster [DBG] 9.16 scrub ok 2024-09-13T22:52:00.138 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.509985+0000 osd.7 (osd.7) 280 : cluster [DBG] 9.17 scrub ok 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.709188+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.711526+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:58.619895+0000 osd.3 (osd.3) 248 : cluster [DBG] 9.2 scrub ok 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:58.747299+0000 osd.3 (osd.3) 249 : cluster [DBG] 9.8 scrub starts 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:58.751767+0000 mgr.y (mgr.14613) 533 : cluster [DBG] pgmap v616: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 166 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 422 op/s; 624/149568 objects misplaced (0.417%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:58.856462+0000 osd.2 (osd.2) 213 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:58.859519+0000 osd.2 (osd.2) 214 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.017897+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-13T22:52:00.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.039630+0000 osd.3 (osd.3) 250 : cluster [DBG] 9.8 scrub ok 2024-09-13T22:52:00.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.202193+0000 osd.4 (osd.4) 183 : cluster [DBG] 9.16 scrub starts 2024-09-13T22:52:00.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.226581+0000 osd.7 (osd.7) 279 : cluster [DBG] 9.17 scrub starts 2024-09-13T22:52:00.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.460061+0000 osd.4 (osd.4) 184 : cluster [DBG] 9.16 scrub ok 2024-09-13T22:52:00.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.509985+0000 osd.7 (osd.7) 280 : cluster [DBG] 9.17 scrub ok 2024-09-13T22:52:00.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.709188+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:52:00.140 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:00 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.711526+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:52:00.156 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:00.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:00.287 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:00.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:58.619895+0000 osd.3 (osd.3) 248 : cluster [DBG] 9.2 scrub ok 2024-09-13T22:52:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:58.747299+0000 osd.3 (osd.3) 249 : cluster [DBG] 9.8 scrub starts 2024-09-13T22:52:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:58.751767+0000 mgr.y (mgr.14613) 533 : cluster [DBG] pgmap v616: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 166 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 422 op/s; 624/149568 objects misplaced (0.417%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T22:52:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:58.856462+0000 osd.2 (osd.2) 213 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:52:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:58.859519+0000 osd.2 (osd.2) 214 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.017897+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.039630+0000 osd.3 (osd.3) 250 : cluster [DBG] 9.8 scrub ok 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.202193+0000 osd.4 (osd.4) 183 : cluster [DBG] 9.16 scrub starts 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.226581+0000 osd.7 (osd.7) 279 : cluster [DBG] 9.17 scrub starts 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.460061+0000 osd.4 (osd.4) 184 : cluster [DBG] 9.16 scrub ok 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.509985+0000 osd.7 (osd.7) 280 : cluster [DBG] 9.17 scrub ok 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.709188+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:52:00.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:00 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.711526+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:52:00.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:00.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:00.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:00.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:00.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:00.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:00.845 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:00.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:00.978 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:00.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:01.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:01.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:52:01.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:01 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.288820+0000 osd.6 (osd.6) 231 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:52:01.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:01 smithi192 bash[24608]: cluster 2024-09-13T22:51:59.289806+0000 osd.6 (osd.6) 232 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:52:01.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:01 smithi192 bash[24608]: cluster 2024-09-13T22:52:00.020271+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-13T22:52:01.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:01 smithi192 bash[24608]: cluster 2024-09-13T22:52:00.249587+0000 osd.4 (osd.4) 185 : cluster [DBG] 9.10 scrub starts 2024-09-13T22:52:01.104 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:52:01.109 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:01.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:01.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.288820+0000 osd.6 (osd.6) 231 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:52:01.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[29889]: cluster 2024-09-13T22:51:59.289806+0000 osd.6 (osd.6) 232 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:52:01.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[29889]: cluster 2024-09-13T22:52:00.020271+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-13T22:52:01.225 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[29889]: cluster 2024-09-13T22:52:00.249587+0000 osd.4 (osd.4) 185 : cluster [DBG] 9.10 scrub starts 2024-09-13T22:52:01.228 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.288820+0000 osd.6 (osd.6) 231 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:52:01.228 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[22840]: cluster 2024-09-13T22:51:59.289806+0000 osd.6 (osd.6) 232 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:52:01.228 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[22840]: cluster 2024-09-13T22:52:00.020271+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-13T22:52:01.229 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:01 smithi178 bash[22840]: cluster 2024-09-13T22:52:00.249587+0000 osd.4 (osd.4) 185 : cluster [DBG] 9.10 scrub starts 2024-09-13T22:52:01.261 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:01.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:01.391 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:01.424 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:01.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:01.521 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:01.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:01.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:01.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:01.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:01.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:01.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:01.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:02.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:02.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:02.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:02.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:00.603527+0000 osd.4 (osd.4) 186 : cluster [DBG] 9.10 scrub ok 2024-09-13T22:52:02.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:00.752300+0000 mgr.y (mgr.14613) 534 : cluster [DBG] pgmap v619: 168 pgs: 1 active+recovering+undersized+remapped, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 624/139902 objects misplaced (0.446%) 2024-09-13T22:52:02.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.170033+0000 osd.1 (osd.1) 260 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:52:02.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.171334+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:52:02.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.188792+0000 osd.7 (osd.7) 281 : cluster [DBG] 9.13 scrub starts 2024-09-13T22:52:02.174 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.201355+0000 osd.4 (osd.4) 187 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:52:02.174 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.204941+0000 osd.4 (osd.4) 188 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:52:02.174 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.540187+0000 osd.7 (osd.7) 282 : cluster [DBG] 9.13 scrub ok 2024-09-13T22:52:02.174 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.773560+0000 osd.0 (osd.0) 279 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:52:02.174 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:02 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.775224+0000 osd.0 (osd.0) 280 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:52:02.190 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:02.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:00.603527+0000 osd.4 (osd.4) 186 : cluster [DBG] 9.10 scrub ok 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:00.752300+0000 mgr.y (mgr.14613) 534 : cluster [DBG] pgmap v619: 168 pgs: 1 active+recovering+undersized+remapped, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 624/139902 objects misplaced (0.446%) 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.170033+0000 osd.1 (osd.1) 260 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.171334+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.188792+0000 osd.7 (osd.7) 281 : cluster [DBG] 9.13 scrub starts 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.201355+0000 osd.4 (osd.4) 187 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:52:02.308 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.204941+0000 osd.4 (osd.4) 188 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.540187+0000 osd.7 (osd.7) 282 : cluster [DBG] 9.13 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.773560+0000 osd.0 (osd.0) 279 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.775224+0000 osd.0 (osd.0) 280 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:00.603527+0000 osd.4 (osd.4) 186 : cluster [DBG] 9.10 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:00.752300+0000 mgr.y (mgr.14613) 534 : cluster [DBG] pgmap v619: 168 pgs: 1 active+recovering+undersized+remapped, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 624/139902 objects misplaced (0.446%) 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.170033+0000 osd.1 (osd.1) 260 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.171334+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.188792+0000 osd.7 (osd.7) 281 : cluster [DBG] 9.13 scrub starts 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.201355+0000 osd.4 (osd.4) 187 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.204941+0000 osd.4 (osd.4) 188 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.540187+0000 osd.7 (osd.7) 282 : cluster [DBG] 9.13 scrub ok 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.773560+0000 osd.0 (osd.0) 279 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:52:02.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:02 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.775224+0000 osd.0 (osd.0) 280 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:52:02.327 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:02.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:02.469 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:02.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:02.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:02.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:02.742 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:02.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:02.891 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:02.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:02.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:03.025 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:03.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:03.156 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.333236+0000 osd.6 (osd.6) 233 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.336550+0000 osd.6 (osd.6) 234 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.754652+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.c scrub starts 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:01.759099+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.c scrub ok 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.194121+0000 osd.1 (osd.1) 262 : cluster [DBG] 9.14 scrub starts 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.607206+0000 osd.1 (osd.1) 263 : cluster [DBG] 9.14 scrub ok 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.799682+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.803007+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:52:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.333236+0000 osd.6 (osd.6) 233 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.336550+0000 osd.6 (osd.6) 234 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.754652+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.c scrub starts 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:01.759099+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.c scrub ok 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.194121+0000 osd.1 (osd.1) 262 : cluster [DBG] 9.14 scrub starts 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.607206+0000 osd.1 (osd.1) 263 : cluster [DBG] 9.14 scrub ok 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.799682+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:52:03.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:03 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.803007+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:52:03.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.333236+0000 osd.6 (osd.6) 233 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.336550+0000 osd.6 (osd.6) 234 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.754652+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.c scrub starts 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:01.759099+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.c scrub ok 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.194121+0000 osd.1 (osd.1) 262 : cluster [DBG] 9.14 scrub starts 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.607206+0000 osd.1 (osd.1) 263 : cluster [DBG] 9.14 scrub ok 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.799682+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:52:03.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:03 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.803007+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:52:03.299 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:03.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:03.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:03.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:03.574 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:03.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:03.703 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:03.809 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:03.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:03.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:03.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:03.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:04.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:04.084 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:52:04.084 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:52:04.085 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T22:52:04.085 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-09-13T22:52:04.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T22:52:04.090 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:04.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:04.233 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:04.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:04.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.336086+0000 osd.6 (osd.6) 235 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:52:04.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.344062+0000 osd.6 (osd.6) 236 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:52:04.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:02.752770+0000 mgr.y (mgr.14613) 535 : cluster [DBG] pgmap v620: 168 pgs: 1 active+recovering+undersized+remapped, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 624/139902 objects misplaced (0.446%) 2024-09-13T22:52:04.350 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.163872+0000 osd.1 (osd.1) 264 : cluster [DBG] 9.0 scrub starts 2024-09-13T22:52:04.351 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.204789+0000 osd.7 (osd.7) 283 : cluster [DBG] 9.3 scrub starts 2024-09-13T22:52:04.351 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.508699+0000 osd.1 (osd.1) 265 : cluster [DBG] 9.0 scrub ok 2024-09-13T22:52:04.351 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.555940+0000 osd.7 (osd.7) 284 : cluster [DBG] 9.3 scrub ok 2024-09-13T22:52:04.351 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: audit 2024-09-13T22:52:03.768640+0000 mon.b (mon.1) 211 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:04.351 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.790573+0000 osd.0 (osd.0) 283 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:52:04.351 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:04 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.795797+0000 osd.0 (osd.0) 284 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:52:04.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.336086+0000 osd.6 (osd.6) 235 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:52:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.344062+0000 osd.6 (osd.6) 236 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:52:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:02.752770+0000 mgr.y (mgr.14613) 535 : cluster [DBG] pgmap v620: 168 pgs: 1 active+recovering+undersized+remapped, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 624/139902 objects misplaced (0.446%) 2024-09-13T22:52:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.163872+0000 osd.1 (osd.1) 264 : cluster [DBG] 9.0 scrub starts 2024-09-13T22:52:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.204789+0000 osd.7 (osd.7) 283 : cluster [DBG] 9.3 scrub starts 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.508699+0000 osd.1 (osd.1) 265 : cluster [DBG] 9.0 scrub ok 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.555940+0000 osd.7 (osd.7) 284 : cluster [DBG] 9.3 scrub ok 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: audit 2024-09-13T22:52:03.768640+0000 mon.b (mon.1) 211 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.790573+0000 osd.0 (osd.0) 283 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.795797+0000 osd.0 (osd.0) 284 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.336086+0000 osd.6 (osd.6) 235 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.344062+0000 osd.6 (osd.6) 236 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:02.752770+0000 mgr.y (mgr.14613) 535 : cluster [DBG] pgmap v620: 168 pgs: 1 active+recovering+undersized+remapped, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 624/139902 objects misplaced (0.446%) 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.163872+0000 osd.1 (osd.1) 264 : cluster [DBG] 9.0 scrub starts 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.204789+0000 osd.7 (osd.7) 283 : cluster [DBG] 9.3 scrub starts 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.508699+0000 osd.1 (osd.1) 265 : cluster [DBG] 9.0 scrub ok 2024-09-13T22:52:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.555940+0000 osd.7 (osd.7) 284 : cluster [DBG] 9.3 scrub ok 2024-09-13T22:52:04.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: audit 2024-09-13T22:52:03.768640+0000 mon.b (mon.1) 211 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:04.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.790573+0000 osd.0 (osd.0) 283 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:52:04.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:04 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.795797+0000 osd.0 (osd.0) 284 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:52:04.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:04.497 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:04.600 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:04.631 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:04.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:04.763 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:04.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:04.894 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:04.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:04.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:05.023 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:05.094 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:52:05.094 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:52:05.109 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.748798+0000 osd.3 (osd.3) 253 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.754465+0000 osd.3 (osd.3) 254 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.823959+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.b scrub starts 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:03.827059+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.b scrub ok 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.168268+0000 osd.1 (osd.1) 266 : cluster [DBG] 9.4 scrub starts 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.215913+0000 osd.4 (osd.4) 189 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.218092+0000 osd.4 (osd.4) 190 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.228879+0000 osd.7 (osd.7) 285 : cluster [DBG] 9.11 scrub starts 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.549629+0000 osd.1 (osd.1) 267 : cluster [DBG] 9.4 scrub ok 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.703003+0000 osd.7 (osd.7) 286 : cluster [DBG] 9.11 scrub ok 2024-09-13T22:52:05.110 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.748798+0000 osd.3 (osd.3) 253 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.754465+0000 osd.3 (osd.3) 254 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.823959+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.b scrub starts 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:03.827059+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.b scrub ok 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.168268+0000 osd.1 (osd.1) 266 : cluster [DBG] 9.4 scrub starts 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.215913+0000 osd.4 (osd.4) 189 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.218092+0000 osd.4 (osd.4) 190 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.228879+0000 osd.7 (osd.7) 285 : cluster [DBG] 9.11 scrub starts 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.549629+0000 osd.1 (osd.1) 267 : cluster [DBG] 9.4 scrub ok 2024-09-13T22:52:05.111 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:05 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.703003+0000 osd.7 (osd.7) 286 : cluster [DBG] 9.11 scrub ok 2024-09-13T22:52:05.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:05.162 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:05.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:05.297 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:05.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.748798+0000 osd.3 (osd.3) 253 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.754465+0000 osd.3 (osd.3) 254 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.823959+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.b scrub starts 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:03.827059+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.b scrub ok 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.168268+0000 osd.1 (osd.1) 266 : cluster [DBG] 9.4 scrub starts 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.215913+0000 osd.4 (osd.4) 189 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.218092+0000 osd.4 (osd.4) 190 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.228879+0000 osd.7 (osd.7) 285 : cluster [DBG] 9.11 scrub starts 2024-09-13T22:52:05.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.549629+0000 osd.1 (osd.1) 267 : cluster [DBG] 9.4 scrub ok 2024-09-13T22:52:05.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.703003+0000 osd.7 (osd.7) 286 : cluster [DBG] 9.11 scrub ok 2024-09-13T22:52:05.426 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:05.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:05.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:05.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:05.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:05.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:05.815 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:52:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:52:05.828 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:05.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:05.951 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:05.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:06.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:06.077 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.476530+0000 osd.5 (osd.5) 309 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.477881+0000 osd.5 (osd.5) 310 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: cluster 2024-09-13T22:52:04.753410+0000 mgr.y (mgr.14613) 536 : cluster [DBG] pgmap v621: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: cluster 2024-09-13T22:52:05.042729+0000 mon.a (mon.0) 1408 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: cluster 2024-09-13T22:52:05.210533+0000 osd.7 (osd.7) 287 : cluster [DBG] 9.15 scrub starts 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: audit 2024-09-13T22:52:05.536010+0000 mgr.y (mgr.14613) 537 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:06.079 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:06 smithi192 bash[24608]: cluster 2024-09-13T22:52:05.624629+0000 osd.7 (osd.7) 288 : cluster [DBG] 9.15 scrub ok 2024-09-13T22:52:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:06.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:06.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:06.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:06.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:06.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.480 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:06.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:06.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:06.606 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.476530+0000 osd.5 (osd.5) 309 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.477881+0000 osd.5 (osd.5) 310 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: cluster 2024-09-13T22:52:04.753410+0000 mgr.y (mgr.14613) 536 : cluster [DBG] pgmap v621: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: cluster 2024-09-13T22:52:05.042729+0000 mon.a (mon.0) 1408 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: cluster 2024-09-13T22:52:05.210533+0000 osd.7 (osd.7) 287 : cluster [DBG] 9.15 scrub starts 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: audit 2024-09-13T22:52:05.536010+0000 mgr.y (mgr.14613) 537 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[22840]: cluster 2024-09-13T22:52:05.624629+0000 osd.7 (osd.7) 288 : cluster [DBG] 9.15 scrub ok 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.476530+0000 osd.5 (osd.5) 309 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.477881+0000 osd.5 (osd.5) 310 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: cluster 2024-09-13T22:52:04.753410+0000 mgr.y (mgr.14613) 536 : cluster [DBG] pgmap v621: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: cluster 2024-09-13T22:52:05.042729+0000 mon.a (mon.0) 1408 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: cluster 2024-09-13T22:52:05.210533+0000 osd.7 (osd.7) 287 : cluster [DBG] 9.15 scrub starts 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: audit 2024-09-13T22:52:05.536010+0000 mgr.y (mgr.14613) 537 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:06 smithi178 bash[29889]: cluster 2024-09-13T22:52:05.624629+0000 osd.7 (osd.7) 288 : cluster [DBG] 9.15 scrub ok 2024-09-13T22:52:06.713 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:06.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:06.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T22:52:06.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:06.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:06.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:06.990 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:06.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:07.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:07.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:07.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:07 smithi192 bash[24608]: cluster 2024-09-13T22:52:05.829986+0000 osd.2 (osd.2) 217 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:52:07.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:07 smithi192 bash[24608]: cluster 2024-09-13T22:52:05.831303+0000 osd.2 (osd.2) 218 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:52:07.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:07 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.329347+0000 osd.7 (osd.7) 289 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:52:07.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:07 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.333681+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:52:07.200 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:07.201 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:07.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[22840]: cluster 2024-09-13T22:52:05.829986+0000 osd.2 (osd.2) 217 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[22840]: cluster 2024-09-13T22:52:05.831303+0000 osd.2 (osd.2) 218 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.329347+0000 osd.7 (osd.7) 289 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.333681+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[29889]: cluster 2024-09-13T22:52:05.829986+0000 osd.2 (osd.2) 217 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[29889]: cluster 2024-09-13T22:52:05.831303+0000 osd.2 (osd.2) 218 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.329347+0000 osd.7 (osd.7) 289 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:52:07.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:07 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.333681+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:52:07.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:07.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:07.400 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:07.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:07.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:07.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:07.674 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:07.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:07.811 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:07.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:07.939 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:07.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:08.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:08.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:08.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:08.184 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.337376+0000 osd.6 (osd.6) 237 : cluster [DBG] 3.a scrub starts 2024-09-13T22:52:08.184 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.349014+0000 osd.6 (osd.6) 238 : cluster [DBG] 3.a scrub ok 2024-09-13T22:52:08.184 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.754124+0000 mgr.y (mgr.14613) 538 : cluster [DBG] pgmap v622: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:52:08.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.781351+0000 osd.3 (osd.3) 255 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:52:08.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:06.783299+0000 osd.3 (osd.3) 256 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:52:08.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.268999+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:52:08.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:08 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.272666+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:52:08.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:08.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:08.325 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:08.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:08.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.337376+0000 osd.6 (osd.6) 237 : cluster [DBG] 3.a scrub starts 2024-09-13T22:52:08.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.349014+0000 osd.6 (osd.6) 238 : cluster [DBG] 3.a scrub ok 2024-09-13T22:52:08.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.754124+0000 mgr.y (mgr.14613) 538 : cluster [DBG] pgmap v622: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.781351+0000 osd.3 (osd.3) 255 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:06.783299+0000 osd.3 (osd.3) 256 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.268999+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.272666+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.337376+0000 osd.6 (osd.6) 237 : cluster [DBG] 3.a scrub starts 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.349014+0000 osd.6 (osd.6) 238 : cluster [DBG] 3.a scrub ok 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.754124+0000 mgr.y (mgr.14613) 538 : cluster [DBG] pgmap v622: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.781351+0000 osd.3 (osd.3) 255 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:06.783299+0000 osd.3 (osd.3) 256 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:52:08.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.268999+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:52:08.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:08 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.272666+0000 osd.7 (osd.7) 292 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:52:08.452 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:08.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:08.585 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:08.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:08.722 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:08.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:08.870 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:08.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:08.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:09.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:09.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.283340+0000 osd.6 (osd.6) 239 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.284897+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.372207+0000 osd.5 (osd.5) 311 : cluster [DBG] 3.c scrub starts 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.375215+0000 osd.5 (osd.5) 312 : cluster [DBG] 3.c scrub ok 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.920366+0000 osd.2 (osd.2) 219 : cluster [DBG] 2.c scrub starts 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:07.923021+0000 osd.2 (osd.2) 220 : cluster [DBG] 2.c scrub ok 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.362115+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub starts 2024-09-13T22:52:09.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:09 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.363237+0000 osd.1 (osd.1) 269 : cluster [DBG] 2.d scrub ok 2024-09-13T22:52:09.134 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:09.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:09.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:09.300 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:09.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:09.396 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:09.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.283340+0000 osd.6 (osd.6) 239 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:52:09.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.284897+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:52:09.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.372207+0000 osd.5 (osd.5) 311 : cluster [DBG] 3.c scrub starts 2024-09-13T22:52:09.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.375215+0000 osd.5 (osd.5) 312 : cluster [DBG] 3.c scrub ok 2024-09-13T22:52:09.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.920366+0000 osd.2 (osd.2) 219 : cluster [DBG] 2.c scrub starts 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:07.923021+0000 osd.2 (osd.2) 220 : cluster [DBG] 2.c scrub ok 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.362115+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub starts 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.363237+0000 osd.1 (osd.1) 269 : cluster [DBG] 2.d scrub ok 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.283340+0000 osd.6 (osd.6) 239 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.284897+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.372207+0000 osd.5 (osd.5) 311 : cluster [DBG] 3.c scrub starts 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.375215+0000 osd.5 (osd.5) 312 : cluster [DBG] 3.c scrub ok 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.920366+0000 osd.2 (osd.2) 219 : cluster [DBG] 2.c scrub starts 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:07.923021+0000 osd.2 (osd.2) 220 : cluster [DBG] 2.c scrub ok 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.362115+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub starts 2024-09-13T22:52:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.363237+0000 osd.1 (osd.1) 269 : cluster [DBG] 2.d scrub ok 2024-09-13T22:52:09.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:09.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:09.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:09.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:09.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:09.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:09.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:09.900 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:09.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:10.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:10.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:10.073 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:52:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:52:10.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:10.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.406451+0000 osd.5 (osd.5) 313 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.410339+0000 osd.5 (osd.5) 314 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.714863+0000 osd.3 (osd.3) 257 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.721125+0000 osd.3 (osd.3) 258 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:08.754657+0000 mgr.y (mgr.14613) 539 : cluster [DBG] pgmap v623: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:09.290411+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:09.294719+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:09.313592+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:52:10.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:10 smithi192 bash[24608]: cluster 2024-09-13T22:52:09.315100+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:52:10.159 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:10.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:10.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.406451+0000 osd.5 (osd.5) 313 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.410339+0000 osd.5 (osd.5) 314 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.714863+0000 osd.3 (osd.3) 257 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.721125+0000 osd.3 (osd.3) 258 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:08.754657+0000 mgr.y (mgr.14613) 539 : cluster [DBG] pgmap v623: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:09.290411+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:52:10.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:09.294719+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:09.313592+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[29889]: cluster 2024-09-13T22:52:09.315100+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.406451+0000 osd.5 (osd.5) 313 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.410339+0000 osd.5 (osd.5) 314 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.714863+0000 osd.3 (osd.3) 257 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.721125+0000 osd.3 (osd.3) 258 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:08.754657+0000 mgr.y (mgr.14613) 539 : cluster [DBG] pgmap v623: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:09.290411+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:09.294719+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:09.313592+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:52:10.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:10 smithi178 bash[22840]: cluster 2024-09-13T22:52:09.315100+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:52:10.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:10.414 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:10.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:10.548 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:10.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:10.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:10.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:10.813 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:10.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:10.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:10.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:11.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:11.076 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:11.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:11.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[29889]: cluster 2024-09-13T22:52:09.438737+0000 osd.5 (osd.5) 315 : cluster [DBG] 9.f scrub starts 2024-09-13T22:52:11.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[29889]: cluster 2024-09-13T22:52:09.829770+0000 osd.5 (osd.5) 316 : cluster [DBG] 9.f scrub ok 2024-09-13T22:52:11.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[29889]: cluster 2024-09-13T22:52:10.257402+0000 osd.4 (osd.4) 193 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:52:11.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[29889]: cluster 2024-09-13T22:52:10.276893+0000 osd.4 (osd.4) 194 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:52:11.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[22840]: cluster 2024-09-13T22:52:09.438737+0000 osd.5 (osd.5) 315 : cluster [DBG] 9.f scrub starts 2024-09-13T22:52:11.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[22840]: cluster 2024-09-13T22:52:09.829770+0000 osd.5 (osd.5) 316 : cluster [DBG] 9.f scrub ok 2024-09-13T22:52:11.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[22840]: cluster 2024-09-13T22:52:10.257402+0000 osd.4 (osd.4) 193 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:52:11.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:11 smithi178 bash[22840]: cluster 2024-09-13T22:52:10.276893+0000 osd.4 (osd.4) 194 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:52:11.211 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:11.317 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:11.337 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:11.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:11.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:11 smithi192 bash[24608]: cluster 2024-09-13T22:52:09.438737+0000 osd.5 (osd.5) 315 : cluster [DBG] 9.f scrub starts 2024-09-13T22:52:11.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:11 smithi192 bash[24608]: cluster 2024-09-13T22:52:09.829770+0000 osd.5 (osd.5) 316 : cluster [DBG] 9.f scrub ok 2024-09-13T22:52:11.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:11 smithi192 bash[24608]: cluster 2024-09-13T22:52:10.257402+0000 osd.4 (osd.4) 193 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:52:11.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:11 smithi192 bash[24608]: cluster 2024-09-13T22:52:10.276893+0000 osd.4 (osd.4) 194 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:52:11.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:11.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:11.613 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:11.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:11.747 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:11.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:11.880 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:11.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:11.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:12.020 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:12.084 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:52:12.084 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:52:12.122 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:52:12.122 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:52:12.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:12.149 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:52:12.149 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:52:12.163 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:12.193 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:52:12.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:52:12.233 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:52:12.234 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:52:12.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:12.276 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:52:12.277 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:52:12.298 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:12.311 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:52:12.311 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:52:12.342 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:52:12.342 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:10.755661+0000 mgr.y (mgr.14613) 540 : cluster [DBG] pgmap v624: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:10.917312+0000 osd.2 (osd.2) 221 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:10.920256+0000 osd.2 (osd.2) 222 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:11.293451+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.e scrub starts 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:11.296818+0000 osd.4 (osd.4) 196 : cluster [DBG] 6.e scrub ok 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:11.322137+0000 osd.1 (osd.1) 272 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: cluster 2024-09-13T22:52:11.324630+0000 osd.1 (osd.1) 273 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:52:12.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[29889]: audit 2024-09-13T22:52:12.049879+0000 mon.a (mon.0) 1409 : audit [DBG] from='client.? v1:172.21.15.178:0/2093077042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:10.755661+0000 mgr.y (mgr.14613) 540 : cluster [DBG] pgmap v624: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:10.917312+0000 osd.2 (osd.2) 221 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:10.920256+0000 osd.2 (osd.2) 222 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:11.293451+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.e scrub starts 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:11.296818+0000 osd.4 (osd.4) 196 : cluster [DBG] 6.e scrub ok 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:11.322137+0000 osd.1 (osd.1) 272 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: cluster 2024-09-13T22:52:11.324630+0000 osd.1 (osd.1) 273 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:52:12.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:12 smithi178 bash[22840]: audit 2024-09-13T22:52:12.049879+0000 mon.a (mon.0) 1409 : audit [DBG] from='client.? v1:172.21.15.178:0/2093077042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:12.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:10.755661+0000 mgr.y (mgr.14613) 540 : cluster [DBG] pgmap v624: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:10.917312+0000 osd.2 (osd.2) 221 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:10.920256+0000 osd.2 (osd.2) 222 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:11.293451+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.e scrub starts 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:11.296818+0000 osd.4 (osd.4) 196 : cluster [DBG] 6.e scrub ok 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:11.322137+0000 osd.1 (osd.1) 272 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: cluster 2024-09-13T22:52:11.324630+0000 osd.1 (osd.1) 273 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:52:12.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:12 smithi192 bash[24608]: audit 2024-09-13T22:52:12.049879+0000 mon.a (mon.0) 1409 : audit [DBG] from='client.? v1:172.21.15.178:0/2093077042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:12.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:12.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:12.555 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:12.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:12.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:12.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:12.812 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:12.915 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:12.941 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:12.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:13.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:13.067 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:13.101 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:52:13.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.193 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:13.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:13.327 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: cluster 2024-09-13T22:52:11.697820+0000 osd.3 (osd.3) 259 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: cluster 2024-09-13T22:52:11.699993+0000 osd.3 (osd.3) 260 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.310784+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.313716+0000 osd.1 (osd.1) 275 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: audit 2024-09-13T22:52:12.420108+0000 mon.c (mon.2) 57 : audit [INF] from='client.? v1:172.21.15.178:0/831652978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: audit 2024-09-13T22:52:12.420793+0000 mon.a (mon.0) 1410 : audit [INF] from='client.25421 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.433133+0000 osd.7 (osd.7) 293 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:52:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:13 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.434778+0000 osd.7 (osd.7) 294 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: cluster 2024-09-13T22:52:11.697820+0000 osd.3 (osd.3) 259 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: cluster 2024-09-13T22:52:11.699993+0000 osd.3 (osd.3) 260 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.310784+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.313716+0000 osd.1 (osd.1) 275 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: audit 2024-09-13T22:52:12.420108+0000 mon.c (mon.2) 57 : audit [INF] from='client.? v1:172.21.15.178:0/831652978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: audit 2024-09-13T22:52:12.420793+0000 mon.a (mon.0) 1410 : audit [INF] from='client.25421 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.433133+0000 osd.7 (osd.7) 293 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:52:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.434778+0000 osd.7 (osd.7) 294 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: cluster 2024-09-13T22:52:11.697820+0000 osd.3 (osd.3) 259 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: cluster 2024-09-13T22:52:11.699993+0000 osd.3 (osd.3) 260 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.310784+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.313716+0000 osd.1 (osd.1) 275 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: audit 2024-09-13T22:52:12.420108+0000 mon.c (mon.2) 57 : audit [INF] from='client.? v1:172.21.15.178:0/831652978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: audit 2024-09-13T22:52:12.420793+0000 mon.a (mon.0) 1410 : audit [INF] from='client.25421 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.433133+0000 osd.7 (osd.7) 293 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:52:13.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:13 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.434778+0000 osd.7 (osd.7) 294 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:52:13.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:13.457 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:13.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.585 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:13.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.709 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:13.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:13.838 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:13.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.966 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:13.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:14.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:14.098 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:14.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:14.231 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.455773+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.457499+0000 osd.5 (osd.5) 318 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.744054+0000 osd.3 (osd.3) 261 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.745628+0000 osd.3 (osd.3) 262 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:12.756170+0000 mgr.y (mgr.14613) 541 : cluster [DBG] pgmap v625: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.009782+0000 osd.2 (osd.2) 223 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.013684+0000 osd.2 (osd.2) 224 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.087901+0000 mon.a (mon.0) 1411 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.087928+0000 mon.a (mon.0) 1412 : cluster [INF] Cluster is now healthy 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: audit 2024-09-13T22:52:13.095907+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25421 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:52:14.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.102368+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.455773+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.457499+0000 osd.5 (osd.5) 318 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.744054+0000 osd.3 (osd.3) 261 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.745628+0000 osd.3 (osd.3) 262 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:12.756170+0000 mgr.y (mgr.14613) 541 : cluster [DBG] pgmap v625: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.009782+0000 osd.2 (osd.2) 223 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.013684+0000 osd.2 (osd.2) 224 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.087901+0000 mon.a (mon.0) 1411 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.087928+0000 mon.a (mon.0) 1412 : cluster [INF] Cluster is now healthy 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: audit 2024-09-13T22:52:13.095907+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25421 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:52:14.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:14 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.102368+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-13T22:52:14.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:14.359 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.455773+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.457499+0000 osd.5 (osd.5) 318 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.744054+0000 osd.3 (osd.3) 261 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.745628+0000 osd.3 (osd.3) 262 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:12.756170+0000 mgr.y (mgr.14613) 541 : cluster [DBG] pgmap v625: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.009782+0000 osd.2 (osd.2) 223 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.013684+0000 osd.2 (osd.2) 224 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:52:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.087901+0000 mon.a (mon.0) 1411 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:52:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.087928+0000 mon.a (mon.0) 1412 : cluster [INF] Cluster is now healthy 2024-09-13T22:52:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: audit 2024-09-13T22:52:13.095907+0000 mon.a (mon.0) 1413 : audit [INF] from='client.25421 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:52:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:14 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.102368+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-13T22:52:14.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:14.492 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:14.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:14.627 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:14.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:14.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:14.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:14.877 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:14.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:14.997 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:14.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:15.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:15.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.995602+0000 osd.3 (osd.3) 263 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:13.998689+0000 osd.3 (osd.3) 264 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.275486+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.e scrub starts 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.277280+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.e scrub ok 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.323067+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.326658+0000 osd.1 (osd.1) 277 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.859439+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:52:15.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.867266+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:52:15.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:15.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.995602+0000 osd.3 (osd.3) 263 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:52:15.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:13.998689+0000 osd.3 (osd.3) 264 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:52:15.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.275486+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.e scrub starts 2024-09-13T22:52:15.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.277280+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.e scrub ok 2024-09-13T22:52:15.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.323067+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:52:15.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.326658+0000 osd.1 (osd.1) 277 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:52:15.248 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.859439+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:52:15.248 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.867266+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:52:15.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.995602+0000 osd.3 (osd.3) 263 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:52:15.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:13.998689+0000 osd.3 (osd.3) 264 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:52:15.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.275486+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.e scrub starts 2024-09-13T22:52:15.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.277280+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.e scrub ok 2024-09-13T22:52:15.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.323067+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:52:15.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.326658+0000 osd.1 (osd.1) 277 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:52:15.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.859439+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:52:15.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:15 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.867266+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:52:15.268 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:15.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:15.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:15.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:15.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:15.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:15.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:52:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:52:15.672 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:15.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:15.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:15.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:15.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:15.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:16.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:16.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:16.122 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T22:52:16.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:16.209 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:16.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.407565+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.413020+0000 osd.5 (osd.5) 320 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: cluster 2024-09-13T22:52:14.756791+0000 mgr.y (mgr.14613) 542 : cluster [DBG] pgmap v627: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: cluster 2024-09-13T22:52:15.481096+0000 osd.7 (osd.7) 295 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: cluster 2024-09-13T22:52:15.489990+0000 osd.7 (osd.7) 296 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: audit 2024-09-13T22:52:15.548590+0000 mgr.y (mgr.14613) 543 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:16.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:16 smithi192 bash[24608]: audit 2024-09-13T22:52:15.549368+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? v1:172.21.15.178:0/1553490769' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:52:16.338 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:16.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.407565+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:52:16.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.413020+0000 osd.5 (osd.5) 320 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:52:16.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: cluster 2024-09-13T22:52:14.756791+0000 mgr.y (mgr.14613) 542 : cluster [DBG] pgmap v627: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:52:16.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: cluster 2024-09-13T22:52:15.481096+0000 osd.7 (osd.7) 295 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: cluster 2024-09-13T22:52:15.489990+0000 osd.7 (osd.7) 296 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: audit 2024-09-13T22:52:15.548590+0000 mgr.y (mgr.14613) 543 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[29889]: audit 2024-09-13T22:52:15.549368+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? v1:172.21.15.178:0/1553490769' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.407565+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.413020+0000 osd.5 (osd.5) 320 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: cluster 2024-09-13T22:52:14.756791+0000 mgr.y (mgr.14613) 542 : cluster [DBG] pgmap v627: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: cluster 2024-09-13T22:52:15.481096+0000 osd.7 (osd.7) 295 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: cluster 2024-09-13T22:52:15.489990+0000 osd.7 (osd.7) 296 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:52:16.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: audit 2024-09-13T22:52:15.548590+0000 mgr.y (mgr.14613) 543 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:16.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:16 smithi178 bash[22840]: audit 2024-09-13T22:52:15.549368+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? v1:172.21.15.178:0/1553490769' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:52:16.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.468 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:16.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:16.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:16.605 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:16.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:16.740 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:16.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:16.862 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:16.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:16.985 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:16.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:17.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:17.125 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:17.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:17.260 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:17.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:17.382 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:17.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:17 smithi192 bash[24608]: cluster 2024-09-13T22:52:15.329703+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:52:17.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:17 smithi192 bash[24608]: cluster 2024-09-13T22:52:15.331466+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:52:17.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:17 smithi192 bash[24608]: cluster 2024-09-13T22:52:15.392276+0000 osd.5 (osd.5) 321 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:52:17.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:17 smithi192 bash[24608]: cluster 2024-09-13T22:52:15.395141+0000 osd.5 (osd.5) 322 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:52:17.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:17 smithi192 bash[24608]: audit 2024-09-13T22:52:16.118401+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? v1:172.21.15.178:0/1553490769' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:52:17.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:17 smithi192 bash[24608]: cluster 2024-09-13T22:52:16.121742+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-13T22:52:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[29889]: cluster 2024-09-13T22:52:15.329703+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:52:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[29889]: cluster 2024-09-13T22:52:15.331466+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:52:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[29889]: cluster 2024-09-13T22:52:15.392276+0000 osd.5 (osd.5) 321 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:52:17.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[29889]: cluster 2024-09-13T22:52:15.395141+0000 osd.5 (osd.5) 322 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[29889]: audit 2024-09-13T22:52:16.118401+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? v1:172.21.15.178:0/1553490769' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[29889]: cluster 2024-09-13T22:52:16.121742+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[22840]: cluster 2024-09-13T22:52:15.329703+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[22840]: cluster 2024-09-13T22:52:15.331466+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[22840]: cluster 2024-09-13T22:52:15.392276+0000 osd.5 (osd.5) 321 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[22840]: cluster 2024-09-13T22:52:15.395141+0000 osd.5 (osd.5) 322 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[22840]: audit 2024-09-13T22:52:16.118401+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? v1:172.21.15.178:0/1553490769' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:52:17.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:17 smithi178 bash[22840]: cluster 2024-09-13T22:52:16.121742+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-13T22:52:17.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:17.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:17.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:17.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:17.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:17.778 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:17.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:17.912 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:17.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:18.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:18.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:18.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:18.180 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:18.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:18.295 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:16.321790+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:16.343334+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:16.757132+0000 mgr.y (mgr.14613) 544 : cluster [DBG] pgmap v629: 168 pgs: 168 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.090769+0000 osd.3 (osd.3) 265 : cluster [DBG] 4.0 deep-scrub starts 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.113696+0000 osd.3 (osd.3) 266 : cluster [DBG] 4.0 deep-scrub ok 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.312581+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.313609+0000 osd.1 (osd.1) 279 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.562400+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.570850+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:52:18.296 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:16.321790+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:16.343334+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:16.757132+0000 mgr.y (mgr.14613) 544 : cluster [DBG] pgmap v629: 168 pgs: 168 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.090769+0000 osd.3 (osd.3) 265 : cluster [DBG] 4.0 deep-scrub starts 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.113696+0000 osd.3 (osd.3) 266 : cluster [DBG] 4.0 deep-scrub ok 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.312581+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.313609+0000 osd.1 (osd.1) 279 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.562400+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:52:18.297 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:18 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.570850+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:52:18.309 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:16.321790+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:52:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:16.343334+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:52:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:16.757132+0000 mgr.y (mgr.14613) 544 : cluster [DBG] pgmap v629: 168 pgs: 168 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:52:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.090769+0000 osd.3 (osd.3) 265 : cluster [DBG] 4.0 deep-scrub starts 2024-09-13T22:52:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.113696+0000 osd.3 (osd.3) 266 : cluster [DBG] 4.0 deep-scrub ok 2024-09-13T22:52:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.312581+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T22:52:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.313609+0000 osd.1 (osd.1) 279 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T22:52:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.562400+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:52:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:18 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.570850+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:52:18.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:18.450 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:18.474 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:52:18.474 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:52:18.474 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171494 9.d e309/251 9:b38bff18:::benchmark_data_smithi178_96665_object72332:head v 309'10567, mlcod=309'10564)", 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.469850+0000", 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.005220878, 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0052491489999999998, 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469850+0000", 2024-09-13T22:52:18.475 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469850+0000", 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469852+0000", 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1319999999999999e-06 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469859+0000", 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4739999999999999e-06 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469866+0000", 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3699999999999997e-06 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.477 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469871+0000", 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.685e-06 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472041+0000", 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0021701820000000001 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472060+0000", 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9054000000000001e-05 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.478 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171496 9.a e309/251 9:5c02d654:::benchmark_data_smithi178_96665_object72334:head v 309'10805, mlcod=309'10801)", 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.472982+0000", 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0020891970000000001, 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0021781159999999999, 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.479 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472982+0000", 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472982+0000", 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472983+0000", 2024-09-13T22:52:18.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.339e-06 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472990+0000", 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3509999999999998e-06 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473001+0000", 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1255e-05 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.481 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473008+0000", 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4080000000000001e-06 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473359+0000", 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00035065299999999998 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473377+0000", 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7852000000000001e-05 2024-09-13T22:52:18.482 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:171498 9.4 9:20a003a9:::benchmark_data_smithi178_96665_object72336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.473025+0000", 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.002046029, 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0021924240000000001, 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:52:18.483 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 171498 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473025+0000", 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473025+0000", 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473027+0000", 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.125e-06 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473034+0000", 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.251e-06 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473048+0000", 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4711e-05 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.485 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473056+0000", 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7419999999999997e-06 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473524+0000", 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00046823999999999998 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.473653+0000", 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000128455 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.486 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 3,4", 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474027+0000", 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00037431399999999999 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171497 9.12 e309/287)", 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.474804+0000", 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.00026702300000000002, 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00047653799999999998, 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "queued for pg", 2024-09-13T22:52:18.487 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474804+0000", 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474804+0000", 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474806+0000", 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2869999999999999e-06 2024-09-13T22:52:18.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474814+0000", 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5709999999999998e-06 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474826+0000", 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2296999999999999e-05 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.474833+0000", 2024-09-13T22:52:18.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4420000000000003e-06 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171482 9.d e309/251 9:bc96ac8b:::benchmark_data_smithi178_96665_object72320:head v 309'10566, mlcod=309'10563)", 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.462166+0000", 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.012905455, 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013161093, 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:52:18.490 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462166+0000", 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462166+0000", 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.491 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462167+0000", 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5820000000000001e-06 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462177+0000", 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.2140000000000002e-06 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462186+0000", 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7540000000000001e-06 2024-09-13T22:52:18.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462193+0000", 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9500000000000004e-06 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462838+0000", 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000645168 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.462855+0000", 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6663e-05 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171487 9.a e309/251 9:5d10b622:::benchmark_data_smithi178_96665_object72325:head v 309'10803, mlcod=309'10801)", 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.464692+0000", 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.010378859000000001, 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010687415, 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.494 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464692+0000", 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464692+0000", 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464693+0000", 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.0800000000000003e-07 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464698+0000", 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.882e-06 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464706+0000", 2024-09-13T22:52:18.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8509999999999999e-06 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464713+0000", 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4450000000000001e-06 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464761+0000", 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8911e-05 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.464779+0000", 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7348999999999999e-05 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171488 9.a e309/251 9:5dccb0ef:::benchmark_data_smithi178_96665_object72326:head v 309'10804, mlcod=309'10801)", 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.467267+0000", 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.00780417, 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0081637350000000001, 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.498 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.467267+0000", 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.467267+0000", 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.467268+0000", 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.367e-06 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.467275+0000", 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.5230000000000001e-06 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.467289+0000", 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3570000000000001e-05 2024-09-13T22:52:18.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.467296+0000", 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.002e-06 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.470133+0000", 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0028374060000000002 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.470155+0000", 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1722999999999999e-05 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:171492 9.14 9:2e9f5fde:::benchmark_data_smithi178_96665_object72330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.468524+0000", 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 0.0065471940000000001, 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0069611619999999999, 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:52:18.502 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 171492 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.468524+0000", 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:52:18.503 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.468524+0000", 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.468526+0000", 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.561e-06 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.468530+0000", 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.5290000000000002e-06 2024-09-13T22:52:18.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.468543+0000", 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2969000000000001e-05 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.468631+0000", 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.8096000000000005e-05 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:52:18.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469264+0000", 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00063310799999999996 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469378+0000", 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000114071 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 3,4", 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.469780+0000", 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00040184299999999998 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:52:18.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.472081+0000", 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0023006620000000002 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout: "num_ops": 8 2024-09-13T22:52:18.507 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:52:18.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:18.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:18.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:18.721 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:18.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:52:18.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:52:18.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:18.854 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:18.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:18.987 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:18.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:19.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:19.113 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:19.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:19.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:19.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:19.366 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.551616+0000 osd.5 (osd.5) 323 : cluster [DBG] 9.a scrub starts 2024-09-13T22:52:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.979978+0000 osd.3 (osd.3) 267 : cluster [DBG] 1.4 deep-scrub starts 2024-09-13T22:52:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: cluster 2024-09-13T22:52:17.983221+0000 osd.3 (osd.3) 268 : cluster [DBG] 1.4 deep-scrub ok 2024-09-13T22:52:19.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: cluster 2024-09-13T22:52:18.121304+0000 osd.0 (osd.0) 287 : cluster [DBG] 9.b scrub starts 2024-09-13T22:52:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: audit 2024-09-13T22:52:18.796745+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: audit 2024-09-13T22:52:18.801834+0000 mon.b (mon.1) 212 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:19 smithi192 bash[24608]: cluster 2024-09-13T22:52:18.836528+0000 osd.0 (osd.0) 288 : cluster [DBG] 9.b scrub ok 2024-09-13T22:52:19.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:19.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:19.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:19.623 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.551616+0000 osd.5 (osd.5) 323 : cluster [DBG] 9.a scrub starts 2024-09-13T22:52:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.979978+0000 osd.3 (osd.3) 267 : cluster [DBG] 1.4 deep-scrub starts 2024-09-13T22:52:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: cluster 2024-09-13T22:52:17.983221+0000 osd.3 (osd.3) 268 : cluster [DBG] 1.4 deep-scrub ok 2024-09-13T22:52:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: cluster 2024-09-13T22:52:18.121304+0000 osd.0 (osd.0) 287 : cluster [DBG] 9.b scrub starts 2024-09-13T22:52:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: audit 2024-09-13T22:52:18.796745+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: audit 2024-09-13T22:52:18.801834+0000 mon.b (mon.1) 212 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[29889]: cluster 2024-09-13T22:52:18.836528+0000 osd.0 (osd.0) 288 : cluster [DBG] 9.b scrub ok 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.551616+0000 osd.5 (osd.5) 323 : cluster [DBG] 9.a scrub starts 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.979978+0000 osd.3 (osd.3) 267 : cluster [DBG] 1.4 deep-scrub starts 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: cluster 2024-09-13T22:52:17.983221+0000 osd.3 (osd.3) 268 : cluster [DBG] 1.4 deep-scrub ok 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: cluster 2024-09-13T22:52:18.121304+0000 osd.0 (osd.0) 287 : cluster [DBG] 9.b scrub starts 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: audit 2024-09-13T22:52:18.796745+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: audit 2024-09-13T22:52:18.801834+0000 mon.b (mon.1) 212 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[22840]: cluster 2024-09-13T22:52:18.836528+0000 osd.0 (osd.0) 288 : cluster [DBG] 9.b scrub ok 2024-09-13T22:52:19.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:19.752 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:19.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:19.879 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:19.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:19.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T22:52:19.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:20.011 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:52:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:52:20.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:20.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:20.154 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:20.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:20.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[29889]: cluster 2024-09-13T22:52:18.300139+0000 osd.5 (osd.5) 324 : cluster [DBG] 9.a scrub ok 2024-09-13T22:52:20.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[29889]: cluster 2024-09-13T22:52:18.757650+0000 mgr.y (mgr.14613) 545 : cluster [DBG] pgmap v630: 168 pgs: 168 active+clean; 1.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 595 op/s 2024-09-13T22:52:20.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[29889]: cluster 2024-09-13T22:52:19.419415+0000 osd.0 (osd.0) 289 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:52:20.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[29889]: cluster 2024-09-13T22:52:19.424507+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:52:20.269 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[22840]: cluster 2024-09-13T22:52:18.300139+0000 osd.5 (osd.5) 324 : cluster [DBG] 9.a scrub ok 2024-09-13T22:52:20.269 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[22840]: cluster 2024-09-13T22:52:18.757650+0000 mgr.y (mgr.14613) 545 : cluster [DBG] pgmap v630: 168 pgs: 168 active+clean; 1.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 595 op/s 2024-09-13T22:52:20.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[22840]: cluster 2024-09-13T22:52:19.419415+0000 osd.0 (osd.0) 289 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:52:20.270 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:20 smithi178 bash[22840]: cluster 2024-09-13T22:52:19.424507+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:52:20.289 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:20.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:20.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:20 smithi192 bash[24608]: cluster 2024-09-13T22:52:18.300139+0000 osd.5 (osd.5) 324 : cluster [DBG] 9.a scrub ok 2024-09-13T22:52:20.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:20 smithi192 bash[24608]: cluster 2024-09-13T22:52:18.757650+0000 mgr.y (mgr.14613) 545 : cluster [DBG] pgmap v630: 168 pgs: 168 active+clean; 1.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 595 op/s 2024-09-13T22:52:20.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:20 smithi192 bash[24608]: cluster 2024-09-13T22:52:19.419415+0000 osd.0 (osd.0) 289 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:52:20.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:20 smithi192 bash[24608]: cluster 2024-09-13T22:52:19.424507+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:52:20.424 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:20.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:20.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:20.664 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:20.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:20.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:20.818 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:20.927 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:20.949 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:20.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:21.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:21.079 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:21.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:21.222 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:21.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:21.336 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:21 smithi192 bash[24608]: cluster 2024-09-13T22:52:20.016982+0000 osd.3 (osd.3) 269 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:52:21.336 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:21 smithi192 bash[24608]: cluster 2024-09-13T22:52:20.021312+0000 osd.3 (osd.3) 270 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:52:21.336 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:21 smithi192 bash[24608]: cluster 2024-09-13T22:52:20.385561+0000 osd.1 (osd.1) 280 : cluster [DBG] 1.3 deep-scrub starts 2024-09-13T22:52:21.336 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:21 smithi192 bash[24608]: cluster 2024-09-13T22:52:20.386848+0000 osd.1 (osd.1) 281 : cluster [DBG] 1.3 deep-scrub ok 2024-09-13T22:52:21.359 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:21.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[29889]: cluster 2024-09-13T22:52:20.016982+0000 osd.3 (osd.3) 269 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:52:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[29889]: cluster 2024-09-13T22:52:20.021312+0000 osd.3 (osd.3) 270 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:52:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[29889]: cluster 2024-09-13T22:52:20.385561+0000 osd.1 (osd.1) 280 : cluster [DBG] 1.3 deep-scrub starts 2024-09-13T22:52:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[29889]: cluster 2024-09-13T22:52:20.386848+0000 osd.1 (osd.1) 281 : cluster [DBG] 1.3 deep-scrub ok 2024-09-13T22:52:21.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[22840]: cluster 2024-09-13T22:52:20.016982+0000 osd.3 (osd.3) 269 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:52:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[22840]: cluster 2024-09-13T22:52:20.021312+0000 osd.3 (osd.3) 270 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:52:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[22840]: cluster 2024-09-13T22:52:20.385561+0000 osd.1 (osd.1) 280 : cluster [DBG] 1.3 deep-scrub starts 2024-09-13T22:52:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:21 smithi178 bash[22840]: cluster 2024-09-13T22:52:20.386848+0000 osd.1 (osd.1) 281 : cluster [DBG] 1.3 deep-scrub ok 2024-09-13T22:52:21.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:21.496 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:21.599 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:21.625 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:21.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:21.752 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:21.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:21.877 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:21.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:21.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:22.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:22.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:22.150 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:22.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:22.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[22840]: cluster 2024-09-13T22:52:20.758537+0000 mgr.y (mgr.14613) 546 : cluster [DBG] pgmap v631: 168 pgs: 168 active+clean; 1.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.141141+0000 osd.0 (osd.0) 291 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.150991+0000 osd.0 (osd.0) 292 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.227263+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 deep-scrub starts 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.230156+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 deep-scrub ok 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[29889]: cluster 2024-09-13T22:52:20.758537+0000 mgr.y (mgr.14613) 546 : cluster [DBG] pgmap v631: 168 pgs: 168 active+clean; 1.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.141141+0000 osd.0 (osd.0) 291 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.150991+0000 osd.0 (osd.0) 292 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:52:22.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.227263+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 deep-scrub starts 2024-09-13T22:52:22.267 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:22 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.230156+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 deep-scrub ok 2024-09-13T22:52:22.283 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:22.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:22.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:22 smithi192 bash[24608]: cluster 2024-09-13T22:52:20.758537+0000 mgr.y (mgr.14613) 546 : cluster [DBG] pgmap v631: 168 pgs: 168 active+clean; 1.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T22:52:22.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:22 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.141141+0000 osd.0 (osd.0) 291 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:52:22.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:22 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.150991+0000 osd.0 (osd.0) 292 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:52:22.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:22 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.227263+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 deep-scrub starts 2024-09-13T22:52:22.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:22 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.230156+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 deep-scrub ok 2024-09-13T22:52:22.421 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:22.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:22.555 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:22.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:22.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:22.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:22.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:22.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:22.948 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:22.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:23.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:23.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:23.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:23.203 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:23.246 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:52:23.246 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:52:23.246 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T22:52:23.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-13T22:52:23.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:23 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.294998+0000 osd.6 (osd.6) 245 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T22:52:23.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:23 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.295879+0000 osd.6 (osd.6) 246 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T22:52:23.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:23 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.299833+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:52:23.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:23 smithi192 bash[24608]: cluster 2024-09-13T22:52:21.302709+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:52:23.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:23 smithi192 bash[24608]: cluster 2024-09-13T22:52:22.270722+0000 osd.4 (osd.4) 201 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:52:23.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:23 smithi192 bash[24608]: cluster 2024-09-13T22:52:22.271822+0000 osd.4 (osd.4) 202 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:52:23.327 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.294998+0000 osd.6 (osd.6) 245 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T22:52:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.295879+0000 osd.6 (osd.6) 246 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T22:52:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.299833+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:52:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[22840]: cluster 2024-09-13T22:52:21.302709+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:52:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[22840]: cluster 2024-09-13T22:52:22.270722+0000 osd.4 (osd.4) 201 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[22840]: cluster 2024-09-13T22:52:22.271822+0000 osd.4 (osd.4) 202 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.294998+0000 osd.6 (osd.6) 245 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.295879+0000 osd.6 (osd.6) 246 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.299833+0000 osd.5 (osd.5) 325 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[29889]: cluster 2024-09-13T22:52:21.302709+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[29889]: cluster 2024-09-13T22:52:22.270722+0000 osd.4 (osd.4) 201 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:52:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:23 smithi178 bash[29889]: cluster 2024-09-13T22:52:22.271822+0000 osd.4 (osd.4) 202 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:52:23.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:23.473 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:23.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:23.608 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:23.609 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:23.609 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:23.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:23.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:23.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:23.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:23.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:23.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:24.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:24.136 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:24.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:24.274 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:24.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:24.387 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[29889]: cluster 2024-09-13T22:52:22.759561+0000 mgr.y (mgr.14613) 547 : cluster [DBG] pgmap v632: 168 pgs: 168 active+clean; 1.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 989 op/s 2024-09-13T22:52:24.387 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[29889]: cluster 2024-09-13T22:52:23.028871+0000 osd.3 (osd.3) 271 : cluster [DBG] 3.b scrub starts 2024-09-13T22:52:24.387 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[29889]: cluster 2024-09-13T22:52:23.033106+0000 osd.3 (osd.3) 272 : cluster [DBG] 3.b scrub ok 2024-09-13T22:52:24.387 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[29889]: cluster 2024-09-13T22:52:23.403980+0000 osd.7 (osd.7) 299 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:52:24.387 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[29889]: cluster 2024-09-13T22:52:23.408632+0000 osd.7 (osd.7) 300 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:52:24.388 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[22840]: cluster 2024-09-13T22:52:22.759561+0000 mgr.y (mgr.14613) 547 : cluster [DBG] pgmap v632: 168 pgs: 168 active+clean; 1.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 989 op/s 2024-09-13T22:52:24.388 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[22840]: cluster 2024-09-13T22:52:23.028871+0000 osd.3 (osd.3) 271 : cluster [DBG] 3.b scrub starts 2024-09-13T22:52:24.388 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[22840]: cluster 2024-09-13T22:52:23.033106+0000 osd.3 (osd.3) 272 : cluster [DBG] 3.b scrub ok 2024-09-13T22:52:24.388 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[22840]: cluster 2024-09-13T22:52:23.403980+0000 osd.7 (osd.7) 299 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:52:24.388 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:24 smithi178 bash[22840]: cluster 2024-09-13T22:52:23.408632+0000 osd.7 (osd.7) 300 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:52:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:24 smithi192 bash[24608]: cluster 2024-09-13T22:52:22.759561+0000 mgr.y (mgr.14613) 547 : cluster [DBG] pgmap v632: 168 pgs: 168 active+clean; 1.6 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 989 op/s 2024-09-13T22:52:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:24 smithi192 bash[24608]: cluster 2024-09-13T22:52:23.028871+0000 osd.3 (osd.3) 271 : cluster [DBG] 3.b scrub starts 2024-09-13T22:52:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:24 smithi192 bash[24608]: cluster 2024-09-13T22:52:23.033106+0000 osd.3 (osd.3) 272 : cluster [DBG] 3.b scrub ok 2024-09-13T22:52:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:24 smithi192 bash[24608]: cluster 2024-09-13T22:52:23.403980+0000 osd.7 (osd.7) 299 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:52:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:24 smithi192 bash[24608]: cluster 2024-09-13T22:52:23.408632+0000 osd.7 (osd.7) 300 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:52:24.415 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:24.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:24.550 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:24.628 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:24.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:24.680 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:24.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:24.806 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:24.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:24.935 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:24.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:25.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:25.074 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:25.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:25.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:25.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:25 smithi192 bash[24608]: cluster 2024-09-13T22:52:24.258930+0000 osd.6 (osd.6) 247 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:52:25.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:25 smithi192 bash[24608]: cluster 2024-09-13T22:52:24.457950+0000 osd.7 (osd.7) 301 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:52:25.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:25 smithi192 bash[24608]: cluster 2024-09-13T22:52:24.462161+0000 osd.7 (osd.7) 302 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:52:25.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:25 smithi192 bash[24608]: cluster 2024-09-13T22:52:25.059128+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:52:25.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:25 smithi192 bash[24608]: cluster 2024-09-13T22:52:25.061694+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:52:25.341 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:25.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[29889]: cluster 2024-09-13T22:52:24.258930+0000 osd.6 (osd.6) 247 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[29889]: cluster 2024-09-13T22:52:24.457950+0000 osd.7 (osd.7) 301 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[29889]: cluster 2024-09-13T22:52:24.462161+0000 osd.7 (osd.7) 302 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[29889]: cluster 2024-09-13T22:52:25.059128+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[29889]: cluster 2024-09-13T22:52:25.061694+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[22840]: cluster 2024-09-13T22:52:24.258930+0000 osd.6 (osd.6) 247 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:52:25.471 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[22840]: cluster 2024-09-13T22:52:24.457950+0000 osd.7 (osd.7) 301 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:52:25.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[22840]: cluster 2024-09-13T22:52:24.462161+0000 osd.7 (osd.7) 302 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:52:25.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[22840]: cluster 2024-09-13T22:52:25.059128+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:52:25.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:25 smithi178 bash[22840]: cluster 2024-09-13T22:52:25.061694+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:52:25.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:25.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:25.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:52:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:52:25.605 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:25.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:25.746 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:25.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:25.882 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:25.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:25.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:26.012 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:26.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:26.148 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:26.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:26.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:26.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:26.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:26 smithi192 bash[24608]: cluster 2024-09-13T22:52:24.284821+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:52:26.396 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:26 smithi192 bash[24608]: cluster 2024-09-13T22:52:24.760670+0000 mgr.y (mgr.14613) 548 : cluster [DBG] pgmap v633: 168 pgs: 168 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T22:52:26.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:26 smithi192 bash[24608]: audit 2024-09-13T22:52:25.557742+0000 mgr.y (mgr.14613) 549 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:26.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:26 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.034986+0000 osd.0 (osd.0) 295 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:52:26.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:26 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.036103+0000 osd.0 (osd.0) 296 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:52:26.410 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:26.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[29889]: cluster 2024-09-13T22:52:24.284821+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:52:26.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[29889]: cluster 2024-09-13T22:52:24.760670+0000 mgr.y (mgr.14613) 548 : cluster [DBG] pgmap v633: 168 pgs: 168 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T22:52:26.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[29889]: audit 2024-09-13T22:52:25.557742+0000 mgr.y (mgr.14613) 549 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:26.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.034986+0000 osd.0 (osd.0) 295 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:52:26.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.036103+0000 osd.0 (osd.0) 296 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:52:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[22840]: cluster 2024-09-13T22:52:24.284821+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:52:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[22840]: cluster 2024-09-13T22:52:24.760670+0000 mgr.y (mgr.14613) 548 : cluster [DBG] pgmap v633: 168 pgs: 168 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T22:52:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[22840]: audit 2024-09-13T22:52:25.557742+0000 mgr.y (mgr.14613) 549 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.034986+0000 osd.0 (osd.0) 295 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:52:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:26 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.036103+0000 osd.0 (osd.0) 296 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:52:26.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:26.546 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:26.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:26.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:26.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:26.800 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:26.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:26.943 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:26.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:27.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:27.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:27.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:27.214 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:27.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:27.339 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:27.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:27.454 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:27 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.515759+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.f scrub starts 2024-09-13T22:52:27.454 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:27 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.526146+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.f scrub ok 2024-09-13T22:52:27.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:27 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.515759+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.f scrub starts 2024-09-13T22:52:27.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:27 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.526146+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.f scrub ok 2024-09-13T22:52:27.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:27.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:27.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:27.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:27 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.515759+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.f scrub starts 2024-09-13T22:52:27.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:27 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.526146+0000 osd.7 (osd.7) 304 : cluster [DBG] 5.f scrub ok 2024-09-13T22:52:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:27.686 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:27.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:27.737 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:27.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:27.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:27.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:27.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:28.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:28.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:28.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:28.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:28.257 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:28 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.436615+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:52:28.257 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:28 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.439465+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:52:28.257 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:28 smithi192 bash[24608]: cluster 2024-09-13T22:52:26.761755+0000 mgr.y (mgr.14613) 550 : cluster [DBG] pgmap v634: 168 pgs: 168 active+clean; 1.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T22:52:28.258 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:28 smithi192 bash[24608]: cluster 2024-09-13T22:52:27.560032+0000 osd.7 (osd.7) 305 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:52:28.258 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:28 smithi192 bash[24608]: cluster 2024-09-13T22:52:27.562602+0000 osd.7 (osd.7) 306 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:52:28.271 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:28.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:28.401 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:28.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:28.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:28.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:28.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.436615+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:52:28.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.439465+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:52:28.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[29889]: cluster 2024-09-13T22:52:26.761755+0000 mgr.y (mgr.14613) 550 : cluster [DBG] pgmap v634: 168 pgs: 168 active+clean; 1.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[29889]: cluster 2024-09-13T22:52:27.560032+0000 osd.7 (osd.7) 305 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[29889]: cluster 2024-09-13T22:52:27.562602+0000 osd.7 (osd.7) 306 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.436615+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.439465+0000 osd.1 (osd.1) 283 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[22840]: cluster 2024-09-13T22:52:26.761755+0000 mgr.y (mgr.14613) 550 : cluster [DBG] pgmap v634: 168 pgs: 168 active+clean; 1.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[22840]: cluster 2024-09-13T22:52:27.560032+0000 osd.7 (osd.7) 305 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:52:28.640 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:28 smithi178 bash[22840]: cluster 2024-09-13T22:52:27.562602+0000 osd.7 (osd.7) 306 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:52:28.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:28.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:28.788 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:28.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:28.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:29.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:29.054 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:29.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:29.181 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:29.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.314 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:29.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:29.415 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:29.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:29.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:29.566 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:29.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:29 smithi192 bash[24608]: cluster 2024-09-13T22:52:27.274945+0000 osd.5 (osd.5) 327 : cluster [DBG] 6.a deep-scrub starts 2024-09-13T22:52:29.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:29 smithi192 bash[24608]: cluster 2024-09-13T22:52:27.309245+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.a deep-scrub ok 2024-09-13T22:52:29.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:29 smithi178 bash[22840]: cluster 2024-09-13T22:52:27.274945+0000 osd.5 (osd.5) 327 : cluster [DBG] 6.a deep-scrub starts 2024-09-13T22:52:29.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:29 smithi178 bash[22840]: cluster 2024-09-13T22:52:27.309245+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.a deep-scrub ok 2024-09-13T22:52:29.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:29 smithi178 bash[29889]: cluster 2024-09-13T22:52:27.274945+0000 osd.5 (osd.5) 327 : cluster [DBG] 6.a deep-scrub starts 2024-09-13T22:52:29.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:29 smithi178 bash[29889]: cluster 2024-09-13T22:52:27.309245+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.a deep-scrub ok 2024-09-13T22:52:29.716 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:29.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:29.855 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:29.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:29.985 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:29.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:30.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:30.098 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:52:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:52:30.118 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:30.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:30.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:30 smithi192 bash[24608]: cluster 2024-09-13T22:52:28.612124+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:52:30.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:30 smithi192 bash[24608]: cluster 2024-09-13T22:52:28.616527+0000 osd.1 (osd.1) 285 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:52:30.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:30 smithi192 bash[24608]: cluster 2024-09-13T22:52:28.762497+0000 mgr.y (mgr.14613) 551 : cluster [DBG] pgmap v635: 168 pgs: 168 active+clean; 1.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 949 op/s 2024-09-13T22:52:30.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:30 smithi192 bash[24608]: cluster 2024-09-13T22:52:29.011082+0000 osd.3 (osd.3) 273 : cluster [DBG] 6.b scrub starts 2024-09-13T22:52:30.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:30 smithi192 bash[24608]: cluster 2024-09-13T22:52:29.017780+0000 osd.3 (osd.3) 274 : cluster [DBG] 6.b scrub ok 2024-09-13T22:52:30.248 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:30.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[22840]: cluster 2024-09-13T22:52:28.612124+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[22840]: cluster 2024-09-13T22:52:28.616527+0000 osd.1 (osd.1) 285 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[22840]: cluster 2024-09-13T22:52:28.762497+0000 mgr.y (mgr.14613) 551 : cluster [DBG] pgmap v635: 168 pgs: 168 active+clean; 1.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 949 op/s 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[22840]: cluster 2024-09-13T22:52:29.011082+0000 osd.3 (osd.3) 273 : cluster [DBG] 6.b scrub starts 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[22840]: cluster 2024-09-13T22:52:29.017780+0000 osd.3 (osd.3) 274 : cluster [DBG] 6.b scrub ok 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[29889]: cluster 2024-09-13T22:52:28.612124+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:52:30.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[29889]: cluster 2024-09-13T22:52:28.616527+0000 osd.1 (osd.1) 285 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:52:30.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[29889]: cluster 2024-09-13T22:52:28.762497+0000 mgr.y (mgr.14613) 551 : cluster [DBG] pgmap v635: 168 pgs: 168 active+clean; 1.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 949 op/s 2024-09-13T22:52:30.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[29889]: cluster 2024-09-13T22:52:29.011082+0000 osd.3 (osd.3) 273 : cluster [DBG] 6.b scrub starts 2024-09-13T22:52:30.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:30 smithi178 bash[29889]: cluster 2024-09-13T22:52:29.017780+0000 osd.3 (osd.3) 274 : cluster [DBG] 6.b scrub ok 2024-09-13T22:52:30.392 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:30.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:30.522 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:30.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:30.657 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:30.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:30.789 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:30.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:30.922 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:30.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:31.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:31.067 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:31.068 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:52:31.069 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:52:31.069 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:52:31.069 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:52:31.069 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:52:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:31.140 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:52:31.140 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:52:31.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:31.182 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:52:31.182 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:52:31.206 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:31.223 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:52:31.223 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:52:31.260 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:52:31.260 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:52:31.297 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:52:31.297 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:52:31.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.107837+0000 osd.0 (osd.0) 297 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.112898+0000 osd.0 (osd.0) 298 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.406322+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.406382+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.558155+0000 osd.3 (osd.3) 275 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.558297+0000 osd.3 (osd.3) 276 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[22840]: audit 2024-09-13T22:52:31.064499+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.178:0/365361169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.107837+0000 osd.0 (osd.0) 297 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.112898+0000 osd.0 (osd.0) 298 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:52:31.312 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.406322+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:31.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.406382+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:31.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.558155+0000 osd.3 (osd.3) 275 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:31.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.558297+0000 osd.3 (osd.3) 276 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:31.313 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:31 smithi178 bash[29889]: audit 2024-09-13T22:52:31.064499+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.178:0/365361169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:31.335 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:52:31.335 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:52:31.351 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:31.377 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:52:31.377 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:52:31.421 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:52:31.421 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:52:31.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:31.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.107837+0000 osd.0 (osd.0) 297 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:52:31.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.112898+0000 osd.0 (osd.0) 298 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:52:31.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.406322+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:31.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.406382+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:31.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.558155+0000 osd.3 (osd.3) 275 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:31.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.558297+0000 osd.3 (osd.3) 276 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:31.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:31 smithi192 bash[24608]: audit 2024-09-13T22:52:31.064499+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.178:0/365361169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:31.487 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:moved 0 / 504 (0%) 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:avg 63 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:stddev 8.7892 -> 8.7892 (expected baseline 7.42462) 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:min osd.2 with 49 -> 49 pgs (0.777778 -> 0.777778 * mean) 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:max osd.7 with 80 -> 80 pgs (1.26984 -> 1.26984 * mean) 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:oload 1000 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:max_change 1 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:max_change_osds 4 2024-09-13T22:52:31.527 INFO:teuthology.orchestra.run.smithi178.stdout:average_utilization 0.0089 2024-09-13T22:52:31.528 INFO:teuthology.orchestra.run.smithi178.stdout:overload_utilization 0.0888 2024-09-13T22:52:31.528 INFO:teuthology.orchestra.run.smithi178.stderr:no change 2024-09-13T22:52:31.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:31.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:31.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:31.749 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:31.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:31.882 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:31.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:31.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:32.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:32.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:32.149 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:32.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:32.284 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:52:32.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:32.398 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:32.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.450315+0000 osd.5 (osd.5) 329 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:32.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.450373+0000 osd.5 (osd.5) 330 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:32.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: cluster 2024-09-13T22:52:30.763990+0000 mgr.y (mgr.14613) 552 : cluster [DBG] pgmap v636: 168 pgs: 168 active+clean; 961 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-13T22:52:32.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: cluster 2024-09-13T22:52:31.221126+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-13T22:52:32.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: cluster 2024-09-13T22:52:31.245903+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:52:32.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: cluster 2024-09-13T22:52:31.258257+0000 osd.5 (osd.5) 332 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:52:32.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: audit 2024-09-13T22:52:31.460536+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? v1:172.21.15.178:0/1632953105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:52:32.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:32 smithi192 bash[24608]: audit 2024-09-13T22:52:31.527569+0000 mgr.y (mgr.14613) 553 : audit [DBG] from='client.15549 v1:172.21.15.178:0/396888382' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:32.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:32.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:32.643 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:32.664 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:32.767 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:32.800 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:32.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:32.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:32.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.450315+0000 osd.5 (osd.5) 329 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.450373+0000 osd.5 (osd.5) 330 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: cluster 2024-09-13T22:52:30.763990+0000 mgr.y (mgr.14613) 552 : cluster [DBG] pgmap v636: 168 pgs: 168 active+clean; 961 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: cluster 2024-09-13T22:52:31.221126+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: cluster 2024-09-13T22:52:31.245903+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: cluster 2024-09-13T22:52:31.258257+0000 osd.5 (osd.5) 332 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:52:32.936 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: audit 2024-09-13T22:52:31.460536+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? v1:172.21.15.178:0/1632953105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[22840]: audit 2024-09-13T22:52:31.527569+0000 mgr.y (mgr.14613) 553 : audit [DBG] from='client.15549 v1:172.21.15.178:0/396888382' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.450315+0000 osd.5 (osd.5) 329 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.450373+0000 osd.5 (osd.5) 330 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: cluster 2024-09-13T22:52:30.763990+0000 mgr.y (mgr.14613) 552 : cluster [DBG] pgmap v636: 168 pgs: 168 active+clean; 961 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: cluster 2024-09-13T22:52:31.221126+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: cluster 2024-09-13T22:52:31.245903+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: cluster 2024-09-13T22:52:31.258257+0000 osd.5 (osd.5) 332 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: audit 2024-09-13T22:52:31.460536+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? v1:172.21.15.178:0/1632953105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:52:32.937 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:32 smithi178 bash[29889]: audit 2024-09-13T22:52:31.527569+0000 mgr.y (mgr.14613) 553 : audit [DBG] from='client.15549 v1:172.21.15.178:0/396888382' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:33.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:33.058 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:33.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:33.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:33.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[22840]: cluster 2024-09-13T22:52:32.051452+0000 osd.3 (osd.3) 277 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T22:52:33.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[22840]: cluster 2024-09-13T22:52:32.054433+0000 osd.3 (osd.3) 278 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T22:52:33.260 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[22840]: cluster 2024-09-13T22:52:32.211455+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:52:33.261 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[22840]: audit 2024-09-13T22:52:32.285940+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.178:0/1632953105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:52:33.261 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[22840]: cluster 2024-09-13T22:52:32.288809+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-13T22:52:33.261 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[22840]: audit 2024-09-13T22:52:33.005142+0000 mon.b (mon.1) 213 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:52:33.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:33.319 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:33.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:33.445 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:33.547 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:33.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[29889]: cluster 2024-09-13T22:52:32.051452+0000 osd.3 (osd.3) 277 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T22:52:33.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[29889]: cluster 2024-09-13T22:52:32.054433+0000 osd.3 (osd.3) 278 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T22:52:33.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[29889]: cluster 2024-09-13T22:52:32.211455+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:52:33.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[29889]: audit 2024-09-13T22:52:32.285940+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.178:0/1632953105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:52:33.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[29889]: cluster 2024-09-13T22:52:32.288809+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-13T22:52:33.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:33 smithi178 bash[29889]: audit 2024-09-13T22:52:33.005142+0000 mon.b (mon.1) 213 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:52:33.567 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:33 smithi192 bash[24608]: cluster 2024-09-13T22:52:32.051452+0000 osd.3 (osd.3) 277 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T22:52:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:33 smithi192 bash[24608]: cluster 2024-09-13T22:52:32.054433+0000 osd.3 (osd.3) 278 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T22:52:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:33 smithi192 bash[24608]: cluster 2024-09-13T22:52:32.211455+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:52:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:33 smithi192 bash[24608]: audit 2024-09-13T22:52:32.285940+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.178:0/1632953105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:52:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:33 smithi192 bash[24608]: cluster 2024-09-13T22:52:32.288809+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-13T22:52:33.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:33 smithi192 bash[24608]: audit 2024-09-13T22:52:33.005142+0000 mon.b (mon.1) 213 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:52:33.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:33.689 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:33.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:33.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:33.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:33.971 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:33.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:34.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T22:52:34.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:34.100 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:34.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:34.241 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:34.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:34.362 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:34 smithi192 bash[24608]: cluster 2024-09-13T22:52:32.764373+0000 mgr.y (mgr.14613) 554 : cluster [DBG] pgmap v639: 168 pgs: 168 active+clean; 899 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s 2024-09-13T22:52:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:34 smithi192 bash[24608]: audit 2024-09-13T22:52:33.828356+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:34 smithi192 bash[24608]: audit 2024-09-13T22:52:33.835236+0000 mon.b (mon.1) 214 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:34 smithi192 bash[24608]: audit 2024-09-13T22:52:34.003815+0000 mon.b (mon.1) 215 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:52:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:34 smithi192 bash[24608]: audit 2024-09-13T22:52:34.004704+0000 mon.b (mon.1) 216 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:52:34.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:34 smithi192 bash[24608]: audit 2024-09-13T22:52:34.013663+0000 mon.a (mon.0) 1425 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:34.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[29889]: cluster 2024-09-13T22:52:32.764373+0000 mgr.y (mgr.14613) 554 : cluster [DBG] pgmap v639: 168 pgs: 168 active+clean; 899 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s 2024-09-13T22:52:34.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[29889]: audit 2024-09-13T22:52:33.828356+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[29889]: audit 2024-09-13T22:52:33.835236+0000 mon.b (mon.1) 214 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[29889]: audit 2024-09-13T22:52:34.003815+0000 mon.b (mon.1) 215 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[29889]: audit 2024-09-13T22:52:34.004704+0000 mon.b (mon.1) 216 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[29889]: audit 2024-09-13T22:52:34.013663+0000 mon.a (mon.0) 1425 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[22840]: cluster 2024-09-13T22:52:32.764373+0000 mgr.y (mgr.14613) 554 : cluster [DBG] pgmap v639: 168 pgs: 168 active+clean; 899 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[22840]: audit 2024-09-13T22:52:33.828356+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[22840]: audit 2024-09-13T22:52:33.835236+0000 mon.b (mon.1) 214 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[22840]: audit 2024-09-13T22:52:34.003815+0000 mon.b (mon.1) 215 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:52:34.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[22840]: audit 2024-09-13T22:52:34.004704+0000 mon.b (mon.1) 216 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:52:34.447 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:34 smithi178 bash[22840]: audit 2024-09-13T22:52:34.013663+0000 mon.a (mon.0) 1425 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:34.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:34.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:34.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:34.629 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:34.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:34.758 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:34.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:34.885 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:34.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:34.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:35.018 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:35.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.148 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:35.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:35.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:35.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:35.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:35 smithi178 bash[29889]: audit 2024-09-13T22:52:34.562170+0000 mon.a (mon.0) 1426 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]: dispatch 2024-09-13T22:52:35.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:35 smithi178 bash[29889]: audit 2024-09-13T22:52:34.563647+0000 mon.b (mon.1) 217 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]: dispatch 2024-09-13T22:52:35.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:35 smithi178 bash[22840]: audit 2024-09-13T22:52:34.562170+0000 mon.a (mon.0) 1426 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]: dispatch 2024-09-13T22:52:35.393 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:35 smithi178 bash[22840]: audit 2024-09-13T22:52:34.563647+0000 mon.b (mon.1) 217 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]: dispatch 2024-09-13T22:52:35.407 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:35.510 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.527 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:35.559 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:35 smithi192 bash[24608]: audit 2024-09-13T22:52:34.562170+0000 mon.a (mon.0) 1426 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]: dispatch 2024-09-13T22:52:35.559 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:35 smithi192 bash[24608]: audit 2024-09-13T22:52:34.563647+0000 mon.b (mon.1) 217 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]: dispatch 2024-09-13T22:52:35.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:35.652 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:35.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.780 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:35.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:52:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:52:35.909 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:35.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:36.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:36.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:36.137 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:36.163 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:36.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:36.293 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:36.352 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T22:52:36.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-13T22:52:36.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:36.416 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:36.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[22840]: cluster 2024-09-13T22:52:34.765295+0000 mgr.y (mgr.14613) 555 : cluster [DBG] pgmap v640: 168 pgs: 168 active+clean; 713 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[22840]: audit 2024-09-13T22:52:35.296777+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]': finished 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[22840]: cluster 2024-09-13T22:52:35.300511+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[22840]: audit 2024-09-13T22:52:35.560303+0000 mgr.y (mgr.14613) 556 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[29889]: cluster 2024-09-13T22:52:34.765295+0000 mgr.y (mgr.14613) 555 : cluster [DBG] pgmap v640: 168 pgs: 168 active+clean; 713 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[29889]: audit 2024-09-13T22:52:35.296777+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]': finished 2024-09-13T22:52:36.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[29889]: cluster 2024-09-13T22:52:35.300511+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-13T22:52:36.529 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:36 smithi178 bash[29889]: audit 2024-09-13T22:52:35.560303+0000 mgr.y (mgr.14613) 556 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:36.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:36.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:36.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:36 smithi192 bash[24608]: cluster 2024-09-13T22:52:34.765295+0000 mgr.y (mgr.14613) 555 : cluster [DBG] pgmap v640: 168 pgs: 168 active+clean; 713 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T22:52:36.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:36 smithi192 bash[24608]: audit 2024-09-13T22:52:35.296777+0000 mon.a (mon.0) 1427 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.e", "id": [7, 2]}]': finished 2024-09-13T22:52:36.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:36 smithi192 bash[24608]: cluster 2024-09-13T22:52:35.300511+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-13T22:52:36.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:36 smithi192 bash[24608]: audit 2024-09-13T22:52:35.560303+0000 mgr.y (mgr.14613) 556 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:36.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:36.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:36.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:36.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:36.928 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:36.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:36.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:52:37.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:37.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:37.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:37.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:37.306 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:37.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:37.436 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:37.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:37.550 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:37 smithi178 bash[22840]: cluster 2024-09-13T22:52:36.302086+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-13T22:52:37.551 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:37 smithi178 bash[29889]: cluster 2024-09-13T22:52:36.302086+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-13T22:52:37.566 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:37.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:37 smithi192 bash[24608]: cluster 2024-09-13T22:52:36.302086+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-13T22:52:37.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:37.698 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:37.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:37.827 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:37.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:37.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:37.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:38.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:38.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:38.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:38.216 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:38.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:38.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:52:38.351 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:38.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:38 smithi192 bash[24608]: cluster 2024-09-13T22:52:36.766025+0000 mgr.y (mgr.14613) 557 : cluster [DBG] pgmap v643: 168 pgs: 168 active+clean; 590 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 0 B/s wr, 890 op/s 2024-09-13T22:52:38.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:38 smithi192 bash[24608]: cluster 2024-09-13T22:52:37.312067+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-13T22:52:38.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:38 smithi192 bash[24608]: cluster 2024-09-13T22:52:37.315809+0000 osd.4 (osd.4) 205 : cluster [DBG] 9.e starting backfill to osd.2 from (0'0,0'0] MAX to 311'11505 2024-09-13T22:52:38.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:38 smithi192 bash[24608]: cluster 2024-09-13T22:52:37.472346+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:38.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:38 smithi192 bash[24608]: cluster 2024-09-13T22:52:37.472423+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:38.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:38.486 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:38.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:38.623 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[22840]: cluster 2024-09-13T22:52:36.766025+0000 mgr.y (mgr.14613) 557 : cluster [DBG] pgmap v643: 168 pgs: 168 active+clean; 590 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 0 B/s wr, 890 op/s 2024-09-13T22:52:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[22840]: cluster 2024-09-13T22:52:37.312067+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-13T22:52:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[22840]: cluster 2024-09-13T22:52:37.315809+0000 osd.4 (osd.4) 205 : cluster [DBG] 9.e starting backfill to osd.2 from (0'0,0'0] MAX to 311'11505 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[22840]: cluster 2024-09-13T22:52:37.472346+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[22840]: cluster 2024-09-13T22:52:37.472423+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[29889]: cluster 2024-09-13T22:52:36.766025+0000 mgr.y (mgr.14613) 557 : cluster [DBG] pgmap v643: 168 pgs: 168 active+clean; 590 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 0 B/s wr, 890 op/s 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[29889]: cluster 2024-09-13T22:52:37.312067+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[29889]: cluster 2024-09-13T22:52:37.315809+0000 osd.4 (osd.4) 205 : cluster [DBG] 9.e starting backfill to osd.2 from (0'0,0'0] MAX to 311'11505 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[29889]: cluster 2024-09-13T22:52:37.472346+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:38 smithi178 bash[29889]: cluster 2024-09-13T22:52:37.472423+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:38.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:38.763 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:38.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:38.903 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:38.905 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:38.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:39.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:39.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:39.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:39.169 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:39.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:39.298 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:39.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:39.435 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:39.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:39.550 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:39 smithi192 bash[24608]: cluster 2024-09-13T22:52:38.334577+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-13T22:52:39.569 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:39.633 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:39 smithi178 bash[29889]: cluster 2024-09-13T22:52:38.334577+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-13T22:52:39.633 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:39 smithi178 bash[22840]: cluster 2024-09-13T22:52:38.334577+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-13T22:52:39.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:39.695 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:39.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:39.827 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:39.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:39.959 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:39.959 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:52:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:52:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:39.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:40.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:40.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:40.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:40.223 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:40.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:40.358 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:40.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:40.501 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:40.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:40.616 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:40 smithi192 bash[24608]: cluster 2024-09-13T22:52:38.766503+0000 mgr.y (mgr.14613) 558 : cluster [DBG] pgmap v646: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 570 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 572 op/s; 456/27954 objects misplaced (1.631%) 2024-09-13T22:52:40.617 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:40 smithi192 bash[24608]: cluster 2024-09-13T22:52:39.330950+0000 mon.a (mon.0) 1432 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:52:40.632 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:40.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:40 smithi178 bash[29889]: cluster 2024-09-13T22:52:38.766503+0000 mgr.y (mgr.14613) 558 : cluster [DBG] pgmap v646: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 570 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 572 op/s; 456/27954 objects misplaced (1.631%) 2024-09-13T22:52:40.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:40 smithi178 bash[29889]: cluster 2024-09-13T22:52:39.330950+0000 mon.a (mon.0) 1432 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:52:40.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:40 smithi178 bash[22840]: cluster 2024-09-13T22:52:38.766503+0000 mgr.y (mgr.14613) 558 : cluster [DBG] pgmap v646: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 570 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 572 op/s; 456/27954 objects misplaced (1.631%) 2024-09-13T22:52:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:40 smithi178 bash[22840]: cluster 2024-09-13T22:52:39.330950+0000 mon.a (mon.0) 1432 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:52:40.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:40.761 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:40.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:40.891 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:40.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:40.941 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:40.941 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:40.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:41.031 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:41.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:41.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:41.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:41.293 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:41.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:41.426 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:41.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:41 smithi178 bash[22840]: cluster 2024-09-13T22:52:40.767220+0000 osd.1 (osd.1) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:41.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:41 smithi178 bash[22840]: cluster 2024-09-13T22:52:40.767285+0000 osd.1 (osd.1) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:41.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:41.552 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:41.576 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:41 smithi178 bash[29889]: cluster 2024-09-13T22:52:40.767220+0000 osd.1 (osd.1) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:41.576 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:41 smithi178 bash[29889]: cluster 2024-09-13T22:52:40.767285+0000 osd.1 (osd.1) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:41.654 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:41.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:41 smithi192 bash[24608]: cluster 2024-09-13T22:52:40.767220+0000 osd.1 (osd.1) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:52:41.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:41 smithi192 bash[24608]: cluster 2024-09-13T22:52:40.767285+0000 osd.1 (osd.1) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:52:41.675 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:41.777 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:41.815 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:41.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:41.949 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:41.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:42.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:42.074 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:42.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:42.206 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:42.300 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:52:42.300 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:52:42.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:42.335 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:42.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:42.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:42 smithi192 bash[24608]: cluster 2024-09-13T22:52:40.767602+0000 mgr.y (mgr.14613) 559 : cluster [DBG] pgmap v647: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 452 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 677 op/s; 456/22266 objects misplaced (2.048%) 2024-09-13T22:52:42.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:42 smithi192 bash[24608]: cluster 2024-09-13T22:52:41.345795+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-13T22:52:42.476 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:42.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:42.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:42 smithi178 bash[29889]: cluster 2024-09-13T22:52:40.767602+0000 mgr.y (mgr.14613) 559 : cluster [DBG] pgmap v647: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 452 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 677 op/s; 456/22266 objects misplaced (2.048%) 2024-09-13T22:52:42.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:42 smithi178 bash[29889]: cluster 2024-09-13T22:52:41.345795+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-13T22:52:42.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:42 smithi178 bash[22840]: cluster 2024-09-13T22:52:40.767602+0000 mgr.y (mgr.14613) 559 : cluster [DBG] pgmap v647: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 452 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 677 op/s; 456/22266 objects misplaced (2.048%) 2024-09-13T22:52:42.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:42 smithi178 bash[22840]: cluster 2024-09-13T22:52:41.345795+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-13T22:52:42.603 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:42.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:42.743 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:42.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:42.877 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:42.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:42.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:43.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:43.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:43.133 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:43.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:43.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:43.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:43.390 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:43.494 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:43.519 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:43.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:43.651 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:43.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:43.780 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:43.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:43.913 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:43.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:44.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:44.044 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:44.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:44.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:44.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:44.296 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:44.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:44.429 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:44.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:44.544 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:44 smithi192 bash[24608]: cluster 2024-09-13T22:52:42.768081+0000 mgr.y (mgr.14613) 560 : cluster [DBG] pgmap v649: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 413 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 471 op/s; 456/20400 objects misplaced (2.235%) 2024-09-13T22:52:44.560 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:44.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:44.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:44 smithi178 bash[29889]: cluster 2024-09-13T22:52:42.768081+0000 mgr.y (mgr.14613) 560 : cluster [DBG] pgmap v649: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 413 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 471 op/s; 456/20400 objects misplaced (2.235%) 2024-09-13T22:52:44.685 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:44 smithi178 bash[22840]: cluster 2024-09-13T22:52:42.768081+0000 mgr.y (mgr.14613) 560 : cluster [DBG] pgmap v649: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 413 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 471 op/s; 456/20400 objects misplaced (2.235%) 2024-09-13T22:52:44.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:44.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:44.843 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:44.858 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Removed 99159 objects 2024-09-13T22:52:44.858 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Clean up completed and total clean up time :102.513 2024-09-13T22:52:44.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:44.967 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:44.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:45.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:45.093 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:45.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:45.225 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:45.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T22:52:45.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:45.354 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:45.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:45.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:45.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:45.601 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:45.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:52:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:52:45.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:45.725 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:45.730 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:45.777 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:52:45.778 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:52:45.794 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:52:45.795 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:52:45.811 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:52:45.811 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:52:45.827 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:52:45.827 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:52:45.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:45.847 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:52:45.847 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:52:45.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:45.868 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:52:45.868 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:52:45.887 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:52:45.888 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:52:45.908 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:52:45.908 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:52:45.950 INFO:teuthology.orchestra.run.smithi178.stdout:moved 5 / 504 (0.992063%) 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:avg 63 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:stddev 8.35165 -> 8.77496 (expected baseline 7.42462) 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:min osd.2 with 50 -> 51 pgs (0.793651 -> 0.809524 * mean) 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:max osd.7 with 79 -> 81 pgs (1.25397 -> 1.28571 * mean) 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:oload 110 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:max_change 0.05 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:max_change_osds 4 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:average_utilization 0.0058 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:overload_utilization 0.0064 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-13T22:52:45.951 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1 weight 1.0000 -> 0.9500 2024-09-13T22:52:45.952 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-13T22:52:45.952 INFO:teuthology.orchestra.run.smithi178.stderr:no change 2024-09-13T22:52:45.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:45.971 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:45.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:46.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:46.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:46.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:46.222 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:46.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:46.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:46.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:46.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:46 smithi192 bash[24608]: cluster 2024-09-13T22:52:44.768877+0000 mgr.y (mgr.14613) 561 : cluster [DBG] pgmap v650: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 184 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 870 op/s; 86/9402 objects misplaced (0.915%) 2024-09-13T22:52:46.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:46 smithi192 bash[24608]: audit 2024-09-13T22:52:45.571313+0000 mgr.y (mgr.14613) 562 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:46.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:46 smithi192 bash[24608]: audit 2024-09-13T22:52:45.760717+0000 mon.a (mon.0) 1434 : audit [DBG] from='client.? v1:172.21.15.178:0/3535755147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:46.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:46.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:46.592 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[23095]: debug 2024-09-13T22:52:45.947+0000 7f089e894640 -1 mgr.server reply reply (3) No such process no change 2024-09-13T22:52:46.592 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[29889]: cluster 2024-09-13T22:52:44.768877+0000 mgr.y (mgr.14613) 561 : cluster [DBG] pgmap v650: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 184 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 870 op/s; 86/9402 objects misplaced (0.915%) 2024-09-13T22:52:46.592 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[29889]: audit 2024-09-13T22:52:45.571313+0000 mgr.y (mgr.14613) 562 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:46.592 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[29889]: audit 2024-09-13T22:52:45.760717+0000 mon.a (mon.0) 1434 : audit [DBG] from='client.? v1:172.21.15.178:0/3535755147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:46.592 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[22840]: cluster 2024-09-13T22:52:44.768877+0000 mgr.y (mgr.14613) 561 : cluster [DBG] pgmap v650: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 184 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 870 op/s; 86/9402 objects misplaced (0.915%) 2024-09-13T22:52:46.593 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[22840]: audit 2024-09-13T22:52:45.571313+0000 mgr.y (mgr.14613) 562 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:46.593 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:46 smithi178 bash[22840]: audit 2024-09-13T22:52:45.760717+0000 mon.a (mon.0) 1434 : audit [DBG] from='client.? v1:172.21.15.178:0/3535755147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:52:46.607 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:46.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:46.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:46.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:46.868 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:46.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:46.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:46.998 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:47.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:47.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:47.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:47.256 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:47.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:47.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:47 smithi192 bash[24608]: audit 2024-09-13T22:52:45.951237+0000 mgr.y (mgr.14613) 563 : audit [DBG] from='client.15561 v1:172.21.15.178:0/3648029763' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:47.387 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:47.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:47.516 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:47.529 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-13T22:52:47.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-13T22:52:47.540 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:47 smithi178 bash[29889]: audit 2024-09-13T22:52:45.951237+0000 mgr.y (mgr.14613) 563 : audit [DBG] from='client.15561 v1:172.21.15.178:0/3648029763' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:47.540 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:47 smithi178 bash[22840]: audit 2024-09-13T22:52:45.951237+0000 mgr.y (mgr.14613) 563 : audit [DBG] from='client.15561 v1:172.21.15.178:0/3648029763' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:47.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:47.645 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:47.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:47.775 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:47.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:47.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:47.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:48.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:48.033 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:48.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:48.163 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:48.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:48.292 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:48.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:48.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:48 smithi192 bash[24608]: cluster 2024-09-13T22:52:46.770107+0000 mgr.y (mgr.14613) 564 : cluster [DBG] pgmap v651: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 45 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 86/2724 objects misplaced (3.157%) 2024-09-13T22:52:48.419 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:48.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:48.532 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:48 smithi178 bash[29889]: cluster 2024-09-13T22:52:46.770107+0000 mgr.y (mgr.14613) 564 : cluster [DBG] pgmap v651: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 45 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 86/2724 objects misplaced (3.157%) 2024-09-13T22:52:48.532 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:48 smithi178 bash[22840]: cluster 2024-09-13T22:52:46.770107+0000 mgr.y (mgr.14613) 564 : cluster [DBG] pgmap v651: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 45 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 86/2724 objects misplaced (3.157%) 2024-09-13T22:52:48.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:48.588 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T22:52:48.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-13T22:52:48.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:48.698 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:48.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:48.828 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:48.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:48.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:48.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:49.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:49.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:49.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:49.207 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:49.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:49.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:49.440 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:49.467 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:49.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:49.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:49.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:49.730 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:49.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:49.862 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:49.868 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:52:49.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:49.973 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:52:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[29889]: cluster 2024-09-13T22:52:48.770649+0000 mgr.y (mgr.14613) 565 : cluster [DBG] pgmap v652: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 33 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[29889]: audit 2024-09-13T22:52:48.851228+0000 mon.a (mon.0) 1435 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[29889]: audit 2024-09-13T22:52:48.857071+0000 mon.b (mon.1) 218 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[29889]: audit 2024-09-13T22:52:49.175407+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.178:0/892200222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[29889]: audit 2024-09-13T22:52:49.175798+0000 mon.a (mon.0) 1436 : audit [INF] from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[22840]: cluster 2024-09-13T22:52:48.770649+0000 mgr.y (mgr.14613) 565 : cluster [DBG] pgmap v652: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 33 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[22840]: audit 2024-09-13T22:52:48.851228+0000 mon.a (mon.0) 1435 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[22840]: audit 2024-09-13T22:52:48.857071+0000 mon.b (mon.1) 218 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[22840]: audit 2024-09-13T22:52:49.175407+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.178:0/892200222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:49.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:49 smithi178 bash[22840]: audit 2024-09-13T22:52:49.175798+0000 mon.a (mon.0) 1436 : audit [INF] from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:49.986 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:49.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:50.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:50.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:50.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:49 smithi192 bash[24608]: cluster 2024-09-13T22:52:48.770649+0000 mgr.y (mgr.14613) 565 : cluster [DBG] pgmap v652: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 33 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s 2024-09-13T22:52:50.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:49 smithi192 bash[24608]: audit 2024-09-13T22:52:48.851228+0000 mon.a (mon.0) 1435 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:52:50.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:49 smithi192 bash[24608]: audit 2024-09-13T22:52:48.857071+0000 mon.b (mon.1) 218 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:52:50.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:49 smithi192 bash[24608]: audit 2024-09-13T22:52:49.175407+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.178:0/892200222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:50.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:49 smithi192 bash[24608]: audit 2024-09-13T22:52:49.175798+0000 mon.a (mon.0) 1436 : audit [INF] from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:52:50.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:50.247 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:50.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:50.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:50.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:50.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:50.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:52:50.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:50.637 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:50.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:50.767 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:50.853 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:50 smithi178 bash[22840]: cluster 2024-09-13T22:52:49.850342+0000 mon.a (mon.0) 1437 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:52:50.853 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:50 smithi178 bash[22840]: audit 2024-09-13T22:52:49.862368+0000 mon.a (mon.0) 1438 : audit [INF] from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:52:50.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:50.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:50.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:50.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:51.027 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:51.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:51.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:51.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:50 smithi192 bash[24608]: cluster 2024-09-13T22:52:49.850342+0000 mon.a (mon.0) 1437 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:52:51.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:50 smithi192 bash[24608]: audit 2024-09-13T22:52:49.862368+0000 mon.a (mon.0) 1438 : audit [INF] from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:52:51.142 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:50 smithi192 bash[24608]: cluster 2024-09-13T22:52:49.868669+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-13T22:52:51.159 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:51.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:50 smithi178 bash[22840]: cluster 2024-09-13T22:52:49.868669+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-13T22:52:51.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:50 smithi178 bash[29889]: cluster 2024-09-13T22:52:49.850342+0000 mon.a (mon.0) 1437 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:52:51.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:50 smithi178 bash[29889]: audit 2024-09-13T22:52:49.862368+0000 mon.a (mon.0) 1438 : audit [INF] from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:52:51.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:50 smithi178 bash[29889]: cluster 2024-09-13T22:52:49.868669+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-13T22:52:51.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:51.287 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:51.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:51.416 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:51.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:51.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:51.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:51.669 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:51.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:51.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:51.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:51.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:51 smithi192 bash[24608]: cluster 2024-09-13T22:52:50.771499+0000 mgr.y (mgr.14613) 566 : cluster [DBG] pgmap v654: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 543 B/s rd, 0 B/s wr, 700 op/s 2024-09-13T22:52:51.930 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:51.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:52.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:52.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:52.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:52.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:51 smithi178 bash[29889]: cluster 2024-09-13T22:52:50.771499+0000 mgr.y (mgr.14613) 566 : cluster [DBG] pgmap v654: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 543 B/s rd, 0 B/s wr, 700 op/s 2024-09-13T22:52:52.172 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:51 smithi178 bash[22840]: cluster 2024-09-13T22:52:50.771499+0000 mgr.y (mgr.14613) 566 : cluster [DBG] pgmap v654: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 543 B/s rd, 0 B/s wr, 700 op/s 2024-09-13T22:52:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:52.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:52.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:52.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:52.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:52.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:52.576 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:52.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:52.706 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:52.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:52.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:52.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:52.967 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:52.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:53.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:52:53.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:53.094 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:53.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:53.223 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:53.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:53.354 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:53.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:53.484 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:53.535 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:53.537 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:53.537 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:53.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:53.608 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:53.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:53.737 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:53.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:53.867 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:53.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:53.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:53.998 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:54.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:54.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:54.127 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:53 smithi178 bash[29889]: cluster 2024-09-13T22:52:52.772463+0000 mgr.y (mgr.14613) 567 : cluster [DBG] pgmap v655: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 660 op/s 2024-09-13T22:52:54.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:53 smithi178 bash[22840]: cluster 2024-09-13T22:52:52.772463+0000 mgr.y (mgr.14613) 567 : cluster [DBG] pgmap v655: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 660 op/s 2024-09-13T22:52:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:54.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:54.257 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:54.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:54.372 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:53 smithi192 bash[24608]: cluster 2024-09-13T22:52:52.772463+0000 mgr.y (mgr.14613) 567 : cluster [DBG] pgmap v655: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 660 op/s 2024-09-13T22:52:54.387 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:54.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:54.516 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:54.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:54.647 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:54.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:54.778 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:54.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:54.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:54.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:55.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:55.038 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:55.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:55.168 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:55.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:55.295 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:55.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:55.423 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:55.526 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:55.553 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:55.566 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:52:55.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:55.685 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:55.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:55.796 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:52:55.796 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:52:55.797 INFO:teuthology.orchestra.run.smithi178.stdout: "num_ops": 0 2024-09-13T22:52:55.797 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:52:55.809 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:55.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:52:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:52:55.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:55.939 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:55.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:56.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:56.068 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:56.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:56.196 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:56.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:56.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:55 smithi192 bash[24608]: cluster 2024-09-13T22:52:54.773391+0000 mgr.y (mgr.14613) 568 : cluster [DBG] pgmap v656: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 294 op/s 2024-09-13T22:52:56.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:55 smithi192 bash[24608]: audit 2024-09-13T22:52:55.582498+0000 mgr.y (mgr.14613) 569 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:56.327 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:56.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:55 smithi178 bash[29889]: cluster 2024-09-13T22:52:54.773391+0000 mgr.y (mgr.14613) 568 : cluster [DBG] pgmap v656: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 294 op/s 2024-09-13T22:52:56.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:55 smithi178 bash[29889]: audit 2024-09-13T22:52:55.582498+0000 mgr.y (mgr.14613) 569 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:56.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:55 smithi178 bash[22840]: cluster 2024-09-13T22:52:54.773391+0000 mgr.y (mgr.14613) 568 : cluster [DBG] pgmap v656: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 294 op/s 2024-09-13T22:52:56.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:55 smithi178 bash[22840]: audit 2024-09-13T22:52:55.582498+0000 mgr.y (mgr.14613) 569 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:52:56.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:56.459 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:56.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:56.588 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:56.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:56.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:56.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:56.847 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:52:56.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:56.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:56.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:57.071 INFO:teuthology.orchestra.run.smithi178.stdout:moved 0 / 504 (0%) 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:avg 63 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:stddev 8.35165 -> 8.35165 (expected baseline 7.42462) 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:min osd.2 with 50 -> 50 pgs (0.793651 -> 0.793651 * mean) 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:max osd.7 with 79 -> 79 pgs (1.25397 -> 1.25397 * mean) 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:oload 1000 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:max_change 3 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:max_change_osds 4 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:average_utilization 0.0047 2024-09-13T22:52:57.072 INFO:teuthology.orchestra.run.smithi178.stdout:overload_utilization 0.0472 2024-09-13T22:52:57.073 INFO:teuthology.orchestra.run.smithi178.stderr:no change 2024-09-13T22:52:57.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:57.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:57.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:57.226 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:57.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:57.357 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:57.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:57.487 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:57.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:57.618 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:57.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:57.749 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:57.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:57.880 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:57.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:57.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:57.984 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_2' removed 2024-09-13T22:52:58.015 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:58.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:58.164 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:52:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:52:58.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:58.279 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:58 smithi192 bash[24608]: cluster 2024-09-13T22:52:56.774200+0000 mgr.y (mgr.14613) 570 : cluster [DBG] pgmap v657: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 71 op/s 2024-09-13T22:52:58.279 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:58 smithi192 bash[24608]: audit 2024-09-13T22:52:57.072362+0000 mgr.y (mgr.14613) 571 : audit [DBG] from='client.25463 v1:172.21.15.178:0/884282452' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:58.279 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:58 smithi192 bash[24608]: audit 2024-09-13T22:52:57.226320+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.178:0/3901456289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:52:58.279 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:58 smithi192 bash[24608]: audit 2024-09-13T22:52:57.984897+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.178:0/3901456289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:52:58.279 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:58 smithi192 bash[24608]: cluster 2024-09-13T22:52:57.989492+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-13T22:52:58.292 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:58.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:58.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[22840]: cluster 2024-09-13T22:52:56.774200+0000 mgr.y (mgr.14613) 570 : cluster [DBG] pgmap v657: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 71 op/s 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[22840]: audit 2024-09-13T22:52:57.072362+0000 mgr.y (mgr.14613) 571 : audit [DBG] from='client.25463 v1:172.21.15.178:0/884282452' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[22840]: audit 2024-09-13T22:52:57.226320+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.178:0/3901456289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[22840]: audit 2024-09-13T22:52:57.984897+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.178:0/3901456289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[22840]: cluster 2024-09-13T22:52:57.989492+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[29889]: cluster 2024-09-13T22:52:56.774200+0000 mgr.y (mgr.14613) 570 : cluster [DBG] pgmap v657: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 71 op/s 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[29889]: audit 2024-09-13T22:52:57.072362+0000 mgr.y (mgr.14613) 571 : audit [DBG] from='client.25463 v1:172.21.15.178:0/884282452' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[29889]: audit 2024-09-13T22:52:57.226320+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.178:0/3901456289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[29889]: audit 2024-09-13T22:52:57.984897+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.178:0/3901456289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:52:58.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:58 smithi178 bash[29889]: cluster 2024-09-13T22:52:57.989492+0000 mon.a (mon.0) 1442 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-13T22:52:58.422 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:52:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:52:58.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:58.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:58.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:58.682 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:52:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:52:58.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:58.814 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:58.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:58.945 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:52:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:58.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:52:59.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:59.076 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:59.133 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T22:52:59.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:59.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:59.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:59.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:52:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:52:59.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:52:59 smithi178 bash[29889]: cluster 2024-09-13T22:52:59.005059+0000 mon.a (mon.0) 1443 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:52:59.356 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:52:59 smithi178 bash[22840]: cluster 2024-09-13T22:52:59.005059+0000 mon.a (mon.0) 1443 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:52:59.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:52:59 smithi192 bash[24608]: cluster 2024-09-13T22:52:59.005059+0000 mon.a (mon.0) 1443 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:52:59.432 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:59.457 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:59.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:59.587 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:52:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:52:59.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:59.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:59.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:59.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:52:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:52:59.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:59.977 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:52:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:52:59.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:00.058 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:52:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:52:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:00.059 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:00 smithi178 bash[22840]: cluster 2024-09-13T22:52:58.775003+0000 mgr.y (mgr.14613) 572 : cluster [DBG] pgmap v659: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:53:00.081 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:00.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:00 smithi178 bash[29889]: cluster 2024-09-13T22:52:58.775003+0000 mgr.y (mgr.14613) 572 : cluster [DBG] pgmap v659: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:53:00.105 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:00.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:00.213 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T22:53:00.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-13T22:53:00.231 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:00.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:00.363 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:00.368 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T22:53:00.368 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T22:53:00.369 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-13T22:53:00.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-13T22:53:00.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:00 smithi192 bash[24608]: cluster 2024-09-13T22:52:58.775003+0000 mgr.y (mgr.14613) 572 : cluster [DBG] pgmap v659: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:53:00.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:00.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:00.526 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:53:00.526 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:53:00.544 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:53:00.544 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:53:00.563 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:53:00.563 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:53:00.580 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:53:00.580 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:53:00.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:00.601 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:53:00.601 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:53:00.620 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:53:00.621 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:53:00.624 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:00.638 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:53:00.640 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:53:00.656 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:53:00.656 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:53:00.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:00.754 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:00.759 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:00.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:00.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:00.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:00.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:01.015 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:01.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:01.144 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:01.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:01.275 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:01.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:01.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:01 smithi192 bash[24608]: audit 2024-09-13T22:53:00.509472+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.178:0/404897057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:01.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:01.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:01 smithi178 bash[29889]: audit 2024-09-13T22:53:00.509472+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.178:0/404897057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:01.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:01 smithi178 bash[22840]: audit 2024-09-13T22:53:00.509472+0000 mon.c (mon.2) 60 : audit [DBG] from='client.? v1:172.21.15.178:0/404897057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:01.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:01.540 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:01.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:01.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:01.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:01.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:01.931 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:01.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:02.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:02.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:02.165 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:02.192 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:02.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:02.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:02.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:02 smithi192 bash[24608]: cluster 2024-09-13T22:53:00.775707+0000 mgr.y (mgr.14613) 573 : cluster [DBG] pgmap v660: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:53:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:02.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:02.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:02 smithi178 bash[22840]: cluster 2024-09-13T22:53:00.775707+0000 mgr.y (mgr.14613) 573 : cluster [DBG] pgmap v660: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:53:02.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:02 smithi178 bash[29889]: cluster 2024-09-13T22:53:00.775707+0000 mgr.y (mgr.14613) 573 : cluster [DBG] pgmap v660: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:53:02.451 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:02.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:02.581 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:02.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:02.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:02.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:02.839 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:02.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:02.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:02.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:03.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:03.086 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:53:03.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:03.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:03.223 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:03.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:03 smithi178 bash[29889]: audit 2024-09-13T22:53:03.026818+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.178:0/1821211599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:53:03.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:03 smithi178 bash[29889]: audit 2024-09-13T22:53:03.027215+0000 mon.a (mon.0) 1444 : audit [INF] from='client.25481 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:53:03.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:03 smithi178 bash[22840]: audit 2024-09-13T22:53:03.026818+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.178:0/1821211599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:53:03.293 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:03 smithi178 bash[22840]: audit 2024-09-13T22:53:03.027215+0000 mon.a (mon.0) 1444 : audit [INF] from='client.25481 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:53:03.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:03.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:03 smithi192 bash[24608]: audit 2024-09-13T22:53:03.026818+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.178:0/1821211599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:53:03.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:03 smithi192 bash[24608]: audit 2024-09-13T22:53:03.027215+0000 mon.a (mon.0) 1444 : audit [INF] from='client.25481 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:53:03.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:03.479 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:03.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:03.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:03.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:03.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:03.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:03.872 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:03.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:03.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:04.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:04.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:04.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:04.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:04.240 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:53:04.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[29889]: cluster 2024-09-13T22:53:02.776180+0000 mgr.y (mgr.14613) 574 : cluster [DBG] pgmap v661: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:53:04.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[29889]: cluster 2024-09-13T22:53:03.069679+0000 mon.a (mon.0) 1445 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:04.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[29889]: audit 2024-09-13T22:53:03.080706+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25481 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:53:04.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[29889]: cluster 2024-09-13T22:53:03.086062+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-13T22:53:04.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[29889]: audit 2024-09-13T22:53:03.874307+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:04.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[29889]: audit 2024-09-13T22:53:03.878531+0000 mon.b (mon.1) 219 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:04.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[22840]: cluster 2024-09-13T22:53:02.776180+0000 mgr.y (mgr.14613) 574 : cluster [DBG] pgmap v661: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:53:04.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[22840]: cluster 2024-09-13T22:53:03.069679+0000 mon.a (mon.0) 1445 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:04.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[22840]: audit 2024-09-13T22:53:03.080706+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25481 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:53:04.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[22840]: cluster 2024-09-13T22:53:03.086062+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-13T22:53:04.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[22840]: audit 2024-09-13T22:53:03.874307+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:04.248 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:04 smithi178 bash[22840]: audit 2024-09-13T22:53:03.878531+0000 mon.b (mon.1) 219 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:04.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:04.391 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:04.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:04 smithi192 bash[24608]: cluster 2024-09-13T22:53:02.776180+0000 mgr.y (mgr.14613) 574 : cluster [DBG] pgmap v661: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:53:04.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:04 smithi192 bash[24608]: cluster 2024-09-13T22:53:03.069679+0000 mon.a (mon.0) 1445 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:04.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:04 smithi192 bash[24608]: audit 2024-09-13T22:53:03.080706+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25481 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:53:04.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:04 smithi192 bash[24608]: cluster 2024-09-13T22:53:03.086062+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-13T22:53:04.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:04 smithi192 bash[24608]: audit 2024-09-13T22:53:03.874307+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:04.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:04 smithi192 bash[24608]: audit 2024-09-13T22:53:03.878531+0000 mon.b (mon.1) 219 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:04.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:04.520 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:04.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:04.652 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:04.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:04.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:04.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:04.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:04.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:05.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:05.042 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:05.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:05.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:05.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:05.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:05 smithi178 bash[29889]: cluster 2024-09-13T22:53:03.725476+0000 osd.6 (osd.6) 249 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:05.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:05 smithi178 bash[29889]: cluster 2024-09-13T22:53:03.725567+0000 osd.6 (osd.6) 250 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:05.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:05 smithi178 bash[29889]: cluster 2024-09-13T22:53:04.085985+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-13T22:53:05.289 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:05 smithi178 bash[22840]: cluster 2024-09-13T22:53:03.725476+0000 osd.6 (osd.6) 249 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:05.289 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:05 smithi178 bash[22840]: cluster 2024-09-13T22:53:03.725567+0000 osd.6 (osd.6) 250 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:05.289 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:05 smithi178 bash[22840]: cluster 2024-09-13T22:53:04.085985+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-13T22:53:05.303 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:05.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:05.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:05 smithi192 bash[24608]: cluster 2024-09-13T22:53:03.725476+0000 osd.6 (osd.6) 249 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:05.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:05 smithi192 bash[24608]: cluster 2024-09-13T22:53:03.725567+0000 osd.6 (osd.6) 250 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:05.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:05 smithi192 bash[24608]: cluster 2024-09-13T22:53:04.085985+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-13T22:53:05.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:05.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:05.561 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:05.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:05.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:53:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:53:05.691 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:05.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:05.824 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:05.927 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:05.954 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:05.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:06.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:53:06.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:06.102 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:06.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:06.215 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:06 smithi178 bash[29889]: cluster 2024-09-13T22:53:04.776863+0000 mgr.y (mgr.14613) 575 : cluster [DBG] pgmap v664: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-13T22:53:06.215 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:06 smithi178 bash[29889]: audit 2024-09-13T22:53:05.592374+0000 mgr.y (mgr.14613) 576 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:06.216 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:06 smithi178 bash[22840]: cluster 2024-09-13T22:53:04.776863+0000 mgr.y (mgr.14613) 575 : cluster [DBG] pgmap v664: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-13T22:53:06.216 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:06 smithi178 bash[22840]: audit 2024-09-13T22:53:05.592374+0000 mgr.y (mgr.14613) 576 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:06.229 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:06.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:06 smithi192 bash[24608]: cluster 2024-09-13T22:53:04.776863+0000 mgr.y (mgr.14613) 575 : cluster [DBG] pgmap v664: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-13T22:53:06.343 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:06 smithi192 bash[24608]: audit 2024-09-13T22:53:05.592374+0000 mgr.y (mgr.14613) 576 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:06.358 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:06.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:06.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:06.558 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:06.558 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:06.559 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:06.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:06.612 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:06.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:06.740 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:06.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:06.870 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:06.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:06.997 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:06.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:07.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:07.125 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:07.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:07.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:07.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:07.368 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:07 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.510805+0000 osd.7 (osd.7) 307 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:07.368 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:07 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.510914+0000 osd.7 (osd.7) 308 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:07.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:07 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.510805+0000 osd.7 (osd.7) 307 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:07.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:07 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.510914+0000 osd.7 (osd.7) 308 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:07.384 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:07.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:07 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.510805+0000 osd.7 (osd.7) 307 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:07.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:07 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.510914+0000 osd.7 (osd.7) 308 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:07.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:07.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:07.618 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:07.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:07.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:07.771 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:07.875 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:07.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:07.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:08.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:08.033 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:08.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:08.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.122881+0000 osd.0 (osd.0) 299 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:53:08.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.125211+0000 osd.0 (osd.0) 300 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.326510+0000 osd.2 (osd.2) 227 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.328886+0000 osd.2 (osd.2) 228 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.539141+0000 osd.5 (osd.5) 333 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.543714+0000 osd.5 (osd.5) 334 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:06.777552+0000 mgr.y (mgr.14613) 577 : cluster [DBG] pgmap v665: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:07.104947+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:07.113990+0000 osd.6 (osd.6) 251 : cluster [DBG] 3.a scrub starts 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:07.118838+0000 osd.6 (osd.6) 252 : cluster [DBG] 3.a scrub ok 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:07.643198+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[22840]: cluster 2024-09-13T22:53:07.644805+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:53:08.146 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.122881+0000 osd.0 (osd.0) 299 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.125211+0000 osd.0 (osd.0) 300 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.326510+0000 osd.2 (osd.2) 227 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.328886+0000 osd.2 (osd.2) 228 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.539141+0000 osd.5 (osd.5) 333 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.543714+0000 osd.5 (osd.5) 334 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:06.777552+0000 mgr.y (mgr.14613) 577 : cluster [DBG] pgmap v665: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:07.104947+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:07.113990+0000 osd.6 (osd.6) 251 : cluster [DBG] 3.a scrub starts 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:07.118838+0000 osd.6 (osd.6) 252 : cluster [DBG] 3.a scrub ok 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:07.643198+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:53:08.147 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:08 smithi178 bash[29889]: cluster 2024-09-13T22:53:07.644805+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:53:08.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:08.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:08.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:08.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:08.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.122881+0000 osd.0 (osd.0) 299 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.125211+0000 osd.0 (osd.0) 300 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.326510+0000 osd.2 (osd.2) 227 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.328886+0000 osd.2 (osd.2) 228 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.539141+0000 osd.5 (osd.5) 333 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.543714+0000 osd.5 (osd.5) 334 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:06.777552+0000 mgr.y (mgr.14613) 577 : cluster [DBG] pgmap v665: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:07.104947+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:07.113990+0000 osd.6 (osd.6) 251 : cluster [DBG] 3.a scrub starts 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:07.118838+0000 osd.6 (osd.6) 252 : cluster [DBG] 3.a scrub ok 2024-09-13T22:53:08.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:07.643198+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:53:08.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:08 smithi192 bash[24608]: cluster 2024-09-13T22:53:07.644805+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:53:08.416 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:08.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:08.546 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:08.588 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:08.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:08.671 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:08.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:08.801 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:08.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:08.932 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:08.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:09.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:09.060 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:09.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:09.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:09 smithi192 bash[24608]: cluster 2024-09-13T22:53:08.051053+0000 osd.3 (osd.3) 279 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:53:09.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:09 smithi192 bash[24608]: cluster 2024-09-13T22:53:08.052571+0000 osd.3 (osd.3) 280 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:53:09.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:09 smithi178 bash[22840]: cluster 2024-09-13T22:53:08.051053+0000 osd.3 (osd.3) 279 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:53:09.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:09 smithi178 bash[22840]: cluster 2024-09-13T22:53:08.052571+0000 osd.3 (osd.3) 280 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:53:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:09 smithi178 bash[29889]: cluster 2024-09-13T22:53:08.051053+0000 osd.3 (osd.3) 279 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:53:09.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:09 smithi178 bash[29889]: cluster 2024-09-13T22:53:08.052571+0000 osd.3 (osd.3) 280 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:53:09.191 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:09.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:09.322 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:09.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:09.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:09.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:09.583 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:09.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:09.713 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:09.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:09.843 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:09.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:53:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:09.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:09.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:09.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:10.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:10.104 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:10.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:10.234 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:10.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:08.128340+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:08.129823+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:08.778335+0000 mgr.y (mgr.14613) 578 : cluster [DBG] pgmap v667: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:09.013279+0000 osd.3 (osd.3) 281 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:09.015202+0000 osd.3 (osd.3) 282 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:09.598959+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[29889]: cluster 2024-09-13T22:53:09.601145+0000 osd.1 (osd.1) 291 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:53:10.350 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:08.128340+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:53:10.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:08.129823+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:53:10.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:08.778335+0000 mgr.y (mgr.14613) 578 : cluster [DBG] pgmap v667: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:53:10.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:09.013279+0000 osd.3 (osd.3) 281 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:53:10.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:09.015202+0000 osd.3 (osd.3) 282 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:53:10.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:09.598959+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:53:10.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:10 smithi178 bash[22840]: cluster 2024-09-13T22:53:09.601145+0000 osd.1 (osd.1) 291 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:53:10.366 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:10.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:08.128340+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:53:10.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:08.129823+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:53:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:08.778335+0000 mgr.y (mgr.14613) 578 : cluster [DBG] pgmap v667: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:53:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:09.013279+0000 osd.3 (osd.3) 281 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:53:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:09.015202+0000 osd.3 (osd.3) 282 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:53:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:09.598959+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:53:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:10 smithi192 bash[24608]: cluster 2024-09-13T22:53:09.601145+0000 osd.1 (osd.1) 291 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:53:10.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:10.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:10.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:10.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:10.630 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:10.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:10.756 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:10.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:10.881 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:10.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:10.961 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:53:10.961 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:53:10.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:11.007 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:11.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:11.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:11.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[22840]: cluster 2024-09-13T22:53:09.969187+0000 osd.3 (osd.3) 283 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[22840]: cluster 2024-09-13T22:53:09.971411+0000 osd.3 (osd.3) 284 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[22840]: cluster 2024-09-13T22:53:10.180876+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[22840]: cluster 2024-09-13T22:53:10.182756+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[29889]: cluster 2024-09-13T22:53:09.969187+0000 osd.3 (osd.3) 283 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[29889]: cluster 2024-09-13T22:53:09.971411+0000 osd.3 (osd.3) 284 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[29889]: cluster 2024-09-13T22:53:10.180876+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:53:11.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:11 smithi178 bash[29889]: cluster 2024-09-13T22:53:10.182756+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:53:11.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:11.267 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:11.371 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:11.395 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:11.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:11 smithi192 bash[24608]: cluster 2024-09-13T22:53:09.969187+0000 osd.3 (osd.3) 283 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:53:11.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:11 smithi192 bash[24608]: cluster 2024-09-13T22:53:09.971411+0000 osd.3 (osd.3) 284 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:53:11.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:11 smithi192 bash[24608]: cluster 2024-09-13T22:53:10.180876+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:53:11.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:11 smithi192 bash[24608]: cluster 2024-09-13T22:53:10.182756+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:53:11.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:11.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:11.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:11.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:11.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:11.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:11.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:11.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:12.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:12.047 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:12.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:12.179 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:12.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:12.303 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:12.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:12 smithi192 bash[24608]: cluster 2024-09-13T22:53:10.779862+0000 mgr.y (mgr.14613) 579 : cluster [DBG] pgmap v668: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:53:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:12 smithi192 bash[24608]: cluster 2024-09-13T22:53:10.949333+0000 osd.3 (osd.3) 285 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:53:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:12 smithi192 bash[24608]: cluster 2024-09-13T22:53:10.951007+0000 osd.3 (osd.3) 286 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:53:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:12 smithi192 bash[24608]: cluster 2024-09-13T22:53:11.228393+0000 osd.0 (osd.0) 305 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:53:12.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:12 smithi192 bash[24608]: cluster 2024-09-13T22:53:11.230780+0000 osd.0 (osd.0) 306 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:53:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[29889]: cluster 2024-09-13T22:53:10.779862+0000 mgr.y (mgr.14613) 579 : cluster [DBG] pgmap v668: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:53:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[29889]: cluster 2024-09-13T22:53:10.949333+0000 osd.3 (osd.3) 285 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:53:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[29889]: cluster 2024-09-13T22:53:10.951007+0000 osd.3 (osd.3) 286 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:53:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[29889]: cluster 2024-09-13T22:53:11.228393+0000 osd.0 (osd.0) 305 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:53:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[29889]: cluster 2024-09-13T22:53:11.230780+0000 osd.0 (osd.0) 306 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:53:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[22840]: cluster 2024-09-13T22:53:10.779862+0000 mgr.y (mgr.14613) 579 : cluster [DBG] pgmap v668: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T22:53:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[22840]: cluster 2024-09-13T22:53:10.949333+0000 osd.3 (osd.3) 285 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:53:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[22840]: cluster 2024-09-13T22:53:10.951007+0000 osd.3 (osd.3) 286 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:53:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[22840]: cluster 2024-09-13T22:53:11.228393+0000 osd.0 (osd.0) 305 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:53:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[22840]: cluster 2024-09-13T22:53:11.230780+0000 osd.0 (osd.0) 306 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:53:12.435 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:moved 15 / 426 (3.52113%) 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:avg 53.25 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:stddev 6.13901 -> 4.94343 (expected baseline 6.82596) 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:min osd.2 with 45 -> 48 pgs (0.84507 -> 0.901408 * mean) 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:max osd.7 with 65 -> 60 pgs (1.22066 -> 1.12676 * mean) 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:oload 110 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:max_change 1 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:max_change_osds 4 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:average_utilization 0.0047 2024-09-13T22:53:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:overload_utilization 0.0052 2024-09-13T22:53:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3 weight 1.0000 -> 0.7973 2024-09-13T22:53:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7 weight 1.0000 -> 0.9090 2024-09-13T22:53:12.500 INFO:teuthology.orchestra.run.smithi178.stderr:no change 2024-09-13T22:53:12.537 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:12.556 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:12.571 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:12 smithi178 bash[23095]: debug 2024-09-13T22:53:12.498+0000 7f089e894640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-13T22:53:12.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:12.677 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:12.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:12.805 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:12.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:12.936 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:12.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:13.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:13.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:13.075 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:53:13.075 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:53:13.091 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:53:13.092 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:53:13.107 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:53:13.107 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:53:13.124 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:53:13.125 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:53:13.129 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_3' created 2024-09-13T22:53:13.159 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:53:13.159 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:53:13.165 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:13.180 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:53:13.181 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:53:13.182 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:13.200 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:53:13.201 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:53:13.221 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:53:13.221 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:53:13.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:13.313 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.171055+0000 osd.6 (osd.6) 253 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.172743+0000 osd.6 (osd.6) 254 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.261011+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.262639+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.340333+0000 osd.4 (osd.4) 206 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.343416+0000 osd.4 (osd.4) 207 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: audit 2024-09-13T22:53:12.498497+0000 mgr.y (mgr.14613) 580 : audit [DBG] from='client.15597 v1:172.21.15.178:0/3139238474' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.607269+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.609430+0000 osd.1 (osd.1) 293 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:53:13.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: audit 2024-09-13T22:53:12.617136+0000 mon.a (mon.0) 1451 : audit [INF] from='client.? v1:172.21.15.178:0/314908269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[29889]: audit 2024-09-13T22:53:13.057834+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.178:0/1473803373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.171055+0000 osd.6 (osd.6) 253 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.172743+0000 osd.6 (osd.6) 254 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.261011+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.262639+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.340333+0000 osd.4 (osd.4) 206 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.343416+0000 osd.4 (osd.4) 207 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: audit 2024-09-13T22:53:12.498497+0000 mgr.y (mgr.14613) 580 : audit [DBG] from='client.15597 v1:172.21.15.178:0/3139238474' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.607269+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.609430+0000 osd.1 (osd.1) 293 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:53:13.373 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: audit 2024-09-13T22:53:12.617136+0000 mon.a (mon.0) 1451 : audit [INF] from='client.? v1:172.21.15.178:0/314908269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-13T22:53:13.374 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:13 smithi178 bash[22840]: audit 2024-09-13T22:53:13.057834+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.178:0/1473803373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.171055+0000 osd.6 (osd.6) 253 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.172743+0000 osd.6 (osd.6) 254 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.261011+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.262639+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.340333+0000 osd.4 (osd.4) 206 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.343416+0000 osd.4 (osd.4) 207 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: audit 2024-09-13T22:53:12.498497+0000 mgr.y (mgr.14613) 580 : audit [DBG] from='client.15597 v1:172.21.15.178:0/3139238474' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.607269+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.609430+0000 osd.1 (osd.1) 293 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: audit 2024-09-13T22:53:12.617136+0000 mon.a (mon.0) 1451 : audit [INF] from='client.? v1:172.21.15.178:0/314908269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-13T22:53:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:13 smithi192 bash[24608]: audit 2024-09-13T22:53:13.057834+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.178:0/1473803373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:13.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:13.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:13.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:13.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:13.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:13.703 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:13.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:13.832 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:13.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:13.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:13.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:14.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:14.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:14.206 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:12.780526+0000 mgr.y (mgr.14613) 581 : cluster [DBG] pgmap v669: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:53:14.206 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.026749+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:53:14.206 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.029280+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: audit 2024-09-13T22:53:13.131420+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? v1:172.21.15.178:0/314908269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.136365+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.217911+0000 osd.0 (osd.0) 309 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.220195+0000 osd.0 (osd.0) 310 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.657987+0000 osd.0 (osd.0) 311 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.658092+0000 osd.0 (osd.0) 312 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:12.780526+0000 mgr.y (mgr.14613) 581 : cluster [DBG] pgmap v669: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.026749+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.029280+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: audit 2024-09-13T22:53:13.131420+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? v1:172.21.15.178:0/314908269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.136365+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.217911+0000 osd.0 (osd.0) 309 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:53:14.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.220195+0000 osd.0 (osd.0) 310 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:53:14.208 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.657987+0000 osd.0 (osd.0) 311 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:14.208 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:14 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.658092+0000 osd.0 (osd.0) 312 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:14.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:14.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:14.351 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:12.780526+0000 mgr.y (mgr.14613) 581 : cluster [DBG] pgmap v669: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T22:53:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.026749+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.029280+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: audit 2024-09-13T22:53:13.131420+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? v1:172.21.15.178:0/314908269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.136365+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.217911+0000 osd.0 (osd.0) 309 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.220195+0000 osd.0 (osd.0) 310 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.657987+0000 osd.0 (osd.0) 311 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:14 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.658092+0000 osd.0 (osd.0) 312 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:14.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.482 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:14.586 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.614 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:14.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.740 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:14.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:14.869 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:14.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:14.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:15.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:15.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:15.131 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:15.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:15.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:15.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:15.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.312416+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:53:15.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:13.314486+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:53:15.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:14.083198+0000 osd.6 (osd.6) 255 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:53:15.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:14.085770+0000 osd.6 (osd.6) 256 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:53:15.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:14.194437+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-13T22:53:15.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:14.256400+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:53:15.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[24608]: cluster 2024-09-13T22:53:14.258470+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:53:15.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.312416+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:13.314486+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:14.083198+0000 osd.6 (osd.6) 255 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:14.085770+0000 osd.6 (osd.6) 256 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:14.194437+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:14.256400+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[22840]: cluster 2024-09-13T22:53:14.258470+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.312416+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:13.314486+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:53:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:14.083198+0000 osd.6 (osd.6) 255 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:53:15.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:14.085770+0000 osd.6 (osd.6) 256 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:53:15.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:14.194437+0000 mon.a (mon.0) 1454 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-13T22:53:15.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:14.256400+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:53:15.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:15 smithi178 bash[29889]: cluster 2024-09-13T22:53:14.258470+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:53:15.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:15.527 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:15.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:15.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:53:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:53:15.658 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:15.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:15.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:15.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T22:53:15.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:15.918 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:15.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:16.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:16.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:16.130 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T22:53:16.130 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-13T22:53:16.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:16.172 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:16.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:16.285 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[22840]: cluster 2024-09-13T22:53:14.781408+0000 mgr.y (mgr.14613) 582 : cluster [DBG] pgmap v672: 158 pgs: 12 creating+peering, 4 unknown, 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 666 B/s rd, 0 op/s 2024-09-13T22:53:16.285 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[22840]: cluster 2024-09-13T22:53:15.186094+0000 mon.a (mon.0) 1455 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:53:16.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[22840]: cluster 2024-09-13T22:53:15.203732+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-13T22:53:16.286 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[22840]: audit 2024-09-13T22:53:15.600598+0000 mgr.y (mgr.14613) 583 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:16.286 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[29889]: cluster 2024-09-13T22:53:14.781408+0000 mgr.y (mgr.14613) 582 : cluster [DBG] pgmap v672: 158 pgs: 12 creating+peering, 4 unknown, 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 666 B/s rd, 0 op/s 2024-09-13T22:53:16.286 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[29889]: cluster 2024-09-13T22:53:15.186094+0000 mon.a (mon.0) 1455 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:53:16.286 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[29889]: cluster 2024-09-13T22:53:15.203732+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-13T22:53:16.286 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:16 smithi178 bash[29889]: audit 2024-09-13T22:53:15.600598+0000 mgr.y (mgr.14613) 583 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:16.298 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:16.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:16 smithi192 bash[24608]: cluster 2024-09-13T22:53:14.781408+0000 mgr.y (mgr.14613) 582 : cluster [DBG] pgmap v672: 158 pgs: 12 creating+peering, 4 unknown, 142 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 666 B/s rd, 0 op/s 2024-09-13T22:53:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:16 smithi192 bash[24608]: cluster 2024-09-13T22:53:15.186094+0000 mon.a (mon.0) 1455 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:53:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:16 smithi192 bash[24608]: cluster 2024-09-13T22:53:15.203732+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-13T22:53:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:16 smithi192 bash[24608]: audit 2024-09-13T22:53:15.600598+0000 mgr.y (mgr.14613) 583 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:16.428 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:16.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:16.622 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-13T22:53:16.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:16.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:16.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:16.815 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:16.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:16.944 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:16.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:17.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:17.070 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:17.143 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:53:17.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:17.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:17.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[29889]: cluster 2024-09-13T22:53:15.748839+0000 osd.5 (osd.5) 335 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[29889]: cluster 2024-09-13T22:53:15.752974+0000 osd.5 (osd.5) 336 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[29889]: cluster 2024-09-13T22:53:16.297436+0000 osd.4 (osd.4) 208 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[29889]: cluster 2024-09-13T22:53:16.301393+0000 osd.4 (osd.4) 209 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[29889]: audit 2024-09-13T22:53:17.055179+0000 mon.a (mon.0) 1457 : audit [INF] from='client.? v1:172.21.15.178:0/1399932287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[22840]: cluster 2024-09-13T22:53:15.748839+0000 osd.5 (osd.5) 335 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:53:17.303 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[22840]: cluster 2024-09-13T22:53:15.752974+0000 osd.5 (osd.5) 336 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:53:17.304 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[22840]: cluster 2024-09-13T22:53:16.297436+0000 osd.4 (osd.4) 208 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:53:17.304 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[22840]: cluster 2024-09-13T22:53:16.301393+0000 osd.4 (osd.4) 209 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:53:17.304 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:17 smithi178 bash[22840]: audit 2024-09-13T22:53:17.055179+0000 mon.a (mon.0) 1457 : audit [INF] from='client.? v1:172.21.15.178:0/1399932287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:17.313 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:17.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:17 smithi192 bash[24608]: cluster 2024-09-13T22:53:15.748839+0000 osd.5 (osd.5) 335 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:53:17.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:17 smithi192 bash[24608]: cluster 2024-09-13T22:53:15.752974+0000 osd.5 (osd.5) 336 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:53:17.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:17 smithi192 bash[24608]: cluster 2024-09-13T22:53:16.297436+0000 osd.4 (osd.4) 208 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:53:17.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:17 smithi192 bash[24608]: cluster 2024-09-13T22:53:16.301393+0000 osd.4 (osd.4) 209 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:53:17.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:17 smithi192 bash[24608]: audit 2024-09-13T22:53:17.055179+0000 mon.a (mon.0) 1457 : audit [INF] from='client.? v1:172.21.15.178:0/1399932287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:17.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.446 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:17.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:17.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:17.574 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:17.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:17.706 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:17.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:17.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:17.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:17.963 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:17.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:18.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:18.099 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:18.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:18.227 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:18.330 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:18.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:16.775366+0000 osd.5 (osd.5) 337 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:16.776753+0000 osd.5 (osd.5) 338 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:16.782069+0000 mgr.y (mgr.14613) 584 : cluster [DBG] pgmap v674: 158 pgs: 12 creating+peering, 146 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:17.002553+0000 osd.6 (osd.6) 257 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:17.004068+0000 osd.6 (osd.6) 258 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:17.104471+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:17.105465+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:17.137450+0000 mon.a (mon.0) 1458 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: audit 2024-09-13T22:53:17.144701+0000 mon.a (mon.0) 1459 : audit [INF] from='client.? v1:172.21.15.178:0/1399932287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:53:18.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[29889]: cluster 2024-09-13T22:53:17.151077+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:16.775366+0000 osd.5 (osd.5) 337 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:16.776753+0000 osd.5 (osd.5) 338 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:16.782069+0000 mgr.y (mgr.14613) 584 : cluster [DBG] pgmap v674: 158 pgs: 12 creating+peering, 146 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:17.002553+0000 osd.6 (osd.6) 257 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:17.004068+0000 osd.6 (osd.6) 258 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:17.104471+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:17.105465+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:17.137450+0000 mon.a (mon.0) 1458 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: audit 2024-09-13T22:53:17.144701+0000 mon.a (mon.0) 1459 : audit [INF] from='client.? v1:172.21.15.178:0/1399932287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:53:18.360 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:18 smithi178 bash[22840]: cluster 2024-09-13T22:53:17.151077+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-13T22:53:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:16.775366+0000 osd.5 (osd.5) 337 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:16.776753+0000 osd.5 (osd.5) 338 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:16.782069+0000 mgr.y (mgr.14613) 584 : cluster [DBG] pgmap v674: 158 pgs: 12 creating+peering, 146 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:17.002553+0000 osd.6 (osd.6) 257 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:17.004068+0000 osd.6 (osd.6) 258 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:17.104471+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:17.105465+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:17.137450+0000 mon.a (mon.0) 1458 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: audit 2024-09-13T22:53:17.144701+0000 mon.a (mon.0) 1459 : audit [INF] from='client.? v1:172.21.15.178:0/1399932287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:53:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:18 smithi192 bash[24608]: cluster 2024-09-13T22:53:17.151077+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-13T22:53:18.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:18.490 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:18.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:18.615 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:18.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:53:18.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:18.744 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:18.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:18.875 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:18.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:18.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:19.004 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:19.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:19.133 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:19.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:19.264 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:19.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:19.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.097100+0000 osd.3 (osd.3) 291 : cluster [DBG] 6.f scrub starts 2024-09-13T22:53:19.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.098300+0000 osd.3 (osd.3) 292 : cluster [DBG] 6.f scrub ok 2024-09-13T22:53:19.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.222772+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:53:19.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.225017+0000 osd.4 (osd.4) 211 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:53:19.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.270769+0000 osd.0 (osd.0) 315 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:53:19.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.273538+0000 osd.0 (osd.0) 316 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.657651+0000 osd.1 (osd.1) 294 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.659091+0000 osd.1 (osd.1) 295 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: audit 2024-09-13T22:53:18.907219+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[29889]: audit 2024-09-13T22:53:18.911254+0000 mon.b (mon.1) 220 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.097100+0000 osd.3 (osd.3) 291 : cluster [DBG] 6.f scrub starts 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.098300+0000 osd.3 (osd.3) 292 : cluster [DBG] 6.f scrub ok 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.222772+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.225017+0000 osd.4 (osd.4) 211 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.270769+0000 osd.0 (osd.0) 315 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.273538+0000 osd.0 (osd.0) 316 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:53:19.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.657651+0000 osd.1 (osd.1) 294 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T22:53:19.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.659091+0000 osd.1 (osd.1) 295 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T22:53:19.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: audit 2024-09-13T22:53:18.907219+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:19.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[22840]: audit 2024-09-13T22:53:18.911254+0000 mon.b (mon.1) 220 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:19.391 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.097100+0000 osd.3 (osd.3) 291 : cluster [DBG] 6.f scrub starts 2024-09-13T22:53:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.098300+0000 osd.3 (osd.3) 292 : cluster [DBG] 6.f scrub ok 2024-09-13T22:53:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.222772+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:53:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.225017+0000 osd.4 (osd.4) 211 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:53:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.270769+0000 osd.0 (osd.0) 315 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:53:19.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.273538+0000 osd.0 (osd.0) 316 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:53:19.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.657651+0000 osd.1 (osd.1) 294 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T22:53:19.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.659091+0000 osd.1 (osd.1) 295 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T22:53:19.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: audit 2024-09-13T22:53:18.907219+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:19.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:19 smithi192 bash[24608]: audit 2024-09-13T22:53:18.911254+0000 mon.b (mon.1) 220 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:19.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:19.525 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:19.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:19.656 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:19.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:19.786 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:19.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:19.918 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:19.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:20.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:20.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:20.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:53:20.110 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:53:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:20.150 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:20.154 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.782915+0000 mgr.y (mgr.14613) 585 : cluster [DBG] pgmap v676: 158 pgs: 10 creating+peering, 148 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2024-09-13T22:53:20.155 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.801374+0000 osd.5 (osd.5) 339 : cluster [DBG] 5.15 deep-scrub starts 2024-09-13T22:53:20.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:20.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:20.301 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:20.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:20.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.782915+0000 mgr.y (mgr.14613) 585 : cluster [DBG] pgmap v676: 158 pgs: 10 creating+peering, 148 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2024-09-13T22:53:20.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.801374+0000 osd.5 (osd.5) 339 : cluster [DBG] 5.15 deep-scrub starts 2024-09-13T22:53:20.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:18.805168+0000 osd.5 (osd.5) 340 : cluster [DBG] 5.15 deep-scrub ok 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.090922+0000 osd.3 (osd.3) 293 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.093497+0000 osd.3 (osd.3) 294 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.227006+0000 osd.4 (osd.4) 212 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.230861+0000 osd.4 (osd.4) 213 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.249932+0000 osd.0 (osd.0) 317 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.251994+0000 osd.0 (osd.0) 318 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.580641+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:53:20.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:20 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.583326+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:18.805168+0000 osd.5 (osd.5) 340 : cluster [DBG] 5.15 deep-scrub ok 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.090922+0000 osd.3 (osd.3) 293 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.093497+0000 osd.3 (osd.3) 294 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.227006+0000 osd.4 (osd.4) 212 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.230861+0000 osd.4 (osd.4) 213 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.249932+0000 osd.0 (osd.0) 317 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.251994+0000 osd.0 (osd.0) 318 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.580641+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.583326+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:53:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.782915+0000 mgr.y (mgr.14613) 585 : cluster [DBG] pgmap v676: 158 pgs: 10 creating+peering, 148 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.801374+0000 osd.5 (osd.5) 339 : cluster [DBG] 5.15 deep-scrub starts 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:18.805168+0000 osd.5 (osd.5) 340 : cluster [DBG] 5.15 deep-scrub ok 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.090922+0000 osd.3 (osd.3) 293 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.093497+0000 osd.3 (osd.3) 294 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.227006+0000 osd.4 (osd.4) 212 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.230861+0000 osd.4 (osd.4) 213 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.249932+0000 osd.0 (osd.0) 317 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.251994+0000 osd.0 (osd.0) 318 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.580641+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:53:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:20 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.583326+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:53:20.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:20.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:20.563 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:20.632 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:20.632 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:20.633 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:20.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:20.695 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:20.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:20.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:20.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:20.953 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:20.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:21.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:21.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:21.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:21.211 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:21.314 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:21.342 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:21.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:21 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.396882+0000 osd.2 (osd.2) 231 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:53:21.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:21 smithi192 bash[24608]: cluster 2024-09-13T22:53:19.398770+0000 osd.2 (osd.2) 232 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:53:21.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:21 smithi192 bash[24608]: cluster 2024-09-13T22:53:20.237178+0000 osd.4 (osd.4) 214 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:53:21.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:21 smithi192 bash[24608]: cluster 2024-09-13T22:53:20.242098+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:53:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.396882+0000 osd.2 (osd.2) 231 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:53:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[29889]: cluster 2024-09-13T22:53:19.398770+0000 osd.2 (osd.2) 232 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:53:21.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[29889]: cluster 2024-09-13T22:53:20.237178+0000 osd.4 (osd.4) 214 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:53:21.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[29889]: cluster 2024-09-13T22:53:20.242098+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:53:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.396882+0000 osd.2 (osd.2) 231 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:53:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[22840]: cluster 2024-09-13T22:53:19.398770+0000 osd.2 (osd.2) 232 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:53:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[22840]: cluster 2024-09-13T22:53:20.237178+0000 osd.4 (osd.4) 214 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:53:21.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:21 smithi178 bash[22840]: cluster 2024-09-13T22:53:20.242098+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:53:21.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:21.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:21.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:21.603 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:21.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:21.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:21.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:21.867 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:21.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:21.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:21.998 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:22.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:22.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:22.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:22.259 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:22.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:22.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:22 smithi178 bash[29889]: cluster 2024-09-13T22:53:20.784803+0000 mgr.y (mgr.14613) 586 : cluster [DBG] pgmap v677: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:53:22.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:22 smithi178 bash[29889]: cluster 2024-09-13T22:53:21.562095+0000 osd.7 (osd.7) 311 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:53:22.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:22 smithi178 bash[29889]: cluster 2024-09-13T22:53:21.566714+0000 osd.7 (osd.7) 312 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:53:22.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:22 smithi178 bash[22840]: cluster 2024-09-13T22:53:20.784803+0000 mgr.y (mgr.14613) 586 : cluster [DBG] pgmap v677: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:53:22.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:22 smithi178 bash[22840]: cluster 2024-09-13T22:53:21.562095+0000 osd.7 (osd.7) 311 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:53:22.378 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:22 smithi178 bash[22840]: cluster 2024-09-13T22:53:21.566714+0000 osd.7 (osd.7) 312 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:53:22.391 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:22.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:22.522 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:22.625 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:22.651 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:22 smithi192 bash[24608]: cluster 2024-09-13T22:53:20.784803+0000 mgr.y (mgr.14613) 586 : cluster [DBG] pgmap v677: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:53:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:22 smithi192 bash[24608]: cluster 2024-09-13T22:53:21.562095+0000 osd.7 (osd.7) 311 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T22:53:22.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:22 smithi192 bash[24608]: cluster 2024-09-13T22:53:21.566714+0000 osd.7 (osd.7) 312 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T22:53:22.672 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:22.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:22.783 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:22.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:22.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:22.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:23.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:23.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:23.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:23.169 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:23.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:23.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:23 smithi192 bash[24608]: cluster 2024-09-13T22:53:22.231337+0000 osd.0 (osd.0) 319 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T22:53:23.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:23 smithi192 bash[24608]: cluster 2024-09-13T22:53:22.232801+0000 osd.0 (osd.0) 320 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T22:53:23.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:23 smithi192 bash[24608]: cluster 2024-09-13T22:53:22.524541+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:53:23.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:23 smithi192 bash[24608]: cluster 2024-09-13T22:53:22.528038+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:53:23.300 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:23.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[22840]: cluster 2024-09-13T22:53:22.231337+0000 osd.0 (osd.0) 319 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T22:53:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[22840]: cluster 2024-09-13T22:53:22.232801+0000 osd.0 (osd.0) 320 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T22:53:23.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[22840]: cluster 2024-09-13T22:53:22.524541+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:53:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[22840]: cluster 2024-09-13T22:53:22.528038+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:53:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[29889]: cluster 2024-09-13T22:53:22.231337+0000 osd.0 (osd.0) 319 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T22:53:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[29889]: cluster 2024-09-13T22:53:22.232801+0000 osd.0 (osd.0) 320 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T22:53:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[29889]: cluster 2024-09-13T22:53:22.524541+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.1c scrub starts 2024-09-13T22:53:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:23 smithi178 bash[29889]: cluster 2024-09-13T22:53:22.528038+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.1c scrub ok 2024-09-13T22:53:23.431 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:23.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:23.561 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:23.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:23.691 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:23.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:23.820 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:23.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:23.950 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:23.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:24.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:24.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:24.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:24.212 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:24.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:24.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:24 smithi192 bash[24608]: cluster 2024-09-13T22:53:22.785493+0000 mgr.y (mgr.14613) 587 : cluster [DBG] pgmap v678: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:53:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:24 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.115040+0000 osd.3 (osd.3) 295 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:53:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:24 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.119011+0000 osd.3 (osd.3) 296 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:53:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:24 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.618426+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.c scrub starts 2024-09-13T22:53:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:24 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.620580+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.c scrub ok 2024-09-13T22:53:24.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[22840]: cluster 2024-09-13T22:53:22.785493+0000 mgr.y (mgr.14613) 587 : cluster [DBG] pgmap v678: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.115040+0000 osd.3 (osd.3) 295 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.119011+0000 osd.3 (osd.3) 296 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.618426+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.c scrub starts 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.620580+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.c scrub ok 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[29889]: cluster 2024-09-13T22:53:22.785493+0000 mgr.y (mgr.14613) 587 : cluster [DBG] pgmap v678: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T22:53:24.457 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.115040+0000 osd.3 (osd.3) 295 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:53:24.458 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.119011+0000 osd.3 (osd.3) 296 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:53:24.458 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.618426+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.c scrub starts 2024-09-13T22:53:24.458 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:24 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.620580+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.c scrub ok 2024-09-13T22:53:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:24.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:24.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:24.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:24.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:24.718 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:24.727 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:24.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:24.857 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:24.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:24.987 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:24.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:25.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:25.118 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:25.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:25.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:25.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:25.381 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:25.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:25.495 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.836111+0000 osd.5 (osd.5) 341 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.838490+0000 osd.5 (osd.5) 342 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.960982+0000 osd.6 (osd.6) 259 : cluster [DBG] 5.1f deep-scrub starts 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:23.962456+0000 osd.6 (osd.6) 260 : cluster [DBG] 5.1f deep-scrub ok 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.237892+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.238936+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.633435+0000 osd.1 (osd.1) 298 : cluster [DBG] 2.a scrub starts 2024-09-13T22:53:25.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.635704+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.a scrub ok 2024-09-13T22:53:25.511 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:25.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:25.625 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.836111+0000 osd.5 (osd.5) 341 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.838490+0000 osd.5 (osd.5) 342 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.960982+0000 osd.6 (osd.6) 259 : cluster [DBG] 5.1f deep-scrub starts 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:23.962456+0000 osd.6 (osd.6) 260 : cluster [DBG] 5.1f deep-scrub ok 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.237892+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.238936+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.633435+0000 osd.1 (osd.1) 298 : cluster [DBG] 2.a scrub starts 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.635704+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.a scrub ok 2024-09-13T22:53:25.626 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.836111+0000 osd.5 (osd.5) 341 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.838490+0000 osd.5 (osd.5) 342 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.960982+0000 osd.6 (osd.6) 259 : cluster [DBG] 5.1f deep-scrub starts 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:23.962456+0000 osd.6 (osd.6) 260 : cluster [DBG] 5.1f deep-scrub ok 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.237892+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.238936+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.633435+0000 osd.1 (osd.1) 298 : cluster [DBG] 2.a scrub starts 2024-09-13T22:53:25.627 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:25 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.635704+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.a scrub ok 2024-09-13T22:53:25.640 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:25.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:25.771 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:25.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:25.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:25.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:53:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:53:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:25.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:26.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:26.028 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 20, 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 600, 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.24767.0:170885 7.e 7:7901dda1:::benchmark_data_smithi178_64772_object65132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:53:26.060 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:43:58.130213+0000", 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 567.92871724600002, 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37978784999999998, 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.24767", 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 170885 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.061 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.062 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.130213+0000", 2024-09-13T22:53:26.064 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.130213+0000", 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.130214+0000", 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2979999999999999e-06 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.065 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.130218+0000", 2024-09-13T22:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0620000000000002e-06 2024-09-13T22:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.130227+0000", 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4279999999999996e-06 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.130233+0000", 2024-09-13T22:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0050000000000001e-06 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.132855+0000", 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0026227450000000001 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.132953+0000", 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7277999999999999e-05 2024-09-13T22:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,5", 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.133055+0000", 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010266400000000001 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.146288+0000", 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013233043 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.146295+0000", 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9659999999999999e-06 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.509600+0000", 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.36330511999999998 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.509875+0000", 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00027439000000000001 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.510001+0000", 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012585100000000001 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.071 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.24767.0:170887 7.e 7:71e5cf30:::benchmark_data_smithi178_64772_object65134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:43:58.131428+0000", 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 567.92750237799999, 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37888057200000003, 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.24767", 2024-09-13T22:53:26.072 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 170887 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.131428+0000", 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.073 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.131428+0000", 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.131429+0000", 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.702e-06 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.131434+0000", 2024-09-13T22:53:26.074 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8679999999999998e-06 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.131445+0000", 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.079e-05 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.075 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.131452+0000", 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9500000000000004e-06 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.143052+0000", 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011600014000000001 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.143166+0000", 2024-09-13T22:53:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000114098 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,5", 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.143286+0000", 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000120229 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.149794+0000", 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0065081280000000002 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.077 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.149888+0000", 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3962000000000006e-05 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.510137+0000", 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.360249123 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.078 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.510234+0000", 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6776999999999999e-05 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.510308+0000", 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3930999999999994e-05 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.079 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.24767.0:170907 7.3 e160/116 7:ca64b79b:::benchmark_data_smithi178_64772_object65154:head v 160'10664, mlcod=160'10662)", 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:43:58.155029+0000", 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 567.90390144100002, 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.36428608899999998, 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.155029+0000", 2024-09-13T22:53:26.080 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.155029+0000", 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.155029+0000", 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3500000000000005e-07 2024-09-13T22:53:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.155035+0000", 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9839999999999998e-06 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.155057+0000", 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1484999999999998e-05 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.155063+0000", 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.699e-06 2024-09-13T22:53:26.083 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.503999+0000", 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.34893670599999999 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.504040+0000", 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0954999999999997e-05 2024-09-13T22:53:26.084 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.519251+0000", 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.015211066000000001 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.085 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:43:58.519315+0000", 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3359000000000001e-05 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.24767.0:189700 7.e 7:76360c71:::benchmark_data_smithi178_64772_object83947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:44:23.329951+0000", 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 542.72897942700001, 2024-09-13T22:53:26.086 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37754470499999998, 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.24767", 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/2830980724", 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 189700 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.087 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.329951+0000", 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.329951+0000", 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.329953+0000", 2024-09-13T22:53:26.088 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3149999999999999e-06 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.329959+0000", 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.178e-06 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.089 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.329973+0000", 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3789e-05 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.329981+0000", 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.1370000000000002e-06 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.330042+0000", 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0871e-05 2024-09-13T22:53:26.090 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.330145+0000", 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000103508 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,5", 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.330219+0000", 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3356999999999996e-05 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.091 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.338210+0000", 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0079908410000000003 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:53:26.092 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.338874+0000", 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00066458800000000003 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.707221+0000", 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.36834707100000003 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.707377+0000", 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00015538300000000001 2024-09-13T22:53:26.093 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:44:23.707495+0000", 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011866699999999999 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.094 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:63752 8.f e193/175 8:ff9242a2:::benchmark_data_smithi178_80607_object63751:head v 193'3982, mlcod=193'3979)", 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:00.506435+0000", 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 445.55249457000002, 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.38840376399999998, 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.095 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506435+0000", 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506435+0000", 2024-09-13T22:53:26.096 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506437+0000", 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.446e-06 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506458+0000", 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1475999999999999e-05 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506467+0000", 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4239999999999993e-06 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506473+0000", 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0190000000000003e-06 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.098 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506528+0000", 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5114999999999997e-05 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.506566+0000", 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.7728000000000002e-05 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.894722+0000", 2024-09-13T22:53:26.099 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.38815654100000002 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.894839+0000", 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000117015 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.100 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:63753 8.a e193/175 8:57f429b0:::benchmark_data_smithi178_80607_object63752:head v 193'4003, mlcod=193'4001)", 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:00.508738+0000", 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 445.55019180699998, 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37877365800000001, 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508738+0000", 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.101 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508738+0000", 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508740+0000", 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.531e-06 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.102 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508766+0000", 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.5865000000000001e-05 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508786+0000", 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0581000000000002e-05 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.103 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508794+0000", 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4599999999999997e-06 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508876+0000", 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2379000000000005e-05 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.508897+0000", 2024-09-13T22:53:26.104 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0658000000000001e-05 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511473+0000", 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0025766529999999999 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.887512+0000", 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37603853100000001 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.105 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:63754 8.f e193/175 8:f032d197:::benchmark_data_smithi178_80607_object63753:head v 193'3983, mlcod=193'3979)", 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:00.511184+0000", 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 445.54774567300001, 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.38829725599999998, 2024-09-13T22:53:26.106 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511184+0000", 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.107 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511184+0000", 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511186+0000", 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7370000000000001e-06 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.108 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511213+0000", 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.7104999999999999e-05 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511225+0000", 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1438e-05 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.109 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511232+0000", 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7870000000000002e-06 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.886866+0000", 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.375633885 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.886963+0000", 2024-09-13T22:53:26.110 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6261000000000006e-05 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.899379+0000", 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.012416177 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.899482+0000", 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000102866 2024-09-13T22:53:26.111 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.112 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:63755 8.1 e193/175 8:872fe947:::benchmark_data_smithi178_80607_object63754:head v 193'4088, mlcod=193'4086)", 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:00.511192+0000", 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 445.54773811299998, 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.39598510999999997, 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511192+0000", 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511192+0000", 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511193+0000", 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3120000000000001e-06 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.114 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511217+0000", 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3368000000000001e-05 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511226+0000", 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4539999999999996e-06 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511233+0000", 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9659999999999999e-06 2024-09-13T22:53:26.115 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.511380+0000", 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014672099999999999 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.887079+0000", 2024-09-13T22:53:26.116 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37569888299999998 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.907080+0000", 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.020001894999999999 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.907177+0000", 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6510999999999998e-05 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.117 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:63764 8.e e193/175 8:738058ca:::benchmark_data_smithi178_80607_object63763:head v 193'4033, mlcod=193'4031)", 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:00.512457+0000", 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 445.54647269999998, 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.38403335300000002, 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.118 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512457+0000", 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512457+0000", 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.119 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512459+0000", 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2500000000000001e-06 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512480+0000", 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1112999999999999e-05 2024-09-13T22:53:26.120 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.887023+0000", 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37454313299999997 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.121 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.887049+0000", 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6489e-05 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.887609+0000", 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00056013999999999999 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.122 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.887677+0000", 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7243000000000007e-05 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.896399+0000", 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0087221610000000008 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.896491+0000", 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1823999999999994e-05 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.123 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25037.0:63758 8.1 e193/175 8:8169c6f8:::benchmark_data_smithi178_80607_object63757:head v 193'4089, mlcod=193'4086)", 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:46:00.512851+0000", 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 445.54607920900003, 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.394354345, 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.124 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512851+0000", 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512851+0000", 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512852+0000", 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.294e-06 2024-09-13T22:53:26.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.512874+0000", 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2395000000000002e-05 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.886749+0000", 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.37387458499999998 2024-09-13T22:53:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.886783+0000", 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.3705999999999999e-05 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.891242+0000", 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0044597059999999999 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.891291+0000", 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8838999999999998e-05 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.907182+0000", 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.015890975000000002 2024-09-13T22:53:26.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:46:00.907205+0000", 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2844999999999999e-05 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:167960 9.0 e308/276 9:006e917c:::benchmark_data_smithi178_96665_object68798:head v 308'7864, mlcod=308'7862)", 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:15.101364+0000", 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 70.957566017999994, 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013882807, 2024-09-13T22:53:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.101364+0000", 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.101364+0000", 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.101365+0000", 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0240000000000001e-06 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.101373+0000", 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7589999999999997e-06 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.101381+0000", 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4249999999999998e-06 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.131 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.101387+0000", 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6370000000000004e-06 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.104224+0000", 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00283731 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.104246+0000", 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1457000000000001e-05 2024-09-13T22:53:26.132 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.115133+0000", 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010887342 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:15.115247+0000", 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000113853 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.133 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:171440 9.6 e309/276 9:67b6f9d5:::benchmark_data_smithi178_96665_object72278:head v 309'8065, mlcod=309'8063)", 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:18.420513+0000", 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 67.638416558000003, 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014998457, 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420513+0000", 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.134 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420513+0000", 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420516+0000", 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.881e-06 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420526+0000", 2024-09-13T22:53:26.135 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.9539999999999999e-06 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420544+0000", 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7411e-05 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420554+0000", 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0389e-05 2024-09-13T22:53:26.136 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420665+0000", 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000111279 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.137 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.420727+0000", 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1790000000000003e-05 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.435321+0000", 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014593538 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:18.435512+0000", 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000191215 2024-09-13T22:53:26.138 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177385 9.16 e309/285 9:6b1640ea:::benchmark_data_smithi178_96665_object78223:head v 309'8238, mlcod=309'8236)", 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.613261+0000", 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.445668593999997, 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.017195999, 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.139 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613261+0000", 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613261+0000", 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.140 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613263+0000", 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.84e-06 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613276+0000", 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2286000000000001e-05 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.141 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613295+0000", 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9513000000000001e-05 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613304+0000", 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.6209999999999993e-06 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.142 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613678+0000", 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00037459899999999998 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613724+0000", 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6161000000000001e-05 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.630421+0000", 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.016696810999999999 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.630457+0000", 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6168000000000003e-05 2024-09-13T22:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177388 9.6 e309/276 9:670f2893:::benchmark_data_smithi178_96665_object78226:head v 309'8244, mlcod=309'8242)", 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.613334+0000", 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.445595769999997, 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.017151454999999999, 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.145 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613334+0000", 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613334+0000", 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613335+0000", 2024-09-13T22:53:26.146 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6099999999999999e-07 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613341+0000", 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.7749999999999998e-06 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613347+0000", 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9460000000000003e-06 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.147 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.613352+0000", 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8899999999999998e-06 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.616692+0000", 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0033402560000000002 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.616885+0000", 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00019304399999999999 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.630461+0000", 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013576423000000001 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.630486+0000", 2024-09-13T22:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4159999999999999e-05 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177391 9.16 e309/285 9:6d3e9e01:::benchmark_data_smithi178_96665_object78229:head v 309'8239, mlcod=309'8236)", 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.614799+0000", 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.444131417000001, 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.015712081999999999, 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.150 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.614799+0000", 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.614799+0000", 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.151 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.614800+0000", 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3149999999999999e-06 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.614808+0000", 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4400000000000005e-06 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.152 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.614820+0000", 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1433e-05 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.614826+0000", 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3319999999999997e-06 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.617102+0000", 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0022755280000000002 2024-09-13T22:53:26.153 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.617206+0000", 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000104187 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.630488+0000", 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.01328173 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.630511+0000", 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3116999999999999e-05 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.155 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177726 9.2 e309/276 9:446feb85:::benchmark_data_smithi178_96665_object78564:head v 309'7957, mlcod=309'7955)", 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.879846+0000", 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.179084099999997, 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.055967199000000002, 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879846+0000", 2024-09-13T22:53:26.156 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879846+0000", 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879848+0000", 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.823e-06 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879856+0000", 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9470000000000003e-06 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879867+0000", 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1640999999999999e-05 2024-09-13T22:53:26.158 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879876+0000", 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2709999999999992e-06 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.159 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879932+0000", 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6159000000000002e-05 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879951+0000", 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9613e-05 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.160 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935770+0000", 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.055818948 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935813+0000", 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2796999999999997e-05 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.161 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177715 9.16 e309/285 9:6924d7e6:::benchmark_data_smithi178_96665_object78553:head v 309'8254, mlcod=309'8251)", 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.879904+0000", 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.179025639000002, 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.067791792000000003, 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879904+0000", 2024-09-13T22:53:26.162 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879904+0000", 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879906+0000", 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.587e-06 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.163 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879916+0000", 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0518e-05 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879925+0000", 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.0110000000000006e-06 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879932+0000", 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0619999999999997e-06 2024-09-13T22:53:26.164 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.879987+0000", 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5838000000000003e-05 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.880007+0000", 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9419000000000002e-05 2024-09-13T22:53:26.165 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.947656+0000", 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.067649036999999995 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.947696+0000", 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.032e-05 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.166 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177728 9.2 e309/276 9:4165320f:::benchmark_data_smithi178_96665_object78566:head v 309'7958, mlcod=309'7955)", 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.882608+0000", 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.176322429999999, 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.058661034000000001, 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.882608+0000", 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.167 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.882608+0000", 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.882609+0000", 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6130000000000001e-06 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.168 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.882616+0000", 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7270000000000001e-06 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935762+0000", 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.053145695 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.169 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935768+0000", 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0070000000000003e-06 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935844+0000", 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6607999999999996e-05 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935865+0000", 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1047999999999999e-05 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.170 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.941238+0000", 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0053730280000000002 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.941269+0000", 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.0307999999999998e-05 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.171 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:177745 9.16 e309/285 9:6fa12bcd:::benchmark_data_smithi178_96665_object78583:head v 309'8255, mlcod=309'8252)", 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:23.895093+0000", 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 62.163837372000003, 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.052633289, 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.172 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.173 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.173 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.895093+0000", 2024-09-13T22:53:26.173 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.173 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.173 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.173 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.895093+0000", 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.895096+0000", 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.9100000000000001e-06 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.895108+0000", 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2918999999999999e-05 2024-09-13T22:53:26.174 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935752+0000", 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.040643948999999999 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.935764+0000", 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1909999999999999e-05 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.175 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.937815+0000", 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0020506919999999998 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.937848+0000", 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.3167000000000003e-05 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.947700+0000", 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0098519560000000003 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.176 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:23.947726+0000", 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.5786e-05 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.15408.0:182960 9.16 e309/285 9:6c99b46f:::benchmark_data_smithi178_96665_object83798:head v 309'8414, mlcod=309'8412)", 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:52:28.614731+0000", 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 57.444198612999998, 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014963028, 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:53:26.177 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.614731+0000", 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.178 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.614731+0000", 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.614737+0000", 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3600000000000004e-06 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.614757+0000", 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0713e-05 2024-09-13T22:53:26.179 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.614801+0000", 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.3105999999999997e-05 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.614824+0000", 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3112000000000002e-05 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:53:26.180 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.615138+0000", 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00031416899999999999 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.615191+0000", 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3170000000000001e-05 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.629620+0000", 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014428942 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:53:26.181 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:52:28.629694+0000", 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4456000000000007e-05 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:53:26.182 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:53:26.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:26.211 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:26.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:26.344 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:26.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:26.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.786463+0000 mgr.y (mgr.14613) 588 : cluster [DBG] pgmap v679: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:53:26.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.806063+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:53:26.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:24.808248+0000 osd.5 (osd.5) 344 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:53:26.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.177285+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.f scrub starts 2024-09-13T22:53:26.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.180761+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub ok 2024-09-13T22:53:26.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: audit 2024-09-13T22:53:25.608688+0000 mgr.y (mgr.14613) 589 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:26.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.672939+0000 osd.1 (osd.1) 300 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:53:26.459 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:26 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.676167+0000 osd.1 (osd.1) 301 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:53:26.470 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.786463+0000 mgr.y (mgr.14613) 588 : cluster [DBG] pgmap v679: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.806063+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:24.808248+0000 osd.5 (osd.5) 344 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.177285+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.f scrub starts 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.180761+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub ok 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: audit 2024-09-13T22:53:25.608688+0000 mgr.y (mgr.14613) 589 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.672939+0000 osd.1 (osd.1) 300 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.676167+0000 osd.1 (osd.1) 301 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:53:26.471 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.786463+0000 mgr.y (mgr.14613) 588 : cluster [DBG] pgmap v679: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.806063+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:24.808248+0000 osd.5 (osd.5) 344 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.177285+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.f scrub starts 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.180761+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub ok 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: audit 2024-09-13T22:53:25.608688+0000 mgr.y (mgr.14613) 589 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.672939+0000 osd.1 (osd.1) 300 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:53:26.472 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:26 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.676167+0000 osd.1 (osd.1) 301 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:53:26.475 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:26.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:26.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:26.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:26.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:26.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:26.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:26.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:26.994 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:26.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:27.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:27.123 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:27.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:27.254 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:27.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:27.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.765081+0000 osd.5 (osd.5) 345 : cluster [DBG] 1.a scrub starts 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.767330+0000 osd.5 (osd.5) 346 : cluster [DBG] 1.a scrub ok 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.929424+0000 osd.6 (osd.6) 261 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:25.931935+0000 osd.6 (osd.6) 262 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.148605+0000 osd.4 (osd.4) 218 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.150948+0000 osd.4 (osd.4) 219 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.614435+0000 osd.7 (osd.7) 315 : cluster [DBG] 5.d scrub starts 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.617440+0000 osd.7 (osd.7) 316 : cluster [DBG] 5.d scrub ok 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.765081+0000 osd.5 (osd.5) 345 : cluster [DBG] 1.a scrub starts 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.767330+0000 osd.5 (osd.5) 346 : cluster [DBG] 1.a scrub ok 2024-09-13T22:53:27.369 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.929424+0000 osd.6 (osd.6) 261 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:53:27.370 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:25.931935+0000 osd.6 (osd.6) 262 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:53:27.370 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.148605+0000 osd.4 (osd.4) 218 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:53:27.370 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.150948+0000 osd.4 (osd.4) 219 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:53:27.370 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.614435+0000 osd.7 (osd.7) 315 : cluster [DBG] 5.d scrub starts 2024-09-13T22:53:27.370 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:27 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.617440+0000 osd.7 (osd.7) 316 : cluster [DBG] 5.d scrub ok 2024-09-13T22:53:27.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:27.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:27.499 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.765081+0000 osd.5 (osd.5) 345 : cluster [DBG] 1.a scrub starts 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.767330+0000 osd.5 (osd.5) 346 : cluster [DBG] 1.a scrub ok 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.929424+0000 osd.6 (osd.6) 261 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:25.931935+0000 osd.6 (osd.6) 262 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.148605+0000 osd.4 (osd.4) 218 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.150948+0000 osd.4 (osd.4) 219 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.614435+0000 osd.7 (osd.7) 315 : cluster [DBG] 5.d scrub starts 2024-09-13T22:53:27.500 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:27 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.617440+0000 osd.7 (osd.7) 316 : cluster [DBG] 5.d scrub ok 2024-09-13T22:53:27.516 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:27.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:27.647 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:27.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:27.771 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:27.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:27.901 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:27.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:28.001 INFO:teuthology.orchestra.run.smithi178.stdout:moved 0 / 474 (0%) 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:avg 59.25 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:stddev 6.31961 -> 6.31961 (expected baseline 7.20026) 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:min osd.1 with 51 -> 51 pgs (0.860759 -> 0.860759 * mean) 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:max osd.7 with 69 -> 69 pgs (1.16456 -> 1.16456 * mean) 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:oload 1000 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:max_change 3 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:max_change_osds 4 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:average_utilization 0.0047 2024-09-13T22:53:28.002 INFO:teuthology.orchestra.run.smithi178.stdout:overload_utilization 0.0467 2024-09-13T22:53:28.003 INFO:teuthology.orchestra.run.smithi178.stderr:no change 2024-09-13T22:53:28.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:28.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:28.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:28.153 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:28.197 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-13T22:53:28.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:28.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.787340+0000 mgr.y (mgr.14613) 590 : cluster [DBG] pgmap v680: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:28.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.913063+0000 osd.6 (osd.6) 263 : cluster [DBG] 2.11 deep-scrub starts 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:26.915091+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.11 deep-scrub ok 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:27.081430+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:27.084346+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:27.176396+0000 osd.4 (osd.4) 220 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:27.178655+0000 osd.4 (osd.4) 221 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:27.656983+0000 osd.7 (osd.7) 317 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: cluster 2024-09-13T22:53:27.660036+0000 osd.7 (osd.7) 318 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.787340+0000 mgr.y (mgr.14613) 590 : cluster [DBG] pgmap v680: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.913063+0000 osd.6 (osd.6) 263 : cluster [DBG] 2.11 deep-scrub starts 2024-09-13T22:53:28.267 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:26.915091+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.11 deep-scrub ok 2024-09-13T22:53:28.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:27.081430+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:53:28.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:27.084346+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:53:28.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:27.176396+0000 osd.4 (osd.4) 220 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:53:28.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:27.178655+0000 osd.4 (osd.4) 221 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:53:28.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:27.656983+0000 osd.7 (osd.7) 317 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T22:53:28.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: cluster 2024-09-13T22:53:27.660036+0000 osd.7 (osd.7) 318 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T22:53:28.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:28.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.787340+0000 mgr.y (mgr.14613) 590 : cluster [DBG] pgmap v680: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.913063+0000 osd.6 (osd.6) 263 : cluster [DBG] 2.11 deep-scrub starts 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:26.915091+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.11 deep-scrub ok 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:27.081430+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:27.084346+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:27.176396+0000 osd.4 (osd.4) 220 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:27.178655+0000 osd.4 (osd.4) 221 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:27.656983+0000 osd.7 (osd.7) 317 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: cluster 2024-09-13T22:53:27.660036+0000 osd.7 (osd.7) 318 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T22:53:28.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:28 smithi192 bash[24608]: audit 2024-09-13T22:53:27.874624+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? v1:172.21.15.178:0/3147890141' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:53:28.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:28.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:28.522 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[29889]: audit 2024-09-13T22:53:27.874624+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? v1:172.21.15.178:0/3147890141' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:53:28.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:28 smithi178 bash[22840]: audit 2024-09-13T22:53:27.874624+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? v1:172.21.15.178:0/3147890141' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:53:28.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:28.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:28.666 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:28.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:28.796 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:28.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:28.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:29.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:29.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:29.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:29.180 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:29.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:29.306 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:29.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:29.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:29 smithi192 bash[24608]: audit 2024-09-13T22:53:28.001177+0000 mgr.y (mgr.14613) 591 : audit [DBG] from='client.25517 v1:172.21.15.178:0/3019413900' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:53:29.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:29 smithi192 bash[24608]: audit 2024-09-13T22:53:28.198416+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? v1:172.21.15.178:0/3147890141' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:53:29.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:29 smithi192 bash[24608]: cluster 2024-09-13T22:53:28.202930+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-13T22:53:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:29.446 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:53:29.447 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:53:29.464 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:53:29.464 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:53:29.482 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:53:29.482 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:53:29.499 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:53:29.500 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:53:29.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T22:53:29.519 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:53:29.519 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:53:29.537 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:53:29.537 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:53:29.538 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[22840]: audit 2024-09-13T22:53:28.001177+0000 mgr.y (mgr.14613) 591 : audit [DBG] from='client.25517 v1:172.21.15.178:0/3019413900' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:53:29.539 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[22840]: audit 2024-09-13T22:53:28.198416+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? v1:172.21.15.178:0/3147890141' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:53:29.539 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[22840]: cluster 2024-09-13T22:53:28.202930+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-13T22:53:29.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[29889]: audit 2024-09-13T22:53:28.001177+0000 mgr.y (mgr.14613) 591 : audit [DBG] from='client.25517 v1:172.21.15.178:0/3019413900' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T22:53:29.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[29889]: audit 2024-09-13T22:53:28.198416+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? v1:172.21.15.178:0/3147890141' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:53:29.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[29889]: cluster 2024-09-13T22:53:28.202930+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-13T22:53:29.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:29.558 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:53:29.559 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:53:29.559 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:29.579 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:53:29.580 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:53:29.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:29.690 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:29.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:29.820 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:29.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:29.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:53:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:29.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:29.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:30.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:30.080 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:30.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:30.214 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:30.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:30.345 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[22840]: cluster 2024-09-13T22:53:28.755056+0000 osd.5 (osd.5) 347 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[22840]: cluster 2024-09-13T22:53:28.757281+0000 osd.5 (osd.5) 348 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[22840]: cluster 2024-09-13T22:53:28.788082+0000 mgr.y (mgr.14613) 592 : cluster [DBG] pgmap v682: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[22840]: cluster 2024-09-13T22:53:29.208671+0000 mon.a (mon.0) 1465 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[22840]: cluster 2024-09-13T22:53:29.208718+0000 mon.a (mon.0) 1466 : cluster [INF] Cluster is now healthy 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[22840]: audit 2024-09-13T22:53:29.429220+0000 mon.a (mon.0) 1467 : audit [DBG] from='client.? v1:172.21.15.178:0/3785542686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[29889]: cluster 2024-09-13T22:53:28.755056+0000 osd.5 (osd.5) 347 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:53:30.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[29889]: cluster 2024-09-13T22:53:28.757281+0000 osd.5 (osd.5) 348 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:53:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[29889]: cluster 2024-09-13T22:53:28.788082+0000 mgr.y (mgr.14613) 592 : cluster [DBG] pgmap v682: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[29889]: cluster 2024-09-13T22:53:29.208671+0000 mon.a (mon.0) 1465 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:53:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[29889]: cluster 2024-09-13T22:53:29.208718+0000 mon.a (mon.0) 1466 : cluster [INF] Cluster is now healthy 2024-09-13T22:53:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:30 smithi178 bash[29889]: audit 2024-09-13T22:53:29.429220+0000 mon.a (mon.0) 1467 : audit [DBG] from='client.? v1:172.21.15.178:0/3785542686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:30.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:30.474 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:30.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:30.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:30 smithi192 bash[24608]: cluster 2024-09-13T22:53:28.755056+0000 osd.5 (osd.5) 347 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:53:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:30 smithi192 bash[24608]: cluster 2024-09-13T22:53:28.757281+0000 osd.5 (osd.5) 348 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:53:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:30 smithi192 bash[24608]: cluster 2024-09-13T22:53:28.788082+0000 mgr.y (mgr.14613) 592 : cluster [DBG] pgmap v682: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:30 smithi192 bash[24608]: cluster 2024-09-13T22:53:29.208671+0000 mon.a (mon.0) 1465 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:53:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:30 smithi192 bash[24608]: cluster 2024-09-13T22:53:29.208718+0000 mon.a (mon.0) 1466 : cluster [INF] Cluster is now healthy 2024-09-13T22:53:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:30 smithi192 bash[24608]: audit 2024-09-13T22:53:29.429220+0000 mon.a (mon.0) 1467 : audit [DBG] from='client.? v1:172.21.15.178:0/3785542686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:30.603 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:30.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:30.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:30.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:30.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:30.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:30.995 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:30.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:31.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:31.123 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:31.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:31.252 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:31.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:31.385 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.109958+0000 osd.4 (osd.4) 222 : cluster [DBG] 5.a scrub starts 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.112226+0000 osd.4 (osd.4) 223 : cluster [DBG] 5.a scrub ok 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.178556+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.182753+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.387122+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.c scrub starts 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.389212+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.c scrub ok 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.712700+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.714792+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.109958+0000 osd.4 (osd.4) 222 : cluster [DBG] 5.a scrub starts 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.112226+0000 osd.4 (osd.4) 223 : cluster [DBG] 5.a scrub ok 2024-09-13T22:53:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.178556+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:53:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.182753+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:53:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.387122+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.c scrub starts 2024-09-13T22:53:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.389212+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.c scrub ok 2024-09-13T22:53:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.712700+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:53:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:31 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.714792+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:53:31.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.518 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:31.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:31.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.109958+0000 osd.4 (osd.4) 222 : cluster [DBG] 5.a scrub starts 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.112226+0000 osd.4 (osd.4) 223 : cluster [DBG] 5.a scrub ok 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.178556+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.182753+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.387122+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.c scrub starts 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.389212+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.c scrub ok 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.712700+0000 osd.7 (osd.7) 319 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:53:31.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:31 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.714792+0000 osd.7 (osd.7) 320 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:53:31.647 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:31.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:31.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:31.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:31.907 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:31.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:32.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:32.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:32.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:32.165 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:32.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:32.289 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:32.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:32.419 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[29889]: cluster 2024-09-13T22:53:30.789641+0000 mgr.y (mgr.14613) 593 : cluster [DBG] pgmap v683: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[29889]: cluster 2024-09-13T22:53:31.353693+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.b scrub starts 2024-09-13T22:53:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[29889]: cluster 2024-09-13T22:53:31.355955+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.b scrub ok 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[29889]: cluster 2024-09-13T22:53:31.676704+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[29889]: cluster 2024-09-13T22:53:31.678021+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[22840]: cluster 2024-09-13T22:53:30.789641+0000 mgr.y (mgr.14613) 593 : cluster [DBG] pgmap v683: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[22840]: cluster 2024-09-13T22:53:31.353693+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.b scrub starts 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[22840]: cluster 2024-09-13T22:53:31.355955+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.b scrub ok 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[22840]: cluster 2024-09-13T22:53:31.676704+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:53:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:32 smithi178 bash[22840]: cluster 2024-09-13T22:53:31.678021+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:53:32.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:32.533 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:32 smithi192 bash[24608]: cluster 2024-09-13T22:53:30.789641+0000 mgr.y (mgr.14613) 593 : cluster [DBG] pgmap v683: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:32.533 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:32 smithi192 bash[24608]: cluster 2024-09-13T22:53:31.353693+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.b scrub starts 2024-09-13T22:53:32.533 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:32 smithi192 bash[24608]: cluster 2024-09-13T22:53:31.355955+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.b scrub ok 2024-09-13T22:53:32.533 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:32 smithi192 bash[24608]: cluster 2024-09-13T22:53:31.676704+0000 osd.7 (osd.7) 321 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:53:32.533 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:32 smithi192 bash[24608]: cluster 2024-09-13T22:53:31.678021+0000 osd.7 (osd.7) 322 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:53:32.547 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:32.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:32.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:32.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:32.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:32.830 DEBUG:teuthology.orchestra.run.smithi178:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-13T22:53:32.833 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T22:53:32.833 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T22:53:32.906 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:hints = 1 2024-09-13T22:53:32.906 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T22:53:32.906 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_111521 2024-09-13T22:53:32.906 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:53:32.907 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-13T22:53:32.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:32.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:32.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:33.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:33.095 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:33.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:33.226 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:33.231 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:53:33.329 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:33.339 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[29889]: cluster 2024-09-13T22:53:31.992566+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:53:33.339 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[29889]: cluster 2024-09-13T22:53:31.994710+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:53:33.339 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[29889]: audit 2024-09-13T22:53:32.484008+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? v1:172.21.15.178:0/1997518796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[29889]: cluster 2024-09-13T22:53:32.684156+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[29889]: cluster 2024-09-13T22:53:32.685387+0000 osd.5 (osd.5) 350 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[22840]: cluster 2024-09-13T22:53:31.992566+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[22840]: cluster 2024-09-13T22:53:31.994710+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[22840]: audit 2024-09-13T22:53:32.484008+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? v1:172.21.15.178:0/1997518796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[22840]: cluster 2024-09-13T22:53:32.684156+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:53:33.340 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:33 smithi178 bash[22840]: cluster 2024-09-13T22:53:32.685387+0000 osd.5 (osd.5) 350 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:53:33.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:33.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:33.483 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:33.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:33.598 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:33 smithi192 bash[24608]: cluster 2024-09-13T22:53:31.992566+0000 osd.6 (osd.6) 265 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:53:33.598 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:33 smithi192 bash[24608]: cluster 2024-09-13T22:53:31.994710+0000 osd.6 (osd.6) 266 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:53:33.598 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:33 smithi192 bash[24608]: audit 2024-09-13T22:53:32.484008+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? v1:172.21.15.178:0/1997518796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:53:33.598 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:33 smithi192 bash[24608]: cluster 2024-09-13T22:53:32.684156+0000 osd.5 (osd.5) 349 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:53:33.598 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:33 smithi192 bash[24608]: cluster 2024-09-13T22:53:32.685387+0000 osd.5 (osd.5) 350 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:53:33.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:33.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:33.737 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:33.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:33.869 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:33.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:33.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:33.907 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 1 15 3127 3112 194.489 194.5 0.00524938 0.0051224 2024-09-13T22:53:33.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:33.999 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:34.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:34.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:34.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:53:34.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:34.266 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:34.371 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:34.397 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: cluster 2024-09-13T22:53:32.790291+0000 mgr.y (mgr.14613) 594 : cluster [DBG] pgmap v684: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: cluster 2024-09-13T22:53:33.071357+0000 osd.4 (osd.4) 224 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: cluster 2024-09-13T22:53:33.072889+0000 osd.4 (osd.4) 225 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: cluster 2024-09-13T22:53:33.221970+0000 mon.a (mon.0) 1469 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: audit 2024-09-13T22:53:33.229807+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? v1:172.21.15.178:0/1997518796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: cluster 2024-09-13T22:53:33.234513+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-13T22:53:34.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: audit 2024-09-13T22:53:33.930366+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:34.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: audit 2024-09-13T22:53:33.933567+0000 mon.b (mon.1) 221 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:34.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:34 smithi192 bash[24608]: audit 2024-09-13T22:53:34.140476+0000 mon.b (mon.1) 222 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:53:34.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:34.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:34.614 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: cluster 2024-09-13T22:53:32.790291+0000 mgr.y (mgr.14613) 594 : cluster [DBG] pgmap v684: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:34.614 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: cluster 2024-09-13T22:53:33.071357+0000 osd.4 (osd.4) 224 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:53:34.614 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: cluster 2024-09-13T22:53:33.072889+0000 osd.4 (osd.4) 225 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:53:34.614 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: cluster 2024-09-13T22:53:33.221970+0000 mon.a (mon.0) 1469 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:34.614 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: audit 2024-09-13T22:53:33.229807+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? v1:172.21.15.178:0/1997518796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:53:34.614 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: cluster 2024-09-13T22:53:33.234513+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: audit 2024-09-13T22:53:33.930366+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: audit 2024-09-13T22:53:33.933567+0000 mon.b (mon.1) 221 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[29889]: audit 2024-09-13T22:53:34.140476+0000 mon.b (mon.1) 222 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: cluster 2024-09-13T22:53:32.790291+0000 mgr.y (mgr.14613) 594 : cluster [DBG] pgmap v684: 158 pgs: 158 active+clean; 583 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: cluster 2024-09-13T22:53:33.071357+0000 osd.4 (osd.4) 224 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: cluster 2024-09-13T22:53:33.072889+0000 osd.4 (osd.4) 225 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: cluster 2024-09-13T22:53:33.221970+0000 mon.a (mon.0) 1469 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: audit 2024-09-13T22:53:33.229807+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? v1:172.21.15.178:0/1997518796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:53:34.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: cluster 2024-09-13T22:53:33.234513+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-13T22:53:34.616 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: audit 2024-09-13T22:53:33.930366+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:34.616 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: audit 2024-09-13T22:53:33.933567+0000 mon.b (mon.1) 221 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:34.616 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:34 smithi178 bash[22840]: audit 2024-09-13T22:53:34.140476+0000 mon.b (mon.1) 222 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:53:34.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:34.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:34.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:34.783 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:34.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:34.906 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 2 16 5784 5768 180.236 166 0.00380992 0.00553631 2024-09-13T22:53:34.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:34.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:34.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:35.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:35.041 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:35.078 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:35.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.170 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:35.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.301 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:35.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:35.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:35 smithi192 bash[24608]: audit 2024-09-13T22:53:35.169732+0000 mon.b (mon.1) 223 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:53:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:35 smithi192 bash[24608]: audit 2024-09-13T22:53:35.171595+0000 mon.b (mon.1) 224 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:53:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:35 smithi192 bash[24608]: audit 2024-09-13T22:53:35.176056+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:35.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:35 smithi178 bash[29889]: audit 2024-09-13T22:53:35.169732+0000 mon.b (mon.1) 223 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:53:35.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:35 smithi178 bash[29889]: audit 2024-09-13T22:53:35.171595+0000 mon.b (mon.1) 224 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:53:35.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:35 smithi178 bash[29889]: audit 2024-09-13T22:53:35.176056+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:35.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:35 smithi178 bash[22840]: audit 2024-09-13T22:53:35.169732+0000 mon.b (mon.1) 223 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:53:35.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:35 smithi178 bash[22840]: audit 2024-09-13T22:53:35.171595+0000 mon.b (mon.1) 224 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:53:35.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:35 smithi178 bash[22840]: audit 2024-09-13T22:53:35.176056+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:53:35.431 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:35.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:35.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:35.684 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:35.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:35.811 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:35.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:53:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:53:35.907 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 3 16 7405 7389 153.922 101.312 0.00179091 0.00571245 2024-09-13T22:53:35.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.940 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:35.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:36.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:36.072 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:36.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:36.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:36.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:36.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:36.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:36 smithi192 bash[24608]: cluster 2024-09-13T22:53:34.791279+0000 mgr.y (mgr.14613) 595 : cluster [DBG] pgmap v686: 158 pgs: 158 active+clean; 93 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 12 MiB/s wr, 185 op/s 2024-09-13T22:53:36.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:36 smithi192 bash[24608]: audit 2024-09-13T22:53:35.619867+0000 mgr.y (mgr.14613) 596 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:36.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:36.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:36 smithi178 bash[29889]: cluster 2024-09-13T22:53:34.791279+0000 mgr.y (mgr.14613) 595 : cluster [DBG] pgmap v686: 158 pgs: 158 active+clean; 93 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 12 MiB/s wr, 185 op/s 2024-09-13T22:53:36.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:36 smithi178 bash[29889]: audit 2024-09-13T22:53:35.619867+0000 mgr.y (mgr.14613) 596 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:36.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:36 smithi178 bash[22840]: cluster 2024-09-13T22:53:34.791279+0000 mgr.y (mgr.14613) 595 : cluster [DBG] pgmap v686: 158 pgs: 158 active+clean; 93 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 12 MiB/s wr, 185 op/s 2024-09-13T22:53:36.446 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:36 smithi178 bash[22840]: audit 2024-09-13T22:53:35.619867+0000 mgr.y (mgr.14613) 596 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:36.460 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:36.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:36.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:36.618 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:53:36.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:36.723 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:36.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:36.857 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:36.907 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 4 16 9905 9889 154.495 156.25 0.00367521 0.00646441 2024-09-13T22:53:36.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:36.976 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:53:36.976 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:53:36.976 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T22:53:36.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T22:53:36.994 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:36.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:37.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:37.122 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:37.226 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:37.266 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:37.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:37.403 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:37.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:37.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:37 smithi192 bash[24608]: cluster 2024-09-13T22:53:35.946685+0000 osd.3 (osd.3) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:37.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:37 smithi192 bash[24608]: cluster 2024-09-13T22:53:35.946782+0000 osd.3 (osd.3) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:37.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:37 smithi192 bash[24608]: cluster 2024-09-13T22:53:36.243780+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-13T22:53:37.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:37.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:37.663 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:37 smithi178 bash[29889]: cluster 2024-09-13T22:53:35.946685+0000 osd.3 (osd.3) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:37 smithi178 bash[29889]: cluster 2024-09-13T22:53:35.946782+0000 osd.3 (osd.3) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:37 smithi178 bash[29889]: cluster 2024-09-13T22:53:36.243780+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-13T22:53:37.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:37 smithi178 bash[22840]: cluster 2024-09-13T22:53:35.946685+0000 osd.3 (osd.3) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:37 smithi178 bash[22840]: cluster 2024-09-13T22:53:35.946782+0000 osd.3 (osd.3) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:37 smithi178 bash[22840]: cluster 2024-09-13T22:53:36.243780+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-13T22:53:37.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:37.794 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:37.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:37.907 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 5 16 12462 12446 155.556 159.812 0.00597282 0.00641936 2024-09-13T22:53:37.923 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:37.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:38.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:38.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:38.128 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:53:38.128 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:53:38.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:38.188 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:38.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:38.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:38 smithi192 bash[24608]: cluster 2024-09-13T22:53:36.791869+0000 mgr.y (mgr.14613) 597 : cluster [DBG] pgmap v688: 158 pgs: 158 active+clean; 178 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 22 MiB/s wr, 356 op/s 2024-09-13T22:53:38.316 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:38.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:38.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:38 smithi178 bash[29889]: cluster 2024-09-13T22:53:36.791869+0000 mgr.y (mgr.14613) 597 : cluster [DBG] pgmap v688: 158 pgs: 158 active+clean; 178 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 22 MiB/s wr, 356 op/s 2024-09-13T22:53:38.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:38 smithi178 bash[22840]: cluster 2024-09-13T22:53:36.791869+0000 mgr.y (mgr.14613) 597 : cluster [DBG] pgmap v688: 158 pgs: 158 active+clean; 178 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 22 MiB/s wr, 356 op/s 2024-09-13T22:53:38.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:38.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:38.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:38.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:38.700 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:38.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:38.824 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:38.907 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 6 16 14466 14450 150.501 125.25 0.00207205 0.00643579 2024-09-13T22:53:38.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:38.952 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:38.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:39.055 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:39.083 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:39.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:39.219 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:39.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:39.349 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:39.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:39.479 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:39.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:39.608 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:39.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:39.737 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:39.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T22:53:39.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:39.868 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:39.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:39.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 7 16 16195 16179 144.428 108.062 0.00404392 0.0069171 2024-09-13T22:53:39.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:53:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:39.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:40.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:40.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:40.129 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:40.165 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:40.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:40.256 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:40.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:40.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:40.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:40.517 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:40.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:40 smithi192 bash[24608]: cluster 2024-09-13T22:53:38.792596+0000 mgr.y (mgr.14613) 598 : cluster [DBG] pgmap v689: 158 pgs: 158 active+clean; 744 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T22:53:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:40.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:40.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:40.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:40 smithi178 bash[29889]: cluster 2024-09-13T22:53:38.792596+0000 mgr.y (mgr.14613) 598 : cluster [DBG] pgmap v689: 158 pgs: 158 active+clean; 744 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T22:53:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:40 smithi178 bash[22840]: cluster 2024-09-13T22:53:38.792596+0000 mgr.y (mgr.14613) 598 : cluster [DBG] pgmap v689: 158 pgs: 158 active+clean; 744 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T22:53:40.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:40.769 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:40.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:40.897 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:40.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:40.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 8 16 18202 18186 142.053 125.438 0.00605577 0.00703425 2024-09-13T22:53:40.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:41.025 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:41.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:41.152 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:41.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:41.281 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:41.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:41.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:41.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.543 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:41.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:41.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:41.669 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:41.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:41.801 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:41.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:41.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 9 16 20527 20511 142.414 145.312 0.00805856 0.00701661 2024-09-13T22:53:41.940 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:41.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:42.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:42.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:42.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:42.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:42.265 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:42.267 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:42.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:42.327 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:42.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:42 smithi192 bash[24608]: cluster 2024-09-13T22:53:40.793290+0000 mgr.y (mgr.14613) 599 : cluster [DBG] pgmap v690: 158 pgs: 158 active+clean; 907 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-13T22:53:42.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:42.462 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:42.565 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:42.574 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:42 smithi178 bash[29889]: cluster 2024-09-13T22:53:40.793290+0000 mgr.y (mgr.14613) 599 : cluster [DBG] pgmap v690: 158 pgs: 158 active+clean; 907 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-13T22:53:42.574 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:42 smithi178 bash[22840]: cluster 2024-09-13T22:53:40.793290+0000 mgr.y (mgr.14613) 599 : cluster [DBG] pgmap v690: 158 pgs: 158 active+clean; 907 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-13T22:53:42.590 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:42.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:42.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:42.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:42.859 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:42.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 10 16 22712 22696 141.827 136.562 0.0069239 0.00704485 2024-09-13T22:53:42.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:42.991 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:42.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:43.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:43.118 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:43.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:43.222 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:43.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:43.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:43.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:43.370 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:43.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:43.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:43.503 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:43.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:43.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:43.633 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:43.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:43.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:43.845 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:53:43.846 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:53:43.868 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:43.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:43.880 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:53:43.880 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:53:43.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 11 16 24397 24381 138.507 105.312 0.00767233 0.00721372 2024-09-13T22:53:43.909 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:53:43.910 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:53:43.949 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:53:43.950 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:53:43.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:43.982 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:53:43.982 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:53:43.998 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:44.021 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:53:44.021 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:53:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:53:44.048 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:53:44.084 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:53:44.085 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:53:44.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:44.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:44.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:44.256 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:44.302 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:44.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:44.378 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:44.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:44 smithi192 bash[24608]: cluster 2024-09-13T22:53:42.793735+0000 mgr.y (mgr.14613) 600 : cluster [DBG] pgmap v691: 158 pgs: 158 active+clean; 907 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 856 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-13T22:53:44.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:44 smithi192 bash[24608]: audit 2024-09-13T22:53:43.805272+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? v1:172.21.15.178:0/4175061802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:44.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:44.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:44 smithi178 bash[22840]: cluster 2024-09-13T22:53:42.793735+0000 mgr.y (mgr.14613) 600 : cluster [DBG] pgmap v691: 158 pgs: 158 active+clean; 907 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 856 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-13T22:53:44.492 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:44 smithi178 bash[22840]: audit 2024-09-13T22:53:43.805272+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? v1:172.21.15.178:0/4175061802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:44.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:44 smithi178 bash[29889]: cluster 2024-09-13T22:53:42.793735+0000 mgr.y (mgr.14613) 600 : cluster [DBG] pgmap v691: 158 pgs: 158 active+clean; 907 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 856 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-13T22:53:44.493 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:44 smithi178 bash[29889]: audit 2024-09-13T22:53:43.805272+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? v1:172.21.15.178:0/4175061802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:53:44.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:44.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:44.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:44.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:44.772 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:44.875 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:44.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:44.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:44.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 12 15 26111 26096 135.897 107.188 0.0100968 0.0073528 2024-09-13T22:53:45.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:45.038 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:45.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:45.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:45.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:45.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:45.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:45.413 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:45.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:45.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:45.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:45.687 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:45.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:45.811 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:45.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:53:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:53:45.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 13 16 27789 27773 133.505 104.812 0.00586329 0.00748444 2024-09-13T22:53:45.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:45.936 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:45.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:46.041 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:46.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:46.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:46.199 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:46.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:46.311 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:46 smithi178 bash[29889]: cluster 2024-09-13T22:53:44.794645+0000 mgr.y (mgr.14613) 601 : cluster [DBG] pgmap v692: 158 pgs: 158 active+clean; 1.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-13T22:53:46.311 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:46 smithi178 bash[29889]: audit 2024-09-13T22:53:45.628411+0000 mgr.y (mgr.14613) 602 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:46.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:46 smithi178 bash[22840]: cluster 2024-09-13T22:53:44.794645+0000 mgr.y (mgr.14613) 601 : cluster [DBG] pgmap v692: 158 pgs: 158 active+clean; 1.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-13T22:53:46.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:46 smithi178 bash[22840]: audit 2024-09-13T22:53:45.628411+0000 mgr.y (mgr.14613) 602 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:46.322 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:46.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:46.453 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:46.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:46.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:46 smithi192 bash[24608]: cluster 2024-09-13T22:53:44.794645+0000 mgr.y (mgr.14613) 601 : cluster [DBG] pgmap v692: 158 pgs: 158 active+clean; 1.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-13T22:53:46.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:46 smithi192 bash[24608]: audit 2024-09-13T22:53:45.628411+0000 mgr.y (mgr.14613) 602 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:46.586 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:46.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:46.714 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:46.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:46.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:46.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 14 16 29474 29458 131.491 105.312 0.00854023 0.00759809 2024-09-13T22:53:46.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:46.963 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:46.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:47.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:47.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:47.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:47.227 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:47.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:47.359 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:47.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:47.486 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:47.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:47.608 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:47.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:47.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:47.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:47.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:47.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 15 16 31113 31097 129.553 102.438 0.0105071 0.00771342 2024-09-13T22:53:47.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:47.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:47.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:48.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:48.125 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:48.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:48.258 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:48.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:48.379 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:48.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:48 smithi178 bash[22840]: cluster 2024-09-13T22:53:46.796135+0000 mgr.y (mgr.14613) 603 : cluster [DBG] pgmap v693: 158 pgs: 158 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 970 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T22:53:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:48 smithi178 bash[22840]: cluster 2024-09-13T22:53:47.421745+0000 osd.5 (osd.5) 351 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:48.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:48 smithi178 bash[22840]: cluster 2024-09-13T22:53:47.421793+0000 osd.5 (osd.5) 352 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:48.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:48 smithi178 bash[29889]: cluster 2024-09-13T22:53:46.796135+0000 mgr.y (mgr.14613) 603 : cluster [DBG] pgmap v693: 158 pgs: 158 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 970 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T22:53:48.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:48 smithi178 bash[29889]: cluster 2024-09-13T22:53:47.421745+0000 osd.5 (osd.5) 351 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:48.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:48 smithi178 bash[29889]: cluster 2024-09-13T22:53:47.421793+0000 osd.5 (osd.5) 352 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:48.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:48.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:48 smithi192 bash[24608]: cluster 2024-09-13T22:53:46.796135+0000 mgr.y (mgr.14613) 603 : cluster [DBG] pgmap v693: 158 pgs: 158 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 970 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T22:53:48.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:48 smithi192 bash[24608]: cluster 2024-09-13T22:53:47.421745+0000 osd.5 (osd.5) 351 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:48.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:48 smithi192 bash[24608]: cluster 2024-09-13T22:53:47.421793+0000 osd.5 (osd.5) 352 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:48.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:48.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:48.639 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:48.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:48.769 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:48.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:48.900 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:48.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:48.908 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 16 16 32678 32662 127.569 97.8125 0.00572265 0.00783311 2024-09-13T22:53:49.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:49.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:49.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:49.159 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:49.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:49.289 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:49.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:49.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:49 smithi178 bash[29889]: cluster 2024-09-13T22:53:48.282609+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-13T22:53:49.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:49 smithi178 bash[29889]: audit 2024-09-13T22:53:48.958230+0000 mon.b (mon.1) 226 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:49.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:49 smithi178 bash[22840]: cluster 2024-09-13T22:53:48.282609+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-13T22:53:49.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:49 smithi178 bash[22840]: audit 2024-09-13T22:53:48.958230+0000 mon.b (mon.1) 226 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:49.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:49 smithi192 bash[24608]: cluster 2024-09-13T22:53:48.282609+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-13T22:53:49.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:49 smithi192 bash[24608]: audit 2024-09-13T22:53:48.958230+0000 mon.b (mon.1) 226 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:53:49.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:49.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:49.562 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:49.571 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:53:49.572 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:53:49.572 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:53:49.572 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:53:49.572 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:53:49.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:49.699 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:49.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:49.823 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:49.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:53:49.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 17 16 34000 33984 124.925 82.625 0.00203394 0.00791501 2024-09-13T22:53:49.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:49.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:49.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:50.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:50.061 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:53:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:50.076 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:50.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:50.213 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:50.302 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:53:50.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:50.340 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:50.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:50.467 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:50.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:50.581 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:50 smithi178 bash[29889]: cluster 2024-09-13T22:53:48.796942+0000 mgr.y (mgr.14613) 604 : cluster [DBG] pgmap v695: 158 pgs: 158 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T22:53:50.581 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:50 smithi178 bash[29889]: audit 2024-09-13T22:53:49.754970+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25465 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:50.581 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:50 smithi178 bash[29889]: audit 2024-09-13T22:53:49.756121+0000 mon.b (mon.1) 227 : audit [INF] from='client.? v1:172.21.15.178:0/2514959953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:50.581 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:50 smithi178 bash[22840]: cluster 2024-09-13T22:53:48.796942+0000 mgr.y (mgr.14613) 604 : cluster [DBG] pgmap v695: 158 pgs: 158 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T22:53:50.581 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:50 smithi178 bash[22840]: audit 2024-09-13T22:53:49.754970+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25465 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:50.581 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:50 smithi178 bash[22840]: audit 2024-09-13T22:53:49.756121+0000 mon.b (mon.1) 227 : audit [INF] from='client.? v1:172.21.15.178:0/2514959953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:50.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:50.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:50 smithi192 bash[24608]: cluster 2024-09-13T22:53:48.796942+0000 mgr.y (mgr.14613) 604 : cluster [DBG] pgmap v695: 158 pgs: 158 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T22:53:50.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:50 smithi192 bash[24608]: audit 2024-09-13T22:53:49.754970+0000 mon.a (mon.0) 1476 : audit [INF] from='client.25465 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:50.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:50 smithi192 bash[24608]: audit 2024-09-13T22:53:49.756121+0000 mon.b (mon.1) 227 : audit [INF] from='client.? v1:172.21.15.178:0/2514959953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:53:50.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:50.726 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:50.828 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:50.860 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:50.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 18 16 35174 35158 122.061 73.375 0.0102106 0.00818699 2024-09-13T22:53:50.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:50.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:50.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:51.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:51.113 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:51.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:51.257 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:51.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:51.390 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:51.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:51 smithi178 bash[29889]: cluster 2024-09-13T22:53:50.297167+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:51.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:51 smithi178 bash[29889]: audit 2024-09-13T22:53:50.299721+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:53:51.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:51 smithi178 bash[29889]: cluster 2024-09-13T22:53:50.308620+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-13T22:53:51.418 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:51 smithi178 bash[22840]: cluster 2024-09-13T22:53:50.297167+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:51.418 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:51 smithi178 bash[22840]: audit 2024-09-13T22:53:50.299721+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:53:51.418 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:51 smithi178 bash[22840]: cluster 2024-09-13T22:53:50.308620+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-13T22:53:51.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:51.523 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:51.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:51.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:51 smithi192 bash[24608]: cluster 2024-09-13T22:53:50.297167+0000 mon.a (mon.0) 1477 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:53:51.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:51 smithi192 bash[24608]: audit 2024-09-13T22:53:50.299721+0000 mon.a (mon.0) 1478 : audit [INF] from='client.25465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:53:51.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:51 smithi192 bash[24608]: cluster 2024-09-13T22:53:50.308620+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-13T22:53:51.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:51.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:51.786 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:51.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:51.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 19 16 36877 36861 121.237 106.438 0.00736916 0.00824215 2024-09-13T22:53:51.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:51.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:52.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.060 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:52.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:52.184 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:52.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:52.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:52.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:52.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:52 smithi178 bash[29889]: cluster 2024-09-13T22:53:50.797826+0000 mgr.y (mgr.14613) 605 : cluster [DBG] pgmap v697: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-13T22:53:52.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:52 smithi178 bash[22840]: cluster 2024-09-13T22:53:50.797826+0000 mgr.y (mgr.14613) 605 : cluster [DBG] pgmap v697: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-13T22:53:52.433 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:52 smithi192 bash[24608]: cluster 2024-09-13T22:53:50.797826+0000 mgr.y (mgr.14613) 605 : cluster [DBG] pgmap v697: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-09-13T22:53:52.463 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:52.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.593 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:52.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.722 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:52.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.895 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:52.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:52.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:53:52.911051+0000 min lat: 0.00112903 max lat: 0.484121 avg lat: 0.00830153 2024-09-13T22:53:52.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:53:52.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 20 15 38535 38520 120.359 103.688 0.00809049 0.00830153 2024-09-13T22:53:52.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T22:53:52.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T22:53:53.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:53.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:53.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:53.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:53.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:53.301 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:53.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:53.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:53 smithi178 bash[29889]: cluster 2024-09-13T22:53:52.119532+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:53.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:53 smithi178 bash[29889]: cluster 2024-09-13T22:53:52.119599+0000 osd.1 (osd.1) 303 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:53.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:53 smithi178 bash[29889]: cluster 2024-09-13T22:53:52.333157+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-13T22:53:53.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:53 smithi178 bash[22840]: cluster 2024-09-13T22:53:52.119532+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:53.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:53 smithi178 bash[22840]: cluster 2024-09-13T22:53:52.119599+0000 osd.1 (osd.1) 303 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:53.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:53 smithi178 bash[22840]: cluster 2024-09-13T22:53:52.333157+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-13T22:53:53.432 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:53.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:53.574 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:53.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:53 smithi192 bash[24608]: cluster 2024-09-13T22:53:52.119532+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:53.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:53 smithi192 bash[24608]: cluster 2024-09-13T22:53:52.119599+0000 osd.1 (osd.1) 303 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:53.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:53 smithi192 bash[24608]: cluster 2024-09-13T22:53:52.333157+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-13T22:53:53.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:53.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:53.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:53.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:53.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 21 16 40137 40121 119.392 100.062 0.00820716 0.00837009 2024-09-13T22:53:53.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:53.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:53.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:54.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:54.112 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:54.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:54.241 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:54.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:54.372 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:54.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:54.486 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:54 smithi178 bash[29889]: cluster 2024-09-13T22:53:52.798286+0000 mgr.y (mgr.14613) 606 : cluster [DBG] pgmap v699: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T22:53:54.487 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:54 smithi178 bash[22840]: cluster 2024-09-13T22:53:52.798286+0000 mgr.y (mgr.14613) 606 : cluster [DBG] pgmap v699: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T22:53:54.504 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:54.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:54.637 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:54.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:54 smithi192 bash[24608]: cluster 2024-09-13T22:53:52.798286+0000 mgr.y (mgr.14613) 606 : cluster [DBG] pgmap v699: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T22:53:54.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:54.767 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:54.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:54.900 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:54.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:54.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:53:54.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 22 16 41832 41816 118.78 105.938 0.00969768 0.00841255 2024-09-13T22:53:55.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:55.032 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:55.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:55.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:55.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:55.290 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:55.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:55.423 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:55.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:55.558 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:55.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:55.691 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:55.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:55.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:53:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:53:55.821 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:55.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 23 16 43419 43403 117.928 99.1875 0.00889811 0.00847441 2024-09-13T22:53:55.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:55.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:55.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:56.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:56.091 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:56.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:56.223 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:56.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:56.355 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:56.458 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:56.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:56.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:56.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:56.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:56 smithi192 bash[24608]: cluster 2024-09-13T22:53:54.799235+0000 mgr.y (mgr.14613) 607 : cluster [DBG] pgmap v700: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-13T22:53:56.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:56 smithi192 bash[24608]: audit 2024-09-13T22:53:55.636546+0000 mgr.y (mgr.14613) 608 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:56.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:53:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:53:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:56 smithi178 bash[29889]: cluster 2024-09-13T22:53:54.799235+0000 mgr.y (mgr.14613) 607 : cluster [DBG] pgmap v700: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-13T22:53:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:56 smithi178 bash[29889]: audit 2024-09-13T22:53:55.636546+0000 mgr.y (mgr.14613) 608 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:56 smithi178 bash[22840]: cluster 2024-09-13T22:53:54.799235+0000 mgr.y (mgr.14613) 607 : cluster [DBG] pgmap v700: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-09-13T22:53:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:56 smithi178 bash[22840]: audit 2024-09-13T22:53:55.636546+0000 mgr.y (mgr.14613) 608 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:53:56.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:56.755 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:56.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:56.890 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:56.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:56.909 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 24 16 45030 45014 117.209 100.688 0.0192297 0.0085265 2024-09-13T22:53:56.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:57.019 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:57.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:57.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:57.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:57.280 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:57.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:57.400 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:57.477 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:57.477 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:57.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:57.527 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:57.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:57.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:57.782 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:57.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:57.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 25 16 45725 45709 114.258 43.4375 0.00223787 0.00855875 2024-09-13T22:53:57.912 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:57.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:58.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:58.052 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:58.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:58.188 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:53:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:53:58.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:58.321 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:58.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:58.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:58.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:58.587 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:53:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[22840]: cluster 2024-09-13T22:53:56.800167+0000 mgr.y (mgr.14613) 609 : cluster [DBG] pgmap v701: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[22840]: cluster 2024-09-13T22:53:56.861986+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[22840]: cluster 2024-09-13T22:53:56.862058+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[22840]: cluster 2024-09-13T22:53:57.359066+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[22840]: cluster 2024-09-13T22:53:58.056491+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[29889]: cluster 2024-09-13T22:53:56.800167+0000 mgr.y (mgr.14613) 609 : cluster [DBG] pgmap v701: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[29889]: cluster 2024-09-13T22:53:56.861986+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:58.592 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[29889]: cluster 2024-09-13T22:53:56.862058+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:58.593 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[29889]: cluster 2024-09-13T22:53:57.359066+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-13T22:53:58.593 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:58 smithi178 bash[29889]: cluster 2024-09-13T22:53:58.056491+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-13T22:53:58.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:58 smithi192 bash[24608]: cluster 2024-09-13T22:53:56.800167+0000 mgr.y (mgr.14613) 609 : cluster [DBG] pgmap v701: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-13T22:53:58.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:58 smithi192 bash[24608]: cluster 2024-09-13T22:53:56.861986+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:58.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:58 smithi192 bash[24608]: cluster 2024-09-13T22:53:56.862058+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:58.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:58 smithi192 bash[24608]: cluster 2024-09-13T22:53:57.359066+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-13T22:53:58.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:58 smithi192 bash[24608]: cluster 2024-09-13T22:53:58.056491+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-13T22:53:58.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:58.715 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:58.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:58.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:58.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 26 16 47056 47040 113.063 83.1875 0.00572349 0.00883855 2024-09-13T22:53:58.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:58.981 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:53:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:58.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:53:59.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:59.115 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:59.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:59.245 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:59.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:59.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:59.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:59.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:53:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:53:59.528 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:53:59.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:59.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:59 smithi178 bash[29889]: cluster 2024-09-13T22:53:59.112084+0000 osd.2 (osd.2) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:59.622 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:53:59 smithi178 bash[29889]: cluster 2024-09-13T22:53:59.112148+0000 osd.2 (osd.2) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:59.622 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:59 smithi178 bash[22840]: cluster 2024-09-13T22:53:59.112084+0000 osd.2 (osd.2) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:59.622 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:53:59 smithi178 bash[22840]: cluster 2024-09-13T22:53:59.112148+0000 osd.2 (osd.2) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:59.634 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:53:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:53:59.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:59 smithi192 bash[24608]: cluster 2024-09-13T22:53:59.112084+0000 osd.2 (osd.2) 237 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:53:59.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:53:59 smithi192 bash[24608]: cluster 2024-09-13T22:53:59.112148+0000 osd.2 (osd.2) 238 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:53:59.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:59.762 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:53:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:53:59.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:59.876 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:53:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:53:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:53:59.900 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:53:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:53:59.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:53:59.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 27 16 48196 48180 111.514 71.25 0.00833979 0.00896258 2024-09-13T22:54:00.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:00.026 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:00.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:00.159 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:00.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:00.292 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:00.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:00.423 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:00.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:00.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:00.573 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:00 smithi178 bash[29889]: cluster 2024-09-13T22:53:58.801248+0000 mgr.y (mgr.14613) 610 : cluster [DBG] pgmap v704: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-13T22:54:00.573 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:00 smithi178 bash[29889]: cluster 2024-09-13T22:53:59.365415+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-13T22:54:00.574 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:00 smithi178 bash[22840]: cluster 2024-09-13T22:53:58.801248+0000 mgr.y (mgr.14613) 610 : cluster [DBG] pgmap v704: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-13T22:54:00.574 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:00 smithi178 bash[22840]: cluster 2024-09-13T22:53:59.365415+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-13T22:54:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:00.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:00 smithi192 bash[24608]: cluster 2024-09-13T22:53:58.801248+0000 mgr.y (mgr.14613) 610 : cluster [DBG] pgmap v704: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-13T22:54:00.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:00 smithi192 bash[24608]: cluster 2024-09-13T22:53:59.365415+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-13T22:54:00.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:00.708 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:00.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:00.833 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:00.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 28 16 49470 49454 110.375 79.625 0.0253526 0.00905428 2024-09-13T22:54:00.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:00.969 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:00.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:01.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:01.098 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:01.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:01.223 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:01.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:01.355 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:01.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:01.485 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:01.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:01.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:01.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.752 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:01.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:01.853 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:01.874 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:01.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:01.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 29 16 50639 50623 109.088 73.0625 0.00513084 0.00914164 2024-09-13T22:54:01.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:02.015 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:02.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:02.147 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:02.251 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:02.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:02.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:02.385 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:02 smithi178 bash[29889]: cluster 2024-09-13T22:54:00.801661+0000 mgr.y (mgr.14613) 611 : cluster [DBG] pgmap v706: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T22:54:02.386 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:02 smithi178 bash[22840]: cluster 2024-09-13T22:54:00.801661+0000 mgr.y (mgr.14613) 611 : cluster [DBG] pgmap v706: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T22:54:02.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:02.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:02.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:02.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:02.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:02 smithi192 bash[24608]: cluster 2024-09-13T22:54:00.801661+0000 mgr.y (mgr.14613) 611 : cluster [DBG] pgmap v706: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T22:54:02.660 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:02.763 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:02.787 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:02.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:02.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 30 16 51777 51761 107.823 71.125 0.0138026 0.00926822 2024-09-13T22:54:02.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:02.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:03.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:03.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:03.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:03.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:03.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.329 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:03.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.460 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:03.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:03.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:03.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.723 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:03.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:03.858 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:03.896 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:54:03.896 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:54:03.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 31 16 53012 52996 106.834 77.1875 0.00937208 0.0093544 2024-09-13T22:54:03.941 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:54:03.942 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:54:03.962 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.976 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:54:03.976 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:54:03.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:03.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:04.017 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:54:04.017 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:54:04.043 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:54:04.044 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:54:04.080 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:54:04.081 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:54:04.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:04.144 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:54:04.145 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:54:04.145 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:54:04.145 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:54:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:04 smithi192 bash[24608]: cluster 2024-09-13T22:54:02.802097+0000 mgr.y (mgr.14613) 612 : cluster [DBG] pgmap v707: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-13T22:54:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:04 smithi192 bash[24608]: cluster 2024-09-13T22:54:03.056809+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-13T22:54:04.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:04 smithi192 bash[24608]: audit 2024-09-13T22:54:03.853970+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.178:0/2780585704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:04.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:04 smithi192 bash[24608]: audit 2024-09-13T22:54:03.981561+0000 mon.b (mon.1) 228 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:04.162 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:04.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[29889]: cluster 2024-09-13T22:54:02.802097+0000 mgr.y (mgr.14613) 612 : cluster [DBG] pgmap v707: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-13T22:54:04.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[29889]: cluster 2024-09-13T22:54:03.056809+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-13T22:54:04.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[29889]: audit 2024-09-13T22:54:03.853970+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.178:0/2780585704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:04.242 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[29889]: audit 2024-09-13T22:54:03.981561+0000 mon.b (mon.1) 228 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:04.242 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[22840]: cluster 2024-09-13T22:54:02.802097+0000 mgr.y (mgr.14613) 612 : cluster [DBG] pgmap v707: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-13T22:54:04.243 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[22840]: cluster 2024-09-13T22:54:03.056809+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-13T22:54:04.243 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[22840]: audit 2024-09-13T22:54:03.853970+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.178:0/2780585704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:04.243 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:04 smithi178 bash[22840]: audit 2024-09-13T22:54:03.981561+0000 mon.b (mon.1) 228 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:04.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:04.297 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:04.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:04.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:04.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:04.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:04.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:04.699 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:04.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:04.829 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:04.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 32 16 54165 54149 105.747 72.0625 0.0073297 0.00945182 2024-09-13T22:54:04.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:04.961 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:04.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:04.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:05.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.095 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:05.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.227 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:05.329 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:05.354 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:05.458 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:05.548 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:54:05.548 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:54:05.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:05.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:05.740 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:05.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:05.874 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:05.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:05.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:54:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:54:05.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 33 16 55338 55322 104.764 73.3125 0.00869985 0.00954127 2024-09-13T22:54:05.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:06.007 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:06.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:06.140 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:06.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:06 smithi178 bash[22840]: cluster 2024-09-13T22:54:04.803171+0000 mgr.y (mgr.14613) 613 : cluster [DBG] pgmap v709: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-13T22:54:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:06 smithi178 bash[22840]: audit 2024-09-13T22:54:05.644434+0000 mgr.y (mgr.14613) 614 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:06 smithi178 bash[29889]: cluster 2024-09-13T22:54:04.803171+0000 mgr.y (mgr.14613) 613 : cluster [DBG] pgmap v709: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-13T22:54:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:06 smithi178 bash[29889]: audit 2024-09-13T22:54:05.644434+0000 mgr.y (mgr.14613) 614 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:06.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:06.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:06.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:06.396 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:06.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:06 smithi192 bash[24608]: cluster 2024-09-13T22:54:04.803171+0000 mgr.y (mgr.14613) 613 : cluster [DBG] pgmap v709: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-13T22:54:06.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:06 smithi192 bash[24608]: audit 2024-09-13T22:54:05.644434+0000 mgr.y (mgr.14613) 614 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:06.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:06.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:06.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:06.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:06.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:06.792 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:06.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:06.910 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 34 16 56425 56409 103.681 67.9375 0.0146602 0.00963997 2024-09-13T22:54:06.922 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:06.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:07.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:07.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:07.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:07.195 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:07.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:07.328 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:07.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:07.461 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:07.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:07.598 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:07.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:07.732 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:07.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:07.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:07.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:07.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 35 16 57744 57728 103.074 82.4375 0.016896 0.00969705 2024-09-13T22:54:07.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:08.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:08.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:08.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:08.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:08 smithi192 bash[24608]: cluster 2024-09-13T22:54:06.804100+0000 mgr.y (mgr.14613) 615 : cluster [DBG] pgmap v710: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 943 op/s 2024-09-13T22:54:08.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:08 smithi192 bash[24608]: cluster 2024-09-13T22:54:08.054968+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-13T22:54:08.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:08.253 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:08.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:08.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:08.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:08.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:08.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:08 smithi178 bash[29889]: cluster 2024-09-13T22:54:06.804100+0000 mgr.y (mgr.14613) 615 : cluster [DBG] pgmap v710: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 943 op/s 2024-09-13T22:54:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:08 smithi178 bash[29889]: cluster 2024-09-13T22:54:08.054968+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-13T22:54:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:08 smithi178 bash[22840]: cluster 2024-09-13T22:54:06.804100+0000 mgr.y (mgr.14613) 615 : cluster [DBG] pgmap v710: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 943 op/s 2024-09-13T22:54:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:08 smithi178 bash[22840]: cluster 2024-09-13T22:54:08.054968+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-13T22:54:08.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:08.781 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:08.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:08.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:08.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:08.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 36 16 58558 58542 101.624 50.875 0.00971388 0.00983355 2024-09-13T22:54:09.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:09.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:09.078 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:54:09.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:09 smithi192 bash[24608]: audit 2024-09-13T22:54:08.859594+0000 mon.a (mon.0) 1486 : audit [INF] from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:54:09.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:09 smithi192 bash[24608]: audit 2024-09-13T22:54:08.860659+0000 mon.b (mon.1) 229 : audit [INF] from='client.? v1:172.21.15.178:0/909531697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:54:09.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:09.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T22:54:09.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:09.316 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:09.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:09.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:09 smithi178 bash[29889]: audit 2024-09-13T22:54:08.859594+0000 mon.a (mon.0) 1486 : audit [INF] from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:54:09.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:09 smithi178 bash[29889]: audit 2024-09-13T22:54:08.860659+0000 mon.b (mon.1) 229 : audit [INF] from='client.? v1:172.21.15.178:0/909531697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:54:09.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:09 smithi178 bash[22840]: audit 2024-09-13T22:54:08.859594+0000 mon.a (mon.0) 1486 : audit [INF] from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:54:09.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:09 smithi178 bash[22840]: audit 2024-09-13T22:54:08.860659+0000 mon.b (mon.1) 229 : audit [INF] from='client.? v1:172.21.15.178:0/909531697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:54:09.448 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:09.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:09.580 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:09.685 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:09.710 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:09.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:09.843 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:09.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 37 16 59685 59669 100.781 70.4375 0.0131006 0.00991705 2024-09-13T22:54:09.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:54:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:54:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:54:09.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:09.970 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:09.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:10.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:54:10.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:10.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:10 smithi192 bash[24608]: cluster 2024-09-13T22:54:08.804936+0000 mgr.y (mgr.14613) 616 : cluster [DBG] pgmap v712: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T22:54:10.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:10 smithi192 bash[24608]: cluster 2024-09-13T22:54:09.065169+0000 mon.a (mon.0) 1487 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:10.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:10 smithi192 bash[24608]: audit 2024-09-13T22:54:09.075891+0000 mon.a (mon.0) 1488 : audit [INF] from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:54:10.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:10 smithi192 bash[24608]: cluster 2024-09-13T22:54:09.083355+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-13T22:54:10.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:10 smithi192 bash[24608]: cluster 2024-09-13T22:54:09.842004+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:54:10.084 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:10 smithi192 bash[24608]: cluster 2024-09-13T22:54:09.848276+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:54:10.099 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:10.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:10.213 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[29889]: cluster 2024-09-13T22:54:08.804936+0000 mgr.y (mgr.14613) 616 : cluster [DBG] pgmap v712: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T22:54:10.213 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[29889]: cluster 2024-09-13T22:54:09.065169+0000 mon.a (mon.0) 1487 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:10.213 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[29889]: audit 2024-09-13T22:54:09.075891+0000 mon.a (mon.0) 1488 : audit [INF] from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:54:10.213 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[29889]: cluster 2024-09-13T22:54:09.083355+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-13T22:54:10.213 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[29889]: cluster 2024-09-13T22:54:09.842004+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:54:10.213 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[29889]: cluster 2024-09-13T22:54:09.848276+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:54:10.214 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[22840]: cluster 2024-09-13T22:54:08.804936+0000 mgr.y (mgr.14613) 616 : cluster [DBG] pgmap v712: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T22:54:10.214 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[22840]: cluster 2024-09-13T22:54:09.065169+0000 mon.a (mon.0) 1487 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:10.214 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[22840]: audit 2024-09-13T22:54:09.075891+0000 mon.a (mon.0) 1488 : audit [INF] from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:54:10.214 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[22840]: cluster 2024-09-13T22:54:09.083355+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-13T22:54:10.214 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[22840]: cluster 2024-09-13T22:54:09.842004+0000 osd.7 (osd.7) 323 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:54:10.214 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:10 smithi178 bash[22840]: cluster 2024-09-13T22:54:09.848276+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:54:10.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:10.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:10.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:10.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:10.496 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:10.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.637 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:10.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:10.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:10.761 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:10.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:10.899 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:10.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:10.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 38 16 60885 60869 100.102 75 0.0129056 0.00998417 2024-09-13T22:54:11.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:11.032 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:11.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T22:54:11.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:11.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:09.586573+0000 osd.5 (osd.5) 353 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:54:11.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:09.589172+0000 osd.5 (osd.5) 354 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:54:11.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.040473+0000 osd.4 (osd.4) 228 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:54:11.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.042591+0000 osd.4 (osd.4) 229 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:54:11.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.519974+0000 osd.7 (osd.7) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:11.150 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.520043+0000 osd.7 (osd.7) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:11.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.857794+0000 osd.7 (osd.7) 327 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:54:11.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:11 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.860575+0000 osd.7 (osd.7) 328 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:54:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:11.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:11.298 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:11.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:11.415 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:09.586573+0000 osd.5 (osd.5) 353 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:09.589172+0000 osd.5 (osd.5) 354 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.040473+0000 osd.4 (osd.4) 228 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.042591+0000 osd.4 (osd.4) 229 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.519974+0000 osd.7 (osd.7) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.520043+0000 osd.7 (osd.7) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.857794+0000 osd.7 (osd.7) 327 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.860575+0000 osd.7 (osd.7) 328 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:09.586573+0000 osd.5 (osd.5) 353 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:09.589172+0000 osd.5 (osd.5) 354 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:54:11.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.040473+0000 osd.4 (osd.4) 228 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:54:11.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.042591+0000 osd.4 (osd.4) 229 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:54:11.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.519974+0000 osd.7 (osd.7) 325 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:11.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.520043+0000 osd.7 (osd.7) 326 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:11.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.857794+0000 osd.7 (osd.7) 327 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:54:11.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:11 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.860575+0000 osd.7 (osd.7) 328 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:54:11.432 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:11.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:11.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:11.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:11.696 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:11.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:11.822 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:11.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 39 15 62173 62158 99.6011 80.5625 0.0184714 0.0100351 2024-09-13T22:54:11.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:11.953 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:11.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:12.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:12.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:12.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:12.219 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:12.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.421763+0000 osd.3 (osd.3) 301 : cluster [DBG] 6.b scrub starts 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.423249+0000 osd.3 (osd.3) 302 : cluster [DBG] 6.b scrub ok 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:10.805463+0000 mgr.y (mgr.14613) 617 : cluster [DBG] pgmap v714: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.004756+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.e scrub starts 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.012117+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.e scrub ok 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.090949+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.223723+0000 osd.1 (osd.1) 304 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.227257+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.379431+0000 osd.3 (osd.3) 303 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:54:12.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.390545+0000 osd.3 (osd.3) 304 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.421763+0000 osd.3 (osd.3) 301 : cluster [DBG] 6.b scrub starts 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.423249+0000 osd.3 (osd.3) 302 : cluster [DBG] 6.b scrub ok 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:10.805463+0000 mgr.y (mgr.14613) 617 : cluster [DBG] pgmap v714: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.004756+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.e scrub starts 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.012117+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.e scrub ok 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.090949+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.223723+0000 osd.1 (osd.1) 304 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.227257+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.379431+0000 osd.3 (osd.3) 303 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:54:12.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:12 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.390545+0000 osd.3 (osd.3) 304 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:54:12.349 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.421763+0000 osd.3 (osd.3) 301 : cluster [DBG] 6.b scrub starts 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.423249+0000 osd.3 (osd.3) 302 : cluster [DBG] 6.b scrub ok 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:10.805463+0000 mgr.y (mgr.14613) 617 : cluster [DBG] pgmap v714: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.004756+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.e scrub starts 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.012117+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.e scrub ok 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.090949+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.223723+0000 osd.1 (osd.1) 304 : cluster [DBG] 3.0 scrub starts 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.227257+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.0 scrub ok 2024-09-13T22:54:12.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.379431+0000 osd.3 (osd.3) 303 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:54:12.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:12 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.390545+0000 osd.3 (osd.3) 304 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:54:12.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:12.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:12.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:12.615 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:12.721 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:12.760 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:12.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:12.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:12.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:12.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:54:12.912806+0000 min lat: 0.00112903 max lat: 0.654557 avg lat: 0.0100768 2024-09-13T22:54:12.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:54:12.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 40 16 63507 63491 99.1937 83.3125 0.00897238 0.0100768 2024-09-13T22:54:12.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:13.030 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:13.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:54:13.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:13.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:13 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.581037+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.d scrub starts 2024-09-13T22:54:13.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:13 smithi192 bash[24608]: cluster 2024-09-13T22:54:11.590685+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.d scrub ok 2024-09-13T22:54:13.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:13 smithi192 bash[24608]: cluster 2024-09-13T22:54:12.213495+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:54:13.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:13 smithi192 bash[24608]: cluster 2024-09-13T22:54:12.215028+0000 osd.1 (osd.1) 307 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:54:13.157 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:13.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:13.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:13.390 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.581037+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.d scrub starts 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[29889]: cluster 2024-09-13T22:54:11.590685+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.d scrub ok 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[29889]: cluster 2024-09-13T22:54:12.213495+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[29889]: cluster 2024-09-13T22:54:12.215028+0000 osd.1 (osd.1) 307 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.581037+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.d scrub starts 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[22840]: cluster 2024-09-13T22:54:11.590685+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.d scrub ok 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[22840]: cluster 2024-09-13T22:54:12.213495+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:54:13.399 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:13 smithi178 bash[22840]: cluster 2024-09-13T22:54:12.215028+0000 osd.1 (osd.1) 307 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:54:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:13.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:13.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:13.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:13.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:13.676 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:13.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.810 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:13.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:13.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:13.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 41 15 64748 64733 98.6674 77.625 0.00902498 0.0101304 2024-09-13T22:54:13.941 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:13.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:14.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:14.073 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:14.109 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:54:14.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:14.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:14.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:14.343 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:14.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:14 smithi192 bash[24608]: cluster 2024-09-13T22:54:12.550361+0000 osd.0 (osd.0) 325 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:54:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:14 smithi192 bash[24608]: cluster 2024-09-13T22:54:12.556036+0000 osd.0 (osd.0) 326 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:54:14.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:14 smithi192 bash[24608]: cluster 2024-09-13T22:54:12.805864+0000 mgr.y (mgr.14613) 618 : cluster [DBG] pgmap v716: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T22:54:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:14 smithi178 bash[29889]: cluster 2024-09-13T22:54:12.550361+0000 osd.0 (osd.0) 325 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:54:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:14 smithi178 bash[29889]: cluster 2024-09-13T22:54:12.556036+0000 osd.0 (osd.0) 326 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:54:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:14 smithi178 bash[29889]: cluster 2024-09-13T22:54:12.805864+0000 mgr.y (mgr.14613) 618 : cluster [DBG] pgmap v716: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T22:54:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:14 smithi178 bash[22840]: cluster 2024-09-13T22:54:12.550361+0000 osd.0 (osd.0) 325 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:54:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:14 smithi178 bash[22840]: cluster 2024-09-13T22:54:12.556036+0000 osd.0 (osd.0) 326 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:54:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:14 smithi178 bash[22840]: cluster 2024-09-13T22:54:12.805864+0000 mgr.y (mgr.14613) 618 : cluster [DBG] pgmap v716: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T22:54:14.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:14.480 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:14.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:14.605 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:14.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:14.739 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:14.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:14.858 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:14.868 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:14.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:14.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 42 16 66033 66017 98.2288 80.25 0.00902729 0.0101753 2024-09-13T22:54:14.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:15.000 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:15.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:15.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:15.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:15.260 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:15.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:15.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:13.518662+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:54:15.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:13.520739+0000 osd.5 (osd.5) 358 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:54:15.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:13.546062+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:13.550425+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:13.691881+0000 osd.6 (osd.6) 267 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:13.691944+0000 osd.6 (osd.6) 268 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[22840]: cluster 2024-09-13T22:54:14.098074+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:13.518662+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:13.520739+0000 osd.5 (osd.5) 358 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:13.546062+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:13.550425+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:13.691881+0000 osd.6 (osd.6) 267 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:13.691944+0000 osd.6 (osd.6) 268 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:15.377 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:15 smithi178 bash[29889]: cluster 2024-09-13T22:54:14.098074+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-13T22:54:15.395 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:15.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:13.518662+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:54:15.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:13.520739+0000 osd.5 (osd.5) 358 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:54:15.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:13.546062+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:54:15.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:13.550425+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:54:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:13.691881+0000 osd.6 (osd.6) 267 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:13.691944+0000 osd.6 (osd.6) 268 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[24608]: cluster 2024-09-13T22:54:14.098074+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-13T22:54:15.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:15.527 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:15.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:15.657 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:15.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:15.791 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:15.881 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:15.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:15.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:54:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:54:15.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 43 16 67258 67242 97.7248 76.5625 0.00287601 0.0102153 2024-09-13T22:54:15.913 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:15.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:16.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:16.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:16.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:16.168 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[22840]: cluster 2024-09-13T22:54:14.807071+0000 mgr.y (mgr.14613) 619 : cluster [DBG] pgmap v718: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[22840]: cluster 2024-09-13T22:54:15.149183+0000 osd.6 (osd.6) 269 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[22840]: cluster 2024-09-13T22:54:15.160549+0000 osd.6 (osd.6) 270 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[22840]: audit 2024-09-13T22:54:15.652451+0000 mgr.y (mgr.14613) 620 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[22840]: cluster 2024-09-13T22:54:15.859044+0000 osd.7 (osd.7) 329 : cluster [DBG] 3.d scrub starts 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[22840]: cluster 2024-09-13T22:54:15.863942+0000 osd.7 (osd.7) 330 : cluster [DBG] 3.d scrub ok 2024-09-13T22:54:16.194 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[29889]: cluster 2024-09-13T22:54:14.807071+0000 mgr.y (mgr.14613) 619 : cluster [DBG] pgmap v718: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T22:54:16.195 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[29889]: cluster 2024-09-13T22:54:15.149183+0000 osd.6 (osd.6) 269 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:54:16.195 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[29889]: cluster 2024-09-13T22:54:15.160549+0000 osd.6 (osd.6) 270 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:54:16.195 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[29889]: audit 2024-09-13T22:54:15.652451+0000 mgr.y (mgr.14613) 620 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:16.195 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[29889]: cluster 2024-09-13T22:54:15.859044+0000 osd.7 (osd.7) 329 : cluster [DBG] 3.d scrub starts 2024-09-13T22:54:16.195 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:16 smithi178 bash[29889]: cluster 2024-09-13T22:54:15.863942+0000 osd.7 (osd.7) 330 : cluster [DBG] 3.d scrub ok 2024-09-13T22:54:16.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:16.310 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:16.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:16 smithi192 bash[24608]: cluster 2024-09-13T22:54:14.807071+0000 mgr.y (mgr.14613) 619 : cluster [DBG] pgmap v718: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T22:54:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:16 smithi192 bash[24608]: cluster 2024-09-13T22:54:15.149183+0000 osd.6 (osd.6) 269 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:54:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:16 smithi192 bash[24608]: cluster 2024-09-13T22:54:15.160549+0000 osd.6 (osd.6) 270 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:54:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:16 smithi192 bash[24608]: audit 2024-09-13T22:54:15.652451+0000 mgr.y (mgr.14613) 620 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:16 smithi192 bash[24608]: cluster 2024-09-13T22:54:15.859044+0000 osd.7 (osd.7) 329 : cluster [DBG] 3.d scrub starts 2024-09-13T22:54:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:16 smithi192 bash[24608]: cluster 2024-09-13T22:54:15.863942+0000 osd.7 (osd.7) 330 : cluster [DBG] 3.d scrub ok 2024-09-13T22:54:16.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:16.443 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:16.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:16.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:16.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:16.702 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:16.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:16.832 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:16.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 44 16 68123 68107 96.7324 54.0625 0.00698029 0.0103315 2024-09-13T22:54:16.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:16.958 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:16.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:17.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:17.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:17.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:17.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:17.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:17.349 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:17.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:17.474 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:17.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:17.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:17.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:17.739 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:17.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:17.869 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:17.911 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 45 16 69191 69175 96.066 66.75 0.0120469 0.0104045 2024-09-13T22:54:17.913 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:17.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:17.996 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:17.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:18.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:18.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:18.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:18.253 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:18.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:18.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:16.807617+0000 mgr.y (mgr.14613) 621 : cluster [DBG] pgmap v719: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 662 B/s rd, 59 MiB/s wr, 944 op/s 2024-09-13T22:54:18.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.057474+0000 osd.4 (osd.4) 232 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:54:18.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.059296+0000 osd.4 (osd.4) 233 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:54:18.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.271512+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:54:18.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.281677+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:54:18.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.847327+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.e scrub starts 2024-09-13T22:54:18.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.857004+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.e scrub ok 2024-09-13T22:54:18.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.058674+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-13T22:54:18.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:16.807617+0000 mgr.y (mgr.14613) 621 : cluster [DBG] pgmap v719: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 662 B/s rd, 59 MiB/s wr, 944 op/s 2024-09-13T22:54:18.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.057474+0000 osd.4 (osd.4) 232 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:54:18.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.059296+0000 osd.4 (osd.4) 233 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:54:18.371 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.271512+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:54:18.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.281677+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:54:18.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.847327+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.e scrub starts 2024-09-13T22:54:18.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.857004+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.e scrub ok 2024-09-13T22:54:18.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:18 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.058674+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-13T22:54:18.381 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:16.807617+0000 mgr.y (mgr.14613) 621 : cluster [DBG] pgmap v719: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 662 B/s rd, 59 MiB/s wr, 944 op/s 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.057474+0000 osd.4 (osd.4) 232 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.059296+0000 osd.4 (osd.4) 233 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.271512+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.281677+0000 osd.2 (osd.2) 240 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.847327+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.e scrub starts 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.857004+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.e scrub ok 2024-09-13T22:54:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:18 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.058674+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-13T22:54:18.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:18.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:18.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:18.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:18.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:18.760 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:18.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:18.890 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:18.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:18.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 46 16 70331 70315 95.5264 71.25 0.00641845 0.0104572 2024-09-13T22:54:18.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:19.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:19.118 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:19.145 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:19.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:19.259 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.989018+0000 osd.0 (osd.0) 329 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:19.259 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:17.989075+0000 osd.0 (osd.0) 330 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:19.259 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.176847+0000 osd.6 (osd.6) 271 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:54:19.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.188206+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:54:19.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.359988+0000 osd.3 (osd.3) 305 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:54:19.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.363737+0000 osd.3 (osd.3) 306 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:54:19.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.819072+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:54:19.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.821899+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:54:19.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:19 smithi192 bash[24608]: audit 2024-09-13T22:54:18.998728+0000 mon.b (mon.1) 230 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:19.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:19.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:19.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.989018+0000 osd.0 (osd.0) 329 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:19.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:17.989075+0000 osd.0 (osd.0) 330 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:19.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.176847+0000 osd.6 (osd.6) 271 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:54:19.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.188206+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:54:19.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.359988+0000 osd.3 (osd.3) 305 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:54:19.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.363737+0000 osd.3 (osd.3) 306 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.819072+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.821899+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[29889]: audit 2024-09-13T22:54:18.998728+0000 mon.b (mon.1) 230 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.989018+0000 osd.0 (osd.0) 329 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:17.989075+0000 osd.0 (osd.0) 330 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.176847+0000 osd.6 (osd.6) 271 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.188206+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.359988+0000 osd.3 (osd.3) 305 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.363737+0000 osd.3 (osd.3) 306 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.819072+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:54:19.395 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.821899+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:54:19.396 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[22840]: audit 2024-09-13T22:54:18.998728+0000 mon.b (mon.1) 230 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:19.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:19.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:19.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:19.593 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:54:19.593 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:54:19.639 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:19.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:19.768 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:19.793 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:19.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:19.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 47 16 70784 70768 94.0963 28.3125 0.00281526 0.0106005 2024-09-13T22:54:19.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:19.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:20.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:20.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:20 smithi192 bash[24608]: cluster 2024-09-13T22:54:18.808509+0000 mgr.y (mgr.14613) 622 : cluster [DBG] pgmap v721: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:54:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:20 smithi192 bash[24608]: cluster 2024-09-13T22:54:19.985178+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:54:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:20 smithi192 bash[24608]: cluster 2024-09-13T22:54:19.987761+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:54:20.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:20.197 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:20.301 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:20.311 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:54:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:54:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:54:20.311 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:20 smithi178 bash[29889]: cluster 2024-09-13T22:54:18.808509+0000 mgr.y (mgr.14613) 622 : cluster [DBG] pgmap v721: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:54:20.311 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:20 smithi178 bash[29889]: cluster 2024-09-13T22:54:19.985178+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:54:20.311 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:20 smithi178 bash[29889]: cluster 2024-09-13T22:54:19.987761+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:54:20.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:20 smithi178 bash[22840]: cluster 2024-09-13T22:54:18.808509+0000 mgr.y (mgr.14613) 622 : cluster [DBG] pgmap v721: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T22:54:20.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:20 smithi178 bash[22840]: cluster 2024-09-13T22:54:19.985178+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:54:20.312 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:20 smithi178 bash[22840]: cluster 2024-09-13T22:54:19.987761+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:54:20.325 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:20.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:20.458 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:20.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:20.597 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:20.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T22:54:20.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:20.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:20.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:20.873 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:20.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:20.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 48 16 71855 71839 93.5303 66.9375 0.0166431 0.0106863 2024-09-13T22:54:20.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:21.008 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:21.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.138 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:21.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:21.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:21.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:21.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:21.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:21 smithi178 bash[29889]: cluster 2024-09-13T22:54:20.091226+0000 osd.4 (osd.4) 234 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:54:21.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:21 smithi178 bash[29889]: cluster 2024-09-13T22:54:20.095459+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:54:21.369 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:21 smithi178 bash[22840]: cluster 2024-09-13T22:54:20.091226+0000 osd.4 (osd.4) 234 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:54:21.369 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:21 smithi178 bash[22840]: cluster 2024-09-13T22:54:20.095459+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:54:21.386 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:21.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:21 smithi192 bash[24608]: cluster 2024-09-13T22:54:20.091226+0000 osd.4 (osd.4) 234 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:54:21.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:21 smithi192 bash[24608]: cluster 2024-09-13T22:54:20.095459+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:54:21.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:21.523 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:21.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:21.654 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:21.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:21.781 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:21.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:21.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:21.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 49 16 72770 72754 92.7885 57.1875 0.0250021 0.0107726 2024-09-13T22:54:21.915 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:21.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:21.923 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:54:21.923 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:54:21.953 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:54:21.954 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:54:21.984 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:54:21.984 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:54:22.017 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:54:22.018 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:54:22.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:22.045 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:22.046 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:54:22.046 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:54:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:22.071 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:54:22.072 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:54:22.111 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:54:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:54:22.137 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:54:22.137 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:54:22.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:22.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:20.809098+0000 mgr.y (mgr.14613) 623 : cluster [DBG] pgmap v722: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.124890+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.126198+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.377137+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.378505+0000 osd.3 (osd.3) 308 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.386748+0000 osd.2 (osd.2) 241 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:54:22.286 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.390326+0000 osd.2 (osd.2) 242 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:54:22.287 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:22 smithi192 bash[24608]: audit 2024-09-13T22:54:21.824891+0000 mon.c (mon.2) 64 : audit [DBG] from='client.? v1:172.21.15.178:0/3317246618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:22.304 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:22.383 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:20.809098+0000 mgr.y (mgr.14613) 623 : cluster [DBG] pgmap v722: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-13T22:54:22.383 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.124890+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:54:22.383 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.126198+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:54:22.383 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.377137+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.378505+0000 osd.3 (osd.3) 308 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.386748+0000 osd.2 (osd.2) 241 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.390326+0000 osd.2 (osd.2) 242 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[22840]: audit 2024-09-13T22:54:21.824891+0000 mon.c (mon.2) 64 : audit [DBG] from='client.? v1:172.21.15.178:0/3317246618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:20.809098+0000 mgr.y (mgr.14613) 623 : cluster [DBG] pgmap v722: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.124890+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.126198+0000 osd.4 (osd.4) 237 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.377137+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.378505+0000 osd.3 (osd.3) 308 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:54:22.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.386748+0000 osd.2 (osd.2) 241 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:54:22.385 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.390326+0000 osd.2 (osd.2) 242 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:54:22.385 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:22 smithi178 bash[29889]: audit 2024-09-13T22:54:21.824891+0000 mon.c (mon.2) 64 : audit [DBG] from='client.? v1:172.21.15.178:0/3317246618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:22.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:22.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.571 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:22.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:22.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:22.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:22.829 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:22.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 50 16 73890 73874 92.3327 70 0.00741122 0.0108257 2024-09-13T22:54:22.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:22.971 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:22.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:23.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:23.098 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:23.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:23.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:23 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.566559+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:54:23.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:23 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.571575+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:54:23.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:23 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.692830+0000 osd.0 (osd.0) 331 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:54:23.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:23 smithi192 bash[24608]: cluster 2024-09-13T22:54:21.696493+0000 osd.0 (osd.0) 332 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:54:23.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:23 smithi192 bash[24608]: cluster 2024-09-13T22:54:22.254096+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:54:23.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:23 smithi192 bash[24608]: cluster 2024-09-13T22:54:22.255556+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:54:23.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:23.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:23.362 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:23.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:23.489 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:23.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.566559+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:54:23.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.571575+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.692830+0000 osd.0 (osd.0) 331 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[29889]: cluster 2024-09-13T22:54:21.696493+0000 osd.0 (osd.0) 332 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[29889]: cluster 2024-09-13T22:54:22.254096+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[29889]: cluster 2024-09-13T22:54:22.255556+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.566559+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.571575+0000 osd.5 (osd.5) 360 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.692830+0000 osd.0 (osd.0) 331 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[22840]: cluster 2024-09-13T22:54:21.696493+0000 osd.0 (osd.0) 332 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[22840]: cluster 2024-09-13T22:54:22.254096+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:54:23.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:23 smithi178 bash[22840]: cluster 2024-09-13T22:54:22.255556+0000 osd.6 (osd.6) 274 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:54:23.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:23.616 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:23.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:23.747 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:23.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:23.876 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:23.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:23.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 51 16 74728 74712 91.5491 52.375 0.0133032 0.0109181 2024-09-13T22:54:23.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:24.012 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:24.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:24.140 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:24.157 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:54:24.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:24.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:24.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:24.403 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:22.568468+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:54:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:22.571560+0000 osd.5 (osd.5) 362 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:54:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:22.809530+0000 mgr.y (mgr.14613) 624 : cluster [DBG] pgmap v723: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 587 B/s rd, 47 MiB/s wr, 760 op/s 2024-09-13T22:54:24.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:23.376315+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:54:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:23.378040+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:54:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:23.401371+0000 osd.3 (osd.3) 309 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:54:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: cluster 2024-09-13T22:54:23.408898+0000 osd.3 (osd.3) 310 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:54:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: audit 2024-09-13T22:54:23.581277+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:24 smithi192 bash[24608]: audit 2024-09-13T22:54:23.582329+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.178:0/208055930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:24.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:22.568468+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:54:24.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:22.571560+0000 osd.5 (osd.5) 362 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:22.809530+0000 mgr.y (mgr.14613) 624 : cluster [DBG] pgmap v723: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 587 B/s rd, 47 MiB/s wr, 760 op/s 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:23.376315+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:23.378040+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:23.401371+0000 osd.3 (osd.3) 309 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: cluster 2024-09-13T22:54:23.408898+0000 osd.3 (osd.3) 310 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: audit 2024-09-13T22:54:23.581277+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:24.411 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[29889]: audit 2024-09-13T22:54:23.582329+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.178:0/208055930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:24.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:22.568468+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:54:24.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:22.571560+0000 osd.5 (osd.5) 362 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:54:24.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:22.809530+0000 mgr.y (mgr.14613) 624 : cluster [DBG] pgmap v723: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 587 B/s rd, 47 MiB/s wr, 760 op/s 2024-09-13T22:54:24.413 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:23.376315+0000 osd.2 (osd.2) 243 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:54:24.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:23.378040+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:54:24.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:23.401371+0000 osd.3 (osd.3) 309 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:54:24.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: cluster 2024-09-13T22:54:23.408898+0000 osd.3 (osd.3) 310 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:54:24.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: audit 2024-09-13T22:54:23.581277+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:24.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:24 smithi178 bash[22840]: audit 2024-09-13T22:54:23.582329+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.178:0/208055930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:24.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:24.536 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:24.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T22:54:24.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:24.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:24.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:24.798 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:24.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:24.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 52 16 75863 75847 91.1526 70.9375 0.0239708 0.0109665 2024-09-13T22:54:24.929 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:24.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:25.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:25.067 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:25.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:25.197 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:25.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:25.310 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.147113+0000 mon.a (mon.0) 1494 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.147133+0000 mon.a (mon.0) 1495 : cluster [INF] Cluster is now healthy 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: audit 2024-09-13T22:54:24.149247+0000 mon.a (mon.0) 1496 : audit [INF] from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.153448+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.162229+0000 osd.4 (osd.4) 238 : cluster [DBG] 5.e scrub starts 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.166599+0000 osd.4 (osd.4) 239 : cluster [DBG] 5.e scrub ok 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.234603+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.235667+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.782827+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:54:25.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.785850+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:54:25.343 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:25.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.147113+0000 mon.a (mon.0) 1494 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:54:25.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.147133+0000 mon.a (mon.0) 1495 : cluster [INF] Cluster is now healthy 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: audit 2024-09-13T22:54:24.149247+0000 mon.a (mon.0) 1496 : audit [INF] from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.153448+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.162229+0000 osd.4 (osd.4) 238 : cluster [DBG] 5.e scrub starts 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.166599+0000 osd.4 (osd.4) 239 : cluster [DBG] 5.e scrub ok 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.234603+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.235667+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.782827+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.785850+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.147113+0000 mon.a (mon.0) 1494 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.147133+0000 mon.a (mon.0) 1495 : cluster [INF] Cluster is now healthy 2024-09-13T22:54:25.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: audit 2024-09-13T22:54:24.149247+0000 mon.a (mon.0) 1496 : audit [INF] from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.153448+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.162229+0000 osd.4 (osd.4) 238 : cluster [DBG] 5.e scrub starts 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.166599+0000 osd.4 (osd.4) 239 : cluster [DBG] 5.e scrub ok 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.234603+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.235667+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.782827+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:54:25.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:25 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.785850+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:54:25.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:25.479 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:25.484 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:54:25.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:25.608 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:25.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:25.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:25.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:25.849 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:54:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:54:25.861 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:25.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:25.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 53 15 77021 77006 90.7994 72.4375 0.00990701 0.0110088 2024-09-13T22:54:25.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:26.001 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:26.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.136 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:26.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.266 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:26.371 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.426339+0000 osd.3 (osd.3) 311 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.431619+0000 osd.3 (osd.3) 312 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.565301+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.573980+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:24.810004+0000 mgr.y (mgr.14613) 625 : cluster [DBG] pgmap v725: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:25.123724+0000 osd.4 (osd.4) 240 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:25.126922+0000 osd.4 (osd.4) 241 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:25.410741+0000 osd.3 (osd.3) 313 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:25.412160+0000 osd.3 (osd.3) 314 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: audit 2024-09-13T22:54:25.660526+0000 mgr.y (mgr.14613) 626 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:26.381 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:25.818385+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[29889]: cluster 2024-09-13T22:54:25.819648+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.426339+0000 osd.3 (osd.3) 311 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.431619+0000 osd.3 (osd.3) 312 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.565301+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.573980+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:24.810004+0000 mgr.y (mgr.14613) 625 : cluster [DBG] pgmap v725: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:25.123724+0000 osd.4 (osd.4) 240 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:25.126922+0000 osd.4 (osd.4) 241 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:25.410741+0000 osd.3 (osd.3) 313 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:25.412160+0000 osd.3 (osd.3) 314 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: audit 2024-09-13T22:54:25.660526+0000 mgr.y (mgr.14613) 626 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:26.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:25.818385+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:54:26.383 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:26 smithi178 bash[22840]: cluster 2024-09-13T22:54:25.819648+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:54:26.398 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:26.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.426339+0000 osd.3 (osd.3) 311 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.431619+0000 osd.3 (osd.3) 312 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.565301+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.573980+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:24.810004+0000 mgr.y (mgr.14613) 625 : cluster [DBG] pgmap v725: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:25.123724+0000 osd.4 (osd.4) 240 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:25.126922+0000 osd.4 (osd.4) 241 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:25.410741+0000 osd.3 (osd.3) 313 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:25.412160+0000 osd.3 (osd.3) 314 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: audit 2024-09-13T22:54:25.660526+0000 mgr.y (mgr.14613) 626 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:25.818385+0000 osd.7 (osd.7) 339 : cluster [DBG] 6.8 scrub starts 2024-09-13T22:54:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:26 smithi192 bash[24608]: cluster 2024-09-13T22:54:25.819648+0000 osd.7 (osd.7) 340 : cluster [DBG] 6.8 scrub ok 2024-09-13T22:54:26.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:26.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:54:26.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:26.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:26.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:26.796 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:26.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 54 16 78151 78135 90.4245 70.5625 0.0284878 0.0110546 2024-09-13T22:54:26.942 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:26.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:27.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:27.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:27.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:27.213 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:27.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:27.351 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:27.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:27 smithi178 bash[29889]: cluster 2024-09-13T22:54:26.165786+0000 osd.1 (osd.1) 308 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T22:54:27.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:27 smithi178 bash[29889]: cluster 2024-09-13T22:54:26.175570+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T22:54:27.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:27 smithi178 bash[22840]: cluster 2024-09-13T22:54:26.165786+0000 osd.1 (osd.1) 308 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T22:54:27.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:27 smithi178 bash[22840]: cluster 2024-09-13T22:54:26.175570+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T22:54:27.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:27.489 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:27.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:54:27.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:27.654 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:27.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:27 smithi192 bash[24608]: cluster 2024-09-13T22:54:26.165786+0000 osd.1 (osd.1) 308 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T22:54:27.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:27 smithi192 bash[24608]: cluster 2024-09-13T22:54:26.175570+0000 osd.1 (osd.1) 309 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T22:54:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:27.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:27.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:27.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:27.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 55 16 79201 79185 89.9735 65.625 0.0244105 0.0111094 2024-09-13T22:54:27.932 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:27.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:28.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.061 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:28.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:28.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:28.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:26.425007+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.c deep-scrub starts 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:26.426356+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.c deep-scrub ok 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:26.810540+0000 mgr.y (mgr.14613) 627 : cluster [DBG] pgmap v726: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.340113+0000 osd.6 (osd.6) 277 : cluster [DBG] 10.7 scrub starts 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.671336+0000 osd.0 (osd.0) 333 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.673483+0000 osd.0 (osd.0) 334 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.846158+0000 osd.7 (osd.7) 341 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.850018+0000 osd.7 (osd.7) 342 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:54:28.315 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:26.425007+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.c deep-scrub starts 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:26.426356+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.c deep-scrub ok 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:26.810540+0000 mgr.y (mgr.14613) 627 : cluster [DBG] pgmap v726: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.340113+0000 osd.6 (osd.6) 277 : cluster [DBG] 10.7 scrub starts 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.671336+0000 osd.0 (osd.0) 333 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.673483+0000 osd.0 (osd.0) 334 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.846158+0000 osd.7 (osd.7) 341 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:54:28.316 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:28 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.850018+0000 osd.7 (osd.7) 342 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:54:28.333 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:28.437 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.465 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:28.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:26.425007+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.c deep-scrub starts 2024-09-13T22:54:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:26.426356+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.c deep-scrub ok 2024-09-13T22:54:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:26.810540+0000 mgr.y (mgr.14613) 627 : cluster [DBG] pgmap v726: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T22:54:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.340113+0000 osd.6 (osd.6) 277 : cluster [DBG] 10.7 scrub starts 2024-09-13T22:54:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.671336+0000 osd.0 (osd.0) 333 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:54:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.673483+0000 osd.0 (osd.0) 334 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:54:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.846158+0000 osd.7 (osd.7) 341 : cluster [DBG] 5.9 scrub starts 2024-09-13T22:54:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:28 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.850018+0000 osd.7 (osd.7) 342 : cluster [DBG] 5.9 scrub ok 2024-09-13T22:54:28.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:28.718 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:28.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:28.857 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:28.912 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 56 16 80203 80187 89.4851 62.625 0.0162449 0.0111704 2024-09-13T22:54:28.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.995 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:28.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:29.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:29.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:29.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.478454+0000 osd.3 (osd.3) 317 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:54:29.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[29889]: cluster 2024-09-13T22:54:27.486847+0000 osd.3 (osd.3) 318 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:54:29.238 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[29889]: cluster 2024-09-13T22:54:28.097079+0000 osd.4 (osd.4) 242 : cluster [DBG] 10.e scrub starts 2024-09-13T22:54:29.238 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[29889]: cluster 2024-09-13T22:54:28.831231+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.b scrub starts 2024-09-13T22:54:29.238 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[29889]: cluster 2024-09-13T22:54:28.840658+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.b scrub ok 2024-09-13T22:54:29.238 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.478454+0000 osd.3 (osd.3) 317 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:54:29.239 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[22840]: cluster 2024-09-13T22:54:27.486847+0000 osd.3 (osd.3) 318 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:54:29.239 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[22840]: cluster 2024-09-13T22:54:28.097079+0000 osd.4 (osd.4) 242 : cluster [DBG] 10.e scrub starts 2024-09-13T22:54:29.239 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[22840]: cluster 2024-09-13T22:54:28.831231+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.b scrub starts 2024-09-13T22:54:29.239 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[22840]: cluster 2024-09-13T22:54:28.840658+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.b scrub ok 2024-09-13T22:54:29.248 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:29.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:29.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:29 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.478454+0000 osd.3 (osd.3) 317 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:54:29.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:29 smithi192 bash[24608]: cluster 2024-09-13T22:54:27.486847+0000 osd.3 (osd.3) 318 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:54:29.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:29 smithi192 bash[24608]: cluster 2024-09-13T22:54:28.097079+0000 osd.4 (osd.4) 242 : cluster [DBG] 10.e scrub starts 2024-09-13T22:54:29.364 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:29 smithi192 bash[24608]: cluster 2024-09-13T22:54:28.831231+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.b scrub starts 2024-09-13T22:54:29.364 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:29 smithi192 bash[24608]: cluster 2024-09-13T22:54:28.840658+0000 osd.7 (osd.7) 344 : cluster [DBG] 2.b scrub ok 2024-09-13T22:54:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:29.498 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:29.501 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:29.526 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:29.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:29.655 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:29.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:29.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:29.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:29.894 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:54:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:54:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:54:29.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:29.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 57 16 80936 80920 88.7117 45.8125 0.00903708 0.0112675 2024-09-13T22:54:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:29.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:30.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.047 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:30.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:30.172 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:30.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:30.409 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[29889]: cluster 2024-09-13T22:54:28.811301+0000 mgr.y (mgr.14613) 628 : cluster [DBG] pgmap v727: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[29889]: cluster 2024-09-13T22:54:29.188642+0000 osd.6 (osd.6) 278 : cluster [DBG] 10.7 scrub ok 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[29889]: cluster 2024-09-13T22:54:29.854764+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[29889]: cluster 2024-09-13T22:54:29.857057+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[22840]: cluster 2024-09-13T22:54:28.811301+0000 mgr.y (mgr.14613) 628 : cluster [DBG] pgmap v727: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[22840]: cluster 2024-09-13T22:54:29.188642+0000 osd.6 (osd.6) 278 : cluster [DBG] 10.7 scrub ok 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[22840]: cluster 2024-09-13T22:54:29.854764+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:54:30.419 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:30 smithi178 bash[22840]: cluster 2024-09-13T22:54:29.857057+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:54:30.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:30.516 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:30.537 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.576 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:30.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:30 smithi192 bash[24608]: cluster 2024-09-13T22:54:28.811301+0000 mgr.y (mgr.14613) 628 : cluster [DBG] pgmap v727: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:30 smithi192 bash[24608]: cluster 2024-09-13T22:54:29.188642+0000 osd.6 (osd.6) 278 : cluster [DBG] 10.7 scrub ok 2024-09-13T22:54:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:30 smithi192 bash[24608]: cluster 2024-09-13T22:54:29.854764+0000 osd.7 (osd.7) 345 : cluster [DBG] 2.1c scrub starts 2024-09-13T22:54:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:30 smithi192 bash[24608]: cluster 2024-09-13T22:54:29.857057+0000 osd.7 (osd.7) 346 : cluster [DBG] 2.1c scrub ok 2024-09-13T22:54:30.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.701 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:30.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:30.827 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:30.917 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:54:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:54:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:54:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:54:30.917 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:54:30.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 58 16 81687 81671 87.9914 46.9375 0.00483877 0.0113294 2024-09-13T22:54:30.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:30.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:30.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:31.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:31.089 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:31.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:31.229 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:31.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:31.342 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:29.583438+0000 osd.5 (osd.5) 365 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:54:31.342 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:29.591209+0000 osd.5 (osd.5) 366 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:54:31.342 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.090191+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:54:31.342 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.090424+0000 osd.1 (osd.1) 310 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:54:31.342 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.093943+0000 osd.1 (osd.1) 311 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.095309+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.238670+0000 osd.4 (osd.4) 245 : cluster [DBG] 10.e scrub ok 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:29.583438+0000 osd.5 (osd.5) 365 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:29.591209+0000 osd.5 (osd.5) 366 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.090191+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.090424+0000 osd.1 (osd.1) 310 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.093943+0000 osd.1 (osd.1) 311 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.095309+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:54:31.343 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:31 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.238670+0000 osd.4 (osd.4) 245 : cluster [DBG] 10.e scrub ok 2024-09-13T22:54:31.360 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:31.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:29.583438+0000 osd.5 (osd.5) 365 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:54:31.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:29.591209+0000 osd.5 (osd.5) 366 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:54:31.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.090191+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.14 scrub starts 2024-09-13T22:54:31.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.090424+0000 osd.1 (osd.1) 310 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:54:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.093943+0000 osd.1 (osd.1) 311 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:54:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.095309+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.14 scrub ok 2024-09-13T22:54:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:31 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.238670+0000 osd.4 (osd.4) 245 : cluster [DBG] 10.e scrub ok 2024-09-13T22:54:31.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:31.492 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:31.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:31.541 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:31.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:31.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:31.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:31.767 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:31.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:31.886 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:31.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:31.917 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 59 16 82555 82539 87.4194 54.25 0.00394177 0.0114205 2024-09-13T22:54:31.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:32.008 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:32.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:32.139 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:32.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:32.270 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:32.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.567937+0000 osd.5 (osd.5) 367 : cluster [DBG] 6.a scrub starts 2024-09-13T22:54:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.580752+0000 osd.5 (osd.5) 368 : cluster [DBG] 6.a scrub ok 2024-09-13T22:54:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:30.812621+0000 mgr.y (mgr.14613) 629 : cluster [DBG] pgmap v728: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:31.066774+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.d scrub starts 2024-09-13T22:54:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:31.067925+0000 osd.1 (osd.1) 313 : cluster [DBG] 2.d scrub ok 2024-09-13T22:54:32.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:31.836536+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:54:32.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:32 smithi192 bash[24608]: cluster 2024-09-13T22:54:31.842936+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:54:32.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:32.454 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.567937+0000 osd.5 (osd.5) 367 : cluster [DBG] 6.a scrub starts 2024-09-13T22:54:32.454 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.580752+0000 osd.5 (osd.5) 368 : cluster [DBG] 6.a scrub ok 2024-09-13T22:54:32.454 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:30.812621+0000 mgr.y (mgr.14613) 629 : cluster [DBG] pgmap v728: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:32.454 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:31.066774+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.d scrub starts 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:31.067925+0000 osd.1 (osd.1) 313 : cluster [DBG] 2.d scrub ok 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:31.836536+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[29889]: cluster 2024-09-13T22:54:31.842936+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.567937+0000 osd.5 (osd.5) 367 : cluster [DBG] 6.a scrub starts 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.580752+0000 osd.5 (osd.5) 368 : cluster [DBG] 6.a scrub ok 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:30.812621+0000 mgr.y (mgr.14613) 629 : cluster [DBG] pgmap v728: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:31.066774+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.d scrub starts 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:31.067925+0000 osd.1 (osd.1) 313 : cluster [DBG] 2.d scrub ok 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:31.836536+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:54:32.455 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:32 smithi178 bash[22840]: cluster 2024-09-13T22:54:31.842936+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:54:32.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:32.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:32.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T22:54:32.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:32.689 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:32.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:32.824 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:32.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:54:32.919206+0000 min lat: 0.00112903 max lat: 0.654557 avg lat: 0.0115358 2024-09-13T22:54:32.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:54:32.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 60 16 83215 83199 86.65 41.25 0.0168539 0.0115358 2024-09-13T22:54:32.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:32.945 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:32.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:33.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:33.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:33.175 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:33.200 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:33.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:33.331 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:33 smithi178 bash[29889]: cluster 2024-09-13T22:54:32.076462+0000 osd.1 (osd.1) 314 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:54:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:33 smithi178 bash[29889]: cluster 2024-09-13T22:54:32.084129+0000 osd.1 (osd.1) 315 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:54:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:33 smithi178 bash[29889]: cluster 2024-09-13T22:54:32.099092+0000 osd.4 (osd.4) 246 : cluster [DBG] 10.1 scrub starts 2024-09-13T22:54:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:33 smithi178 bash[22840]: cluster 2024-09-13T22:54:32.076462+0000 osd.1 (osd.1) 314 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:54:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:33 smithi178 bash[22840]: cluster 2024-09-13T22:54:32.084129+0000 osd.1 (osd.1) 315 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:54:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:33 smithi178 bash[22840]: cluster 2024-09-13T22:54:32.099092+0000 osd.4 (osd.4) 246 : cluster [DBG] 10.1 scrub starts 2024-09-13T22:54:33.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:33.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:33.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:33 smithi192 bash[24608]: cluster 2024-09-13T22:54:32.076462+0000 osd.1 (osd.1) 314 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:54:33.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:33 smithi192 bash[24608]: cluster 2024-09-13T22:54:32.084129+0000 osd.1 (osd.1) 315 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:54:33.474 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:33 smithi192 bash[24608]: cluster 2024-09-13T22:54:32.099092+0000 osd.4 (osd.4) 246 : cluster [DBG] 10.1 scrub starts 2024-09-13T22:54:33.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:33.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:33.710 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:33.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:33.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:33.864 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:33.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:33.917 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 61 16 84036 84020 86.0706 51.3125 0.013993 0.0116132 2024-09-13T22:54:33.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:34.003 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:34.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:34.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:34.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:34.254 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:34.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:34.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:34 smithi192 bash[24608]: cluster 2024-09-13T22:54:32.813148+0000 mgr.y (mgr.14613) 630 : cluster [DBG] pgmap v729: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:34.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:34 smithi192 bash[24608]: cluster 2024-09-13T22:54:33.328853+0000 osd.6 (osd.6) 279 : cluster [DBG] 10.0 scrub starts 2024-09-13T22:54:34.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:34 smithi192 bash[24608]: audit 2024-09-13T22:54:34.026863+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:34.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:34 smithi192 bash[24608]: audit 2024-09-13T22:54:34.033380+0000 mon.b (mon.1) 232 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:34.397 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:34.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[29889]: cluster 2024-09-13T22:54:32.813148+0000 mgr.y (mgr.14613) 630 : cluster [DBG] pgmap v729: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:34.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[29889]: cluster 2024-09-13T22:54:33.328853+0000 osd.6 (osd.6) 279 : cluster [DBG] 10.0 scrub starts 2024-09-13T22:54:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[29889]: audit 2024-09-13T22:54:34.026863+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[29889]: audit 2024-09-13T22:54:34.033380+0000 mon.b (mon.1) 232 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[22840]: cluster 2024-09-13T22:54:32.813148+0000 mgr.y (mgr.14613) 630 : cluster [DBG] pgmap v729: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 996 op/s 2024-09-13T22:54:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[22840]: cluster 2024-09-13T22:54:33.328853+0000 osd.6 (osd.6) 279 : cluster [DBG] 10.0 scrub starts 2024-09-13T22:54:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[22840]: audit 2024-09-13T22:54:34.026863+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:34 smithi178 bash[22840]: audit 2024-09-13T22:54:34.033380+0000 mon.b (mon.1) 232 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:34.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:34.525 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:34.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:34.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:34.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:34.792 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:34.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:34.917 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 62 16 84933 84917 85.5866 56.0625 0.0185836 0.0116788 2024-09-13T22:54:34.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:34.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:35.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:35.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:35.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:35.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:35.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:35.301 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.132502+0000 osd.4 (osd.4) 247 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:54:35.301 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.133443+0000 osd.4 (osd.4) 248 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:54:35.301 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.291197+0000 osd.6 (osd.6) 280 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T22:54:35.301 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.302933+0000 osd.6 (osd.6) 281 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T22:54:35.301 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.324144+0000 osd.4 (osd.4) 249 : cluster [DBG] 10.1 scrub ok 2024-09-13T22:54:35.301 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.773859+0000 osd.0 (osd.0) 335 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.778824+0000 osd.0 (osd.0) 336 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.092500+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.097338+0000 osd.1 (osd.1) 317 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.132502+0000 osd.4 (osd.4) 247 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.133443+0000 osd.4 (osd.4) 248 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.291197+0000 osd.6 (osd.6) 280 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.302933+0000 osd.6 (osd.6) 281 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.324144+0000 osd.4 (osd.4) 249 : cluster [DBG] 10.1 scrub ok 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.773859+0000 osd.0 (osd.0) 335 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.778824+0000 osd.0 (osd.0) 336 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:54:35.302 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.092500+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:54:35.303 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:35 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.097338+0000 osd.1 (osd.1) 317 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:54:35.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:35.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:35.464 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:35.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:35.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.132502+0000 osd.4 (osd.4) 247 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.133443+0000 osd.4 (osd.4) 248 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.291197+0000 osd.6 (osd.6) 280 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.302933+0000 osd.6 (osd.6) 281 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.324144+0000 osd.4 (osd.4) 249 : cluster [DBG] 10.1 scrub ok 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.773859+0000 osd.0 (osd.0) 335 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.778824+0000 osd.0 (osd.0) 336 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.092500+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:54:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.097338+0000 osd.1 (osd.1) 317 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:54:35.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.753 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:35.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:35.854 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:35.884 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:35.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:35.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 63 16 85950 85934 85.237 63.5625 0.0103565 0.0117254 2024-09-13T22:54:35.987 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:36.018 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:36.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:36.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:54:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:54:36.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:36.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:36.301 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:36.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.431728+0000 osd.5 (osd.5) 369 : cluster [DBG] 3.c scrub starts 2024-09-13T22:54:36.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.437154+0000 osd.5 (osd.5) 370 : cluster [DBG] 3.c scrub ok 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.510066+0000 osd.2 (osd.2) 245 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.511487+0000 osd.2 (osd.2) 246 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.648828+0000 osd.6 (osd.6) 282 : cluster [DBG] 10.0 scrub ok 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:34.814187+0000 mgr.y (mgr.14613) 631 : cluster [DBG] pgmap v730: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 864 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.110787+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub starts 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.115916+0000 osd.4 (osd.4) 251 : cluster [DBG] 2.f scrub ok 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: audit 2024-09-13T22:54:35.296345+0000 mon.b (mon.1) 233 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.338946+0000 osd.3 (osd.3) 319 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.340599+0000 osd.3 (osd.3) 320 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.463509+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:35.464725+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: audit 2024-09-13T22:54:35.672558+0000 mgr.y (mgr.14613) 632 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:36.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:36.095611+0000 osd.1 (osd.1) 318 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[29889]: cluster 2024-09-13T22:54:36.099466+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.431728+0000 osd.5 (osd.5) 369 : cluster [DBG] 3.c scrub starts 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.437154+0000 osd.5 (osd.5) 370 : cluster [DBG] 3.c scrub ok 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.510066+0000 osd.2 (osd.2) 245 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.511487+0000 osd.2 (osd.2) 246 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.648828+0000 osd.6 (osd.6) 282 : cluster [DBG] 10.0 scrub ok 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:34.814187+0000 mgr.y (mgr.14613) 631 : cluster [DBG] pgmap v730: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 864 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.110787+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub starts 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.115916+0000 osd.4 (osd.4) 251 : cluster [DBG] 2.f scrub ok 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: audit 2024-09-13T22:54:35.296345+0000 mon.b (mon.1) 233 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:54:36.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.338946+0000 osd.3 (osd.3) 319 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:54:36.352 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.340599+0000 osd.3 (osd.3) 320 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:54:36.352 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.463509+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:54:36.352 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:35.464725+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:54:36.352 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: audit 2024-09-13T22:54:35.672558+0000 mgr.y (mgr.14613) 632 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:36.352 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:36.095611+0000 osd.1 (osd.1) 318 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:54:36.352 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:36 smithi178 bash[22840]: cluster 2024-09-13T22:54:36.099466+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:54:36.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.431728+0000 osd.5 (osd.5) 369 : cluster [DBG] 3.c scrub starts 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.437154+0000 osd.5 (osd.5) 370 : cluster [DBG] 3.c scrub ok 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.510066+0000 osd.2 (osd.2) 245 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.511487+0000 osd.2 (osd.2) 246 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.648828+0000 osd.6 (osd.6) 282 : cluster [DBG] 10.0 scrub ok 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:34.814187+0000 mgr.y (mgr.14613) 631 : cluster [DBG] pgmap v730: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 864 B/s rd, 55 MiB/s wr, 874 op/s 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.110787+0000 osd.4 (osd.4) 250 : cluster [DBG] 2.f scrub starts 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.115916+0000 osd.4 (osd.4) 251 : cluster [DBG] 2.f scrub ok 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: audit 2024-09-13T22:54:35.296345+0000 mon.b (mon.1) 233 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.338946+0000 osd.3 (osd.3) 319 : cluster [DBG] 2.19 scrub starts 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.340599+0000 osd.3 (osd.3) 320 : cluster [DBG] 2.19 scrub ok 2024-09-13T22:54:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.463509+0000 osd.5 (osd.5) 371 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:54:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:35.464725+0000 osd.5 (osd.5) 372 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:54:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: audit 2024-09-13T22:54:35.672558+0000 mgr.y (mgr.14613) 632 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:36.095611+0000 osd.1 (osd.1) 318 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:54:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:36 smithi192 bash[24608]: cluster 2024-09-13T22:54:36.099466+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:54:36.439 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:36.536 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:54:36.536 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:54:36.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:36.573 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:36.596 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:54:36.597 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:54:36.632 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:54:36.633 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:54:36.671 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:54:36.672 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:54:36.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:36.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:36.711 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:54:36.711 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:54:36.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:36.740 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:54:36.740 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:54:36.776 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:54:36.776 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:54:36.808 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:54:36.809 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:54:36.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:36.848 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:36.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 64 16 86819 86803 84.7537 54.3125 0.0140285 0.0117941 2024-09-13T22:54:36.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:36.979 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:36.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:37.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:37.108 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:37.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:37.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:37.307 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:54:37.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:37.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:37.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[29889]: audit 2024-09-13T22:54:36.277194+0000 mon.b (mon.1) 234 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[29889]: audit 2024-09-13T22:54:36.279087+0000 mon.b (mon.1) 235 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[29889]: audit 2024-09-13T22:54:36.288341+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[29889]: audit 2024-09-13T22:54:36.469163+0000 mon.a (mon.0) 1500 : audit [DBG] from='client.? v1:172.21.15.178:0/1448472804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[29889]: audit 2024-09-13T22:54:36.853323+0000 mon.c (mon.2) 65 : audit [INF] from='client.? v1:172.21.15.178:0/1968647993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[29889]: audit 2024-09-13T22:54:36.853871+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25598 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[22840]: audit 2024-09-13T22:54:36.277194+0000 mon.b (mon.1) 234 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[22840]: audit 2024-09-13T22:54:36.279087+0000 mon.b (mon.1) 235 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[22840]: audit 2024-09-13T22:54:36.288341+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[22840]: audit 2024-09-13T22:54:36.469163+0000 mon.a (mon.0) 1500 : audit [DBG] from='client.? v1:172.21.15.178:0/1448472804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:37.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[22840]: audit 2024-09-13T22:54:36.853323+0000 mon.c (mon.2) 65 : audit [INF] from='client.? v1:172.21.15.178:0/1968647993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:54:37.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:37 smithi178 bash[22840]: audit 2024-09-13T22:54:36.853871+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25598 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:54:37.461 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:54:37.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:37.518 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:37.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:37 smithi192 bash[24608]: audit 2024-09-13T22:54:36.277194+0000 mon.b (mon.1) 234 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:54:37.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:37 smithi192 bash[24608]: audit 2024-09-13T22:54:36.279087+0000 mon.b (mon.1) 235 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:54:37.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:37 smithi192 bash[24608]: audit 2024-09-13T22:54:36.288341+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:37.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:37 smithi192 bash[24608]: audit 2024-09-13T22:54:36.469163+0000 mon.a (mon.0) 1500 : audit [DBG] from='client.? v1:172.21.15.178:0/1448472804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:37.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:37 smithi192 bash[24608]: audit 2024-09-13T22:54:36.853323+0000 mon.c (mon.2) 65 : audit [INF] from='client.? v1:172.21.15.178:0/1968647993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:54:37.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:37 smithi192 bash[24608]: audit 2024-09-13T22:54:36.853871+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25598 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:54:37.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:37.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:37.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:37.792 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:37.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:37.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 65 16 87832 87816 84.4238 63.3125 0.00766231 0.0118401 2024-09-13T22:54:37.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:37.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:38.023 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:38.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:38.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:38.174 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:38.277 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:36.814710+0000 mgr.y (mgr.14613) 633 : cluster [DBG] pgmap v731: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 767 B/s rd, 52 MiB/s wr, 840 op/s 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:37.150195+0000 osd.3 (osd.3) 321 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:37.150265+0000 osd.3 (osd.3) 322 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:37.278501+0000 osd.3 (osd.3) 323 : cluster [DBG] 1.4 scrub starts 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:37.279935+0000 osd.3 (osd.3) 324 : cluster [DBG] 1.4 scrub ok 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:37.286117+0000 mon.a (mon.0) 1502 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: audit 2024-09-13T22:54:37.296959+0000 mon.a (mon.0) 1503 : audit [INF] from='client.25598 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:54:38.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[29889]: cluster 2024-09-13T22:54:37.300721+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-13T22:54:38.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:36.814710+0000 mgr.y (mgr.14613) 633 : cluster [DBG] pgmap v731: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 767 B/s rd, 52 MiB/s wr, 840 op/s 2024-09-13T22:54:38.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:37.150195+0000 osd.3 (osd.3) 321 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:38.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:37.150265+0000 osd.3 (osd.3) 322 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:38.294 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:37.278501+0000 osd.3 (osd.3) 323 : cluster [DBG] 1.4 scrub starts 2024-09-13T22:54:38.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:37.279935+0000 osd.3 (osd.3) 324 : cluster [DBG] 1.4 scrub ok 2024-09-13T22:54:38.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:37.286117+0000 mon.a (mon.0) 1502 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:38.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: audit 2024-09-13T22:54:37.296959+0000 mon.a (mon.0) 1503 : audit [INF] from='client.25598 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:54:38.295 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:38 smithi178 bash[22840]: cluster 2024-09-13T22:54:37.300721+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-13T22:54:38.306 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:38.412 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:38.437 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:38.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:38.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:36.814710+0000 mgr.y (mgr.14613) 633 : cluster [DBG] pgmap v731: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 767 B/s rd, 52 MiB/s wr, 840 op/s 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:37.150195+0000 osd.3 (osd.3) 321 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:37.150265+0000 osd.3 (osd.3) 322 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:37.278501+0000 osd.3 (osd.3) 323 : cluster [DBG] 1.4 scrub starts 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:37.279935+0000 osd.3 (osd.3) 324 : cluster [DBG] 1.4 scrub ok 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:37.286117+0000 mon.a (mon.0) 1502 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: audit 2024-09-13T22:54:37.296959+0000 mon.a (mon.0) 1503 : audit [INF] from='client.25598 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:54:38.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:38 smithi192 bash[24608]: cluster 2024-09-13T22:54:37.300721+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-13T22:54:38.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:38.697 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:38.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T22:54:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:38.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:38.822 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:38.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 66 16 88549 88533 83.8236 44.8125 0.00416793 0.0118978 2024-09-13T22:54:38.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:38.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:38.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:39.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:39.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:39.196 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:39.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:39.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:39.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.454 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:39.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.597 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:39.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.739 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:39.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:39.863 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:39.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 67 16 89190 89174 83.1705 40.0625 0.00274932 0.0120047 2024-09-13T22:54:39.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.991 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:39.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:40.094 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:40.098 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:54:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:54:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:54:40.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:40.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:40.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:40.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:40.402 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:40.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:40 smithi178 bash[29889]: cluster 2024-09-13T22:54:38.815371+0000 mgr.y (mgr.14613) 634 : cluster [DBG] pgmap v733: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 865 op/s 2024-09-13T22:54:40.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:40 smithi178 bash[22840]: cluster 2024-09-13T22:54:38.815371+0000 mgr.y (mgr.14613) 634 : cluster [DBG] pgmap v733: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 865 op/s 2024-09-13T22:54:40.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:40.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:40.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:40.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:40 smithi192 bash[24608]: cluster 2024-09-13T22:54:38.815371+0000 mgr.y (mgr.14613) 634 : cluster [DBG] pgmap v733: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 865 op/s 2024-09-13T22:54:40.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:40.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:40.909 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:40.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 68 16 90060 90044 82.747 54.375 0.0162931 0.01208 2024-09-13T22:54:40.939 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:40.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:41.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:41.074 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:41.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:41.222 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:41.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:41.354 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:41.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:41.490 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:41.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:41.629 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:41.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:41.773 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:41.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:41.916 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:41.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 69 16 91158 91142 82.5422 68.625 0.00693388 0.0121106 2024-09-13T22:54:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:41.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:42.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:42.037 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:54:42.037 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:54:42.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:42.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:54:42.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:42.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:54:42.192 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:42.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:42.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:42.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:42 smithi192 bash[24608]: cluster 2024-09-13T22:54:40.815909+0000 mgr.y (mgr.14613) 635 : cluster [DBG] pgmap v734: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-13T22:54:42.437 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:42 smithi178 bash[22840]: cluster 2024-09-13T22:54:40.815909+0000 mgr.y (mgr.14613) 635 : cluster [DBG] pgmap v734: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-13T22:54:42.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:42 smithi178 bash[29889]: cluster 2024-09-13T22:54:40.815909+0000 mgr.y (mgr.14613) 635 : cluster [DBG] pgmap v734: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-13T22:54:42.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:42.463 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:42.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:42.596 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:42.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:42.731 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:42.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:42.868 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:42.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:42.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 70 16 92174 92158 82.2702 63.5 0.00806894 0.0121495 2024-09-13T22:54:42.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:43.003 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:43.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:43.138 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:43.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:43.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T22:54:43.269 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:43.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:43.398 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:43.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:43.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:43.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:43.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:43.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:43.805 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:43.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:43.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 71 16 93227 93211 82.0383 65.8125 0.00798633 0.012184 2024-09-13T22:54:43.941 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:43.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:44.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:44.063 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:44.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:44.189 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:44.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:44.321 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:44.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:44.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:44 smithi192 bash[24608]: cluster 2024-09-13T22:54:42.816363+0000 mgr.y (mgr.14613) 636 : cluster [DBG] pgmap v735: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-13T22:54:44.451 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:44.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:44.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:44 smithi178 bash[29889]: cluster 2024-09-13T22:54:42.816363+0000 mgr.y (mgr.14613) 636 : cluster [DBG] pgmap v735: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-13T22:54:44.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:44 smithi178 bash[22840]: cluster 2024-09-13T22:54:42.816363+0000 mgr.y (mgr.14613) 636 : cluster [DBG] pgmap v735: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 848 op/s 2024-09-13T22:54:44.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:44.720 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:44.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:44.856 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:44.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 72 16 93711 93695 81.3191 30.25 0.0133913 0.0122927 2024-09-13T22:54:44.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:44.984 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:44.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:45.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:45.114 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:45.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:45.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:45.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:45.380 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:45.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:45.503 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:45.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:45.629 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:45.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:45.757 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:45.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:45.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:54:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:54:45.889 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:45.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:45.918 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 73 16 94815 94799 81.1502 69 0.00988718 0.0123173 2024-09-13T22:54:45.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:46.021 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:46.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:46.161 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:46.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:46.302 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:46.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:46 smithi178 bash[29889]: cluster 2024-09-13T22:54:44.817611+0000 mgr.y (mgr.14613) 637 : cluster [DBG] pgmap v736: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 903 op/s 2024-09-13T22:54:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:46 smithi178 bash[29889]: audit 2024-09-13T22:54:45.684558+0000 mgr.y (mgr.14613) 638 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:46 smithi178 bash[22840]: cluster 2024-09-13T22:54:44.817611+0000 mgr.y (mgr.14613) 637 : cluster [DBG] pgmap v736: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 903 op/s 2024-09-13T22:54:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:46 smithi178 bash[22840]: audit 2024-09-13T22:54:45.684558+0000 mgr.y (mgr.14613) 638 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:46.435 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:46.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:46.550 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:46 smithi192 bash[24608]: cluster 2024-09-13T22:54:44.817611+0000 mgr.y (mgr.14613) 637 : cluster [DBG] pgmap v736: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 903 op/s 2024-09-13T22:54:46.550 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:46 smithi192 bash[24608]: audit 2024-09-13T22:54:45.684558+0000 mgr.y (mgr.14613) 638 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.580 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:46.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:46.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:46.710 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:46.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:46.839 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:46.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 74 16 95915 95899 80.9826 68.75 0.0213237 0.0123435 2024-09-13T22:54:46.943 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:46.971 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:46.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:47.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:47.109 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:47.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:47.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T22:54:47.237 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:47.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:47.374 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:47.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:47.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:47.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:47.627 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:47.627 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:47.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:47.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:47.778 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:47.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:47.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 75 16 96990 96974 80.7986 67.1875 0.0102215 0.0123718 2024-09-13T22:54:47.920 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:47.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:48.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:48.060 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:48.108 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:54:48.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:48.189 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:48.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.330 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:48.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:48 smithi192 bash[24608]: cluster 2024-09-13T22:54:46.818258+0000 mgr.y (mgr.14613) 639 : cluster [DBG] pgmap v737: 158 pgs: 158 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-13T22:54:48.469 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:48.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:48.580 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:48 smithi178 bash[29889]: cluster 2024-09-13T22:54:46.818258+0000 mgr.y (mgr.14613) 639 : cluster [DBG] pgmap v737: 158 pgs: 158 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-13T22:54:48.581 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:48 smithi178 bash[22840]: cluster 2024-09-13T22:54:46.818258+0000 mgr.y (mgr.14613) 639 : cluster [DBG] pgmap v737: 158 pgs: 158 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-13T22:54:48.596 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:48.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:48.732 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:48.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.861 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:48.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:48.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 76 15 98097 98082 80.6466 69.25 0.0111433 0.0123956 2024-09-13T22:54:48.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:48.999 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:49.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:49.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:49.263 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:49.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.401 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:49.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.531 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:49.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:49.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:49.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:49.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:49.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:49.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 77 16 99186 99170 80.4823 68 0.0127625 0.0124202 2024-09-13T22:54:49.930 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:49.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:50.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:50.062 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:50.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:54:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:54:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:50 smithi178 bash[29889]: cluster 2024-09-13T22:54:48.819418+0000 mgr.y (mgr.14613) 640 : cluster [DBG] pgmap v738: 158 pgs: 158 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:50 smithi178 bash[29889]: audit 2024-09-13T22:54:49.051655+0000 mon.a (mon.0) 1505 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:50 smithi178 bash[29889]: audit 2024-09-13T22:54:49.055213+0000 mon.b (mon.1) 236 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:50 smithi178 bash[22840]: cluster 2024-09-13T22:54:48.819418+0000 mgr.y (mgr.14613) 640 : cluster [DBG] pgmap v738: 158 pgs: 158 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:50 smithi178 bash[22840]: audit 2024-09-13T22:54:49.051655+0000 mon.a (mon.0) 1505 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:50.174 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:50 smithi178 bash[22840]: audit 2024-09-13T22:54:49.055213+0000 mon.b (mon.1) 236 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:50.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:50.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:50.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:50 smithi192 bash[24608]: cluster 2024-09-13T22:54:48.819418+0000 mgr.y (mgr.14613) 640 : cluster [DBG] pgmap v738: 158 pgs: 158 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-13T22:54:50.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:50 smithi192 bash[24608]: audit 2024-09-13T22:54:49.051655+0000 mon.a (mon.0) 1505 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:54:50.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:50 smithi192 bash[24608]: audit 2024-09-13T22:54:49.055213+0000 mon.b (mon.1) 236 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:54:50.320 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:50.348 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:54:50.348 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:54:50.424 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:54:50.425 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:54:50.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:50.450 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:54:50.467 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:54:50.513 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:54:50.513 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:54:50.547 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:54:50.547 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:54:50.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:50.578 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:54:50.579 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:54:50.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:50.618 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:54:50.618 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:54:50.665 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:54:50.665 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:54:50.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:50.719 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:50.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:50.852 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:50.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 78 16 100174 100158 80.242 61.75 0.0123664 0.0124577 2024-09-13T22:54:50.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:50.981 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:50.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:51.056 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:54:51.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:51.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:51.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:51.239 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:51 smithi178 bash[29889]: audit 2024-09-13T22:54:50.271238+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.178:0/3774805628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:51.240 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:51 smithi178 bash[29889]: audit 2024-09-13T22:54:50.644126+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.178:0/3981713640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:51.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:51 smithi178 bash[22840]: audit 2024-09-13T22:54:50.271238+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.178:0/3774805628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:51.240 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:51 smithi178 bash[22840]: audit 2024-09-13T22:54:50.644126+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.178:0/3981713640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:51.250 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:51.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:51.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:51.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:51 smithi192 bash[24608]: audit 2024-09-13T22:54:50.271238+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.178:0/3774805628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:54:51.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:51 smithi192 bash[24608]: audit 2024-09-13T22:54:50.644126+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? v1:172.21.15.178:0/3981713640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:54:51.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:51.506 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:51.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:51.645 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:51.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:51.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:51.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:51.910 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:51.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:51.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 79 16 100763 100747 79.6923 36.8125 0.00978482 0.0125435 2024-09-13T22:54:52.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:52.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:52.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:52.154 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:52 smithi192 bash[24608]: cluster 2024-09-13T22:54:50.823222+0000 mgr.y (mgr.14613) 641 : cluster [DBG] pgmap v739: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 881 op/s 2024-09-13T22:54:52.154 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:52 smithi192 bash[24608]: cluster 2024-09-13T22:54:51.050285+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:52.154 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:52 smithi192 bash[24608]: audit 2024-09-13T22:54:51.057149+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? v1:172.21.15.178:0/3981713640' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:54:52.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:52 smithi192 bash[24608]: cluster 2024-09-13T22:54:51.062560+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-13T22:54:52.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:52.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:52.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:52.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:52.417 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:54:52.417 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:54:52.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:101232 10.6 10:67dce959:::benchmark_data_smithi178_111521_object101231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:54:52.403356+0000", 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016558928000000001, 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016597655999999999, 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 101232 2024-09-13T22:54:52.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403356+0000", 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403356+0000", 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403357+0000", 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2300000000000001e-06 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403425+0000", 2024-09-13T22:54:52.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8219999999999994e-05 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403434+0000", 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7649999999999999e-06 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403440+0000", 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1260000000000003e-06 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403493+0000", 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3006e-05 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403629+0000", 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013572899999999999 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-09-13T22:54:52.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.403761+0000", 2024-09-13T22:54:52.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013259799999999999 2024-09-13T22:54:52.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.406718+0000", 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029561219999999998 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.408481+0000", 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001763464 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:54:52.424 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:101236 10.6 10:6999d257:::benchmark_data_smithi178_111521_object101235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:54:52.412018+0000", 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0078965500000000004, 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080283319999999991, 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:54:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 101236 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412018+0000", 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412018+0000", 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412021+0000", 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412063+0000", 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2524999999999997e-05 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412072+0000", 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5790000000000004e-06 2024-09-13T22:54:52.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412079+0000", 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8129999999999998e-06 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412134+0000", 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5693999999999999e-05 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:54:52.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412266+0000", 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013210399999999999 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.412429+0000", 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016224100000000001 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.415201+0000", 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027719960000000001 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.416028+0000", 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00082719599999999999 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25538.0:101226 10.7 e344/322 10:e94911e2:::benchmark_data_smithi178_111521_object101225:head v 344'6468, mlcod=344'6466)", 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:54:52.419502+0000", 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00041322599999999999, 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063703200000000005, 2024-09-13T22:54:52.430 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419502+0000", 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419502+0000", 2024-09-13T22:54:52.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419504+0000", 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419533+0000", 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8979000000000002e-05 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419550+0000", 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7533e-05 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419558+0000", 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2619999999999998e-06 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419653+0000", 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4210999999999996e-05 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:54:52.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:54:52.419673+0000", 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9984999999999999e-05 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 3 2024-09-13T22:54:52.434 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[22840]: cluster 2024-09-13T22:54:50.823222+0000 mgr.y (mgr.14613) 641 : cluster [DBG] pgmap v739: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 881 op/s 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[22840]: cluster 2024-09-13T22:54:51.050285+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[22840]: audit 2024-09-13T22:54:51.057149+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? v1:172.21.15.178:0/3981713640' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[22840]: cluster 2024-09-13T22:54:51.062560+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[29889]: cluster 2024-09-13T22:54:50.823222+0000 mgr.y (mgr.14613) 641 : cluster [DBG] pgmap v739: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 881 op/s 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[29889]: cluster 2024-09-13T22:54:51.050285+0000 mon.a (mon.0) 1507 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:54:52.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[29889]: audit 2024-09-13T22:54:51.057149+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? v1:172.21.15.178:0/3981713640' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:54:52.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:52 smithi178 bash[29889]: cluster 2024-09-13T22:54:51.062560+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-13T22:54:52.449 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:52.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:52.591 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:52.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:52.718 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:52.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:52.848 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:52.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:54:52.920946+0000 min lat: 0.00112903 max lat: 0.654557 avg lat: 0.0125835 2024-09-13T22:54:52.920 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:54:52.920 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 80 16 101717 101701 79.4414 59.625 0.015034 0.0125835 2024-09-13T22:54:52.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:52.976 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:52.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:53.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:53.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:53 smithi178 bash[29889]: cluster 2024-09-13T22:54:51.764232+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:53 smithi178 bash[29889]: cluster 2024-09-13T22:54:51.764280+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:53 smithi178 bash[29889]: cluster 2024-09-13T22:54:52.072588+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-13T22:54:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:53 smithi178 bash[22840]: cluster 2024-09-13T22:54:51.764232+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:53 smithi178 bash[22840]: cluster 2024-09-13T22:54:51.764280+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:53 smithi178 bash[22840]: cluster 2024-09-13T22:54:52.072588+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-13T22:54:53.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:53.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:53.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:53.336 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:53 smithi192 bash[24608]: cluster 2024-09-13T22:54:51.764232+0000 osd.5 (osd.5) 373 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:54:53.337 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:53 smithi192 bash[24608]: cluster 2024-09-13T22:54:51.764280+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:54:53.337 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:53 smithi192 bash[24608]: cluster 2024-09-13T22:54:52.072588+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-13T22:54:53.361 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:53.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:53.489 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:53.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:53.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:54:53.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:53.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T22:54:53.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:53.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:53.901 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:53.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:53.921 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 81 16 102405 102389 78.9915 43 0.00432177 0.0126522 2024-09-13T22:54:54.006 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:54.027 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:54.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:54.154 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:54.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:54 smithi178 bash[29889]: cluster 2024-09-13T22:54:52.823815+0000 mgr.y (mgr.14613) 642 : cluster [DBG] pgmap v742: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 728 op/s 2024-09-13T22:54:54.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:54 smithi178 bash[22840]: cluster 2024-09-13T22:54:52.823815+0000 mgr.y (mgr.14613) 642 : cluster [DBG] pgmap v742: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 728 op/s 2024-09-13T22:54:54.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:54.279 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:54.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:54.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:54 smithi192 bash[24608]: cluster 2024-09-13T22:54:52.823815+0000 mgr.y (mgr.14613) 642 : cluster [DBG] pgmap v742: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 728 op/s 2024-09-13T22:54:54.407 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:54.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:54.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:54.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:54.670 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:54.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:54.805 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:54.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:54:54.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:54.920 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 82 16 103221 103205 78.6501 51 0.0186296 0.0127099 2024-09-13T22:54:54.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:54.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:55.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:55.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:55.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:55.196 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:55.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:54:55.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:55.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:55.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:55.470 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:55.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.612 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:55.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:55.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:55.748 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:55.853 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:55.876 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:55.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:55.919 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 83 16 104228 104212 78.4608 62.9375 0.00530987 0.0127407 2024-09-13T22:54:55.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:55.999 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:56.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:56.078 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:54:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:54:56.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:56.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:56.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:56.252 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:56.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:56.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:56 smithi192 bash[24608]: cluster 2024-09-13T22:54:54.824716+0000 mgr.y (mgr.14613) 643 : cluster [DBG] pgmap v743: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:54:56.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:56 smithi192 bash[24608]: audit 2024-09-13T22:54:55.692613+0000 mgr.y (mgr.14613) 644 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:56.379 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:56.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:56.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:56.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:56.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:56 smithi178 bash[29889]: cluster 2024-09-13T22:54:54.824716+0000 mgr.y (mgr.14613) 643 : cluster [DBG] pgmap v743: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:54:56.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:56 smithi178 bash[29889]: audit 2024-09-13T22:54:55.692613+0000 mgr.y (mgr.14613) 644 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:56 smithi178 bash[22840]: cluster 2024-09-13T22:54:54.824716+0000 mgr.y (mgr.14613) 643 : cluster [DBG] pgmap v743: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T22:54:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:56 smithi178 bash[22840]: audit 2024-09-13T22:54:55.692613+0000 mgr.y (mgr.14613) 644 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:54:56.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T22:54:56.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:56.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:56.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:56.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 84 16 105210 105194 78.2549 61.375 0.0479588 0.012773 2024-09-13T22:54:56.936 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:56.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:57.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:57.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:57.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:57.203 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:57.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:57.342 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:57.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:57.477 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:57.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:57.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:57.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:57.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:57.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:57.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:57.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:57.922 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 85 16 106307 106291 78.1408 68.5625 0.011789 0.0127934 2024-09-13T22:54:57.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:58.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:54:58.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:54:58.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:58.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:54:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:54:58.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:58.271 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:58.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:58.382 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:54:58 smithi178 bash[29889]: cluster 2024-09-13T22:54:56.825248+0000 mgr.y (mgr.14613) 645 : cluster [DBG] pgmap v744: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 701 op/s 2024-09-13T22:54:58.382 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:54:58 smithi178 bash[22840]: cluster 2024-09-13T22:54:56.825248+0000 mgr.y (mgr.14613) 645 : cluster [DBG] pgmap v744: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 701 op/s 2024-09-13T22:54:58.405 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:58.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:54:58 smithi192 bash[24608]: cluster 2024-09-13T22:54:56.825248+0000 mgr.y (mgr.14613) 645 : cluster [DBG] pgmap v744: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 701 op/s 2024-09-13T22:54:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:58.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:58.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:58.600 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:54:58.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:58.678 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:58.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:58.816 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:58.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:58.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 86 16 107389 107373 78.0185 67.625 0.0116326 0.0128133 2024-09-13T22:54:58.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:58.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:59.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:59.084 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:54:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:54:59.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:59.219 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:54:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:54:59.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:59.371 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:59.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:59.504 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:54:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:54:59.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:59.631 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:54:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:54:59.735 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:59.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:54:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:54:59.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:59.902 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:54:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:54:59.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:54:59.922 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 87 16 108473 108457 77.9004 67.75 0.00851384 0.012832 2024-09-13T22:54:59.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:54:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:54:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:00.009 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:00.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:00.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:00.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:00 smithi192 bash[24608]: cluster 2024-09-13T22:54:58.826022+0000 mgr.y (mgr.14613) 646 : cluster [DBG] pgmap v745: 158 pgs: 158 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-13T22:55:00.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:00.276 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:00.277 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:00.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:00.317 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:00.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:00.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:00 smithi178 bash[29889]: cluster 2024-09-13T22:54:58.826022+0000 mgr.y (mgr.14613) 646 : cluster [DBG] pgmap v745: 158 pgs: 158 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-13T22:55:00.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:00 smithi178 bash[22840]: cluster 2024-09-13T22:54:58.826022+0000 mgr.y (mgr.14613) 646 : cluster [DBG] pgmap v745: 158 pgs: 158 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-13T22:55:00.439 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:00.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:00.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:00.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:00.698 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:00.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:00.829 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:00.922 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 88 16 109100 109084 77.4605 39.1875 0.0146318 0.0129055 2024-09-13T22:55:00.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:00.969 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:00.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:01.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:01.101 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:01.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:01.248 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:01.304 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:01.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:01.376 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:01.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:01.500 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:01.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:01.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:01.625 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:01.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:01.754 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:01.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:01.895 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:01.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:01.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 89 16 109742 109726 77.041 40.125 0.00556183 0.0129756 2024-09-13T22:55:02.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.034 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:02.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:02.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:02.144 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:02 smithi192 bash[24608]: cluster 2024-09-13T22:55:00.826735+0000 mgr.y (mgr.14613) 647 : cluster [DBG] pgmap v746: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 945 op/s 2024-09-13T22:55:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.166 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:02.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:02.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:02.292 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:02.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:02.420 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:02.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:02 smithi178 bash[29889]: cluster 2024-09-13T22:55:00.826735+0000 mgr.y (mgr.14613) 647 : cluster [DBG] pgmap v746: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 945 op/s 2024-09-13T22:55:02.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:02 smithi178 bash[22840]: cluster 2024-09-13T22:55:00.826735+0000 mgr.y (mgr.14613) 647 : cluster [DBG] pgmap v746: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 945 op/s 2024-09-13T22:55:02.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:02.555 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:02.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:02.684 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:02.719 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:55:02.719 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:55:02.719 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:55:02.719 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:55:02.719 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:55:02.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:02.819 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:02.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total time run: 90.0075 2024-09-13T22:55:02.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total writes made: 110733 2024-09-13T22:55:02.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Write size: 65536 2024-09-13T22:55:02.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object size: 65536 2024-09-13T22:55:02.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Bandwidth (MB/sec): 76.8916 2024-09-13T22:55:02.923 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Bandwidth: 30.803 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max bandwidth (MB/sec): 194.5 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min bandwidth (MB/sec): 28.3125 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average IOPS: 1230 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev IOPS: 492.847 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max IOPS: 3112 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min IOPS: 453 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average Latency(s): 0.0130016 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Latency(s): 0.0229376 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max latency(s): 0.654557 2024-09-13T22:55:02.924 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min latency(s): 0.00112903 2024-09-13T22:55:02.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:02.928 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T22:55:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:02.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:02.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:03.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:03.080 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:03.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:03.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:03.194 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:03 smithi192 bash[24608]: cluster 2024-09-13T22:55:02.515181+0000 osd.4 (osd.4) 252 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:03.194 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:03 smithi192 bash[24608]: cluster 2024-09-13T22:55:02.515248+0000 osd.4 (osd.4) 253 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:03.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:03.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:03.324 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:03 smithi178 bash[29889]: cluster 2024-09-13T22:55:02.515181+0000 osd.4 (osd.4) 252 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:03.324 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:03 smithi178 bash[29889]: cluster 2024-09-13T22:55:02.515248+0000 osd.4 (osd.4) 253 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:03.325 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:03 smithi178 bash[22840]: cluster 2024-09-13T22:55:02.515181+0000 osd.4 (osd.4) 252 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:03.325 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:03 smithi178 bash[22840]: cluster 2024-09-13T22:55:02.515248+0000 osd.4 (osd.4) 253 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:03.345 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:03.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:03.482 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:03.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:03.619 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:03.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:03.751 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:03.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:03.890 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:03.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:03.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:04.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T22:55:04.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:04.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:04.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:04 smithi192 bash[24608]: cluster 2024-09-13T22:55:02.827275+0000 mgr.y (mgr.14613) 648 : cluster [DBG] pgmap v747: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 952 B/s rd, 54 MiB/s wr, 859 op/s 2024-09-13T22:55:04.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:04 smithi192 bash[24608]: cluster 2024-09-13T22:55:03.105003+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-13T22:55:04.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:04 smithi192 bash[24608]: audit 2024-09-13T22:55:04.071227+0000 mon.b (mon.1) 237 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:04.416 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:04 smithi178 bash[29889]: cluster 2024-09-13T22:55:02.827275+0000 mgr.y (mgr.14613) 648 : cluster [DBG] pgmap v747: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 952 B/s rd, 54 MiB/s wr, 859 op/s 2024-09-13T22:55:04.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:04 smithi178 bash[29889]: cluster 2024-09-13T22:55:03.105003+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-13T22:55:04.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:04 smithi178 bash[29889]: audit 2024-09-13T22:55:04.071227+0000 mon.b (mon.1) 237 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:04.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:04 smithi178 bash[22840]: cluster 2024-09-13T22:55:02.827275+0000 mgr.y (mgr.14613) 648 : cluster [DBG] pgmap v747: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 952 B/s rd, 54 MiB/s wr, 859 op/s 2024-09-13T22:55:04.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:04 smithi178 bash[22840]: cluster 2024-09-13T22:55:03.105003+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-13T22:55:04.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:04 smithi178 bash[22840]: audit 2024-09-13T22:55:04.071227+0000 mon.b (mon.1) 237 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:04.442 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:04.550 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:04.578 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:04.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.698 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:04.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:04.823 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:04.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.959 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:04.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:05.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.095 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:05.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:05.223 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:05.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:05.358 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:05.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.489 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:05.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:05.631 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:05.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.746 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:55:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:55:05.761 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:05.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:05.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:06.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:06.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:06.105 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:55:06.105 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:55:06.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:06.146 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:55:06.146 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:55:06.156 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:06.190 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:55:06.191 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:55:06.218 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:55:06.218 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:55:06.221 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:06 smithi178 bash[29889]: cluster 2024-09-13T22:55:04.828960+0000 mgr.y (mgr.14613) 649 : cluster [DBG] pgmap v749: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 909 op/s 2024-09-13T22:55:06.221 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:06 smithi178 bash[29889]: audit 2024-09-13T22:55:05.702548+0000 mgr.y (mgr.14613) 650 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:06.221 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:06 smithi178 bash[29889]: audit 2024-09-13T22:55:06.051263+0000 mon.a (mon.0) 1512 : audit [DBG] from='client.? v1:172.21.15.178:0/701689019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:06.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:06 smithi178 bash[22840]: cluster 2024-09-13T22:55:04.828960+0000 mgr.y (mgr.14613) 649 : cluster [DBG] pgmap v749: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 909 op/s 2024-09-13T22:55:06.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:06 smithi178 bash[22840]: audit 2024-09-13T22:55:05.702548+0000 mgr.y (mgr.14613) 650 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:06.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:06 smithi178 bash[22840]: audit 2024-09-13T22:55:06.051263+0000 mon.a (mon.0) 1512 : audit [DBG] from='client.? v1:172.21.15.178:0/701689019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:06.258 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:55:06.258 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:55:06.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:06.288 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:55:06.288 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:55:06.289 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:06.319 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:55:06.319 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:55:06.352 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:55:06.352 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:55:06.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:06.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:06 smithi192 bash[24608]: cluster 2024-09-13T22:55:04.828960+0000 mgr.y (mgr.14613) 649 : cluster [DBG] pgmap v749: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 909 op/s 2024-09-13T22:55:06.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:06 smithi192 bash[24608]: audit 2024-09-13T22:55:05.702548+0000 mgr.y (mgr.14613) 650 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:06.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:06 smithi192 bash[24608]: audit 2024-09-13T22:55:06.051263+0000 mon.a (mon.0) 1512 : audit [DBG] from='client.? v1:172.21.15.178:0/701689019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:06.417 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:06.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:06.543 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:06.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.687 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:06.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:06.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:06.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:06.920 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:06.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:06.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:07.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:07.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:07.124 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:55:07.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:07.214 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:07.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:07.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:07 smithi178 bash[22840]: audit 2024-09-13T22:55:06.523181+0000 mon.a (mon.0) 1513 : audit [INF] from='client.25537 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:55:07.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:07 smithi178 bash[22840]: audit 2024-09-13T22:55:06.524093+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.178:0/2560550765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:55:07.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:07 smithi178 bash[29889]: audit 2024-09-13T22:55:06.523181+0000 mon.a (mon.0) 1513 : audit [INF] from='client.25537 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:55:07.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:07 smithi178 bash[29889]: audit 2024-09-13T22:55:06.524093+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.178:0/2560550765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:55:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.341 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:07.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:07.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:07 smithi192 bash[24608]: audit 2024-09-13T22:55:06.523181+0000 mon.a (mon.0) 1513 : audit [INF] from='client.25537 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:55:07.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:07 smithi192 bash[24608]: audit 2024-09-13T22:55:06.524093+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.178:0/2560550765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:55:07.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:07.475 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:07.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:07.601 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:07.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:07.724 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:07.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:07.859 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:07.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:07.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:08.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:08.103 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:08.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:08.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:08.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:08.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:08.378 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:08.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:08 smithi192 bash[24608]: cluster 2024-09-13T22:55:06.829907+0000 mgr.y (mgr.14613) 651 : cluster [DBG] pgmap v750: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 907 op/s 2024-09-13T22:55:08.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:08 smithi192 bash[24608]: cluster 2024-09-13T22:55:07.109716+0000 mon.a (mon.0) 1514 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:55:08.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:08 smithi192 bash[24608]: audit 2024-09-13T22:55:07.120838+0000 mon.a (mon.0) 1515 : audit [INF] from='client.25537 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:55:08.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:08 smithi192 bash[24608]: cluster 2024-09-13T22:55:07.124250+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-13T22:55:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:08.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:08.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:08.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:08.654 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[29889]: cluster 2024-09-13T22:55:06.829907+0000 mgr.y (mgr.14613) 651 : cluster [DBG] pgmap v750: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 907 op/s 2024-09-13T22:55:08.654 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[29889]: cluster 2024-09-13T22:55:07.109716+0000 mon.a (mon.0) 1514 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:55:08.654 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[29889]: audit 2024-09-13T22:55:07.120838+0000 mon.a (mon.0) 1515 : audit [INF] from='client.25537 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:55:08.654 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[29889]: cluster 2024-09-13T22:55:07.124250+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-13T22:55:08.654 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[22840]: cluster 2024-09-13T22:55:06.829907+0000 mgr.y (mgr.14613) 651 : cluster [DBG] pgmap v750: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 907 op/s 2024-09-13T22:55:08.655 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[22840]: cluster 2024-09-13T22:55:07.109716+0000 mon.a (mon.0) 1514 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:55:08.655 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[22840]: audit 2024-09-13T22:55:07.120838+0000 mon.a (mon.0) 1515 : audit [INF] from='client.25537 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:55:08.655 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:08 smithi178 bash[22840]: cluster 2024-09-13T22:55:07.124250+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-13T22:55:08.682 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:08.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:08.809 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:08.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:08.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:08.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:08.971 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:55:09.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.076 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:09.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:09.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[29889]: cluster 2024-09-13T22:55:07.809814+0000 osd.2 (osd.2) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[29889]: cluster 2024-09-13T22:55:07.809929+0000 osd.2 (osd.2) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[29889]: cluster 2024-09-13T22:55:08.164310+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-13T22:55:09.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[22840]: cluster 2024-09-13T22:55:07.809814+0000 osd.2 (osd.2) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:09.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[22840]: cluster 2024-09-13T22:55:07.809929+0000 osd.2 (osd.2) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:09.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[22840]: cluster 2024-09-13T22:55:08.164310+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-13T22:55:09.200 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:09.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T22:55:09.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.323 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:09.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:09 smithi192 bash[24608]: cluster 2024-09-13T22:55:07.809814+0000 osd.2 (osd.2) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:09.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:09 smithi192 bash[24608]: cluster 2024-09-13T22:55:07.809929+0000 osd.2 (osd.2) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:09.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:09 smithi192 bash[24608]: cluster 2024-09-13T22:55:08.164310+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-13T22:55:09.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.450 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:09.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:09.585 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:09.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:09.717 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:09.821 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:09.831 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:55:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:55:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.855 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:09.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:09.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.987 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:09.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:09.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:10.095 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:10.119 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:10.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:10.251 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:10.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:10 smithi192 bash[24608]: cluster 2024-09-13T22:55:08.428720+0000 osd.6 (osd.6) 283 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:55:10.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:10 smithi192 bash[24608]: cluster 2024-09-13T22:55:08.441560+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:55:10.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:10 smithi192 bash[24608]: cluster 2024-09-13T22:55:08.832930+0000 mgr.y (mgr.14613) 652 : cluster [DBG] pgmap v753: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 35 MiB/s wr, 1.33k op/s 2024-09-13T22:55:10.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:10.382 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:10.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:10 smithi178 bash[29889]: cluster 2024-09-13T22:55:08.428720+0000 osd.6 (osd.6) 283 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:55:10.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:10 smithi178 bash[29889]: cluster 2024-09-13T22:55:08.441560+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:55:10.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:10 smithi178 bash[29889]: cluster 2024-09-13T22:55:08.832930+0000 mgr.y (mgr.14613) 652 : cluster [DBG] pgmap v753: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 35 MiB/s wr, 1.33k op/s 2024-09-13T22:55:10.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:10 smithi178 bash[22840]: cluster 2024-09-13T22:55:08.428720+0000 osd.6 (osd.6) 283 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:55:10.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:10 smithi178 bash[22840]: cluster 2024-09-13T22:55:08.441560+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:55:10.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:10 smithi178 bash[22840]: cluster 2024-09-13T22:55:08.832930+0000 mgr.y (mgr.14613) 652 : cluster [DBG] pgmap v753: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 35 MiB/s wr, 1.33k op/s 2024-09-13T22:55:10.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:10.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:10.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:10.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:10.898 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:10.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:11.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:11.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:11.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:11.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:11.302 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:11.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:11 smithi192 bash[24608]: cluster 2024-09-13T22:55:09.506705+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:55:11.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:11 smithi192 bash[24608]: cluster 2024-09-13T22:55:09.516209+0000 osd.1 (osd.1) 321 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:55:11.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:11.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:11 smithi178 bash[29889]: cluster 2024-09-13T22:55:09.506705+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:55:11.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:11 smithi178 bash[29889]: cluster 2024-09-13T22:55:09.516209+0000 osd.1 (osd.1) 321 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:55:11.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:11 smithi178 bash[22840]: cluster 2024-09-13T22:55:09.506705+0000 osd.1 (osd.1) 320 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:55:11.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:11 smithi178 bash[22840]: cluster 2024-09-13T22:55:09.516209+0000 osd.1 (osd.1) 321 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:55:11.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:11.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:11.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:11.675 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:11.710 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:11.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:11.844 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:11.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:11.971 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:11.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:11.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:55:12.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:12.095 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:12.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:55:12.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:12.221 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:12.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:10.833732+0000 mgr.y (mgr.14613) 653 : cluster [DBG] pgmap v754: 158 pgs: 158 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.19k op/s 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:10.920608+0000 osd.0 (osd.0) 337 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:10.922909+0000 osd.0 (osd.0) 338 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:10.927045+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:10.933578+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:11.301519+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:11.307528+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:11.629416+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:11.632157+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:11.846709+0000 osd.0 (osd.0) 339 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:55:12.340 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:12 smithi192 bash[24608]: cluster 2024-09-13T22:55:11.848201+0000 osd.0 (osd.0) 340 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:55:12.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:10.833732+0000 mgr.y (mgr.14613) 653 : cluster [DBG] pgmap v754: 158 pgs: 158 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.19k op/s 2024-09-13T22:55:12.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:10.920608+0000 osd.0 (osd.0) 337 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:10.922909+0000 osd.0 (osd.0) 338 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:10.927045+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:10.933578+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:11.301519+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:11.307528+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:11.629416+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:11.632157+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:11.846709+0000 osd.0 (osd.0) 339 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[29889]: cluster 2024-09-13T22:55:11.848201+0000 osd.0 (osd.0) 340 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:10.833732+0000 mgr.y (mgr.14613) 653 : cluster [DBG] pgmap v754: 158 pgs: 158 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.19k op/s 2024-09-13T22:55:12.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:10.920608+0000 osd.0 (osd.0) 337 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:10.922909+0000 osd.0 (osd.0) 338 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:10.927045+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.1f scrub starts 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:10.933578+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.1f scrub ok 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:11.301519+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:11.307528+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:11.629416+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.0 scrub starts 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:11.632157+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.0 scrub ok 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:11.846709+0000 osd.0 (osd.0) 339 : cluster [DBG] 5.3 scrub starts 2024-09-13T22:55:12.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:12 smithi178 bash[22840]: cluster 2024-09-13T22:55:11.848201+0000 osd.0 (osd.0) 340 : cluster [DBG] 5.3 scrub ok 2024-09-13T22:55:12.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:12.502 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:12.607 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:12.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:12.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:12.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:12.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:12.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:12.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:13.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:13.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:13.120 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:55:13.121 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:55:13.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:13.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:13.305 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:13.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:13.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:13 smithi192 bash[24608]: cluster 2024-09-13T22:55:12.361006+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:55:13.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:13 smithi192 bash[24608]: cluster 2024-09-13T22:55:12.370199+0000 osd.4 (osd.4) 255 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:55:13.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:13 smithi178 bash[29889]: cluster 2024-09-13T22:55:12.361006+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:55:13.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:13 smithi178 bash[29889]: cluster 2024-09-13T22:55:12.370199+0000 osd.4 (osd.4) 255 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:55:13.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:13 smithi178 bash[22840]: cluster 2024-09-13T22:55:12.361006+0000 osd.4 (osd.4) 254 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:55:13.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:13 smithi178 bash[22840]: cluster 2024-09-13T22:55:12.370199+0000 osd.4 (osd.4) 255 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:55:13.437 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:13.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:13.568 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:13.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.694 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:13.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.829 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:13.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:13.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:13.983 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:14.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:14.110 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:14.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:14.239 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:14.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:14.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:14 smithi178 bash[29889]: cluster 2024-09-13T22:55:12.834602+0000 mgr.y (mgr.14613) 654 : cluster [DBG] pgmap v755: 158 pgs: 158 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 2.4 MiB/s wr, 731 op/s 2024-09-13T22:55:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:14 smithi178 bash[29889]: cluster 2024-09-13T22:55:13.387920+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:55:14.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:14 smithi178 bash[29889]: cluster 2024-09-13T22:55:13.390947+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:55:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:14 smithi178 bash[22840]: cluster 2024-09-13T22:55:12.834602+0000 mgr.y (mgr.14613) 654 : cluster [DBG] pgmap v755: 158 pgs: 158 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 2.4 MiB/s wr, 731 op/s 2024-09-13T22:55:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:14 smithi178 bash[22840]: cluster 2024-09-13T22:55:13.387920+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:55:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:14 smithi178 bash[22840]: cluster 2024-09-13T22:55:13.390947+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:55:14.474 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:14.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:14.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:14 smithi192 bash[24608]: cluster 2024-09-13T22:55:12.834602+0000 mgr.y (mgr.14613) 654 : cluster [DBG] pgmap v755: 158 pgs: 158 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 2.4 MiB/s wr, 731 op/s 2024-09-13T22:55:14.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:14 smithi192 bash[24608]: cluster 2024-09-13T22:55:13.387920+0000 osd.4 (osd.4) 256 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:55:14.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:14 smithi192 bash[24608]: cluster 2024-09-13T22:55:13.390947+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:55:14.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T22:55:14.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:14.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:14.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:14.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:14.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:14.931 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:14.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:15.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:15.056 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:15.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:15.185 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:15.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:15.323 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:13.398069+0000 osd.6 (osd.6) 285 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:13.409565+0000 osd.6 (osd.6) 286 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:13.906948+0000 osd.3 (osd.3) 327 : cluster [DBG] 3.b scrub starts 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:13.915379+0000 osd.3 (osd.3) 328 : cluster [DBG] 3.b scrub ok 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:14.325501+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:14.331255+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:14.830621+0000 osd.0 (osd.0) 341 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:55:15.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[24608]: cluster 2024-09-13T22:55:14.852492+0000 osd.0 (osd.0) 342 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:55:15.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:13.398069+0000 osd.6 (osd.6) 285 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:55:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:13.409565+0000 osd.6 (osd.6) 286 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:55:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:13.906948+0000 osd.3 (osd.3) 327 : cluster [DBG] 3.b scrub starts 2024-09-13T22:55:15.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:13.915379+0000 osd.3 (osd.3) 328 : cluster [DBG] 3.b scrub ok 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:14.325501+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:14.331255+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:14.830621+0000 osd.0 (osd.0) 341 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[22840]: cluster 2024-09-13T22:55:14.852492+0000 osd.0 (osd.0) 342 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:13.398069+0000 osd.6 (osd.6) 285 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:13.409565+0000 osd.6 (osd.6) 286 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:13.906948+0000 osd.3 (osd.3) 327 : cluster [DBG] 3.b scrub starts 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:13.915379+0000 osd.3 (osd.3) 328 : cluster [DBG] 3.b scrub ok 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:14.325501+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:14.331255+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:14.830621+0000 osd.0 (osd.0) 341 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:55:15.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:15 smithi178 bash[29889]: cluster 2024-09-13T22:55:14.852492+0000 osd.0 (osd.0) 342 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:55:15.460 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:15.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:15.591 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:15.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:15.733 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:15.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:15.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:15.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:15.997 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:15.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:16.017 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:16.017 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:16.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:16.110 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:55:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:55:16.128 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:16.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:16.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:16.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:16.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:16.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:16 smithi192 bash[24608]: cluster 2024-09-13T22:55:14.399801+0000 osd.6 (osd.6) 287 : cluster [DBG] 10.d scrub starts 2024-09-13T22:55:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:16 smithi192 bash[24608]: cluster 2024-09-13T22:55:14.835442+0000 mgr.y (mgr.14613) 655 : cluster [DBG] pgmap v756: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T22:55:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:16 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.308461+0000 osd.5 (osd.5) 379 : cluster [DBG] 10.9 scrub starts 2024-09-13T22:55:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:16 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.393127+0000 osd.4 (osd.4) 258 : cluster [DBG] 5.a scrub starts 2024-09-13T22:55:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:16 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.398479+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.a scrub ok 2024-09-13T22:55:16.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:16 smithi192 bash[24608]: audit 2024-09-13T22:55:15.712683+0000 mgr.y (mgr.14613) 656 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:16.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:16.531 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:16.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:16.647 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[29889]: cluster 2024-09-13T22:55:14.399801+0000 osd.6 (osd.6) 287 : cluster [DBG] 10.d scrub starts 2024-09-13T22:55:16.647 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[29889]: cluster 2024-09-13T22:55:14.835442+0000 mgr.y (mgr.14613) 655 : cluster [DBG] pgmap v756: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T22:55:16.647 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.308461+0000 osd.5 (osd.5) 379 : cluster [DBG] 10.9 scrub starts 2024-09-13T22:55:16.647 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.393127+0000 osd.4 (osd.4) 258 : cluster [DBG] 5.a scrub starts 2024-09-13T22:55:16.647 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.398479+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.a scrub ok 2024-09-13T22:55:16.647 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[29889]: audit 2024-09-13T22:55:15.712683+0000 mgr.y (mgr.14613) 656 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:16.648 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[22840]: cluster 2024-09-13T22:55:14.399801+0000 osd.6 (osd.6) 287 : cluster [DBG] 10.d scrub starts 2024-09-13T22:55:16.648 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[22840]: cluster 2024-09-13T22:55:14.835442+0000 mgr.y (mgr.14613) 655 : cluster [DBG] pgmap v756: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T22:55:16.648 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.308461+0000 osd.5 (osd.5) 379 : cluster [DBG] 10.9 scrub starts 2024-09-13T22:55:16.648 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.393127+0000 osd.4 (osd.4) 258 : cluster [DBG] 5.a scrub starts 2024-09-13T22:55:16.648 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.398479+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.a scrub ok 2024-09-13T22:55:16.648 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:16 smithi178 bash[22840]: audit 2024-09-13T22:55:15.712683+0000 mgr.y (mgr.14613) 656 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:16.667 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:16.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.797 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:16.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:16.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:16.917 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:16.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:17.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:17.049 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:17.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.185 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:17.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:17.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:17.310 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:17.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:17 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.314401+0000 osd.1 (osd.1) 322 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:55:17.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:17 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.317827+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:55:17.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:17 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.789413+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.c scrub starts 2024-09-13T22:55:17.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:17 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.790368+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.c scrub ok 2024-09-13T22:55:17.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:17 smithi192 bash[24608]: cluster 2024-09-13T22:55:15.941981+0000 osd.6 (osd.6) 288 : cluster [DBG] 10.d scrub ok 2024-09-13T22:55:17.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:17 smithi192 bash[24608]: cluster 2024-09-13T22:55:16.263308+0000 osd.1 (osd.1) 324 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:55:17.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:17.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:17.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:17.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.314401+0000 osd.1 (osd.1) 322 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:55:17.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.317827+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:55:17.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.789413+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.c scrub starts 2024-09-13T22:55:17.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.790368+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.c scrub ok 2024-09-13T22:55:17.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[22840]: cluster 2024-09-13T22:55:15.941981+0000 osd.6 (osd.6) 288 : cluster [DBG] 10.d scrub ok 2024-09-13T22:55:17.612 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[22840]: cluster 2024-09-13T22:55:16.263308+0000 osd.1 (osd.1) 324 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:55:17.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.314401+0000 osd.1 (osd.1) 322 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:55:17.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.317827+0000 osd.1 (osd.1) 323 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:55:17.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.789413+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.c scrub starts 2024-09-13T22:55:17.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.790368+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.c scrub ok 2024-09-13T22:55:17.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[29889]: cluster 2024-09-13T22:55:15.941981+0000 osd.6 (osd.6) 288 : cluster [DBG] 10.d scrub ok 2024-09-13T22:55:17.613 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:17 smithi178 bash[29889]: cluster 2024-09-13T22:55:16.263308+0000 osd.1 (osd.1) 324 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:55:17.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:17.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:17.767 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:17.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:17.892 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:17.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:17.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:18.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:18.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:18.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:18.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:18.258 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:18 smithi192 bash[24608]: cluster 2024-09-13T22:55:16.266193+0000 osd.1 (osd.1) 325 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:55:18.258 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:18 smithi192 bash[24608]: cluster 2024-09-13T22:55:16.835860+0000 mgr.y (mgr.14613) 657 : cluster [DBG] pgmap v757: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T22:55:18.258 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:18 smithi192 bash[24608]: cluster 2024-09-13T22:55:16.965998+0000 osd.5 (osd.5) 380 : cluster [DBG] 10.9 scrub ok 2024-09-13T22:55:18.258 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:18 smithi192 bash[24608]: cluster 2024-09-13T22:55:17.348337+0000 osd.4 (osd.4) 260 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:55:18.259 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:18 smithi192 bash[24608]: cluster 2024-09-13T22:55:17.356876+0000 osd.4 (osd.4) 261 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:55:18.299 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:18.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[29889]: cluster 2024-09-13T22:55:16.266193+0000 osd.1 (osd.1) 325 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:55:18.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[29889]: cluster 2024-09-13T22:55:16.835860+0000 mgr.y (mgr.14613) 657 : cluster [DBG] pgmap v757: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[29889]: cluster 2024-09-13T22:55:16.965998+0000 osd.5 (osd.5) 380 : cluster [DBG] 10.9 scrub ok 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[29889]: cluster 2024-09-13T22:55:17.348337+0000 osd.4 (osd.4) 260 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[29889]: cluster 2024-09-13T22:55:17.356876+0000 osd.4 (osd.4) 261 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[22840]: cluster 2024-09-13T22:55:16.266193+0000 osd.1 (osd.1) 325 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[22840]: cluster 2024-09-13T22:55:16.835860+0000 mgr.y (mgr.14613) 657 : cluster [DBG] pgmap v757: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[22840]: cluster 2024-09-13T22:55:16.965998+0000 osd.5 (osd.5) 380 : cluster [DBG] 10.9 scrub ok 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[22840]: cluster 2024-09-13T22:55:17.348337+0000 osd.4 (osd.4) 260 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:55:18.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:18 smithi178 bash[22840]: cluster 2024-09-13T22:55:17.356876+0000 osd.4 (osd.4) 261 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:55:18.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:18.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:18.633 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:18.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:18.770 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:18.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:18.905 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:18.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:19.009 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:19.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:19.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:19.170 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:19.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:19.303 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:19 smithi192 bash[24608]: cluster 2024-09-13T22:55:18.674394+0000 osd.7 (osd.7) 351 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:55:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:19 smithi192 bash[24608]: cluster 2024-09-13T22:55:18.683678+0000 osd.7 (osd.7) 352 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:55:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:19 smithi192 bash[24608]: cluster 2024-09-13T22:55:18.815166+0000 osd.0 (osd.0) 343 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:55:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:19 smithi192 bash[24608]: cluster 2024-09-13T22:55:18.819416+0000 osd.0 (osd.0) 344 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:55:19.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:19 smithi192 bash[24608]: audit 2024-09-13T22:55:19.090798+0000 mon.b (mon.1) 239 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:19.408 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:19.426 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:55:19.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:19.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[29889]: cluster 2024-09-13T22:55:18.674394+0000 osd.7 (osd.7) 351 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[29889]: cluster 2024-09-13T22:55:18.683678+0000 osd.7 (osd.7) 352 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[29889]: cluster 2024-09-13T22:55:18.815166+0000 osd.0 (osd.0) 343 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[29889]: cluster 2024-09-13T22:55:18.819416+0000 osd.0 (osd.0) 344 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[29889]: audit 2024-09-13T22:55:19.090798+0000 mon.b (mon.1) 239 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[22840]: cluster 2024-09-13T22:55:18.674394+0000 osd.7 (osd.7) 351 : cluster [DBG] 6.5 scrub starts 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[22840]: cluster 2024-09-13T22:55:18.683678+0000 osd.7 (osd.7) 352 : cluster [DBG] 6.5 scrub ok 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[22840]: cluster 2024-09-13T22:55:18.815166+0000 osd.0 (osd.0) 343 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[22840]: cluster 2024-09-13T22:55:18.819416+0000 osd.0 (osd.0) 344 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:55:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[22840]: audit 2024-09-13T22:55:19.090798+0000 mon.b (mon.1) 239 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:19.732 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:19.835 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:19.863 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:19.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:19.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:19.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:20.096 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:55:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:55:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:20.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:20.127 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:20.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:20.265 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:20.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:20.474 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:20.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:20.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:20.587 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:20 smithi178 bash[29889]: cluster 2024-09-13T22:55:18.836618+0000 mgr.y (mgr.14613) 658 : cluster [DBG] pgmap v758: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-13T22:55:20.588 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:20 smithi178 bash[29889]: cluster 2024-09-13T22:55:19.305403+0000 osd.5 (osd.5) 381 : cluster [DBG] 10.6 scrub starts 2024-09-13T22:55:20.588 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:20 smithi178 bash[22840]: cluster 2024-09-13T22:55:18.836618+0000 mgr.y (mgr.14613) 658 : cluster [DBG] pgmap v758: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-13T22:55:20.588 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:20 smithi178 bash[22840]: cluster 2024-09-13T22:55:19.305403+0000 osd.5 (osd.5) 381 : cluster [DBG] 10.6 scrub starts 2024-09-13T22:55:20.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:20 smithi192 bash[24608]: cluster 2024-09-13T22:55:18.836618+0000 mgr.y (mgr.14613) 658 : cluster [DBG] pgmap v758: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-13T22:55:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:20 smithi192 bash[24608]: cluster 2024-09-13T22:55:19.305403+0000 osd.5 (osd.5) 381 : cluster [DBG] 10.6 scrub starts 2024-09-13T22:55:20.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:20.725 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:20.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:20.854 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:20.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:20.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:20.991 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:20.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:21.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:21.122 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:21.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:21.261 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:55:21.338 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:21.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:21.465 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:21.517 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:19.316515+0000 osd.1 (osd.1) 326 : cluster [DBG] 10.2 scrub starts 2024-09-13T22:55:21.517 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:19.751375+0000 osd.2 (osd.2) 251 : cluster [DBG] 5.b scrub starts 2024-09-13T22:55:21.517 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:19.752815+0000 osd.2 (osd.2) 252 : cluster [DBG] 5.b scrub ok 2024-09-13T22:55:21.517 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:19.896519+0000 osd.3 (osd.3) 329 : cluster [DBG] 2.e scrub starts 2024-09-13T22:55:21.517 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:19.899946+0000 osd.3 (osd.3) 330 : cluster [DBG] 2.e scrub ok 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: audit 2024-09-13T22:55:20.735884+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.178:0/1939467022' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: audit 2024-09-13T22:55:20.736285+0000 mon.a (mon.0) 1518 : audit [INF] from='client.25640 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:20.803763+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:20.805532+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[29889]: cluster 2024-09-13T22:55:20.862916+0000 osd.0 (osd.0) 345 : cluster [DBG] 10.4 scrub starts 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:19.316515+0000 osd.1 (osd.1) 326 : cluster [DBG] 10.2 scrub starts 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:19.751375+0000 osd.2 (osd.2) 251 : cluster [DBG] 5.b scrub starts 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:19.752815+0000 osd.2 (osd.2) 252 : cluster [DBG] 5.b scrub ok 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:19.896519+0000 osd.3 (osd.3) 329 : cluster [DBG] 2.e scrub starts 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:19.899946+0000 osd.3 (osd.3) 330 : cluster [DBG] 2.e scrub ok 2024-09-13T22:55:21.518 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: audit 2024-09-13T22:55:20.735884+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.178:0/1939467022' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:21.519 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: audit 2024-09-13T22:55:20.736285+0000 mon.a (mon.0) 1518 : audit [INF] from='client.25640 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:21.519 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:20.803763+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:55:21.519 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:20.805532+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:55:21.519 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:21 smithi178 bash[22840]: cluster 2024-09-13T22:55:20.862916+0000 osd.0 (osd.0) 345 : cluster [DBG] 10.4 scrub starts 2024-09-13T22:55:21.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:21.587 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:19.316515+0000 osd.1 (osd.1) 326 : cluster [DBG] 10.2 scrub starts 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:19.751375+0000 osd.2 (osd.2) 251 : cluster [DBG] 5.b scrub starts 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:19.752815+0000 osd.2 (osd.2) 252 : cluster [DBG] 5.b scrub ok 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:19.896519+0000 osd.3 (osd.3) 329 : cluster [DBG] 2.e scrub starts 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:19.899946+0000 osd.3 (osd.3) 330 : cluster [DBG] 2.e scrub ok 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: audit 2024-09-13T22:55:20.735884+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.178:0/1939467022' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:21.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: audit 2024-09-13T22:55:20.736285+0000 mon.a (mon.0) 1518 : audit [INF] from='client.25640 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:20.803763+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.1d scrub starts 2024-09-13T22:55:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:20.805532+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.1d scrub ok 2024-09-13T22:55:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:21 smithi192 bash[24608]: cluster 2024-09-13T22:55:20.862916+0000 osd.0 (osd.0) 345 : cluster [DBG] 10.4 scrub starts 2024-09-13T22:55:21.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:21.721 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:21.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:21.855 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:21.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:21.979 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:21.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:22.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:22.242 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:22.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:22.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:22.371 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:22.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:22.385 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:55:22.385 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:55:22.404 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:55:22.405 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:55:22.428 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:55:22.428 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:55:22.455 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:55:22.456 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:55:22.472 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:55:22.473 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:55:22.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:22.493 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:55:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:55:22.497 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:22.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:22.515 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:55:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:55:22.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:22.622 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:22.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:20.837745+0000 mgr.y (mgr.14613) 659 : cluster [DBG] pgmap v759: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 915 op/s 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.209919+0000 osd.5 (osd.5) 382 : cluster [DBG] 10.6 scrub ok 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.252018+0000 mon.a (mon.0) 1519 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.252047+0000 mon.a (mon.0) 1520 : cluster [INF] Cluster is now healthy 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: audit 2024-09-13T22:55:21.257652+0000 mon.a (mon.0) 1521 : audit [INF] from='client.25640 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.263241+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.363381+0000 osd.1 (osd.1) 327 : cluster [DBG] 10.2 scrub ok 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.390953+0000 osd.1 (osd.1) 328 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:22.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.391030+0000 osd.1 (osd.1) 329 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.430040+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.2 deep-scrub starts 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.433985+0000 osd.1 (osd.1) 331 : cluster [DBG] 4.2 deep-scrub ok 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.713406+0000 osd.7 (osd.7) 355 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.721047+0000 osd.7 (osd.7) 356 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:20.837745+0000 mgr.y (mgr.14613) 659 : cluster [DBG] pgmap v759: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 915 op/s 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.209919+0000 osd.5 (osd.5) 382 : cluster [DBG] 10.6 scrub ok 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.252018+0000 mon.a (mon.0) 1519 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.252047+0000 mon.a (mon.0) 1520 : cluster [INF] Cluster is now healthy 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: audit 2024-09-13T22:55:21.257652+0000 mon.a (mon.0) 1521 : audit [INF] from='client.25640 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.263241+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.363381+0000 osd.1 (osd.1) 327 : cluster [DBG] 10.2 scrub ok 2024-09-13T22:55:22.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.390953+0000 osd.1 (osd.1) 328 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:22.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.391030+0000 osd.1 (osd.1) 329 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:22.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.430040+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.2 deep-scrub starts 2024-09-13T22:55:22.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.433985+0000 osd.1 (osd.1) 331 : cluster [DBG] 4.2 deep-scrub ok 2024-09-13T22:55:22.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.713406+0000 osd.7 (osd.7) 355 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:55:22.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:22 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.721047+0000 osd.7 (osd.7) 356 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:55:22.725 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:22.749 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:22.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:22.819 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:55:22.820 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:55:22.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:20.837745+0000 mgr.y (mgr.14613) 659 : cluster [DBG] pgmap v759: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 915 op/s 2024-09-13T22:55:22.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.209919+0000 osd.5 (osd.5) 382 : cluster [DBG] 10.6 scrub ok 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.252018+0000 mon.a (mon.0) 1519 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.252047+0000 mon.a (mon.0) 1520 : cluster [INF] Cluster is now healthy 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: audit 2024-09-13T22:55:21.257652+0000 mon.a (mon.0) 1521 : audit [INF] from='client.25640 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.263241+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.363381+0000 osd.1 (osd.1) 327 : cluster [DBG] 10.2 scrub ok 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.390953+0000 osd.1 (osd.1) 328 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.391030+0000 osd.1 (osd.1) 329 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.430040+0000 osd.1 (osd.1) 330 : cluster [DBG] 4.2 deep-scrub starts 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.433985+0000 osd.1 (osd.1) 331 : cluster [DBG] 4.2 deep-scrub ok 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.713406+0000 osd.7 (osd.7) 355 : cluster [DBG] 5.1a scrub starts 2024-09-13T22:55:22.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:22 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.721047+0000 osd.7 (osd.7) 356 : cluster [DBG] 5.1a scrub ok 2024-09-13T22:55:22.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:22.886 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:22.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:22.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:23.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:23.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:23.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:23.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:23.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:23.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:23.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[29889]: cluster 2024-09-13T22:55:21.519886+0000 osd.6 (osd.6) 289 : cluster [DBG] 10.a scrub starts 2024-09-13T22:55:23.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[29889]: cluster 2024-09-13T22:55:22.272311+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[29889]: cluster 2024-09-13T22:55:22.289101+0000 osd.4 (osd.4) 262 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[29889]: cluster 2024-09-13T22:55:22.292408+0000 osd.4 (osd.4) 263 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[29889]: audit 2024-09-13T22:55:22.367441+0000 mon.a (mon.0) 1524 : audit [DBG] from='client.? v1:172.21.15.178:0/4126717586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[29889]: cluster 2024-09-13T22:55:22.640807+0000 osd.0 (osd.0) 346 : cluster [DBG] 10.4 scrub ok 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[22840]: cluster 2024-09-13T22:55:21.519886+0000 osd.6 (osd.6) 289 : cluster [DBG] 10.a scrub starts 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[22840]: cluster 2024-09-13T22:55:22.272311+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[22840]: cluster 2024-09-13T22:55:22.289101+0000 osd.4 (osd.4) 262 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[22840]: cluster 2024-09-13T22:55:22.292408+0000 osd.4 (osd.4) 263 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[22840]: audit 2024-09-13T22:55:22.367441+0000 mon.a (mon.0) 1524 : audit [DBG] from='client.? v1:172.21.15.178:0/4126717586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:23 smithi178 bash[22840]: cluster 2024-09-13T22:55:22.640807+0000 osd.0 (osd.0) 346 : cluster [DBG] 10.4 scrub ok 2024-09-13T22:55:23.492 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:23.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:23.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:23.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:23 smithi192 bash[24608]: cluster 2024-09-13T22:55:21.519886+0000 osd.6 (osd.6) 289 : cluster [DBG] 10.a scrub starts 2024-09-13T22:55:23.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:23 smithi192 bash[24608]: cluster 2024-09-13T22:55:22.272311+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-13T22:55:23.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:23 smithi192 bash[24608]: cluster 2024-09-13T22:55:22.289101+0000 osd.4 (osd.4) 262 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:55:23.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:23 smithi192 bash[24608]: cluster 2024-09-13T22:55:22.292408+0000 osd.4 (osd.4) 263 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:55:23.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:23 smithi192 bash[24608]: audit 2024-09-13T22:55:22.367441+0000 mon.a (mon.0) 1524 : audit [DBG] from='client.? v1:172.21.15.178:0/4126717586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:23.607 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:23 smithi192 bash[24608]: cluster 2024-09-13T22:55:22.640807+0000 osd.0 (osd.0) 346 : cluster [DBG] 10.4 scrub ok 2024-09-13T22:55:23.631 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:23.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:23.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:23.763 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:23.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:23.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:23.901 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:23.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:24.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:24.044 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:24.150 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:24.177 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:24.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:24.303 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:24 smithi192 bash[24608]: cluster 2024-09-13T22:55:22.612514+0000 osd.6 (osd.6) 290 : cluster [DBG] 10.a scrub ok 2024-09-13T22:55:24.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:24 smithi192 bash[24608]: cluster 2024-09-13T22:55:22.838253+0000 mgr.y (mgr.14613) 660 : cluster [DBG] pgmap v762: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s 2024-09-13T22:55:24.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:24.439 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:24.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:24.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:24 smithi178 bash[29889]: cluster 2024-09-13T22:55:22.612514+0000 osd.6 (osd.6) 290 : cluster [DBG] 10.a scrub ok 2024-09-13T22:55:24.566 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:24 smithi178 bash[29889]: cluster 2024-09-13T22:55:22.838253+0000 mgr.y (mgr.14613) 660 : cluster [DBG] pgmap v762: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s 2024-09-13T22:55:24.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:24 smithi178 bash[22840]: cluster 2024-09-13T22:55:22.612514+0000 osd.6 (osd.6) 290 : cluster [DBG] 10.a scrub ok 2024-09-13T22:55:24.566 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:24 smithi178 bash[22840]: cluster 2024-09-13T22:55:22.838253+0000 mgr.y (mgr.14613) 660 : cluster [DBG] pgmap v762: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s 2024-09-13T22:55:24.566 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:24.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:24.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:24.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:24.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:24.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:24.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:24.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:25.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:25.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:25.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:25.234 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:25.327 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:131242 10.f 10:f294dc0c:::benchmark_data_smithi178_111521_object20506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:25.325181+0000", 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0032609819999999999, 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032915219999999999, 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:55:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131242 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325181+0000", 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325181+0000", 2024-09-13T22:55:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325182+0000", 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.47e-07 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325185+0000", 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.05e-06 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325191+0000", 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8839999999999999e-06 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325196+0000", 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.13e-06 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325243+0000", 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6483999999999997e-05 2024-09-13T22:55:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325340+0000", 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7250999999999994e-05 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325472+0000", 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013241300000000001 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:131227 10.7 10:ea9ed3ae:::benchmark_data_smithi178_111521_object20491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:25.317280+0000", 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011162421, 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011268525999999999, 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:55:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:55:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:55:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131227 2024-09-13T22:55:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:55:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:55:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317280+0000", 2024-09-13T22:55:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:55:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317280+0000", 2024-09-13T22:55:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317281+0000", 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.719e-06 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317286+0000", 2024-09-13T22:55:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7199999999999997e-06 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317296+0000", 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4439999999999997e-06 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317302+0000", 2024-09-13T22:55:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2389999999999999e-06 2024-09-13T22:55:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:55:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317353+0000", 2024-09-13T22:55:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1125999999999998e-05 2024-09-13T22:55:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317476+0000", 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012278199999999999 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,5", 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.317584+0000", 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108151 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323578+0000", 2024-09-13T22:55:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059936829999999996 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323594+0000", 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6963999999999999e-05 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:55:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:131233 10.7 10:e04f75fe:::benchmark_data_smithi178_111521_object20497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T22:55:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:25.319245+0000", 2024-09-13T22:55:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0091971120000000003, 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093791540000000007, 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:55:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:55:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131233 2024-09-13T22:55:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:55:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.319245+0000", 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.319245+0000", 2024-09-13T22:55:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.319246+0000", 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.46e-06 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.319254+0000", 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.542e-06 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.319263+0000", 2024-09-13T22:55:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.251e-06 2024-09-13T22:55:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:55:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.319270+0000", 2024-09-13T22:55:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5989999999999998e-06 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.320615+0000", 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013449689999999999 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:55:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.320729+0000", 2024-09-13T22:55:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114047 2024-09-13T22:55:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,5", 2024-09-13T22:55:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.320844+0000", 2024-09-13T22:55:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011481600000000001 2024-09-13T22:55:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:55:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.324243+0000", 2024-09-13T22:55:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033991410000000001 2024-09-13T22:55:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:55:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:131236 10.d 10:bbd9cd09:::benchmark_data_smithi178_111521_object20500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:25.323606+0000", 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0048362930000000002, 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050896240000000001, 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:55:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131236 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:55:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323606+0000", 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323606+0000", 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323608+0000", 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.854e-06 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:55:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323613+0000", 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2159999999999997e-06 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323624+0000", 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0679999999999999e-05 2024-09-13T22:55:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323631+0000", 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3329999999999999e-06 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:55:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323687+0000", 2024-09-13T22:55:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5680000000000002e-05 2024-09-13T22:55:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:55:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323785+0000", 2024-09-13T22:55:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8326999999999996e-05 2024-09-13T22:55:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-09-13T22:55:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.323894+0000", 2024-09-13T22:55:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109273 2024-09-13T22:55:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:55:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.326894+0000", 2024-09-13T22:55:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002999444 2024-09-13T22:55:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:55:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:131241 10.a 10:5000ffab:::benchmark_data_smithi178_111521_object20505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T22:55:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:25.325135+0000", 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0033070299999999999, 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036289809999999999, 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131241 2024-09-13T22:55:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325135+0000", 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325135+0000", 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325137+0000", 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.528e-06 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325141+0000", 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6709999999999998e-06 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325151+0000", 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7259999999999997e-06 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325158+0000", 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.63e-06 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325207+0000", 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9251e-05 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325306+0000", 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8803999999999998e-05 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:55:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,5", 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:25.325406+0000", 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100653 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 5 2024-09-13T22:55:25.368 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:55:25.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:25.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:25 smithi192 bash[24608]: cluster 2024-09-13T22:55:23.793762+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:55:25.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:25 smithi192 bash[24608]: cluster 2024-09-13T22:55:23.798018+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:55:25.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:25.494 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:25 smithi178 bash[22840]: cluster 2024-09-13T22:55:23.793762+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:55:25.494 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:25 smithi178 bash[22840]: cluster 2024-09-13T22:55:23.798018+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:55:25.495 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:25 smithi178 bash[29889]: cluster 2024-09-13T22:55:23.793762+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:55:25.495 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:25 smithi178 bash[29889]: cluster 2024-09-13T22:55:23.798018+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:55:25.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:25.537 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:25.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:25.673 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:25.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:25.806 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:25.907 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:55:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:55:25.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:25.931 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:25.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:26.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:26.058 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:26.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:26.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:26.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:26.234 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:26.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:26.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:26.364 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:26.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:26.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:26.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:26 smithi192 bash[24608]: cluster 2024-09-13T22:55:24.438571+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.15 deep-scrub starts 2024-09-13T22:55:26.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:26 smithi192 bash[24608]: cluster 2024-09-13T22:55:24.439965+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 deep-scrub ok 2024-09-13T22:55:26.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:26 smithi192 bash[24608]: cluster 2024-09-13T22:55:24.839157+0000 mgr.y (mgr.14613) 661 : cluster [DBG] pgmap v763: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-09-13T22:55:26.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:26 smithi192 bash[24608]: cluster 2024-09-13T22:55:24.872604+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:55:26.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:26 smithi192 bash[24608]: cluster 2024-09-13T22:55:24.878060+0000 osd.3 (osd.3) 332 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:55:26.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:26 smithi192 bash[24608]: audit 2024-09-13T22:55:25.720711+0000 mgr.y (mgr.14613) 662 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:26.637 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:26.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:26.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T22:55:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[29889]: cluster 2024-09-13T22:55:24.438571+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.15 deep-scrub starts 2024-09-13T22:55:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[29889]: cluster 2024-09-13T22:55:24.439965+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 deep-scrub ok 2024-09-13T22:55:26.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[29889]: cluster 2024-09-13T22:55:24.839157+0000 mgr.y (mgr.14613) 661 : cluster [DBG] pgmap v763: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[29889]: cluster 2024-09-13T22:55:24.872604+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[29889]: cluster 2024-09-13T22:55:24.878060+0000 osd.3 (osd.3) 332 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[29889]: audit 2024-09-13T22:55:25.720711+0000 mgr.y (mgr.14613) 662 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[22840]: cluster 2024-09-13T22:55:24.438571+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.15 deep-scrub starts 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[22840]: cluster 2024-09-13T22:55:24.439965+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 deep-scrub ok 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[22840]: cluster 2024-09-13T22:55:24.839157+0000 mgr.y (mgr.14613) 661 : cluster [DBG] pgmap v763: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[22840]: cluster 2024-09-13T22:55:24.872604+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.1b scrub starts 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[22840]: cluster 2024-09-13T22:55:24.878060+0000 osd.3 (osd.3) 332 : cluster [DBG] 2.1b scrub ok 2024-09-13T22:55:26.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:26 smithi178 bash[22840]: audit 2024-09-13T22:55:25.720711+0000 mgr.y (mgr.14613) 662 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:26.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:26.769 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:26.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:26.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:26.900 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:55:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:26.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:27.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:27.042 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:27.150 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:27.175 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:27.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:27.308 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:27.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:27.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[29889]: cluster 2024-09-13T22:55:25.519759+0000 osd.6 (osd.6) 291 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:55:27.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[29889]: cluster 2024-09-13T22:55:25.529891+0000 osd.6 (osd.6) 292 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:55:27.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.345407+0000 osd.5 (osd.5) 383 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:55:27.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.348989+0000 osd.5 (osd.5) 384 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.885744+0000 osd.7 (osd.7) 357 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.889221+0000 osd.7 (osd.7) 358 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[22840]: cluster 2024-09-13T22:55:25.519759+0000 osd.6 (osd.6) 291 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[22840]: cluster 2024-09-13T22:55:25.529891+0000 osd.6 (osd.6) 292 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.345407+0000 osd.5 (osd.5) 383 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.348989+0000 osd.5 (osd.5) 384 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.885744+0000 osd.7 (osd.7) 357 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:55:27.424 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:27 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.889221+0000 osd.7 (osd.7) 358 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:55:27.438 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:27.453 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:55:27.453 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:55:27.453 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-13T22:55:27.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T22:55:27.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:55:27.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:27.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:27 smithi192 bash[24608]: cluster 2024-09-13T22:55:25.519759+0000 osd.6 (osd.6) 291 : cluster [DBG] 5.1f scrub starts 2024-09-13T22:55:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:27 smithi192 bash[24608]: cluster 2024-09-13T22:55:25.529891+0000 osd.6 (osd.6) 292 : cluster [DBG] 5.1f scrub ok 2024-09-13T22:55:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:27 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.345407+0000 osd.5 (osd.5) 383 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:55:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:27 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.348989+0000 osd.5 (osd.5) 384 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:55:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:27 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.885744+0000 osd.7 (osd.7) 357 : cluster [DBG] 1.0 scrub starts 2024-09-13T22:55:27.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:27 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.889221+0000 osd.7 (osd.7) 358 : cluster [DBG] 1.0 scrub ok 2024-09-13T22:55:27.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:27.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:27.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:27.906 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:27.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:28.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:28.051 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:28.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:28.188 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:28.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:28.323 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:28.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.464000+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.471631+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[22840]: cluster 2024-09-13T22:55:26.839670+0000 mgr.y (mgr.14613) 663 : cluster [DBG] pgmap v764: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 597 op/s 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[22840]: cluster 2024-09-13T22:55:27.202921+0000 osd.4 (osd.4) 264 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[22840]: cluster 2024-09-13T22:55:27.209015+0000 osd.4 (osd.4) 265 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.464000+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:55:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.471631+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:55:28.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[29889]: cluster 2024-09-13T22:55:26.839670+0000 mgr.y (mgr.14613) 663 : cluster [DBG] pgmap v764: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 597 op/s 2024-09-13T22:55:28.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[29889]: cluster 2024-09-13T22:55:27.202921+0000 osd.4 (osd.4) 264 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:55:28.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:28 smithi178 bash[29889]: cluster 2024-09-13T22:55:27.209015+0000 osd.4 (osd.4) 265 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:55:28.455 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:28.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:28.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:28 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.464000+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:55:28.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:28 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.471631+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:55:28.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:28 smithi192 bash[24608]: cluster 2024-09-13T22:55:26.839670+0000 mgr.y (mgr.14613) 663 : cluster [DBG] pgmap v764: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 597 op/s 2024-09-13T22:55:28.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:28 smithi192 bash[24608]: cluster 2024-09-13T22:55:27.202921+0000 osd.4 (osd.4) 264 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:55:28.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:28 smithi192 bash[24608]: cluster 2024-09-13T22:55:27.209015+0000 osd.4 (osd.4) 265 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:55:28.593 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:28.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:28.732 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:28.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:28.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:28.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:28.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:29.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:29.172 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:29.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:27.842730+0000 osd.3 (osd.3) 333 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:55:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:27.845502+0000 osd.3 (osd.3) 334 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:55:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:27.936063+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:55:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:27.942881+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:55:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.198447+0000 osd.4 (osd.4) 266 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:55:29.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.200977+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:55:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.324643+0000 osd.5 (osd.5) 385 : cluster [DBG] 10.3 scrub starts 2024-09-13T22:55:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.845981+0000 osd.7 (osd.7) 359 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:55:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.849436+0000 osd.7 (osd.7) 360 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:55:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.998410+0000 osd.0 (osd.0) 347 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:55:29.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:29 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.003270+0000 osd.0 (osd.0) 348 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:55:29.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.439 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:29.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.552 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:27.842730+0000 osd.3 (osd.3) 333 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:27.845502+0000 osd.3 (osd.3) 334 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:27.936063+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:27.942881+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.198447+0000 osd.4 (osd.4) 266 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.200977+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.324643+0000 osd.5 (osd.5) 385 : cluster [DBG] 10.3 scrub starts 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.845981+0000 osd.7 (osd.7) 359 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.849436+0000 osd.7 (osd.7) 360 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.998410+0000 osd.0 (osd.0) 347 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:55:29.553 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.003270+0000 osd.0 (osd.0) 348 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:27.842730+0000 osd.3 (osd.3) 333 : cluster [DBG] 5.13 scrub starts 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:27.845502+0000 osd.3 (osd.3) 334 : cluster [DBG] 5.13 scrub ok 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:27.936063+0000 osd.2 (osd.2) 255 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:27.942881+0000 osd.2 (osd.2) 256 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.198447+0000 osd.4 (osd.4) 266 : cluster [DBG] 5.1c scrub starts 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.200977+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.1c scrub ok 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.324643+0000 osd.5 (osd.5) 385 : cluster [DBG] 10.3 scrub starts 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.845981+0000 osd.7 (osd.7) 359 : cluster [DBG] 3.15 scrub starts 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.849436+0000 osd.7 (osd.7) 360 : cluster [DBG] 3.15 scrub ok 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.998410+0000 osd.0 (osd.0) 347 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:55:29.554 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.003270+0000 osd.0 (osd.0) 348 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:55:29.574 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:29.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.733 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:29.839 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:29.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:29.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:29.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:30.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:30.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:30.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:30.133 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:30.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:55:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:55:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:30.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:30.268 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:30.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:30.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:30.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:30.641 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.497997+0000 osd.6 (osd.6) 293 : cluster [DBG] 10.b scrub starts 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[22840]: cluster 2024-09-13T22:55:28.840390+0000 mgr.y (mgr.14613) 664 : cluster [DBG] pgmap v765: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.339204+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.340950+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.497997+0000 osd.6 (osd.6) 293 : cluster [DBG] 10.b scrub starts 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[29889]: cluster 2024-09-13T22:55:28.840390+0000 mgr.y (mgr.14613) 664 : cluster [DBG] pgmap v765: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s 2024-09-13T22:55:30.650 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.339204+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:55:30.651 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:30 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.340950+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:55:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:30 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.497997+0000 osd.6 (osd.6) 293 : cluster [DBG] 10.b scrub starts 2024-09-13T22:55:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:30 smithi192 bash[24608]: cluster 2024-09-13T22:55:28.840390+0000 mgr.y (mgr.14613) 664 : cluster [DBG] pgmap v765: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s 2024-09-13T22:55:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:30 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.339204+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:55:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:30 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.340950+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:55:30.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:30.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:30.803 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:30.909 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:30.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:30.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:31.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:31.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:31.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:31.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:31.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:31.329 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:31.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:31.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.485415+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub starts 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.486535+0000 osd.1 (osd.1) 337 : cluster [DBG] 5.c scrub ok 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.813649+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:29.816208+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.119219+0000 osd.6 (osd.6) 294 : cluster [DBG] 10.b scrub ok 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.127684+0000 osd.5 (osd.5) 388 : cluster [DBG] 10.3 scrub ok 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.303816+0000 osd.5 (osd.5) 389 : cluster [DBG] 1.a scrub starts 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.307487+0000 osd.5 (osd.5) 390 : cluster [DBG] 1.a scrub ok 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.785656+0000 osd.7 (osd.7) 361 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:55:31.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:31 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.790110+0000 osd.7 (osd.7) 362 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:55:31.458 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:31.556 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:55:31.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:31.574 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.485415+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub starts 2024-09-13T22:55:31.574 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.486535+0000 osd.1 (osd.1) 337 : cluster [DBG] 5.c scrub ok 2024-09-13T22:55:31.574 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.813649+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:55:31.575 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:29.816208+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:55:31.575 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.119219+0000 osd.6 (osd.6) 294 : cluster [DBG] 10.b scrub ok 2024-09-13T22:55:31.575 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.127684+0000 osd.5 (osd.5) 388 : cluster [DBG] 10.3 scrub ok 2024-09-13T22:55:31.575 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.303816+0000 osd.5 (osd.5) 389 : cluster [DBG] 1.a scrub starts 2024-09-13T22:55:31.575 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.307487+0000 osd.5 (osd.5) 390 : cluster [DBG] 1.a scrub ok 2024-09-13T22:55:31.575 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.785656+0000 osd.7 (osd.7) 361 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:55:31.576 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.790110+0000 osd.7 (osd.7) 362 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:55:31.576 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.485415+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub starts 2024-09-13T22:55:31.576 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.486535+0000 osd.1 (osd.1) 337 : cluster [DBG] 5.c scrub ok 2024-09-13T22:55:31.576 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.813649+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.8 scrub starts 2024-09-13T22:55:31.576 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:29.816208+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.8 scrub ok 2024-09-13T22:55:31.577 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.119219+0000 osd.6 (osd.6) 294 : cluster [DBG] 10.b scrub ok 2024-09-13T22:55:31.577 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.127684+0000 osd.5 (osd.5) 388 : cluster [DBG] 10.3 scrub ok 2024-09-13T22:55:31.577 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.303816+0000 osd.5 (osd.5) 389 : cluster [DBG] 1.a scrub starts 2024-09-13T22:55:31.577 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.307487+0000 osd.5 (osd.5) 390 : cluster [DBG] 1.a scrub ok 2024-09-13T22:55:31.577 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.785656+0000 osd.7 (osd.7) 361 : cluster [DBG] 3.13 scrub starts 2024-09-13T22:55:31.577 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:31 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.790110+0000 osd.7 (osd.7) 362 : cluster [DBG] 3.13 scrub ok 2024-09-13T22:55:31.597 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:31.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:31.730 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:31.730 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:31.731 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:31.731 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:31.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:31.873 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:31.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:31.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.008 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:32.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:32.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:32.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:32.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:32.277 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:32.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:32.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:32.510 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:32.520 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.525670+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:55:32.520 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.533703+0000 osd.6 (osd.6) 296 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:55:32.520 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:30.841405+0000 mgr.y (mgr.14613) 665 : cluster [DBG] pgmap v766: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 890 op/s 2024-09-13T22:55:32.521 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.147242+0000 osd.4 (osd.4) 268 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:55:32.521 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.153149+0000 osd.4 (osd.4) 269 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:55:32.521 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.846565+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.f scrub starts 2024-09-13T22:55:32.521 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.848109+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.f scrub ok 2024-09-13T22:55:32.521 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.525670+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:55:32.521 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.533703+0000 osd.6 (osd.6) 296 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:55:32.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:30.841405+0000 mgr.y (mgr.14613) 665 : cluster [DBG] pgmap v766: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 890 op/s 2024-09-13T22:55:32.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.147242+0000 osd.4 (osd.4) 268 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:55:32.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.153149+0000 osd.4 (osd.4) 269 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:55:32.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.846565+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.f scrub starts 2024-09-13T22:55:32.522 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:32 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.848109+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.f scrub ok 2024-09-13T22:55:32.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:32.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:32.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:32.693 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.525670+0000 osd.6 (osd.6) 295 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:55:32.693 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.533703+0000 osd.6 (osd.6) 296 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:55:32.693 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:30.841405+0000 mgr.y (mgr.14613) 665 : cluster [DBG] pgmap v766: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 890 op/s 2024-09-13T22:55:32.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.147242+0000 osd.4 (osd.4) 268 : cluster [DBG] 6.2 scrub starts 2024-09-13T22:55:32.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.153149+0000 osd.4 (osd.4) 269 : cluster [DBG] 6.2 scrub ok 2024-09-13T22:55:32.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.846565+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.f scrub starts 2024-09-13T22:55:32.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:32 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.848109+0000 osd.7 (osd.7) 364 : cluster [DBG] 5.f scrub ok 2024-09-13T22:55:32.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:32.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:32.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:32.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:32.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:33.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.075 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:33.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:33.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:33.206 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:33.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:33.344 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:33.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:33.482 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:33.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:33.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.458678+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:55:33.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.459907+0000 osd.1 (osd.1) 339 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:55:33.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.570920+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:31.579365+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:32.031110+0000 osd.0 (osd.0) 349 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:32.033124+0000 osd.0 (osd.0) 350 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:32.844796+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[29889]: cluster 2024-09-13T22:55:32.846641+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.458678+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.459907+0000 osd.1 (osd.1) 339 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:55:33.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.570920+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:55:33.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:31.579365+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:55:33.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:32.031110+0000 osd.0 (osd.0) 349 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:55:33.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:32.033124+0000 osd.0 (osd.0) 350 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:55:33.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:32.844796+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:55:33.598 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:33 smithi178 bash[22840]: cluster 2024-09-13T22:55:32.846641+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:55:33.611 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:33.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:33.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:33.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:33.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.458678+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.459907+0000 osd.1 (osd.1) 339 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.570920+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:31.579365+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:32.031110+0000 osd.0 (osd.0) 349 : cluster [DBG] 5.1e scrub starts 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:32.033124+0000 osd.0 (osd.0) 350 : cluster [DBG] 5.1e scrub ok 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:32.844796+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.4 scrub starts 2024-09-13T22:55:33.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:33 smithi192 bash[24608]: cluster 2024-09-13T22:55:32.846641+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.4 scrub ok 2024-09-13T22:55:33.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:33.867 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:33.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:33.991 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:33.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:34.094 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:34.139 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:34.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:34.281 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:34.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:34.421 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:34.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:34 smithi178 bash[29889]: cluster 2024-09-13T22:55:32.841885+0000 mgr.y (mgr.14613) 666 : cluster [DBG] pgmap v767: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 807 op/s 2024-09-13T22:55:34.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:34 smithi178 bash[29889]: audit 2024-09-13T22:55:34.129523+0000 mon.a (mon.0) 1525 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:34.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:34 smithi178 bash[29889]: audit 2024-09-13T22:55:34.137351+0000 mon.b (mon.1) 240 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:34 smithi178 bash[22840]: cluster 2024-09-13T22:55:32.841885+0000 mgr.y (mgr.14613) 666 : cluster [DBG] pgmap v767: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 807 op/s 2024-09-13T22:55:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:34 smithi178 bash[22840]: audit 2024-09-13T22:55:34.129523+0000 mon.a (mon.0) 1525 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:34 smithi178 bash[22840]: audit 2024-09-13T22:55:34.137351+0000 mon.b (mon.1) 240 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:34.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:34.535 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:34 smithi192 bash[24608]: cluster 2024-09-13T22:55:32.841885+0000 mgr.y (mgr.14613) 666 : cluster [DBG] pgmap v767: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 807 op/s 2024-09-13T22:55:34.535 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:34 smithi192 bash[24608]: audit 2024-09-13T22:55:34.129523+0000 mon.a (mon.0) 1525 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:34.535 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:34 smithi192 bash[24608]: audit 2024-09-13T22:55:34.137351+0000 mon.b (mon.1) 240 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:34.550 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:34.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:34.675 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:34.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:34.808 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:34.914 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:34.937 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:34.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:35.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:35.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:35.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:35.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:35.199 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:35.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:35.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:35.330 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:35.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:35 smithi192 bash[24608]: cluster 2024-09-13T22:55:34.120285+0000 osd.0 (osd.0) 351 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:55:35.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:35 smithi192 bash[24608]: cluster 2024-09-13T22:55:34.128459+0000 osd.0 (osd.0) 352 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:55:35.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:35 smithi178 bash[29889]: cluster 2024-09-13T22:55:34.120285+0000 osd.0 (osd.0) 351 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:55:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:35 smithi178 bash[29889]: cluster 2024-09-13T22:55:34.128459+0000 osd.0 (osd.0) 352 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:55:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:35 smithi178 bash[22840]: cluster 2024-09-13T22:55:34.120285+0000 osd.0 (osd.0) 351 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:55:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:35 smithi178 bash[22840]: cluster 2024-09-13T22:55:34.128459+0000 osd.0 (osd.0) 352 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:55:35.749 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:35.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:35.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:35.879 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:35.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:35.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:36.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:36.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:36.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:55:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:55:36.153 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:36.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:36.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:36 smithi192 bash[24608]: cluster 2024-09-13T22:55:34.727891+0000 osd.3 (osd.3) 337 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:55:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:36 smithi192 bash[24608]: cluster 2024-09-13T22:55:34.729316+0000 osd.3 (osd.3) 338 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:55:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:36 smithi192 bash[24608]: cluster 2024-09-13T22:55:34.843304+0000 mgr.y (mgr.14613) 667 : cluster [DBG] pgmap v768: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T22:55:36.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:36 smithi192 bash[24608]: audit 2024-09-13T22:55:35.728838+0000 mgr.y (mgr.14613) 668 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:36.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:36.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:36.525 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:55:36.526 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:55:36.526 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:55:36.526 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:55:36.526 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:55:36.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[29889]: cluster 2024-09-13T22:55:34.727891+0000 osd.3 (osd.3) 337 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:55:36.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[29889]: cluster 2024-09-13T22:55:34.729316+0000 osd.3 (osd.3) 338 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:55:36.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[29889]: cluster 2024-09-13T22:55:34.843304+0000 mgr.y (mgr.14613) 667 : cluster [DBG] pgmap v768: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T22:55:36.586 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[29889]: audit 2024-09-13T22:55:35.728838+0000 mgr.y (mgr.14613) 668 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:36.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[22840]: cluster 2024-09-13T22:55:34.727891+0000 osd.3 (osd.3) 337 : cluster [DBG] 5.17 scrub starts 2024-09-13T22:55:36.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[22840]: cluster 2024-09-13T22:55:34.729316+0000 osd.3 (osd.3) 338 : cluster [DBG] 5.17 scrub ok 2024-09-13T22:55:36.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[22840]: cluster 2024-09-13T22:55:34.843304+0000 mgr.y (mgr.14613) 667 : cluster [DBG] pgmap v768: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T22:55:36.587 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:36 smithi178 bash[22840]: audit 2024-09-13T22:55:35.728838+0000 mgr.y (mgr.14613) 668 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:36.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:36.638 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:36.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:36.788 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:36.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:36.928 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:36.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:36.981 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:55:36.981 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:55:37.012 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:55:37.013 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:55:37.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:37.052 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:55:37.053 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:55:37.064 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:37.096 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:55:37.096 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:55:37.159 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:55:37.160 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:55:37.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:37.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:37.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:37.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:37.352 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:55:37.370 INFO:teuthology.orchestra.run.smithi178.stderr:marked out osd.4. 2024-09-13T22:55:37.415 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:37.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:35.529518+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.a scrub starts 2024-09-13T22:55:37.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:35.535248+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.a scrub ok 2024-09-13T22:55:37.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:35.743615+0000 osd.3 (osd.3) 339 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:55:37.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:35.746024+0000 osd.3 (osd.3) 340 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:55:37.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:36.126005+0000 osd.0 (osd.0) 353 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:55:37.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:36.129387+0000 osd.0 (osd.0) 354 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:55:37.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: audit 2024-09-13T22:55:36.425226+0000 mon.b (mon.1) 241 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:55:37.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:36.778020+0000 osd.7 (osd.7) 367 : cluster [DBG] 10.c scrub starts 2024-09-13T22:55:37.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: audit 2024-09-13T22:55:36.946620+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? v1:172.21.15.178:0/2673594668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:37.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: audit 2024-09-13T22:55:37.231838+0000 mon.a (mon.0) 1526 : audit [INF] from='client.25555 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:55:37.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: cluster 2024-09-13T22:55:37.231942+0000 mon.a (mon.0) 1527 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T22:55:37.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: audit 2024-09-13T22:55:37.232729+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.178:0/3553790666' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:55:37.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[29889]: audit 2024-09-13T22:55:37.261634+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? v1:172.21.15.178:0/1339014303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:55:37.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: debug 2024-09-13T22:55:37.227+0000 7f8b8e05d640 -1 mon.a@0(leader).osd e349 definitely_dead 0 2024-09-13T22:55:37.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:35.529518+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.a scrub starts 2024-09-13T22:55:37.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:35.535248+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.a scrub ok 2024-09-13T22:55:37.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:35.743615+0000 osd.3 (osd.3) 339 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:55:37.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:35.746024+0000 osd.3 (osd.3) 340 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:55:37.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:36.126005+0000 osd.0 (osd.0) 353 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:55:37.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:36.129387+0000 osd.0 (osd.0) 354 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:55:37.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: audit 2024-09-13T22:55:36.425226+0000 mon.b (mon.1) 241 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:55:37.427 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:36.778020+0000 osd.7 (osd.7) 367 : cluster [DBG] 10.c scrub starts 2024-09-13T22:55:37.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: audit 2024-09-13T22:55:36.946620+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? v1:172.21.15.178:0/2673594668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:37.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: audit 2024-09-13T22:55:37.231838+0000 mon.a (mon.0) 1526 : audit [INF] from='client.25555 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:55:37.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: cluster 2024-09-13T22:55:37.231942+0000 mon.a (mon.0) 1527 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T22:55:37.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: audit 2024-09-13T22:55:37.232729+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.178:0/3553790666' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:55:37.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:37 smithi178 bash[22840]: audit 2024-09-13T22:55:37.261634+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? v1:172.21.15.178:0/1339014303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:55:37.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:37.495 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:55:37.495 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:55:37.511 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:55:37.511 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:55:37.529 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:55:37.529 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:55:37.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:37.555 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:37.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:37.681 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:37.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:37.808 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:35.529518+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.a scrub starts 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:35.535248+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.a scrub ok 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:35.743615+0000 osd.3 (osd.3) 339 : cluster [DBG] 3.18 scrub starts 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:35.746024+0000 osd.3 (osd.3) 340 : cluster [DBG] 3.18 scrub ok 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:36.126005+0000 osd.0 (osd.0) 353 : cluster [DBG] 2.13 scrub starts 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:36.129387+0000 osd.0 (osd.0) 354 : cluster [DBG] 2.13 scrub ok 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: audit 2024-09-13T22:55:36.425226+0000 mon.b (mon.1) 241 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:55:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:36.778020+0000 osd.7 (osd.7) 367 : cluster [DBG] 10.c scrub starts 2024-09-13T22:55:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: audit 2024-09-13T22:55:36.946620+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? v1:172.21.15.178:0/2673594668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: audit 2024-09-13T22:55:37.231838+0000 mon.a (mon.0) 1526 : audit [INF] from='client.25555 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:55:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: cluster 2024-09-13T22:55:37.231942+0000 mon.a (mon.0) 1527 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T22:55:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: audit 2024-09-13T22:55:37.232729+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.178:0/3553790666' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-13T22:55:37.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:37 smithi192 bash[24608]: audit 2024-09-13T22:55:37.261634+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? v1:172.21.15.178:0/1339014303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:55:37.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:37.936 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:37.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:37.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T22:55:38.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:38.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:38.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:38.187 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:38.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:38.313 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:38.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:38.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:38.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:36.579816+0000 osd.1 (osd.1) 342 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:36.587992+0000 osd.1 (osd.1) 343 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:36.843763+0000 mgr.y (mgr.14613) 669 : cluster [DBG] pgmap v769: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 918 op/s 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:37.079440+0000 osd.4 (osd.4) 270 : cluster [DBG] 6.e scrub starts 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:37.084440+0000 osd.4 (osd.4) 271 : cluster [DBG] 6.e scrub ok 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:37.344146+0000 mon.a (mon.0) 1529 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: audit 2024-09-13T22:55:37.353244+0000 mon.a (mon.0) 1530 : audit [INF] from='client.25555 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: audit 2024-09-13T22:55:37.353425+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? v1:172.21.15.178:0/1339014303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:37.373242+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-09-13T22:55:38.442 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: audit 2024-09-13T22:55:37.662740+0000 mon.b (mon.1) 244 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:55:38.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: audit 2024-09-13T22:55:37.664200+0000 mon.b (mon.1) 245 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:55:38.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: audit 2024-09-13T22:55:37.671699+0000 mon.a (mon.0) 1533 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:38.443 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:38 smithi192 bash[24608]: cluster 2024-09-13T22:55:38.078002+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-09-13T22:55:38.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:38.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:38.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:36.579816+0000 osd.1 (osd.1) 342 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T22:55:38.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:36.587992+0000 osd.1 (osd.1) 343 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T22:55:38.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:36.843763+0000 mgr.y (mgr.14613) 669 : cluster [DBG] pgmap v769: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 918 op/s 2024-09-13T22:55:38.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:37.079440+0000 osd.4 (osd.4) 270 : cluster [DBG] 6.e scrub starts 2024-09-13T22:55:38.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:37.084440+0000 osd.4 (osd.4) 271 : cluster [DBG] 6.e scrub ok 2024-09-13T22:55:38.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:37.344146+0000 mon.a (mon.0) 1529 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: audit 2024-09-13T22:55:37.353244+0000 mon.a (mon.0) 1530 : audit [INF] from='client.25555 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: audit 2024-09-13T22:55:37.353425+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? v1:172.21.15.178:0/1339014303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:37.373242+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: audit 2024-09-13T22:55:37.662740+0000 mon.b (mon.1) 244 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: audit 2024-09-13T22:55:37.664200+0000 mon.b (mon.1) 245 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: audit 2024-09-13T22:55:37.671699+0000 mon.a (mon.0) 1533 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[29889]: cluster 2024-09-13T22:55:38.078002+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:36.579816+0000 osd.1 (osd.1) 342 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:36.587992+0000 osd.1 (osd.1) 343 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T22:55:38.557 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:36.843763+0000 mgr.y (mgr.14613) 669 : cluster [DBG] pgmap v769: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 918 op/s 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:37.079440+0000 osd.4 (osd.4) 270 : cluster [DBG] 6.e scrub starts 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:37.084440+0000 osd.4 (osd.4) 271 : cluster [DBG] 6.e scrub ok 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:37.344146+0000 mon.a (mon.0) 1529 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: audit 2024-09-13T22:55:37.353244+0000 mon.a (mon.0) 1530 : audit [INF] from='client.25555 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: audit 2024-09-13T22:55:37.353425+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? v1:172.21.15.178:0/1339014303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:37.373242+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: audit 2024-09-13T22:55:37.662740+0000 mon.b (mon.1) 244 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: audit 2024-09-13T22:55:37.664200+0000 mon.b (mon.1) 245 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: audit 2024-09-13T22:55:37.671699+0000 mon.a (mon.0) 1533 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:38.558 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:38 smithi178 bash[22840]: cluster 2024-09-13T22:55:38.078002+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-09-13T22:55:38.571 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:38.674 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:38.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:38.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:38.830 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:38.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:38.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:38.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:39.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:39.107 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:39.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:39.251 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:39.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:39.385 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:39.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:39.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[22840]: cluster 2024-09-13T22:55:37.472888+0000 osd.5 (osd.5) 391 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:55:39.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[22840]: cluster 2024-09-13T22:55:38.796718+0000 osd.7 (osd.7) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:39.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[22840]: cluster 2024-09-13T22:55:38.796826+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:39.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.084759+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-09-13T22:55:39.498 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.086832+0000 osd.7 (osd.7) 370 : cluster [DBG] 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 349'8751 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.091309+0000 osd.7 (osd.7) 371 : cluster [DBG] 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 349'8851 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[29889]: cluster 2024-09-13T22:55:37.472888+0000 osd.5 (osd.5) 391 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[29889]: cluster 2024-09-13T22:55:38.796718+0000 osd.7 (osd.7) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[29889]: cluster 2024-09-13T22:55:38.796826+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.084759+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.086832+0000 osd.7 (osd.7) 370 : cluster [DBG] 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 349'8751 2024-09-13T22:55:39.499 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.091309+0000 osd.7 (osd.7) 371 : cluster [DBG] 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 349'8851 2024-09-13T22:55:39.523 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:39.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:39 smithi192 bash[24608]: cluster 2024-09-13T22:55:37.472888+0000 osd.5 (osd.5) 391 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:55:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:39 smithi192 bash[24608]: cluster 2024-09-13T22:55:38.796718+0000 osd.7 (osd.7) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:39 smithi192 bash[24608]: cluster 2024-09-13T22:55:38.796826+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:39 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.084759+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-09-13T22:55:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:39 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.086832+0000 osd.7 (osd.7) 370 : cluster [DBG] 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 349'8751 2024-09-13T22:55:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:39 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.091309+0000 osd.7 (osd.7) 371 : cluster [DBG] 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 349'8851 2024-09-13T22:55:39.706 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:39.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:39.837 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:39.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:39.968 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:39.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:40.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:40.107 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:40.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:55:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:55:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:40.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:40.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:40.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:40.372 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:40.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.520 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:40.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:40.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:40.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:38.844690+0000 mgr.y (mgr.14613) 670 : cluster [DBG] pgmap v772: 158 pgs: 6 remapped+peering, 152 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 970 op/s; 63 B/s, 4 objects/s recovering 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.083853+0000 osd.5 (osd.5) 392 : cluster [DBG] 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 349'8928 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.085384+0000 osd.0 (osd.0) 355 : cluster [DBG] 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 349'9197 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.090398+0000 osd.6 (osd.6) 299 : cluster [DBG] 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 349'8873 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.091865+0000 osd.1 (osd.1) 344 : cluster [DBG] 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 349'9080 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.092337+0000 osd.6 (osd.6) 300 : cluster [DBG] 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 349'8747 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:39.110447+0000 osd.6 (osd.6) 301 : cluster [DBG] 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 349'8939 2024-09-13T22:55:40.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:40 smithi192 bash[24608]: cluster 2024-09-13T22:55:40.081645+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-09-13T22:55:40.647 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:40.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:38.844690+0000 mgr.y (mgr.14613) 670 : cluster [DBG] pgmap v772: 158 pgs: 6 remapped+peering, 152 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 970 op/s; 63 B/s, 4 objects/s recovering 2024-09-13T22:55:40.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.083853+0000 osd.5 (osd.5) 392 : cluster [DBG] 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 349'8928 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.085384+0000 osd.0 (osd.0) 355 : cluster [DBG] 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 349'9197 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.090398+0000 osd.6 (osd.6) 299 : cluster [DBG] 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 349'8873 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.091865+0000 osd.1 (osd.1) 344 : cluster [DBG] 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 349'9080 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.092337+0000 osd.6 (osd.6) 300 : cluster [DBG] 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 349'8747 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:39.110447+0000 osd.6 (osd.6) 301 : cluster [DBG] 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 349'8939 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[22840]: cluster 2024-09-13T22:55:40.081645+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:38.844690+0000 mgr.y (mgr.14613) 670 : cluster [DBG] pgmap v772: 158 pgs: 6 remapped+peering, 152 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 970 op/s; 63 B/s, 4 objects/s recovering 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.083853+0000 osd.5 (osd.5) 392 : cluster [DBG] 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 349'8928 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.085384+0000 osd.0 (osd.0) 355 : cluster [DBG] 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 349'9197 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.090398+0000 osd.6 (osd.6) 299 : cluster [DBG] 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 349'8873 2024-09-13T22:55:40.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.091865+0000 osd.1 (osd.1) 344 : cluster [DBG] 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 349'9080 2024-09-13T22:55:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.092337+0000 osd.6 (osd.6) 300 : cluster [DBG] 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 349'8747 2024-09-13T22:55:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:39.110447+0000 osd.6 (osd.6) 301 : cluster [DBG] 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 349'8939 2024-09-13T22:55:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:40 smithi178 bash[29889]: cluster 2024-09-13T22:55:40.081645+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-09-13T22:55:40.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:40.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:40.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:40.906 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:40.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:41.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:41.047 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:41.154 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:41.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:41.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:41.321 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:41.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:41.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:41.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:41.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:41.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:41.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:41.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:41.849 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:41.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:41.979 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:41.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:42.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:42.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:42.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:42.269 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:42.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:42.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:42 smithi192 bash[24608]: cluster 2024-09-13T22:55:40.564735+0000 osd.6 (osd.6) 302 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:42.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:42 smithi192 bash[24608]: cluster 2024-09-13T22:55:40.564802+0000 osd.6 (osd.6) 303 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:42.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:42 smithi192 bash[24608]: cluster 2024-09-13T22:55:40.845320+0000 mgr.y (mgr.14613) 671 : cluster [DBG] pgmap v775: 158 pgs: 1 active+undersized+remapped, 9 remapped+peering, 1 active+remapped+backfill_toofull, 147 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 824 op/s; 4925/234552 objects misplaced (2.100%); 346 B/s, 19 objects/s recovering 2024-09-13T22:55:42.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:42 smithi192 bash[24608]: cluster 2024-09-13T22:55:41.088208+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-09-13T22:55:42.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:42 smithi192 bash[24608]: cluster 2024-09-13T22:55:41.366683+0000 mon.a (mon.0) 1538 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:55:42.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[29889]: cluster 2024-09-13T22:55:40.564735+0000 osd.6 (osd.6) 302 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[29889]: cluster 2024-09-13T22:55:40.564802+0000 osd.6 (osd.6) 303 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[29889]: cluster 2024-09-13T22:55:40.845320+0000 mgr.y (mgr.14613) 671 : cluster [DBG] pgmap v775: 158 pgs: 1 active+undersized+remapped, 9 remapped+peering, 1 active+remapped+backfill_toofull, 147 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 824 op/s; 4925/234552 objects misplaced (2.100%); 346 B/s, 19 objects/s recovering 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[29889]: cluster 2024-09-13T22:55:41.088208+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[29889]: cluster 2024-09-13T22:55:41.366683+0000 mon.a (mon.0) 1538 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[22840]: cluster 2024-09-13T22:55:40.564735+0000 osd.6 (osd.6) 302 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[22840]: cluster 2024-09-13T22:55:40.564802+0000 osd.6 (osd.6) 303 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:42.434 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[22840]: cluster 2024-09-13T22:55:40.845320+0000 mgr.y (mgr.14613) 671 : cluster [DBG] pgmap v775: 158 pgs: 1 active+undersized+remapped, 9 remapped+peering, 1 active+remapped+backfill_toofull, 147 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 824 op/s; 4925/234552 objects misplaced (2.100%); 346 B/s, 19 objects/s recovering 2024-09-13T22:55:42.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[22840]: cluster 2024-09-13T22:55:41.088208+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-09-13T22:55:42.435 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:42 smithi178 bash[22840]: cluster 2024-09-13T22:55:41.366683+0000 mon.a (mon.0) 1538 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:55:42.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:42.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:42.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:42.676 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:42.752 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:55:42.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:42.804 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:42.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:42.946 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:42.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:55:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:42.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:42.990 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:55:43.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:43.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:43.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.241 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:43.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.373 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:43.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:43.501 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:43.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:43.606 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.636 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:43.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:43.768 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:43.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.905 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:43.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:44.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:44.121 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:44 smithi178 bash[29889]: cluster 2024-09-13T22:55:42.845781+0000 mgr.y (mgr.14613) 672 : cluster [DBG] pgmap v777: 158 pgs: 1 active+undersized+remapped, 9 remapped+peering, 1 active+remapped+backfill_toofull, 147 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 691 op/s; 4925/234552 objects misplaced (2.100%); 290 B/s, 16 objects/s recovering 2024-09-13T22:55:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:44 smithi178 bash[29889]: cluster 2024-09-13T22:55:43.453120+0000 osd.3 (osd.3) 341 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:44 smithi178 bash[29889]: cluster 2024-09-13T22:55:43.453176+0000 osd.3 (osd.3) 342 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:44 smithi178 bash[22840]: cluster 2024-09-13T22:55:42.845781+0000 mgr.y (mgr.14613) 672 : cluster [DBG] pgmap v777: 158 pgs: 1 active+undersized+remapped, 9 remapped+peering, 1 active+remapped+backfill_toofull, 147 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 691 op/s; 4925/234552 objects misplaced (2.100%); 290 B/s, 16 objects/s recovering 2024-09-13T22:55:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:44 smithi178 bash[22840]: cluster 2024-09-13T22:55:43.453120+0000 osd.3 (osd.3) 341 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:44 smithi178 bash[22840]: cluster 2024-09-13T22:55:43.453176+0000 osd.3 (osd.3) 342 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:44.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:44.254 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:44.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:44.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:44 smithi192 bash[24608]: cluster 2024-09-13T22:55:42.845781+0000 mgr.y (mgr.14613) 672 : cluster [DBG] pgmap v777: 158 pgs: 1 active+undersized+remapped, 9 remapped+peering, 1 active+remapped+backfill_toofull, 147 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 691 op/s; 4925/234552 objects misplaced (2.100%); 290 B/s, 16 objects/s recovering 2024-09-13T22:55:44.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:44 smithi192 bash[24608]: cluster 2024-09-13T22:55:43.453120+0000 osd.3 (osd.3) 341 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:44.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:44 smithi192 bash[24608]: cluster 2024-09-13T22:55:43.453176+0000 osd.3 (osd.3) 342 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:44.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:44.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:44.517 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:44.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:44.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:44.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:44.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:44.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:44.932 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:44.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:45.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:45.070 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:45.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:45.185 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:45 smithi178 bash[22840]: cluster 2024-09-13T22:55:44.107228+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-09-13T22:55:45.187 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:45 smithi178 bash[29889]: cluster 2024-09-13T22:55:44.107228+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-09-13T22:55:45.207 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:45.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:45.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:45.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:45 smithi192 bash[24608]: cluster 2024-09-13T22:55:44.107228+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-09-13T22:55:45.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:45.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:45.604 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:45.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:45.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:45.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:45.878 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:45.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:45.967 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:55:45.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:55:45.968 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 5, 6, 7] 2024-09-13T22:55:45.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd out 6 2024-09-13T22:55:45.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:46.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:55:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:55:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.144 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:46.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:46.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:46.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:46.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:46 smithi192 bash[24608]: cluster 2024-09-13T22:55:44.846242+0000 mgr.y (mgr.14613) 673 : cluster [DBG] pgmap v779: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 720 op/s; 35024/223602 objects misplaced (15.664%); 22 MiB/s, 360 objects/s recovering 2024-09-13T22:55:46.389 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:46 smithi192 bash[24608]: audit 2024-09-13T22:55:45.736904+0000 mgr.y (mgr.14613) 674 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:46.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:46 smithi178 bash[29889]: cluster 2024-09-13T22:55:44.846242+0000 mgr.y (mgr.14613) 673 : cluster [DBG] pgmap v779: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 720 op/s; 35024/223602 objects misplaced (15.664%); 22 MiB/s, 360 objects/s recovering 2024-09-13T22:55:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:46 smithi178 bash[29889]: audit 2024-09-13T22:55:45.736904+0000 mgr.y (mgr.14613) 674 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:46 smithi178 bash[22840]: cluster 2024-09-13T22:55:44.846242+0000 mgr.y (mgr.14613) 673 : cluster [DBG] pgmap v779: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 720 op/s; 35024/223602 objects misplaced (15.664%); 22 MiB/s, 360 objects/s recovering 2024-09-13T22:55:46.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:46 smithi178 bash[22840]: audit 2024-09-13T22:55:45.736904+0000 mgr.y (mgr.14613) 674 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:46.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:46.535 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:46.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:46.666 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:46.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:46.806 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:46.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:46.938 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:46.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:47.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:47.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:47.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:47.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:47.184 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:47.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:47.287 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:47.309 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:47.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:47.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:47 smithi192 bash[24608]: cluster 2024-09-13T22:55:46.233015+0000 osd.0 (osd.0) 356 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:47.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:47 smithi192 bash[24608]: cluster 2024-09-13T22:55:46.233064+0000 osd.0 (osd.0) 357 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:47.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:47.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:47 smithi178 bash[22840]: cluster 2024-09-13T22:55:46.233015+0000 osd.0 (osd.0) 356 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:47.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:47 smithi178 bash[22840]: cluster 2024-09-13T22:55:46.233064+0000 osd.0 (osd.0) 357 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:47.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:47 smithi178 bash[29889]: cluster 2024-09-13T22:55:46.233015+0000 osd.0 (osd.0) 356 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:55:47.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:47 smithi178 bash[29889]: cluster 2024-09-13T22:55:46.233064+0000 osd.0 (osd.0) 357 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:55:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:47.750 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:47.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:47.823 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:47.826 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:47.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:47.882 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:47.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:47.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:48.014 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:48.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:48.160 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:48.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:48.288 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:48.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:48 smithi192 bash[24608]: cluster 2024-09-13T22:55:46.848723+0000 mgr.y (mgr.14613) 675 : cluster [DBG] pgmap v780: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 756 B/s rd, 0 B/s wr, 763 op/s; 34709/220572 objects misplaced (15.736%); 19 MiB/s, 307 objects/s recovering 2024-09-13T22:55:48.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:48 smithi192 bash[24608]: cluster 2024-09-13T22:55:47.131810+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-09-13T22:55:48.390 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:48 smithi192 bash[24608]: cluster 2024-09-13T22:55:48.075141+0000 mon.a (mon.0) 1541 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:55:48.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:48.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:48 smithi178 bash[29889]: cluster 2024-09-13T22:55:46.848723+0000 mgr.y (mgr.14613) 675 : cluster [DBG] pgmap v780: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 756 B/s rd, 0 B/s wr, 763 op/s; 34709/220572 objects misplaced (15.736%); 19 MiB/s, 307 objects/s recovering 2024-09-13T22:55:48.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:48 smithi178 bash[29889]: cluster 2024-09-13T22:55:47.131810+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-09-13T22:55:48.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:48 smithi178 bash[29889]: cluster 2024-09-13T22:55:48.075141+0000 mon.a (mon.0) 1541 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:55:48.402 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:48 smithi178 bash[22840]: cluster 2024-09-13T22:55:46.848723+0000 mgr.y (mgr.14613) 675 : cluster [DBG] pgmap v780: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 756 B/s rd, 0 B/s wr, 763 op/s; 34709/220572 objects misplaced (15.736%); 19 MiB/s, 307 objects/s recovering 2024-09-13T22:55:48.402 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:48 smithi178 bash[22840]: cluster 2024-09-13T22:55:47.131810+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-09-13T22:55:48.402 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:48 smithi178 bash[22840]: cluster 2024-09-13T22:55:48.075141+0000 mon.a (mon.0) 1541 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:55:48.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:48.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:48.558 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:48.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:48.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:48.703 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T22:55:48.703 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:55:48.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:48.817 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:48.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:48.950 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:48.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:49.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:49.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:49.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.226 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:49.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:49.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:49.356 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:49.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:49.488 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:49.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:49.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:49.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:49.743 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:49.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:55:49.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:49.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:49.995 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:49.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:50.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:50.111 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:55:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:55:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:50.127 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:50.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:50.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:50.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:50.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:50.401 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:50 smithi178 bash[22840]: cluster 2024-09-13T22:55:48.849513+0000 mgr.y (mgr.14613) 676 : cluster [DBG] pgmap v782: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1k op/s; 30571/211290 objects misplaced (14.469%); 40 MiB/s, 643 objects/s recovering 2024-09-13T22:55:50.401 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:50 smithi178 bash[22840]: audit 2024-09-13T22:55:49.157705+0000 mon.a (mon.0) 1542 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:50.401 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:50 smithi178 bash[22840]: audit 2024-09-13T22:55:49.163472+0000 mon.b (mon.1) 246 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:50.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:50 smithi178 bash[29889]: cluster 2024-09-13T22:55:48.849513+0000 mgr.y (mgr.14613) 676 : cluster [DBG] pgmap v782: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1k op/s; 30571/211290 objects misplaced (14.469%); 40 MiB/s, 643 objects/s recovering 2024-09-13T22:55:50.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:50 smithi178 bash[29889]: audit 2024-09-13T22:55:49.157705+0000 mon.a (mon.0) 1542 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:50.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:50 smithi178 bash[29889]: audit 2024-09-13T22:55:49.163472+0000 mon.b (mon.1) 246 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:50.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:50.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:50 smithi192 bash[24608]: cluster 2024-09-13T22:55:48.849513+0000 mgr.y (mgr.14613) 676 : cluster [DBG] pgmap v782: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1k op/s; 30571/211290 objects misplaced (14.469%); 40 MiB/s, 643 objects/s recovering 2024-09-13T22:55:50.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:50 smithi192 bash[24608]: audit 2024-09-13T22:55:49.157705+0000 mon.a (mon.0) 1542 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:55:50.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:50 smithi192 bash[24608]: audit 2024-09-13T22:55:49.163472+0000 mon.b (mon.1) 246 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:55:50.526 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:50.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:50.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T22:55:50.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:50.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:50.802 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:50.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:50.939 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:50.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:51.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:51.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:51.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:51.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:51.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:51.344 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:51.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:51.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:51.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:51.611 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:51.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:51.742 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:51.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:51.883 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:51.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:51.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:52.017 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:52.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:52.143 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:52.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:52.268 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:52.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:52.392 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:52.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:52 smithi192 bash[24608]: cluster 2024-09-13T22:55:50.850099+0000 mgr.y (mgr.14613) 677 : cluster [DBG] pgmap v783: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 30038/209691 objects misplaced (14.325%); 39 MiB/s, 623 objects/s recovering 2024-09-13T22:55:52.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:52 smithi178 bash[29889]: cluster 2024-09-13T22:55:50.850099+0000 mgr.y (mgr.14613) 677 : cluster [DBG] pgmap v783: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 30038/209691 objects misplaced (14.325%); 39 MiB/s, 623 objects/s recovering 2024-09-13T22:55:52.410 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:52 smithi178 bash[22840]: cluster 2024-09-13T22:55:50.850099+0000 mgr.y (mgr.14613) 677 : cluster [DBG] pgmap v783: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 30038/209691 objects misplaced (14.325%); 39 MiB/s, 623 objects/s recovering 2024-09-13T22:55:52.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:52.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:52.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:52.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:52.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:52.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:52.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:52.918 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:52.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:53.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:53.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:53.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:53.187 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:53.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:53.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:53.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:53.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.450 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:53.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:53.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:53.575 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:53.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:53.706 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:53.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:53.834 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:53.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:53.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:54.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:54.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:54.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:54.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:54.201 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:54 smithi178 bash[22840]: cluster 2024-09-13T22:55:52.850679+0000 mgr.y (mgr.14613) 678 : cluster [DBG] pgmap v784: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 585 B/s rd, 0 B/s wr, 608 op/s; 29810/207645 objects misplaced (14.356%); 21 MiB/s, 334 objects/s recovering 2024-09-13T22:55:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:54.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:54.333 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:54.438 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:54.465 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:54.468 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:54 smithi178 bash[29889]: cluster 2024-09-13T22:55:52.850679+0000 mgr.y (mgr.14613) 678 : cluster [DBG] pgmap v784: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 585 B/s rd, 0 B/s wr, 608 op/s; 29810/207645 objects misplaced (14.356%); 21 MiB/s, 334 objects/s recovering 2024-09-13T22:55:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:54.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:54.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:54 smithi192 bash[24608]: cluster 2024-09-13T22:55:52.850679+0000 mgr.y (mgr.14613) 678 : cluster [DBG] pgmap v784: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 585 B/s rd, 0 B/s wr, 608 op/s; 29810/207645 objects misplaced (14.356%); 21 MiB/s, 334 objects/s recovering 2024-09-13T22:55:54.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:54.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:54.736 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:54.844 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:54.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:54.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:54.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:55.005 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:55.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:55.137 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:55.241 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:55.269 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:55.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:55.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:55.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:55.518 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:55:55.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:55.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:55.660 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:55.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:55.785 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:55.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:55.907 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:55:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:55:55.916 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:55.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:56.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:56.046 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:56.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:56.174 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:56.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:56.287 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:56 smithi192 bash[24608]: cluster 2024-09-13T22:55:54.853355+0000 mgr.y (mgr.14613) 679 : cluster [DBG] pgmap v785: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 24418/198102 objects misplaced (12.326%); 44 MiB/s, 701 objects/s recovering 2024-09-13T22:55:56.287 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:56 smithi192 bash[24608]: audit 2024-09-13T22:55:55.745107+0000 mgr.y (mgr.14613) 680 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:56.299 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:56.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:56.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:56.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:56.559 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:56.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:56.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:56.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:56.826 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:56.892 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:56 smithi178 bash[22840]: cluster 2024-09-13T22:55:54.853355+0000 mgr.y (mgr.14613) 679 : cluster [DBG] pgmap v785: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 24418/198102 objects misplaced (12.326%); 44 MiB/s, 701 objects/s recovering 2024-09-13T22:55:56.892 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:56 smithi178 bash[22840]: audit 2024-09-13T22:55:55.745107+0000 mgr.y (mgr.14613) 680 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:56.892 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:56 smithi178 bash[29889]: cluster 2024-09-13T22:55:54.853355+0000 mgr.y (mgr.14613) 679 : cluster [DBG] pgmap v785: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 24418/198102 objects misplaced (12.326%); 44 MiB/s, 701 objects/s recovering 2024-09-13T22:55:56.892 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:56 smithi178 bash[29889]: audit 2024-09-13T22:55:55.745107+0000 mgr.y (mgr.14613) 680 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:55:56.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:56.962 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:56.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:57.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:57.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:57.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:57.235 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:57.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:57.340 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:57.376 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:57.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:57.511 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:57.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:57.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:57.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:57.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:57.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:57.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:57.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:58.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:58.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:58.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:58.172 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:58.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:58.296 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:55:58.400 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:58.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:58 smithi192 bash[24608]: cluster 2024-09-13T22:55:56.853931+0000 mgr.y (mgr.14613) 681 : cluster [DBG] pgmap v786: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 826 op/s; 24157/195780 objects misplaced (12.339%); 44 MiB/s, 702 objects/s recovering 2024-09-13T22:55:58.423 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:58.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:58 smithi178 bash[29889]: cluster 2024-09-13T22:55:56.853931+0000 mgr.y (mgr.14613) 681 : cluster [DBG] pgmap v786: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 826 op/s; 24157/195780 objects misplaced (12.339%); 44 MiB/s, 702 objects/s recovering 2024-09-13T22:55:58.445 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:58 smithi178 bash[22840]: cluster 2024-09-13T22:55:56.853931+0000 mgr.y (mgr.14613) 681 : cluster [DBG] pgmap v786: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 826 op/s; 24157/195780 objects misplaced (12.339%); 44 MiB/s, 702 objects/s recovering 2024-09-13T22:55:58.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:58.547 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:55:58.547 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:55:58.552 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:58.592 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:55:58.592 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:55:58.639 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:55:58.639 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:55:58.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:58.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:58.699 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:55:58.699 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:55:58.738 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:55:58.738 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:55:58.769 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:55:58.769 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:55:58.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:58.814 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:55:58.814 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:55:58.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:55:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:55:58.863 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:55:58.864 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:55:58.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:58.956 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:55:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:58.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:55:59.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:59.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:59.176 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[22840]: debug 2024-09-13T22:55:59.103+0000 7f8b8e05d640 -1 mon.a@0(leader).osd e356 definitely_dead 0 2024-09-13T22:55:59.185 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:55:59.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:59.189 INFO:teuthology.orchestra.run.smithi178.stderr:marked out osd.6. 2024-09-13T22:55:59.217 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:55:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:55:59.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:59.341 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:55:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:55:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:59 smithi192 bash[24608]: audit 2024-09-13T22:55:58.494827+0000 mon.b (mon.1) 247 : audit [DBG] from='client.? v1:172.21.15.178:0/3308726941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:59 smithi192 bash[24608]: audit 2024-09-13T22:55:59.093905+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.178:0/3269732513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:59.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:59 smithi192 bash[24608]: audit 2024-09-13T22:55:59.107919+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.178:0/2371956926' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-13T22:55:59.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:59 smithi192 bash[24608]: audit 2024-09-13T22:55:59.108637+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25670 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-13T22:55:59.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:55:59 smithi192 bash[24608]: cluster 2024-09-13T22:55:59.108856+0000 mon.a (mon.0) 1545 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T22:55:59.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:59.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:55:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:55:59.564 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[29889]: audit 2024-09-13T22:55:58.494827+0000 mon.b (mon.1) 247 : audit [DBG] from='client.? v1:172.21.15.178:0/3308726941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[29889]: audit 2024-09-13T22:55:59.093905+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.178:0/3269732513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[29889]: audit 2024-09-13T22:55:59.107919+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.178:0/2371956926' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[29889]: audit 2024-09-13T22:55:59.108637+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25670 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[29889]: cluster 2024-09-13T22:55:59.108856+0000 mon.a (mon.0) 1545 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[22840]: audit 2024-09-13T22:55:58.494827+0000 mon.b (mon.1) 247 : audit [DBG] from='client.? v1:172.21.15.178:0/3308726941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[22840]: audit 2024-09-13T22:55:59.093905+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.178:0/3269732513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[22840]: audit 2024-09-13T22:55:59.107919+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.178:0/2371956926' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[22840]: audit 2024-09-13T22:55:59.108637+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25670 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-13T22:55:59.565 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[22840]: cluster 2024-09-13T22:55:59.108856+0000 mon.a (mon.0) 1545 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T22:55:59.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:59.588 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:55:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:55:59.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:59.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:59.815 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:59.824 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:55:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:55:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:55:59.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:55:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:55:59.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:59.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:55:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:55:59.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:00.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:00.083 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:00.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:00.231 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:00.339 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:00.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[29889]: cluster 2024-09-13T22:55:58.856007+0000 mgr.y (mgr.14613) 682 : cluster [DBG] pgmap v787: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 941 op/s; 20112/187485 objects misplaced (10.727%); 56 MiB/s, 892 objects/s recovering 2024-09-13T22:56:00.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[29889]: cluster 2024-09-13T22:55:59.176562+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:56:00.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[29889]: audit 2024-09-13T22:55:59.184031+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? v1:172.21.15.178:0/3269732513' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:56:00.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[29889]: audit 2024-09-13T22:55:59.184199+0000 mon.a (mon.0) 1548 : audit [INF] from='client.25670 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T22:56:00.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[29889]: cluster 2024-09-13T22:55:59.190249+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-09-13T22:56:00.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[22840]: cluster 2024-09-13T22:55:58.856007+0000 mgr.y (mgr.14613) 682 : cluster [DBG] pgmap v787: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 941 op/s; 20112/187485 objects misplaced (10.727%); 56 MiB/s, 892 objects/s recovering 2024-09-13T22:56:00.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[22840]: cluster 2024-09-13T22:55:59.176562+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:56:00.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[22840]: audit 2024-09-13T22:55:59.184031+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? v1:172.21.15.178:0/3269732513' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:56:00.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[22840]: audit 2024-09-13T22:55:59.184199+0000 mon.a (mon.0) 1548 : audit [INF] from='client.25670 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T22:56:00.351 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:00 smithi178 bash[22840]: cluster 2024-09-13T22:55:59.190249+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-09-13T22:56:00.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:00.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:00.475 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:00 smithi192 bash[24608]: cluster 2024-09-13T22:55:58.856007+0000 mgr.y (mgr.14613) 682 : cluster [DBG] pgmap v787: 158 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 150 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 941 op/s; 20112/187485 objects misplaced (10.727%); 56 MiB/s, 892 objects/s recovering 2024-09-13T22:56:00.476 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:00 smithi192 bash[24608]: cluster 2024-09-13T22:55:59.176562+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:56:00.476 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:00 smithi192 bash[24608]: audit 2024-09-13T22:55:59.184031+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? v1:172.21.15.178:0/3269732513' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:56:00.476 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:00 smithi192 bash[24608]: audit 2024-09-13T22:55:59.184199+0000 mon.a (mon.0) 1548 : audit [INF] from='client.25670 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T22:56:00.476 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:00 smithi192 bash[24608]: cluster 2024-09-13T22:55:59.190249+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-09-13T22:56:00.492 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:00.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:00.622 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:00.725 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:00.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:00.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:00.878 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:00.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:00.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.005 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:01.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.135 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:01.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:01 smithi178 bash[29889]: cluster 2024-09-13T22:56:00.190494+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-09-13T22:56:01.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:01 smithi178 bash[29889]: cluster 2024-09-13T22:56:00.217752+0000 osd.6 (osd.6) 304 : cluster [DBG] 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 356'9932 2024-09-13T22:56:01.249 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:01 smithi178 bash[29889]: cluster 2024-09-13T22:56:00.242400+0000 osd.6 (osd.6) 305 : cluster [INF] 10.e continuing backfill to osd.5 from (341'5200,356'9932] 10:7d52c34a:::benchmark_data_smithi178_111521_object66294:head to 356'9932 2024-09-13T22:56:01.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:01 smithi178 bash[22840]: cluster 2024-09-13T22:56:00.190494+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-09-13T22:56:01.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:01 smithi178 bash[22840]: cluster 2024-09-13T22:56:00.217752+0000 osd.6 (osd.6) 304 : cluster [DBG] 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 356'9932 2024-09-13T22:56:01.249 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:01 smithi178 bash[22840]: cluster 2024-09-13T22:56:00.242400+0000 osd.6 (osd.6) 305 : cluster [INF] 10.e continuing backfill to osd.5 from (341'5200,356'9932] 10:7d52c34a:::benchmark_data_smithi178_111521_object66294:head to 356'9932 2024-09-13T22:56:01.268 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:01.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:01.405 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:01.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.535 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:01.639 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:01 smithi192 bash[24608]: cluster 2024-09-13T22:56:00.190494+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-09-13T22:56:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:01 smithi192 bash[24608]: cluster 2024-09-13T22:56:00.217752+0000 osd.6 (osd.6) 304 : cluster [DBG] 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 356'9932 2024-09-13T22:56:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:01 smithi192 bash[24608]: cluster 2024-09-13T22:56:00.242400+0000 osd.6 (osd.6) 305 : cluster [INF] 10.e continuing backfill to osd.5 from (341'5200,356'9932] 10:7d52c34a:::benchmark_data_smithi178_111521_object66294:head to 356'9932 2024-09-13T22:56:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:01.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:01.798 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:01.902 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:01.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:01.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:02.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:02.063 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:02.168 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:02.188 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:02.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:00.856695+0000 mgr.y (mgr.14613) 683 : cluster [DBG] pgmap v790: 158 pgs: 1 peering, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 149 active+clean; 3.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 899 op/s; 19644/186081 objects misplaced (10.557%); 59 MiB/s, 941 objects/s recovering 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.199749+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e359: 8 total, 8 up, 6 in 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.200385+0000 osd.3 (osd.3) 343 : cluster [DBG] 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 356'9843 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.201488+0000 osd.1 (osd.1) 345 : cluster [DBG] 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 356'9998 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.201738+0000 osd.0 (osd.0) 358 : cluster [INF] 10.d continuing backfill to osd.3 from (341'5060,356'9826] 10:b933206a:::benchmark_data_smithi178_111521_object53980:head to 356'9826 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.201829+0000 osd.0 (osd.0) 359 : cluster [DBG] 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 356'9826 2024-09-13T22:56:02.301 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.202499+0000 osd.0 (osd.0) 360 : cluster [DBG] 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 356'10276 2024-09-13T22:56:02.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.204202+0000 osd.3 (osd.3) 344 : cluster [DBG] 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 356'10225 2024-09-13T22:56:02.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.209280+0000 osd.1 (osd.1) 346 : cluster [DBG] 10.b starting backfill to osd.7 from (0'0,0'0] MAX to 356'10084 2024-09-13T22:56:02.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:02 smithi192 bash[24608]: cluster 2024-09-13T22:56:01.214923+0000 osd.1 (osd.1) 347 : cluster [INF] 10.0 continuing backfill to osd.7 from (341'5252,356'9998] MIN to 356'9998 2024-09-13T22:56:02.315 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:02.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:02.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:00.856695+0000 mgr.y (mgr.14613) 683 : cluster [DBG] pgmap v790: 158 pgs: 1 peering, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 149 active+clean; 3.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 899 op/s; 19644/186081 objects misplaced (10.557%); 59 MiB/s, 941 objects/s recovering 2024-09-13T22:56:02.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.199749+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e359: 8 total, 8 up, 6 in 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.200385+0000 osd.3 (osd.3) 343 : cluster [DBG] 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 356'9843 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.201488+0000 osd.1 (osd.1) 345 : cluster [DBG] 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 356'9998 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.201738+0000 osd.0 (osd.0) 358 : cluster [INF] 10.d continuing backfill to osd.3 from (341'5060,356'9826] 10:b933206a:::benchmark_data_smithi178_111521_object53980:head to 356'9826 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.201829+0000 osd.0 (osd.0) 359 : cluster [DBG] 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 356'9826 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.202499+0000 osd.0 (osd.0) 360 : cluster [DBG] 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 356'10276 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.204202+0000 osd.3 (osd.3) 344 : cluster [DBG] 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 356'10225 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.209280+0000 osd.1 (osd.1) 346 : cluster [DBG] 10.b starting backfill to osd.7 from (0'0,0'0] MAX to 356'10084 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[29889]: cluster 2024-09-13T22:56:01.214923+0000 osd.1 (osd.1) 347 : cluster [INF] 10.0 continuing backfill to osd.7 from (341'5252,356'9998] MIN to 356'9998 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:00.856695+0000 mgr.y (mgr.14613) 683 : cluster [DBG] pgmap v790: 158 pgs: 1 peering, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 149 active+clean; 3.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 899 op/s; 19644/186081 objects misplaced (10.557%); 59 MiB/s, 941 objects/s recovering 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.199749+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e359: 8 total, 8 up, 6 in 2024-09-13T22:56:02.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.200385+0000 osd.3 (osd.3) 343 : cluster [DBG] 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 356'9843 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.201488+0000 osd.1 (osd.1) 345 : cluster [DBG] 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 356'9998 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.201738+0000 osd.0 (osd.0) 358 : cluster [INF] 10.d continuing backfill to osd.3 from (341'5060,356'9826] 10:b933206a:::benchmark_data_smithi178_111521_object53980:head to 356'9826 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.201829+0000 osd.0 (osd.0) 359 : cluster [DBG] 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 356'9826 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.202499+0000 osd.0 (osd.0) 360 : cluster [DBG] 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 356'10276 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.204202+0000 osd.3 (osd.3) 344 : cluster [DBG] 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 356'10225 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.209280+0000 osd.1 (osd.1) 346 : cluster [DBG] 10.b starting backfill to osd.7 from (0'0,0'0] MAX to 356'10084 2024-09-13T22:56:02.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:02 smithi178 bash[22840]: cluster 2024-09-13T22:56:01.214923+0000 osd.1 (osd.1) 347 : cluster [INF] 10.0 continuing backfill to osd.7 from (341'5252,356'9998] MIN to 356'9998 2024-09-13T22:56:02.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:02.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:02.568 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:02.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:02.690 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:02.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:02.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:02.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:02.956 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:02.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:03.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:03.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:03.107 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:161449 10.1 10:83484fa6:::benchmark_data_smithi178_111521_object50713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:03.088942+0000", 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018975906000000001, 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019002593000000002, 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:56:03.108 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161449 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.088942+0000", 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.088942+0000", 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.088944+0000", 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9019999999999999e-06 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.088951+0000", 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2049999999999999e-06 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.088964+0000", 2024-09-13T22:56:03.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.382e-05 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.088973+0000", 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8249999999999994e-06 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.089044+0000", 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0696000000000002e-05 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:03.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.089205+0000", 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016091200000000001 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.089360+0000", 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015484299999999999 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.094374+0000", 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050139119999999997 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.094438+0000", 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4448000000000004e-05 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.103986+0000", 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0095481179999999995 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.113 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:161462 10.c 10:39cb2c53:::benchmark_data_smithi178_111521_object50726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:03.101538+0000", 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0063792329999999998, 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0064961330000000003, 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161462 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:03.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101538+0000", 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101538+0000", 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101541+0000", 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8849999999999999e-06 2024-09-13T22:56:03.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101547+0000", 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101562+0000", 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5600999999999999e-05 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101570+0000", 2024-09-13T22:56:03.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2949999999999996e-06 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101644+0000", 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4396999999999999e-05 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101791+0000", 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014668999999999999 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.101935+0000", 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014441399999999999 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106885+0000", 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049493250000000001 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106906+0000", 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1852e-05 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.118 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25538.0:161450 10.8 e360/322 10:171ddb4b:::benchmark_data_smithi178_111521_object50714:head v 360'9989, mlcod=360'9987)", 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:03.106214+0000", 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0017034179999999999, 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019034849999999999, 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106214+0000", 2024-09-13T22:56:03.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106214+0000", 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106216+0000", 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.542e-06 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:03.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106222+0000", 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106235+0000", 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2638e-05 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106242+0000", 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3339999999999996e-06 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106392+0000", 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014950099999999999 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106410+0000", 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7972999999999999e-05 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25538.0:161463 10.9 e360/322 10:9d2dbef9:::benchmark_data_smithi178_111521_object50727:head v 360'10196, mlcod=360'10194)", 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:03.106286+0000", 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.001631052, 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018837280000000001, 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106286+0000", 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106286+0000", 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106287+0000", 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8999999999999995e-07 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106292+0000", 2024-09-13T22:56:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6650000000000002e-06 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106299+0000", 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4560000000000003e-06 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106305+0000", 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4199999999999998e-06 2024-09-13T22:56:03.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106351+0000", 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5763999999999998e-05 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:03.106371+0000", 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0242e-05 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:03.126 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-09-13T22:56:03.127 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 4 2024-09-13T22:56:03.127 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:56:03.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:03.244 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:03.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:03.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:03 smithi178 bash[29889]: cluster 2024-09-13T22:56:02.204902+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e360: 8 total, 8 up, 6 in 2024-09-13T22:56:03.356 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:03 smithi178 bash[22840]: cluster 2024-09-13T22:56:02.204902+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e360: 8 total, 8 up, 6 in 2024-09-13T22:56:03.382 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:03.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:03 smithi192 bash[24608]: cluster 2024-09-13T22:56:02.204902+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e360: 8 total, 8 up, 6 in 2024-09-13T22:56:03.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:03.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:03.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:03.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:03.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:03.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:03.769 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:03.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:03.893 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:03.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:03.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:04.017 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:04.130 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:04.155 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:04.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:04.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:04 smithi192 bash[24608]: cluster 2024-09-13T22:56:02.857282+0000 mgr.y (mgr.14613) 684 : cluster [DBG] pgmap v793: 158 pgs: 3 remapped+peering, 1 peering, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 147 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 6.7 KiB/s rd, 55 KiB/s wr, 280 op/s; 17826/184215 objects misplaced (9.677%); 599 KiB/s, 11 objects/s recovering 2024-09-13T22:56:04.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:04 smithi192 bash[24608]: cluster 2024-09-13T22:56:03.204967+0000 mon.a (mon.0) 1553 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T22:56:04.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:04 smithi192 bash[24608]: cluster 2024-09-13T22:56:03.220832+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e361: 8 total, 8 up, 6 in 2024-09-13T22:56:04.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:04 smithi192 bash[24608]: audit 2024-09-13T22:56:04.181699+0000 mon.a (mon.0) 1555 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:04.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:04 smithi192 bash[24608]: audit 2024-09-13T22:56:04.184370+0000 mon.b (mon.1) 248 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.287 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:04.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:04.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:04.412 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[29889]: cluster 2024-09-13T22:56:02.857282+0000 mgr.y (mgr.14613) 684 : cluster [DBG] pgmap v793: 158 pgs: 3 remapped+peering, 1 peering, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 147 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 6.7 KiB/s rd, 55 KiB/s wr, 280 op/s; 17826/184215 objects misplaced (9.677%); 599 KiB/s, 11 objects/s recovering 2024-09-13T22:56:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[29889]: cluster 2024-09-13T22:56:03.204967+0000 mon.a (mon.0) 1553 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T22:56:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[29889]: cluster 2024-09-13T22:56:03.220832+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e361: 8 total, 8 up, 6 in 2024-09-13T22:56:04.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[29889]: audit 2024-09-13T22:56:04.181699+0000 mon.a (mon.0) 1555 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:04.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[29889]: audit 2024-09-13T22:56:04.184370+0000 mon.b (mon.1) 248 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[22840]: cluster 2024-09-13T22:56:02.857282+0000 mgr.y (mgr.14613) 684 : cluster [DBG] pgmap v793: 158 pgs: 3 remapped+peering, 1 peering, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 147 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 6.7 KiB/s rd, 55 KiB/s wr, 280 op/s; 17826/184215 objects misplaced (9.677%); 599 KiB/s, 11 objects/s recovering 2024-09-13T22:56:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[22840]: cluster 2024-09-13T22:56:03.204967+0000 mon.a (mon.0) 1553 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T22:56:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[22840]: cluster 2024-09-13T22:56:03.220832+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e361: 8 total, 8 up, 6 in 2024-09-13T22:56:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[22840]: audit 2024-09-13T22:56:04.181699+0000 mon.a (mon.0) 1555 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:04.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:04 smithi178 bash[22840]: audit 2024-09-13T22:56:04.184370+0000 mon.b (mon.1) 248 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:04.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T22:56:04.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:04.542 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:04.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:04.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:04.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.806 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:04.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:04.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:04.936 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:04.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:05.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:05.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:05.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:05.212 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:05.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:05.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:05 smithi178 bash[29889]: cluster 2024-09-13T22:56:05.224157+0000 mon.a (mon.0) 1556 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:05.328 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:05 smithi178 bash[22840]: cluster 2024-09-13T22:56:05.224157+0000 mon.a (mon.0) 1556 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:05.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:05.485 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:05.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:05.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:56:05.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:05.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:56:05.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:05 smithi192 bash[24608]: cluster 2024-09-13T22:56:05.224157+0000 mon.a (mon.0) 1556 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:05.721 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:05.779 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:05.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:05.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:05.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:06.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.053 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:06.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:06.154 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:06.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:56:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:56:06.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:06.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:06.315 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:06.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:06.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:06.549 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:06.560 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:06 smithi178 bash[22840]: cluster 2024-09-13T22:56:04.858328+0000 mgr.y (mgr.14613) 685 : cluster [DBG] pgmap v795: 158 pgs: 3 remapped+peering, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 146 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 12 KiB/s rd, 94 KiB/s wr, 410 op/s; 34070/180546 objects misplaced (18.871%); 30 MiB/s, 501 objects/s recovering 2024-09-13T22:56:06.560 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:06 smithi178 bash[22840]: audit 2024-09-13T22:56:05.759073+0000 mgr.y (mgr.14613) 686 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:06.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:06 smithi178 bash[29889]: cluster 2024-09-13T22:56:04.858328+0000 mgr.y (mgr.14613) 685 : cluster [DBG] pgmap v795: 158 pgs: 3 remapped+peering, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 146 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 12 KiB/s rd, 94 KiB/s wr, 410 op/s; 34070/180546 objects misplaced (18.871%); 30 MiB/s, 501 objects/s recovering 2024-09-13T22:56:06.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:06 smithi178 bash[29889]: audit 2024-09-13T22:56:05.759073+0000 mgr.y (mgr.14613) 686 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:06 smithi192 bash[24608]: cluster 2024-09-13T22:56:04.858328+0000 mgr.y (mgr.14613) 685 : cluster [DBG] pgmap v795: 158 pgs: 3 remapped+peering, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 146 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 12 KiB/s rd, 94 KiB/s wr, 410 op/s; 34070/180546 objects misplaced (18.871%); 30 MiB/s, 501 objects/s recovering 2024-09-13T22:56:06.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:06 smithi192 bash[24608]: audit 2024-09-13T22:56:05.759073+0000 mgr.y (mgr.14613) 686 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:06.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:06.728 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:06.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:06.867 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:06.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:06.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:06.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:07.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:07.145 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:07.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:07.272 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:07.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:07.401 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:07.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:07 smithi178 bash[29889]: cluster 2024-09-13T22:56:07.277480+0000 mon.a (mon.0) 1557 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T22:56:07.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:07 smithi178 bash[22840]: cluster 2024-09-13T22:56:07.277480+0000 mon.a (mon.0) 1557 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T22:56:07.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:07.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:07 smithi192 bash[24608]: cluster 2024-09-13T22:56:07.277480+0000 mon.a (mon.0) 1557 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T22:56:07.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:07.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:07.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:07.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:07.797 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:07.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:07.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:08.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:08.058 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:08.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:08.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:08.315 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:08.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:08.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:08 smithi192 bash[24608]: cluster 2024-09-13T22:56:06.859004+0000 mgr.y (mgr.14613) 687 : cluster [DBG] pgmap v796: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 9.7 KiB/s rd, 73 KiB/s wr, 484 op/s; 41914/177579 objects misplaced (23.603%); 31 MiB/s, 505 objects/s recovering 2024-09-13T22:56:08.451 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:08.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:08.582 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:08.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:08.634 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7] out_osds: [4, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:56:08.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T22:56:08.644 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:08 smithi178 bash[29889]: cluster 2024-09-13T22:56:06.859004+0000 mgr.y (mgr.14613) 687 : cluster [DBG] pgmap v796: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 9.7 KiB/s rd, 73 KiB/s wr, 484 op/s; 41914/177579 objects misplaced (23.603%); 31 MiB/s, 505 objects/s recovering 2024-09-13T22:56:08.645 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:08 smithi178 bash[22840]: cluster 2024-09-13T22:56:06.859004+0000 mgr.y (mgr.14613) 687 : cluster [DBG] pgmap v796: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 9.7 KiB/s rd, 73 KiB/s wr, 484 op/s; 41914/177579 objects misplaced (23.603%); 31 MiB/s, 505 objects/s recovering 2024-09-13T22:56:08.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:08.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:08.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:08.856 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:08.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:08.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.001 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:09.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:09.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:09.130 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:09.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:09.262 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:09.335 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:56:09.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:09.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:09.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:09.523 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:09.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:09.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:09.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:09.783 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:09.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:09.924 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:09.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:10.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:10.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:10.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:10.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:56:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:56:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:56:10.186 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:10.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:10.320 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:10.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:10.461 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:10.483 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:10.483 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:10.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:10.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:10 smithi192 bash[24608]: cluster 2024-09-13T22:56:08.859714+0000 mgr.y (mgr.14613) 688 : cluster [DBG] pgmap v797: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 4.7 KiB/s rd, 29 KiB/s wr, 593 op/s; 39605/170691 objects misplaced (23.203%); 31 MiB/s, 506 objects/s recovering 2024-09-13T22:56:10.585 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:10.630 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:10 smithi178 bash[29889]: cluster 2024-09-13T22:56:08.859714+0000 mgr.y (mgr.14613) 688 : cluster [DBG] pgmap v797: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 4.7 KiB/s rd, 29 KiB/s wr, 593 op/s; 39605/170691 objects misplaced (23.203%); 31 MiB/s, 506 objects/s recovering 2024-09-13T22:56:10.630 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:10 smithi178 bash[22840]: cluster 2024-09-13T22:56:08.859714+0000 mgr.y (mgr.14613) 688 : cluster [DBG] pgmap v797: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 4.7 KiB/s rd, 29 KiB/s wr, 593 op/s; 39605/170691 objects misplaced (23.203%); 31 MiB/s, 506 objects/s recovering 2024-09-13T22:56:10.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:10.718 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:10.821 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:10.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:10.972 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:10.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:11.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:11.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:11.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:11.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:11.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:11.348 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:11.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:11.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:11.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:11.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:11.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:11.725 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:11.828 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:11.863 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:11.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:11.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:12.007 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:12.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:12.134 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:12.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:12.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:12.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:12.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:12 smithi192 bash[24608]: cluster 2024-09-13T22:56:10.860329+0000 mgr.y (mgr.14613) 689 : cluster [DBG] pgmap v798: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 4.2 KiB/s rd, 25 KiB/s wr, 634 op/s; 38110/167862 objects misplaced (22.703%); 32 MiB/s, 517 objects/s recovering 2024-09-13T22:56:12.400 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:12.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:12.513 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:12.532 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:12.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:12.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:12.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:12.780 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:12 smithi178 bash[29889]: cluster 2024-09-13T22:56:10.860329+0000 mgr.y (mgr.14613) 689 : cluster [DBG] pgmap v798: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 4.2 KiB/s rd, 25 KiB/s wr, 634 op/s; 38110/167862 objects misplaced (22.703%); 32 MiB/s, 517 objects/s recovering 2024-09-13T22:56:12.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:12.781 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:12 smithi178 bash[22840]: cluster 2024-09-13T22:56:10.860329+0000 mgr.y (mgr.14613) 689 : cluster [DBG] pgmap v798: 158 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 4.2 KiB/s rd, 25 KiB/s wr, 634 op/s; 38110/167862 objects misplaced (22.703%); 32 MiB/s, 517 objects/s recovering 2024-09-13T22:56:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:12.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:12.910 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:12.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:13.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:13.038 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:13.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:13.163 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:13.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:13.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:13.437 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:13.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:13.563 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:13.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:13.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:13 smithi192 bash[24608]: cluster 2024-09-13T22:56:12.560135+0000 osd.5 (osd.5) 393 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:13.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:13 smithi192 bash[24608]: cluster 2024-09-13T22:56:12.560188+0000 osd.5 (osd.5) 394 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:13.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:13 smithi192 bash[24608]: cluster 2024-09-13T22:56:13.082110+0000 mon.a (mon.0) 1558 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:13 smithi178 bash[29889]: cluster 2024-09-13T22:56:12.560135+0000 osd.5 (osd.5) 393 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:13 smithi178 bash[29889]: cluster 2024-09-13T22:56:12.560188+0000 osd.5 (osd.5) 394 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:13 smithi178 bash[29889]: cluster 2024-09-13T22:56:13.082110+0000 mon.a (mon.0) 1558 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:13 smithi178 bash[22840]: cluster 2024-09-13T22:56:12.560135+0000 osd.5 (osd.5) 393 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:13 smithi178 bash[22840]: cluster 2024-09-13T22:56:12.560188+0000 osd.5 (osd.5) 394 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:13 smithi178 bash[22840]: cluster 2024-09-13T22:56:13.082110+0000 mon.a (mon.0) 1558 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:13.694 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:13.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:13.825 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:13.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:13.972 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:13.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:14.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:14.100 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:14.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:14.244 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:14.295 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:56:14.295 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T22:56:14.295 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T22:56:14.295 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T22:56:14.295 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:56:14.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:14.373 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:14.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:14.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:14.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:14.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:14 smithi192 bash[24608]: cluster 2024-09-13T22:56:12.860974+0000 mgr.y (mgr.14613) 690 : cluster [DBG] pgmap v799: 158 pgs: 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 147 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 706 op/s; 36539/163152 objects misplaced (22.396%); 31 MiB/s, 501 objects/s recovering 2024-09-13T22:56:14.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:14 smithi192 bash[24608]: cluster 2024-09-13T22:56:13.356637+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e362: 8 total, 8 up, 6 in 2024-09-13T22:56:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:14.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:14.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:14 smithi178 bash[22840]: cluster 2024-09-13T22:56:12.860974+0000 mgr.y (mgr.14613) 690 : cluster [DBG] pgmap v799: 158 pgs: 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 147 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 706 op/s; 36539/163152 objects misplaced (22.396%); 31 MiB/s, 501 objects/s recovering 2024-09-13T22:56:14.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:14 smithi178 bash[22840]: cluster 2024-09-13T22:56:13.356637+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e362: 8 total, 8 up, 6 in 2024-09-13T22:56:14.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:14 smithi178 bash[29889]: cluster 2024-09-13T22:56:12.860974+0000 mgr.y (mgr.14613) 690 : cluster [DBG] pgmap v799: 158 pgs: 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 147 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 706 op/s; 36539/163152 objects misplaced (22.396%); 31 MiB/s, 501 objects/s recovering 2024-09-13T22:56:14.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:14 smithi178 bash[29889]: cluster 2024-09-13T22:56:13.356637+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e362: 8 total, 8 up, 6 in 2024-09-13T22:56:14.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:14.763 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:14.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:14.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:14.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:15.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:15.027 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:15.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:15.156 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:15.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:15.292 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:15.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:15.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:15.413 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:15.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:15.541 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:15.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:15.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:15.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:15.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:15.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:15.920 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:15.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:16.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:16.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:56:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:56:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:16.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:16.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:16.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:16.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:16.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:16.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T22:56:16.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:16.449 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:16.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:16.584 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:16.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:16 smithi192 bash[24608]: cluster 2024-09-13T22:56:14.861936+0000 mgr.y (mgr.14613) 691 : cluster [DBG] pgmap v801: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 842 op/s; 32328/155295 objects misplaced (20.817%); 33 MiB/s, 526 objects/s recovering 2024-09-13T22:56:16.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:16 smithi178 bash[29889]: cluster 2024-09-13T22:56:14.861936+0000 mgr.y (mgr.14613) 691 : cluster [DBG] pgmap v801: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 842 op/s; 32328/155295 objects misplaced (20.817%); 33 MiB/s, 526 objects/s recovering 2024-09-13T22:56:16.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:16 smithi178 bash[22840]: cluster 2024-09-13T22:56:14.861936+0000 mgr.y (mgr.14613) 691 : cluster [DBG] pgmap v801: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 842 op/s; 32328/155295 objects misplaced (20.817%); 33 MiB/s, 526 objects/s recovering 2024-09-13T22:56:16.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:16.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:16.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:16.839 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:16.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:16.943 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:16.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:16.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:17.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:17.119 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:17.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:17.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:17.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:17.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:17.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:17.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:17 smithi192 bash[24608]: audit 2024-09-13T22:56:15.769260+0000 mgr.y (mgr.14613) 692 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:17.510 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:17.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:17.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:17 smithi178 bash[29889]: audit 2024-09-13T22:56:15.769260+0000 mgr.y (mgr.14613) 692 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:17 smithi178 bash[22840]: audit 2024-09-13T22:56:15.769260+0000 mgr.y (mgr.14613) 692 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:17.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:17.789 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:17.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:17.916 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:17.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:18.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:18.042 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:18.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:18.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:18.168 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:18.274 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:18.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:18 smithi192 bash[24608]: cluster 2024-09-13T22:56:16.862543+0000 mgr.y (mgr.14613) 693 : cluster [DBG] pgmap v802: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 29957/152079 objects misplaced (19.698%); 40 MiB/s, 635 objects/s recovering 2024-09-13T22:56:18.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:18 smithi192 bash[24608]: cluster 2024-09-13T22:56:18.084070+0000 mon.a (mon.0) 1560 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:18.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:18.432 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:18.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:18.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:18 smithi178 bash[29889]: cluster 2024-09-13T22:56:16.862543+0000 mgr.y (mgr.14613) 693 : cluster [DBG] pgmap v802: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 29957/152079 objects misplaced (19.698%); 40 MiB/s, 635 objects/s recovering 2024-09-13T22:56:18.545 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:18 smithi178 bash[29889]: cluster 2024-09-13T22:56:18.084070+0000 mon.a (mon.0) 1560 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:18.545 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:18 smithi178 bash[22840]: cluster 2024-09-13T22:56:16.862543+0000 mgr.y (mgr.14613) 693 : cluster [DBG] pgmap v802: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 29957/152079 objects misplaced (19.698%); 40 MiB/s, 635 objects/s recovering 2024-09-13T22:56:18.545 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:18 smithi178 bash[22840]: cluster 2024-09-13T22:56:18.084070+0000 mon.a (mon.0) 1560 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:18.562 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:18.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:18.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:18.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:18.849 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:18.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:18.979 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:18.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:19.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:19.107 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:19.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:19.260 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:19.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:19.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:19.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:19.531 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:19.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:19.664 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:19.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:19.801 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:19.905 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:19.941 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:19.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:20.049 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:20.061 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:56:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:56:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:56:20.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:20.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:20.213 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:20.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:20.333 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[29889]: cluster 2024-09-13T22:56:18.863372+0000 mgr.y (mgr.14613) 694 : cluster [DBG] pgmap v803: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 831 op/s; 28015/145779 objects misplaced (19.217%); 39 MiB/s, 624 objects/s recovering 2024-09-13T22:56:20.333 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[29889]: cluster 2024-09-13T22:56:19.183510+0000 osd.2 (osd.2) 257 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[29889]: cluster 2024-09-13T22:56:19.183568+0000 osd.2 (osd.2) 258 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[29889]: audit 2024-09-13T22:56:19.208420+0000 mon.a (mon.0) 1561 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[29889]: audit 2024-09-13T22:56:19.211454+0000 mon.b (mon.1) 249 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[29889]: cluster 2024-09-13T22:56:19.384120+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e363: 8 total, 8 up, 6 in 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[22840]: cluster 2024-09-13T22:56:18.863372+0000 mgr.y (mgr.14613) 694 : cluster [DBG] pgmap v803: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 831 op/s; 28015/145779 objects misplaced (19.217%); 39 MiB/s, 624 objects/s recovering 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[22840]: cluster 2024-09-13T22:56:19.183510+0000 osd.2 (osd.2) 257 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[22840]: cluster 2024-09-13T22:56:19.183568+0000 osd.2 (osd.2) 258 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[22840]: audit 2024-09-13T22:56:19.208420+0000 mon.a (mon.0) 1561 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:20.334 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[22840]: audit 2024-09-13T22:56:19.211454+0000 mon.b (mon.1) 249 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:20.335 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:20 smithi178 bash[22840]: cluster 2024-09-13T22:56:19.384120+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e363: 8 total, 8 up, 6 in 2024-09-13T22:56:20.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:20.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:20.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:20.624 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:20 smithi192 bash[24608]: cluster 2024-09-13T22:56:18.863372+0000 mgr.y (mgr.14613) 694 : cluster [DBG] pgmap v803: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 3.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 831 op/s; 28015/145779 objects misplaced (19.217%); 39 MiB/s, 624 objects/s recovering 2024-09-13T22:56:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:20 smithi192 bash[24608]: cluster 2024-09-13T22:56:19.183510+0000 osd.2 (osd.2) 257 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:20 smithi192 bash[24608]: cluster 2024-09-13T22:56:19.183568+0000 osd.2 (osd.2) 258 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:20 smithi192 bash[24608]: audit 2024-09-13T22:56:19.208420+0000 mon.a (mon.0) 1561 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:20 smithi192 bash[24608]: audit 2024-09-13T22:56:19.211454+0000 mon.b (mon.1) 249 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:20.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:20 smithi192 bash[24608]: cluster 2024-09-13T22:56:19.384120+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e363: 8 total, 8 up, 6 in 2024-09-13T22:56:20.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:20.757 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:20.862 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:20.887 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:20.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:20.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.014 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:21.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.141 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:21.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.252 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T22:56:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.276 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:21.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.403 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:21.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:21.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:21.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:21.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:21.656 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:21.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:21.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:21.905 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:21.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:21.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:22.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:22.032 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:22.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:22.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:22.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:22.284 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:22.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:22.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:22 smithi192 bash[24608]: cluster 2024-09-13T22:56:20.863940+0000 mgr.y (mgr.14613) 695 : cluster [DBG] pgmap v805: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 826 op/s; 26297/143328 objects misplaced (18.347%); 49 MiB/s, 779 objects/s recovering 2024-09-13T22:56:22.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:22 smithi192 bash[24608]: cluster 2024-09-13T22:56:21.221332+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e364: 8 total, 8 up, 6 in 2024-09-13T22:56:22.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:22.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:22 smithi178 bash[29889]: cluster 2024-09-13T22:56:20.863940+0000 mgr.y (mgr.14613) 695 : cluster [DBG] pgmap v805: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 826 op/s; 26297/143328 objects misplaced (18.347%); 49 MiB/s, 779 objects/s recovering 2024-09-13T22:56:22.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:22 smithi178 bash[29889]: cluster 2024-09-13T22:56:21.221332+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e364: 8 total, 8 up, 6 in 2024-09-13T22:56:22.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:22 smithi178 bash[22840]: cluster 2024-09-13T22:56:20.863940+0000 mgr.y (mgr.14613) 695 : cluster [DBG] pgmap v805: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 826 op/s; 26297/143328 objects misplaced (18.347%); 49 MiB/s, 779 objects/s recovering 2024-09-13T22:56:22.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:22 smithi178 bash[22840]: cluster 2024-09-13T22:56:21.221332+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e364: 8 total, 8 up, 6 in 2024-09-13T22:56:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:22.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:22.557 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:22.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:22.681 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:22.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:22.806 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:22.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:22.931 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:22.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:23.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:23.058 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:23.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:23.183 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:23.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:23.309 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:23.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:23 smithi192 bash[24608]: cluster 2024-09-13T22:56:22.229634+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e365: 8 total, 8 up, 6 in 2024-09-13T22:56:23.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:23.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:23 smithi178 bash[29889]: cluster 2024-09-13T22:56:22.229634+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e365: 8 total, 8 up, 6 in 2024-09-13T22:56:23.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:23 smithi178 bash[22840]: cluster 2024-09-13T22:56:22.229634+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e365: 8 total, 8 up, 6 in 2024-09-13T22:56:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:23.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:23.572 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:23.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:23.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:23.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:23.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:23.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:23.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:23.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:24.081 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:24.105 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:24.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:24.235 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:24.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:24.360 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:24.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:24.485 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:24.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:24.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:24 smithi178 bash[29889]: cluster 2024-09-13T22:56:22.864406+0000 mgr.y (mgr.14613) 696 : cluster [DBG] pgmap v808: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 733 op/s; 24833/138885 objects misplaced (17.880%); 28 MiB/s, 443 objects/s recovering 2024-09-13T22:56:24.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:24 smithi178 bash[29889]: cluster 2024-09-13T22:56:23.241352+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e366: 8 total, 8 up, 6 in 2024-09-13T22:56:24.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:24 smithi178 bash[22840]: cluster 2024-09-13T22:56:22.864406+0000 mgr.y (mgr.14613) 696 : cluster [DBG] pgmap v808: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 733 op/s; 24833/138885 objects misplaced (17.880%); 28 MiB/s, 443 objects/s recovering 2024-09-13T22:56:24.597 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:24 smithi178 bash[22840]: cluster 2024-09-13T22:56:23.241352+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e366: 8 total, 8 up, 6 in 2024-09-13T22:56:24.609 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:24 smithi192 bash[24608]: cluster 2024-09-13T22:56:22.864406+0000 mgr.y (mgr.14613) 696 : cluster [DBG] pgmap v808: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 2.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 733 op/s; 24833/138885 objects misplaced (17.880%); 28 MiB/s, 443 objects/s recovering 2024-09-13T22:56:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:24 smithi192 bash[24608]: cluster 2024-09-13T22:56:23.241352+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e366: 8 total, 8 up, 6 in 2024-09-13T22:56:24.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:24.733 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:24.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:24.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:24.986 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:24.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:25.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.113 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:25.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:25.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:25.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:25.364 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:25.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.493 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:25.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.604 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:25 smithi192 bash[24608]: cluster 2024-09-13T22:56:24.252774+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e367: 8 total, 8 up, 6 in 2024-09-13T22:56:25.617 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:25 smithi178 bash[29889]: cluster 2024-09-13T22:56:24.252774+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e367: 8 total, 8 up, 6 in 2024-09-13T22:56:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:25 smithi178 bash[22840]: cluster 2024-09-13T22:56:24.252774+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e367: 8 total, 8 up, 6 in 2024-09-13T22:56:25.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:25.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:25.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:25.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.979 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:56:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:56:25.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:25.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:26.012 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T22:56:26.012 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-09-13T22:56:26.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-09-13T22:56:26.012 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T22:56:26.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14454.0:2091 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 93919878334720 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:47:02.870214+0000", 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 563.14133825199997, 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0801809200000001, 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14454", 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/1055532036", 2024-09-13T22:56:26.013 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2091 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870214+0000", 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870214+0000", 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870217+0000", 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3530000000000002e-06 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870223+0000", 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.737e-06 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870240+0000", 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6146000000000001e-05 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870249+0000", 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2310000000000002e-06 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870323+0000", 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3876999999999998e-05 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.870328+0000", 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2009999999999998e-06 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.889003+0000", 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018674671 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:02.889013+0000", 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9199999999999999e-06 2024-09-13T22:56:26.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.944098+0000", 2024-09-13T22:56:26.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.055085737 2024-09-13T22:56:26.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-09-13T22:56:26.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.945224+0000", 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001126223 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.948523+0000", 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032982889999999998 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.948613+0000", 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0489999999999996e-05 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:47:03.950395+0000", 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017820449999999999 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:50594 9.e 9:72312fee:::benchmark_data_smithi178_96665_object50593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T22:56:26.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:02.997409+0000", 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.01414374400002, 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2948786999999999, 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50594 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997409+0000", 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997409+0000", 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997410+0000", 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4499999999999997e-07 2024-09-13T22:56:26.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997437+0000", 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7441e-05 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997442+0000", 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1669999999999998e-06 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997446+0000", 2024-09-13T22:56:26.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.692e-06 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997501+0000", 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5148999999999997e-05 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997512+0000", 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0994e-05 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948178+0000", 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95066591700000003 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948196+0000", 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8479e-05 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.285075+0000", 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.336878732 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.285200+0000", 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000124738 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.285308+0000", 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010803599999999999 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.290670+0000", 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0053625319999999997 2024-09-13T22:56:26.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.290720+0000", 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0068999999999998e-05 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.290731+0000", 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0053e-05 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.290767+0000", 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6313000000000003e-05 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.292288+0000", 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001520743 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.030 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:50596 9.e 9:72918480:::benchmark_data_smithi178_96665_object50595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:02.997567+0000", 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.01398596400003, 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.300856155, 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50596 2024-09-13T22:56:26.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997567+0000", 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997567+0000", 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997568+0000", 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1900000000000001e-07 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997595+0000", 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7223e-05 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997601+0000", 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3999999999999997e-06 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997614+0000", 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2863e-05 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997653+0000", 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8621999999999997e-05 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:02.997660+0000", 2024-09-13T22:56:26.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1289999999999998e-06 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948235+0000", 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95057557000000004 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948248+0000", 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2478e-05 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.288918+0000", 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3406700499999999 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.289028+0000", 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000110339 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:56:26.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.289145+0000", 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116644 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.292411+0000", 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003266017 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.292421+0000", 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0288e-05 2024-09-13T22:56:26.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.296102+0000", 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036807939999999998 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.296168+0000", 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6130999999999994e-05 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.298423+0000", 2024-09-13T22:56:26.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022546879999999999 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:50621 9.e 9:70fac94a:::benchmark_data_smithi178_96665_object50620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:03.016632+0000", 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 382.99492049700001, 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2836801740000001, 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.039 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50621 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016632+0000", 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016632+0000", 2024-09-13T22:56:26.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016634+0000", 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7910000000000001e-06 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016699+0000", 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5246000000000005e-05 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016708+0000", 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7679999999999997e-06 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016717+0000", 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1719999999999996e-06 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016776+0000", 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-05 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.016783+0000", 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8369999999999999e-06 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948282+0000", 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93149892700000003 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948294+0000", 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2398000000000001e-05 2024-09-13T22:56:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.292448+0000", 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.344153674 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.292569+0000", 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121043 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:56:26.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.292692+0000", 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012267499999999999 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.296052+0000", 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033597969999999999 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.298512+0000", 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024601559999999998 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.298607+0000", 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5295000000000002e-05 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.298673+0000", 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5623000000000005e-05 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.300312+0000", 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001639562 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:50643 9.e 9:7c639100:::benchmark_data_smithi178_96665_object50642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:03.042632+0000", 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 382.968920164, 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2596080079999998, 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50643 2024-09-13T22:56:26.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042632+0000", 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042632+0000", 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042634+0000", 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.124e-06 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042661+0000", 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7197000000000001e-05 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042670+0000", 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8939999999999999e-06 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042675+0000", 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8949999999999997e-06 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042727+0000", 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2887999999999999e-05 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.042735+0000", 2024-09-13T22:56:26.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8890000000000002e-06 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948329+0000", 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90559367400000002 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:03.948339+0000", 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0324e-05 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.294174+0000", 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.345834591 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.294292+0000", 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118047 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.294401+0000", 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010907 2024-09-13T22:56:26.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.298454+0000", 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040527389999999996 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.298558+0000", 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010388 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.300378+0000", 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018200849999999999 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.300456+0000", 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8121000000000006e-05 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:05.302240+0000", 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00178459 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.055 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:81930 9.3 9:c4f80ed4:::benchmark_data_smithi178_96665_object81929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:39.245097+0000", 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 346.76645529400002, 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.021278889, 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.056 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 81930 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245097+0000", 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245097+0000", 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245099+0000", 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.984e-06 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245162+0000", 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2284000000000002e-05 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245173+0000", 2024-09-13T22:56:26.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1844e-05 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245181+0000", 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3019999999999997e-06 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375686+0000", 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13050566 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375697+0000", 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.046e-05 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.266294+0000", 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89059677699999995 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.266376+0000", 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2577999999999994e-05 2024-09-13T22:56:26.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:81936 9.3 9:c55ee236:::benchmark_data_smithi178_96665_object81935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:39.245193+0000", 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 346.766359626, 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.037228751, 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 81936 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245193+0000", 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245193+0000", 2024-09-13T22:56:26.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245194+0000", 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245294+0000", 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9735000000000006e-05 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245300+0000", 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4590000000000003e-06 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245306+0000", 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4299999999999997e-06 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375721+0000", 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.130415165 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375728+0000", 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0049999999999998e-06 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.266400+0000", 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89067224499999997 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.266530+0000", 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000129651 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.266805+0000", 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027555 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.272281+0000", 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054755530000000002 2024-09-13T22:56:26.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.273861+0000", 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001580372 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.282309+0000", 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008448367 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.282382+0000", 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2939000000000001e-05 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.282422+0000", 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9376999999999999e-05 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:81937 9.3 9:c26de276:::benchmark_data_smithi178_96665_object81936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:39.245313+0000", 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 346.76623980400001, 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.043580299, 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.069 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 81937 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245313+0000", 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245313+0000", 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245314+0000", 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2900000000000002e-07 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245410+0000", 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5748000000000006e-05 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245416+0000", 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6680000000000004e-06 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245422+0000", 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5679999999999999e-06 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375750+0000", 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.130328099 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375756+0000", 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9179999999999999e-06 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.267846+0000", 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89209067799999997 2024-09-13T22:56:26.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.267984+0000", 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137229 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.268182+0000", 2024-09-13T22:56:26.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019838000000000001 2024-09-13T22:56:26.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.274574+0000", 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063917150000000001 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.276924+0000", 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023499829999999999 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.288786+0000", 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011862607000000001 2024-09-13T22:56:26.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.288852+0000", 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5351000000000005e-05 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.288893+0000", 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1425999999999999e-05 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.077 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:81940 9.3 9:c3db6408:::benchmark_data_smithi178_96665_object81939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:39.245432+0000", 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 346.76612042599999, 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0451760649999999, 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 81940 2024-09-13T22:56:26.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245432+0000", 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245432+0000", 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245433+0000", 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8400000000000004e-07 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245482+0000", 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9144000000000002e-05 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245488+0000", 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7450000000000001e-06 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.245493+0000", 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7820000000000001e-06 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375780+0000", 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.130287509 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:39.375787+0000", 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5239999999999997e-06 2024-09-13T22:56:26.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.269983+0000", 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89419669000000002 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.270174+0000", 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000190939 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.270575+0000", 2024-09-13T22:56:26.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040100200000000002 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.274609+0000", 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040335520000000001 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.276965+0000", 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023558780000000001 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.290499+0000", 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01353473 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.290561+0000", 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1950000000000001e-05 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:40.290608+0000", 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6935999999999999e-05 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82301 9.15 9:ae51c13b:::benchmark_data_smithi178_96665_object82300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.293237+0000", 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.71831584400002, 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.06041958, 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82301 2024-09-13T22:56:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293237+0000", 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293237+0000", 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293239+0000", 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1569999999999998e-06 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293298+0000", 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9249999999999997e-05 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293312+0000", 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4058e-05 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293320+0000", 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0050000000000004e-06 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293379+0000", 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8937000000000002e-05 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.293390+0000", 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0716e-05 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.248418+0000", 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95502841199999999 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.248430+0000", 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1797000000000001e-05 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.340463+0000", 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0920331560000001 2024-09-13T22:56:26.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.340622+0000", 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000158543 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.340956+0000", 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000334042 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.342435+0000", 2024-09-13T22:56:26.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014787520000000001 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.344698+0000", 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022636650000000002 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.348466+0000", 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037675009999999999 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.352590+0000", 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041239750000000002 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.353466+0000", 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00087592699999999998 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.353531+0000", 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4949000000000004e-05 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.353656+0000", 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012573799999999999 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82305 9.e 9:7ca2e929:::benchmark_data_smithi178_96665_object82304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.295321+0000", 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.71623165199998, 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97409713600000003, 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82305 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295321+0000", 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295321+0000", 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295323+0000", 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.663e-06 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295475+0000", 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015279900000000001 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295488+0000", 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2918000000000001e-05 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295496+0000", 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9019999999999999e-06 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295568+0000", 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2230999999999993e-05 2024-09-13T22:56:26.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295579+0000", 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0531e-05 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258515+0000", 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96293554000000003 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258665+0000", 2024-09-13T22:56:26.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015012099999999999 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258790+0000", 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012527 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.262458+0000", 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036681790000000001 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.264986+0000", 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025282600000000001 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.269300+0000", 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004314081 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.269383+0000", 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2563000000000002e-05 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.269418+0000", 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5077999999999999e-05 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82306 9.17 9:eb100991:::benchmark_data_smithi178_96665_object82305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.295889+0000", 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.71566413599999, 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0547956319999998, 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82306 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295889+0000", 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295889+0000", 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295890+0000", 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.499e-06 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295946+0000", 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6103000000000001e-05 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295957+0000", 2024-09-13T22:56:26.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0501e-05 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.295967+0000", 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0654e-05 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.296043+0000", 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5727000000000002e-05 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.296055+0000", 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1788e-05 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.243954+0000", 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94789873000000002 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.243966+0000", 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2349e-05 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.331434+0000", 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0874679389999999 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.331529+0000", 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5124000000000006e-05 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.331807+0000", 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027783099999999998 2024-09-13T22:56:26.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.333466+0000", 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016591430000000001 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.345596+0000", 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012130444000000001 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.350347+0000", 2024-09-13T22:56:26.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047503019999999996 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.350392+0000", 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5082e-05 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.350431+0000", 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9209999999999999e-05 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.350542+0000", 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111326 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.350684+0000", 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014187999999999999 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82311 9.13 9:cdf9f136:::benchmark_data_smithi178_96665_object82310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.298082+0000", 2024-09-13T22:56:26.108 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.71347046400001, 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0251555890000001, 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82311 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298082+0000", 2024-09-13T22:56:26.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298082+0000", 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298084+0000", 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.522e-06 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298128+0000", 2024-09-13T22:56:26.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4286000000000003e-05 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298138+0000", 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0220999999999999e-05 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298144+0000", 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4060000000000004e-06 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298205+0000", 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1542000000000001e-05 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.298222+0000", 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6852000000000001e-05 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.246218+0000", 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94799642100000003 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.246231+0000", 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2271e-05 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.307631+0000", 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0614002119999999 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.307793+0000", 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000161898 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3,5,6", 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.308071+0000", 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027793300000000001 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.313072+0000", 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005001457 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.313088+0000", 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5619000000000001e-05 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.316592+0000", 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003504119 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.317818+0000", 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00122587 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.323023+0000", 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0052047960000000002 2024-09-13T22:56:26.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.323098+0000", 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5575999999999996e-05 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.323238+0000", 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013958800000000001 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82313 9.11 9:8883a0cf:::benchmark_data_smithi178_96665_object82312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.304718+0000", 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.70683456, 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95352132199999995, 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82313 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304718+0000", 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304718+0000", 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304725+0000", 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3830000000000003e-06 2024-09-13T22:56:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304782+0000", 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6833999999999998e-05 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304794+0000", 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1990999999999999e-05 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304800+0000", 2024-09-13T22:56:26.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0000000000000002e-06 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304857+0000", 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6969999999999998e-05 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.304865+0000", 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9950000000000005e-06 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.252946+0000", 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94808077499999999 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.253093+0000", 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014648100000000001 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.253385+0000", 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000292785 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.256315+0000", 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002929526 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.256781+0000", 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000465928 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258143+0000", 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013623680000000001 2024-09-13T22:56:26.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258197+0000", 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4311999999999998e-05 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258239+0000", 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1974000000000002e-05 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82319 9.e 9:7c7ac9c3:::benchmark_data_smithi178_96665_object82318:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.312279+0000", 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.69927377400001, 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95729555499999996, 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82319 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312279+0000", 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312279+0000", 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312281+0000", 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7519999999999999e-06 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312365+0000", 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4130999999999998e-05 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312375+0000", 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0416e-05 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312380+0000", 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2240000000000003e-06 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312427+0000", 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6758000000000002e-05 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.312435+0000", 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6190000000000002e-06 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.260505+0000", 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94807057800000005 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.260622+0000", 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116272 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.260781+0000", 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015969199999999999 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.265017+0000", 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004235702 2024-09-13T22:56:26.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.265090+0000", 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2910000000000005e-05 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.269500+0000", 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004410069 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.269538+0000", 2024-09-13T22:56:26.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8442000000000002e-05 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.269574+0000", 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5989999999999999e-05 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82329 9.13 9:ce5f1a8a:::benchmark_data_smithi178_96665_object82328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:41.380225+0000", 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 344.63132792900001, 2024-09-13T22:56:26.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9521563959999999, 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82329 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380225+0000", 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380225+0000", 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380226+0000", 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3200000000000003e-07 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380334+0000", 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108344 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380341+0000", 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4739999999999999e-06 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380346+0000", 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4660000000000001e-06 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380412+0000", 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6421999999999995e-05 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:41.380420+0000", 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2239999999999998e-06 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.246257+0000", 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86583728900000001 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.246265+0000", 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8189999999999992e-06 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.309065+0000", 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.062799904 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.309210+0000", 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000145795 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3,5,6", 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.309449+0000", 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000238139 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.313138+0000", 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036891910000000001 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.313160+0000", 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1946999999999998e-05 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.318237+0000", 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005077341 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.323292+0000", 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050553339999999999 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.332180+0000", 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0088872550000000002 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.332251+0000", 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1305999999999998e-05 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.332381+0000", 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000130214 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.138 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82332 9.15 9:a849583e:::benchmark_data_smithi178_96665_object82331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:42.258624+0000", 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 343.75292848100003, 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1106333799999999, 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.139 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82332 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258624+0000", 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258624+0000", 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258626+0000", 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.81e-06 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258781+0000", 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015478200000000001 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258794+0000", 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3713000000000001e-05 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258803+0000", 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1790000000000008e-06 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258860+0000", 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7377999999999999e-05 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.258867+0000", 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6599999999999998e-06 2024-09-13T22:56:26.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.342471+0000", 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083604706 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.342602+0000", 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131011 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.342853+0000", 2024-09-13T22:56:26.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025060399999999997 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.347744+0000", 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048906510000000002 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.348510+0000", 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00076682599999999997 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.353021+0000", 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045104510000000004 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.369037+0000", 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016015773000000001 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.369077+0000", 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0679999999999997e-05 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.369144+0000", 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6153000000000002e-05 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.369258+0000", 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114003 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.146 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15408.0:82340 9.13 9:cb6a64cc:::benchmark_data_smithi178_96665_object82339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:50:42.287358+0000", 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 343.72419470199998, 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.045597739, 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15408", 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 82340 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287358+0000", 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287358+0000", 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287360+0000", 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3120000000000001e-06 2024-09-13T22:56:26.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287409+0000", 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9038e-05 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287423+0000", 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4089e-05 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287433+0000", 2024-09-13T22:56:26.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5040000000000008e-06 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287507+0000", 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3771999999999998e-05 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:42.287514+0000", 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6090000000000003e-06 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.310489+0000", 2024-09-13T22:56:26.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0229747929999999 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.310645+0000", 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015608899999999999 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3,5,6", 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.310909+0000", 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026381000000000001 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T22:56:26.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.313165+0000", 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022564460000000001 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.315133+0000", 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001968029 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.318275+0000", 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031420269999999999 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.323666+0000", 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0053904510000000001 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.332843+0000", 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0091774579999999994 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.332887+0000", 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3284000000000001e-05 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:50:43.332956+0000", 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9028000000000004e-05 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14454.0:3620 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93919878342784 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:59.186338+0000", 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 26.825214836000001, 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.052682637, 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:56:26.154 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14454", 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/1055532036", 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3620 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186338+0000", 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186338+0000", 2024-09-13T22:56:26.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186340+0000", 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4080000000000001e-06 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186345+0000", 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2800000000000003e-06 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186356+0000", 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0859999999999999e-05 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186365+0000", 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9029999999999993e-06 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186463+0000", 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7717e-05 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186468+0000", 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3070000000000002e-06 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.196658+0000", 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010189693 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.196665+0000", 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8789999999999997e-06 2024-09-13T22:56:26.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.229544+0000", 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.032879519 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.229872+0000", 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000327559 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.233454+0000", 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035819319999999999 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.233556+0000", 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102394 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.239020+0000", 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054641860000000002 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14454.0:3621 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 93919878334720 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:55:59.186390+0000", 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 26.825163125, 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.052909479, 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14454", 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/1055532036", 2024-09-13T22:56:26.161 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3621 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186390+0000", 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186390+0000", 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186390+0000", 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6700000000000003e-07 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186393+0000", 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3549999999999999e-06 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186398+0000", 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4190000000000001e-06 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186404+0000", 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9460000000000003e-06 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186461+0000", 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6789000000000001e-05 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.186467+0000", 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9780000000000002e-06 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.193228+0000", 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0067609039999999999 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:55:59.193237+0000", 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7799999999999995e-06 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.231857+0000", 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.03862001 2024-09-13T22:56:26.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.232176+0000", 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031897000000000002 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.236020+0000", 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038436719999999998 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.236101+0000", 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0587999999999995e-05 2024-09-13T22:56:26.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:00.239299+0000", 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031983010000000002 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T22:56:26.167 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T22:56:26.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:26.190 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:26.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:26.320 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:26.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:26 smithi192 bash[24608]: cluster 2024-09-13T22:56:24.864846+0000 mgr.y (mgr.14613) 697 : cluster [DBG] pgmap v811: 158 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 149 active+clean; 2.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 923 op/s; 21086/132264 objects misplaced (15.942%); 39 MiB/s, 621 objects/s recovering 2024-09-13T22:56:26.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:26 smithi192 bash[24608]: cluster 2024-09-13T22:56:25.261040+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e368: 8 total, 8 up, 6 in 2024-09-13T22:56:26.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:26.451 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:26.536 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:26 smithi178 bash[29889]: cluster 2024-09-13T22:56:24.864846+0000 mgr.y (mgr.14613) 697 : cluster [DBG] pgmap v811: 158 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 149 active+clean; 2.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 923 op/s; 21086/132264 objects misplaced (15.942%); 39 MiB/s, 621 objects/s recovering 2024-09-13T22:56:26.536 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:26 smithi178 bash[29889]: cluster 2024-09-13T22:56:25.261040+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e368: 8 total, 8 up, 6 in 2024-09-13T22:56:26.536 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:26 smithi178 bash[22840]: cluster 2024-09-13T22:56:24.864846+0000 mgr.y (mgr.14613) 697 : cluster [DBG] pgmap v811: 158 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 149 active+clean; 2.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 923 op/s; 21086/132264 objects misplaced (15.942%); 39 MiB/s, 621 objects/s recovering 2024-09-13T22:56:26.536 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:26 smithi178 bash[22840]: cluster 2024-09-13T22:56:25.261040+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e368: 8 total, 8 up, 6 in 2024-09-13T22:56:26.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:26.578 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:26.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:26.699 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:26.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:26.829 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:26.876 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:56:26.876 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:56:26.894 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:56:26.894 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:56:26.927 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:56:26.928 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:56:26.932 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:26.944 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:56:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:56:26.952 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:26.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:26.966 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:56:26.966 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:56:26.987 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:56:26.987 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:56:27.006 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:56:27.006 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:56:27.026 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:56:27.027 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:56:27.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:27.083 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:27.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T22:56:27.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:27.211 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:27.276 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T22:56:27.283 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:56:27.298 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[22840]: audit 2024-09-13T22:56:25.778468+0000 mgr.y (mgr.14613) 698 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:27.298 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[22840]: cluster 2024-09-13T22:56:26.266488+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e369: 8 total, 8 up, 6 in 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[22840]: audit 2024-09-13T22:56:26.856313+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.178:0/1190851629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[22840]: audit 2024-09-13T22:56:27.061000+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? v1:172.21.15.178:0/4117599387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[22840]: audit 2024-09-13T22:56:27.067194+0000 mon.c (mon.2) 70 : audit [INF] from='client.? v1:172.21.15.178:0/1567166346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[22840]: audit 2024-09-13T22:56:27.067460+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25688 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[29889]: audit 2024-09-13T22:56:25.778468+0000 mgr.y (mgr.14613) 698 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[29889]: cluster 2024-09-13T22:56:26.266488+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e369: 8 total, 8 up, 6 in 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[29889]: audit 2024-09-13T22:56:26.856313+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.178:0/1190851629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[29889]: audit 2024-09-13T22:56:27.061000+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? v1:172.21.15.178:0/4117599387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:56:27.299 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[29889]: audit 2024-09-13T22:56:27.067194+0000 mon.c (mon.2) 70 : audit [INF] from='client.? v1:172.21.15.178:0/1567166346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:56:27.300 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:27 smithi178 bash[29889]: audit 2024-09-13T22:56:27.067460+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25688 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:56:27.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:27.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:27 smithi192 bash[24608]: audit 2024-09-13T22:56:25.778468+0000 mgr.y (mgr.14613) 698 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:27.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:27 smithi192 bash[24608]: cluster 2024-09-13T22:56:26.266488+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e369: 8 total, 8 up, 6 in 2024-09-13T22:56:27.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:27 smithi192 bash[24608]: audit 2024-09-13T22:56:26.856313+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.178:0/1190851629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:56:27.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:27 smithi192 bash[24608]: audit 2024-09-13T22:56:27.061000+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? v1:172.21.15.178:0/4117599387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:56:27.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:27 smithi192 bash[24608]: audit 2024-09-13T22:56:27.067194+0000 mon.c (mon.2) 70 : audit [INF] from='client.? v1:172.21.15.178:0/1567166346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:56:27.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:27 smithi192 bash[24608]: audit 2024-09-13T22:56:27.067460+0000 mon.a (mon.0) 1570 : audit [INF] from='client.25688 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:56:27.344 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:27.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:27.474 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:27.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:27.601 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:27.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:27.734 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:27.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:27.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:27.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:27.988 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:27.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:28.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:28.126 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:28.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:28.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:28.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:28.388 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:28.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: cluster 2024-09-13T22:56:26.865214+0000 mgr.y (mgr.14613) 699 : cluster [DBG] pgmap v814: 158 pgs: 1 peering, 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 714 op/s; 18459/130326 objects misplaced (14.164%); 64 MiB/s, 1.03k objects/s recovering 2024-09-13T22:56:28.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: cluster 2024-09-13T22:56:27.270495+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:56:28.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: audit 2024-09-13T22:56:27.277143+0000 mon.a (mon.0) 1572 : audit [INF] from='client.? v1:172.21.15.178:0/4117599387' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: audit 2024-09-13T22:56:27.277278+0000 mon.a (mon.0) 1573 : audit [INF] from='client.25688 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: cluster 2024-09-13T22:56:27.281975+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e370: 8 total, 8 up, 6 in 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: cluster 2024-09-13T22:56:27.398753+0000 osd.2 (osd.2) 259 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: cluster 2024-09-13T22:56:27.405950+0000 osd.2 (osd.2) 260 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[22840]: cluster 2024-09-13T22:56:27.813820+0000 osd.0 (osd.0) 361 : cluster [DBG] 10.8 scrub starts 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: cluster 2024-09-13T22:56:26.865214+0000 mgr.y (mgr.14613) 699 : cluster [DBG] pgmap v814: 158 pgs: 1 peering, 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 714 op/s; 18459/130326 objects misplaced (14.164%); 64 MiB/s, 1.03k objects/s recovering 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: cluster 2024-09-13T22:56:27.270495+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: audit 2024-09-13T22:56:27.277143+0000 mon.a (mon.0) 1572 : audit [INF] from='client.? v1:172.21.15.178:0/4117599387' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: audit 2024-09-13T22:56:27.277278+0000 mon.a (mon.0) 1573 : audit [INF] from='client.25688 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: cluster 2024-09-13T22:56:27.281975+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e370: 8 total, 8 up, 6 in 2024-09-13T22:56:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: cluster 2024-09-13T22:56:27.398753+0000 osd.2 (osd.2) 259 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:56:28.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: cluster 2024-09-13T22:56:27.405950+0000 osd.2 (osd.2) 260 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:56:28.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:28 smithi178 bash[29889]: cluster 2024-09-13T22:56:27.813820+0000 osd.0 (osd.0) 361 : cluster [DBG] 10.8 scrub starts 2024-09-13T22:56:28.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:28.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:28.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: cluster 2024-09-13T22:56:26.865214+0000 mgr.y (mgr.14613) 699 : cluster [DBG] pgmap v814: 158 pgs: 1 peering, 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 714 op/s; 18459/130326 objects misplaced (14.164%); 64 MiB/s, 1.03k objects/s recovering 2024-09-13T22:56:28.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: cluster 2024-09-13T22:56:27.270495+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:56:28.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: audit 2024-09-13T22:56:27.277143+0000 mon.a (mon.0) 1572 : audit [INF] from='client.? v1:172.21.15.178:0/4117599387' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:56:28.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: audit 2024-09-13T22:56:27.277278+0000 mon.a (mon.0) 1573 : audit [INF] from='client.25688 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:56:28.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: cluster 2024-09-13T22:56:27.281975+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e370: 8 total, 8 up, 6 in 2024-09-13T22:56:28.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: cluster 2024-09-13T22:56:27.398753+0000 osd.2 (osd.2) 259 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:56:28.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: cluster 2024-09-13T22:56:27.405950+0000 osd.2 (osd.2) 260 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:56:28.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:28 smithi192 bash[24608]: cluster 2024-09-13T22:56:27.813820+0000 osd.0 (osd.0) 361 : cluster [DBG] 10.8 scrub starts 2024-09-13T22:56:28.645 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:28.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:28.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:28.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:28.899 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:28.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:29.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:29.025 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:29.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:29.152 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:29.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:29.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:29.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:29.403 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:29 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.036367+0000 osd.3 (osd.3) 345 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:56:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:29 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.051736+0000 osd.3 (osd.3) 346 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:56:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:29 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.433658+0000 osd.2 (osd.2) 261 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:56:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:29 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.444323+0000 osd.2 (osd.2) 262 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:56:29.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:29 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.768600+0000 osd.0 (osd.0) 362 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:56:29.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:29 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.769560+0000 osd.0 (osd.0) 363 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:56:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:29.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:29.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:29.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:29.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:29.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.036367+0000 osd.3 (osd.3) 345 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.051736+0000 osd.3 (osd.3) 346 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.433658+0000 osd.2 (osd.2) 261 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.444323+0000 osd.2 (osd.2) 262 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.768600+0000 osd.0 (osd.0) 362 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.769560+0000 osd.0 (osd.0) 363 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.036367+0000 osd.3 (osd.3) 345 : cluster [DBG] 2.1e scrub starts 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.051736+0000 osd.3 (osd.3) 346 : cluster [DBG] 2.1e scrub ok 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.433658+0000 osd.2 (osd.2) 261 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.444323+0000 osd.2 (osd.2) 262 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.768600+0000 osd.0 (osd.0) 362 : cluster [DBG] 1.6 scrub starts 2024-09-13T22:56:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.769560+0000 osd.0 (osd.0) 363 : cluster [DBG] 1.6 scrub ok 2024-09-13T22:56:29.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:29.784 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:29.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:29.914 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:29.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:30.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:30.031 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:56:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:56:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:56:30.050 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:30.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:30.176 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:30.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:30.300 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:30.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:30.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:30.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:30.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:30.632 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:30 smithi178 bash[29889]: cluster 2024-09-13T22:56:28.865893+0000 mgr.y (mgr.14613) 700 : cluster [DBG] pgmap v816: 158 pgs: 1 peering, 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 460 op/s; 17764/125907 objects misplaced (14.109%); 32 MiB/s, 518 objects/s recovering 2024-09-13T22:56:30.632 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:30 smithi178 bash[29889]: cluster 2024-09-13T22:56:29.145388+0000 osd.0 (osd.0) 364 : cluster [DBG] 10.8 scrub ok 2024-09-13T22:56:30.632 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:30 smithi178 bash[29889]: cluster 2024-09-13T22:56:30.085269+0000 osd.7 (osd.7) 372 : cluster [DBG] 10.c scrub starts 2024-09-13T22:56:30.633 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:30 smithi178 bash[22840]: cluster 2024-09-13T22:56:28.865893+0000 mgr.y (mgr.14613) 700 : cluster [DBG] pgmap v816: 158 pgs: 1 peering, 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 460 op/s; 17764/125907 objects misplaced (14.109%); 32 MiB/s, 518 objects/s recovering 2024-09-13T22:56:30.633 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:30 smithi178 bash[22840]: cluster 2024-09-13T22:56:29.145388+0000 osd.0 (osd.0) 364 : cluster [DBG] 10.8 scrub ok 2024-09-13T22:56:30.633 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:30 smithi178 bash[22840]: cluster 2024-09-13T22:56:30.085269+0000 osd.7 (osd.7) 372 : cluster [DBG] 10.c scrub starts 2024-09-13T22:56:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:30 smithi192 bash[24608]: cluster 2024-09-13T22:56:28.865893+0000 mgr.y (mgr.14613) 700 : cluster [DBG] pgmap v816: 158 pgs: 1 peering, 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 460 op/s; 17764/125907 objects misplaced (14.109%); 32 MiB/s, 518 objects/s recovering 2024-09-13T22:56:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:30 smithi192 bash[24608]: cluster 2024-09-13T22:56:29.145388+0000 osd.0 (osd.0) 364 : cluster [DBG] 10.8 scrub ok 2024-09-13T22:56:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:30 smithi192 bash[24608]: cluster 2024-09-13T22:56:30.085269+0000 osd.7 (osd.7) 372 : cluster [DBG] 10.c scrub starts 2024-09-13T22:56:30.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.689 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:30.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:30.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:30.814 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:30.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:30.941 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:30.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:31.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:31.078 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:31.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:31.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:31.326 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:31.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:31.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:31 smithi178 bash[22840]: cluster 2024-09-13T22:56:30.154086+0000 osd.3 (osd.3) 347 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:56:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:31 smithi178 bash[22840]: cluster 2024-09-13T22:56:30.160964+0000 osd.3 (osd.3) 348 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:56:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:31 smithi178 bash[22840]: cluster 2024-09-13T22:56:30.832004+0000 osd.7 (osd.7) 373 : cluster [DBG] 10.c scrub ok 2024-09-13T22:56:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:31 smithi178 bash[29889]: cluster 2024-09-13T22:56:30.154086+0000 osd.3 (osd.3) 347 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:56:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:31 smithi178 bash[29889]: cluster 2024-09-13T22:56:30.160964+0000 osd.3 (osd.3) 348 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:56:31.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:31 smithi178 bash[29889]: cluster 2024-09-13T22:56:30.832004+0000 osd.7 (osd.7) 373 : cluster [DBG] 10.c scrub ok 2024-09-13T22:56:31.496 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:31.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:31.608 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:31 smithi192 bash[24608]: cluster 2024-09-13T22:56:30.154086+0000 osd.3 (osd.3) 347 : cluster [DBG] 1.7 scrub starts 2024-09-13T22:56:31.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:31 smithi192 bash[24608]: cluster 2024-09-13T22:56:30.160964+0000 osd.3 (osd.3) 348 : cluster [DBG] 1.7 scrub ok 2024-09-13T22:56:31.609 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:31 smithi192 bash[24608]: cluster 2024-09-13T22:56:30.832004+0000 osd.7 (osd.7) 373 : cluster [DBG] 10.c scrub ok 2024-09-13T22:56:31.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:31.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:31.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:31.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:31.882 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:31.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:31.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:32.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:32.113 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:32.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T22:56:32.152 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:32.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:32.282 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:32.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:32.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:32 smithi178 bash[29889]: cluster 2024-09-13T22:56:30.867025+0000 mgr.y (mgr.14613) 701 : cluster [DBG] pgmap v817: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 151 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 407 op/s; 17168/124944 objects misplaced (13.741%); 29 MiB/s, 471 objects/s recovering 2024-09-13T22:56:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:32 smithi178 bash[29889]: cluster 2024-09-13T22:56:31.793052+0000 osd.4 (osd.4) 272 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:32 smithi178 bash[29889]: cluster 2024-09-13T22:56:31.793113+0000 osd.4 (osd.4) 273 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:32 smithi178 bash[22840]: cluster 2024-09-13T22:56:30.867025+0000 mgr.y (mgr.14613) 701 : cluster [DBG] pgmap v817: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 151 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 407 op/s; 17168/124944 objects misplaced (13.741%); 29 MiB/s, 471 objects/s recovering 2024-09-13T22:56:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:32 smithi178 bash[22840]: cluster 2024-09-13T22:56:31.793052+0000 osd.4 (osd.4) 272 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:32 smithi178 bash[22840]: cluster 2024-09-13T22:56:31.793113+0000 osd.4 (osd.4) 273 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:32.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:32.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:32.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:32 smithi192 bash[24608]: cluster 2024-09-13T22:56:30.867025+0000 mgr.y (mgr.14613) 701 : cluster [DBG] pgmap v817: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 151 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 407 op/s; 17168/124944 objects misplaced (13.741%); 29 MiB/s, 471 objects/s recovering 2024-09-13T22:56:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:32 smithi192 bash[24608]: cluster 2024-09-13T22:56:31.793052+0000 osd.4 (osd.4) 272 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:32 smithi192 bash[24608]: cluster 2024-09-13T22:56:31.793113+0000 osd.4 (osd.4) 273 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:32.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:32.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:32.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:32.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:32.921 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:32.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:33.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:33.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:33.154 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:33.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:33.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:33.319 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:33.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:33.444 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:33.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:33.571 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:33.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:33 smithi192 bash[24608]: cluster 2024-09-13T22:56:31.528564+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:56:33.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:33 smithi192 bash[24608]: cluster 2024-09-13T22:56:31.538236+0000 osd.2 (osd.2) 264 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:56:33.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:33 smithi192 bash[24608]: cluster 2024-09-13T22:56:32.299145+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e371: 8 total, 8 up, 6 in 2024-09-13T22:56:33.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:33 smithi192 bash[24608]: cluster 2024-09-13T22:56:33.101915+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e372: 8 total, 8 up, 6 in 2024-09-13T22:56:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:33.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[29889]: cluster 2024-09-13T22:56:31.528564+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:56:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[29889]: cluster 2024-09-13T22:56:31.538236+0000 osd.2 (osd.2) 264 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:56:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[29889]: cluster 2024-09-13T22:56:32.299145+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e371: 8 total, 8 up, 6 in 2024-09-13T22:56:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[29889]: cluster 2024-09-13T22:56:33.101915+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e372: 8 total, 8 up, 6 in 2024-09-13T22:56:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[22840]: cluster 2024-09-13T22:56:31.528564+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:56:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[22840]: cluster 2024-09-13T22:56:31.538236+0000 osd.2 (osd.2) 264 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:56:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[22840]: cluster 2024-09-13T22:56:32.299145+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e371: 8 total, 8 up, 6 in 2024-09-13T22:56:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:33 smithi178 bash[22840]: cluster 2024-09-13T22:56:33.101915+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e372: 8 total, 8 up, 6 in 2024-09-13T22:56:33.695 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:33.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:33.821 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:33.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:33.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:33.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:34.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:56:34.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:34.075 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:34.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:56:34.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:34.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:34.333 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:34.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[29889]: cluster 2024-09-13T22:56:32.869488+0000 mgr.y (mgr.14613) 702 : cluster [DBG] pgmap v819: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 2.5 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 440 op/s; 16833/121539 objects misplaced (13.850%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:56:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[29889]: cluster 2024-09-13T22:56:33.126138+0000 osd.5 (osd.5) 395 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:56:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[29889]: cluster 2024-09-13T22:56:33.129821+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:56:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[29889]: cluster 2024-09-13T22:56:34.096669+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e373: 8 total, 8 up, 6 in 2024-09-13T22:56:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[29889]: audit 2024-09-13T22:56:34.227681+0000 mon.b (mon.1) 250 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[22840]: cluster 2024-09-13T22:56:32.869488+0000 mgr.y (mgr.14613) 702 : cluster [DBG] pgmap v819: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 2.5 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 440 op/s; 16833/121539 objects misplaced (13.850%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:56:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[22840]: cluster 2024-09-13T22:56:33.126138+0000 osd.5 (osd.5) 395 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:56:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[22840]: cluster 2024-09-13T22:56:33.129821+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:56:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[22840]: cluster 2024-09-13T22:56:34.096669+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e373: 8 total, 8 up, 6 in 2024-09-13T22:56:34.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:34 smithi178 bash[22840]: audit 2024-09-13T22:56:34.227681+0000 mon.b (mon.1) 250 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:34.437 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:34.494 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:34.606 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:34.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:34 smithi192 bash[24608]: cluster 2024-09-13T22:56:32.869488+0000 mgr.y (mgr.14613) 702 : cluster [DBG] pgmap v819: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 2.5 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 440 op/s; 16833/121539 objects misplaced (13.850%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T22:56:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:34 smithi192 bash[24608]: cluster 2024-09-13T22:56:33.126138+0000 osd.5 (osd.5) 395 : cluster [DBG] 6.7 scrub starts 2024-09-13T22:56:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:34 smithi192 bash[24608]: cluster 2024-09-13T22:56:33.129821+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.7 scrub ok 2024-09-13T22:56:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:34 smithi192 bash[24608]: cluster 2024-09-13T22:56:34.096669+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e373: 8 total, 8 up, 6 in 2024-09-13T22:56:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:34 smithi192 bash[24608]: audit 2024-09-13T22:56:34.227681+0000 mon.b (mon.1) 250 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:34.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:34.767 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:34.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:34.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:34.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:34.999 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:35.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:35.023 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:35.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:35.155 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:35.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:35.282 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:35.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:35.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:35 smithi178 bash[29889]: cluster 2024-09-13T22:56:34.084475+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:56:35.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:35 smithi178 bash[29889]: cluster 2024-09-13T22:56:34.086417+0000 osd.5 (osd.5) 398 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:56:35.398 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:35 smithi178 bash[22840]: cluster 2024-09-13T22:56:34.084475+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:56:35.398 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:35 smithi178 bash[22840]: cluster 2024-09-13T22:56:34.086417+0000 osd.5 (osd.5) 398 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:56:35.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:35.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:35.545 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:35.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:35 smithi192 bash[24608]: cluster 2024-09-13T22:56:34.084475+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.18 scrub starts 2024-09-13T22:56:35.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:35 smithi192 bash[24608]: cluster 2024-09-13T22:56:34.086417+0000 osd.5 (osd.5) 398 : cluster [DBG] 2.18 scrub ok 2024-09-13T22:56:35.676 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:35.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:35.805 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:35.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:35.930 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:35.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:36.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:56:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:56:36.053 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:36.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:36.183 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:36.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:36.319 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:36.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:36.459 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:36.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.585 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:36.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:36 smithi192 bash[24608]: cluster 2024-09-13T22:56:34.870069+0000 mgr.y (mgr.14613) 703 : cluster [DBG] pgmap v822: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 2.3 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 700 op/s; 14328/113199 objects misplaced (12.657%); 15 MiB/s, 243 objects/s recovering 2024-09-13T22:56:36.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:36 smithi192 bash[24608]: cluster 2024-09-13T22:56:35.130508+0000 osd.5 (osd.5) 399 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:56:36.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:36 smithi192 bash[24608]: cluster 2024-09-13T22:56:35.136325+0000 osd.5 (osd.5) 400 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:56:36.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:36 smithi192 bash[24608]: cluster 2024-09-13T22:56:35.312659+0000 mon.a (mon.0) 1578 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[22840]: cluster 2024-09-13T22:56:34.870069+0000 mgr.y (mgr.14613) 703 : cluster [DBG] pgmap v822: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 2.3 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 700 op/s; 14328/113199 objects misplaced (12.657%); 15 MiB/s, 243 objects/s recovering 2024-09-13T22:56:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[22840]: cluster 2024-09-13T22:56:35.130508+0000 osd.5 (osd.5) 399 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:56:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[22840]: cluster 2024-09-13T22:56:35.136325+0000 osd.5 (osd.5) 400 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:56:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[22840]: cluster 2024-09-13T22:56:35.312659+0000 mon.a (mon.0) 1578 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:36.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[29889]: cluster 2024-09-13T22:56:34.870069+0000 mgr.y (mgr.14613) 703 : cluster [DBG] pgmap v822: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 2.3 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 700 op/s; 14328/113199 objects misplaced (12.657%); 15 MiB/s, 243 objects/s recovering 2024-09-13T22:56:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[29889]: cluster 2024-09-13T22:56:35.130508+0000 osd.5 (osd.5) 399 : cluster [DBG] 2.12 scrub starts 2024-09-13T22:56:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[29889]: cluster 2024-09-13T22:56:35.136325+0000 osd.5 (osd.5) 400 : cluster [DBG] 2.12 scrub ok 2024-09-13T22:56:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:36 smithi178 bash[29889]: cluster 2024-09-13T22:56:35.312659+0000 mon.a (mon.0) 1578 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:36.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:36.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:36.853 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:36.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.987 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:36.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:37.039 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:37.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:37.130 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:37.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:37.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:37.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:37.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:37.496 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[22840]: audit 2024-09-13T22:56:35.785311+0000 mgr.y (mgr.14613) 704 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:37.497 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[22840]: cluster 2024-09-13T22:56:36.089446+0000 osd.5 (osd.5) 401 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:56:37.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:37.525 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:37.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[22840]: cluster 2024-09-13T22:56:36.090902+0000 osd.5 (osd.5) 402 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[22840]: cluster 2024-09-13T22:56:36.678168+0000 osd.1 (osd.1) 348 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[22840]: cluster 2024-09-13T22:56:36.678214+0000 osd.1 (osd.1) 349 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[29889]: audit 2024-09-13T22:56:35.785311+0000 mgr.y (mgr.14613) 704 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[29889]: cluster 2024-09-13T22:56:36.089446+0000 osd.5 (osd.5) 401 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[29889]: cluster 2024-09-13T22:56:36.090902+0000 osd.5 (osd.5) 402 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[29889]: cluster 2024-09-13T22:56:36.678168+0000 osd.1 (osd.1) 348 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:37.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:37 smithi178 bash[29889]: cluster 2024-09-13T22:56:36.678214+0000 osd.1 (osd.1) 349 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:37.652 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:37 smithi192 bash[24608]: audit 2024-09-13T22:56:35.785311+0000 mgr.y (mgr.14613) 704 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:37 smithi192 bash[24608]: cluster 2024-09-13T22:56:36.089446+0000 osd.5 (osd.5) 401 : cluster [DBG] 5.15 scrub starts 2024-09-13T22:56:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:37 smithi192 bash[24608]: cluster 2024-09-13T22:56:36.090902+0000 osd.5 (osd.5) 402 : cluster [DBG] 5.15 scrub ok 2024-09-13T22:56:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:37 smithi192 bash[24608]: cluster 2024-09-13T22:56:36.678168+0000 osd.1 (osd.1) 348 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:37.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:37 smithi192 bash[24608]: cluster 2024-09-13T22:56:36.678214+0000 osd.1 (osd.1) 349 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:37.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:37.780 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:37.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:37.908 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:37.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:38.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:38.036 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:38.061 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:38.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:38.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:38.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:38.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:38.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:38.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[29889]: cluster 2024-09-13T22:56:36.870710+0000 mgr.y (mgr.14613) 705 : cluster [DBG] pgmap v823: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 2.2 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 862 op/s; 13973/109320 objects misplaced (12.782%); 11 MiB/s, 170 objects/s recovering 2024-09-13T22:56:38.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[29889]: cluster 2024-09-13T22:56:37.049095+0000 osd.5 (osd.5) 403 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[29889]: cluster 2024-09-13T22:56:37.051139+0000 osd.5 (osd.5) 404 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[29889]: cluster 2024-09-13T22:56:37.339855+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e374: 8 total, 8 up, 6 in 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[29889]: audit 2024-09-13T22:56:37.872667+0000 mon.b (mon.1) 251 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[22840]: cluster 2024-09-13T22:56:36.870710+0000 mgr.y (mgr.14613) 705 : cluster [DBG] pgmap v823: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 2.2 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 862 op/s; 13973/109320 objects misplaced (12.782%); 11 MiB/s, 170 objects/s recovering 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[22840]: cluster 2024-09-13T22:56:37.049095+0000 osd.5 (osd.5) 403 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[22840]: cluster 2024-09-13T22:56:37.051139+0000 osd.5 (osd.5) 404 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:56:38.428 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[22840]: cluster 2024-09-13T22:56:37.339855+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e374: 8 total, 8 up, 6 in 2024-09-13T22:56:38.429 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:38 smithi178 bash[22840]: audit 2024-09-13T22:56:37.872667+0000 mon.b (mon.1) 251 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:56:38.456 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:38.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:38.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:38 smithi192 bash[24608]: cluster 2024-09-13T22:56:36.870710+0000 mgr.y (mgr.14613) 705 : cluster [DBG] pgmap v823: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 2.2 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 862 op/s; 13973/109320 objects misplaced (12.782%); 11 MiB/s, 170 objects/s recovering 2024-09-13T22:56:38.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:38 smithi192 bash[24608]: cluster 2024-09-13T22:56:37.049095+0000 osd.5 (osd.5) 403 : cluster [DBG] 1.8 scrub starts 2024-09-13T22:56:38.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:38 smithi192 bash[24608]: cluster 2024-09-13T22:56:37.051139+0000 osd.5 (osd.5) 404 : cluster [DBG] 1.8 scrub ok 2024-09-13T22:56:38.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:38 smithi192 bash[24608]: cluster 2024-09-13T22:56:37.339855+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e374: 8 total, 8 up, 6 in 2024-09-13T22:56:38.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:38 smithi192 bash[24608]: audit 2024-09-13T22:56:37.872667+0000 mon.b (mon.1) 251 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:56:38.583 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:38.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:38.705 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:38.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:38.829 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:38.932 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:38.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:38.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:39.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:39.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:39.089 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:39.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:39.212 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:39.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:39.340 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:39.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:39.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:39.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[29889]: audit 2024-09-13T22:56:38.759441+0000 mon.b (mon.1) 252 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:56:39.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[29889]: audit 2024-09-13T22:56:38.760255+0000 mon.b (mon.1) 253 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:56:39.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[29889]: audit 2024-09-13T22:56:38.763825+0000 mon.a (mon.0) 1580 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:39.544 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[22840]: audit 2024-09-13T22:56:38.759441+0000 mon.b (mon.1) 252 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:56:39.544 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[22840]: audit 2024-09-13T22:56:38.760255+0000 mon.b (mon.1) 253 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:56:39.545 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[22840]: audit 2024-09-13T22:56:38.763825+0000 mon.a (mon.0) 1580 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:39.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:39.595 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:39 smithi192 bash[24608]: audit 2024-09-13T22:56:38.759441+0000 mon.b (mon.1) 252 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:56:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:39 smithi192 bash[24608]: audit 2024-09-13T22:56:38.760255+0000 mon.b (mon.1) 253 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:56:39.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:39 smithi192 bash[24608]: audit 2024-09-13T22:56:38.763825+0000 mon.a (mon.0) 1580 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:56:39.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:39.720 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:39.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:39.847 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:39.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:39.978 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:39.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:40.083 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:40.092 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:56:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:56:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:56:40.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:40.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:40.252 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:40.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:40.387 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:40.494 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:40.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:40.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:40.658 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:40.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:40.794 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:40.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:40.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:40 smithi192 bash[24608]: cluster 2024-09-13T22:56:38.871603+0000 mgr.y (mgr.14613) 706 : cluster [DBG] pgmap v825: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 11253/102213 objects misplaced (11.009%); 24 MiB/s, 386 objects/s recovering 2024-09-13T22:56:40.927 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:40.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:40.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:40 smithi178 bash[29889]: cluster 2024-09-13T22:56:38.871603+0000 mgr.y (mgr.14613) 706 : cluster [DBG] pgmap v825: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 11253/102213 objects misplaced (11.009%); 24 MiB/s, 386 objects/s recovering 2024-09-13T22:56:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:40 smithi178 bash[22840]: cluster 2024-09-13T22:56:38.871603+0000 mgr.y (mgr.14613) 706 : cluster [DBG] pgmap v825: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 11253/102213 objects misplaced (11.009%); 24 MiB/s, 386 objects/s recovering 2024-09-13T22:56:41.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:41.055 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:41.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:41.179 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:41.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:41.310 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:41.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:41.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:41.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:41.568 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:41.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:41.700 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:41.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:41.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:41.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:41.990 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:41.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:42.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:42.119 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:42.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:42.246 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:42.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:42.390 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:42.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:42.527 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:42.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:42.670 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:42.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:42.802 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:42.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:42.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[29889]: cluster 2024-09-13T22:56:40.872334+0000 mgr.y (mgr.14613) 707 : cluster [DBG] pgmap v826: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 2.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 10961/100521 objects misplaced (10.904%); 19 MiB/s, 298 objects/s recovering 2024-09-13T22:56:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[29889]: cluster 2024-09-13T22:56:41.963288+0000 osd.5 (osd.5) 405 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[29889]: cluster 2024-09-13T22:56:41.965743+0000 osd.5 (osd.5) 406 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[29889]: cluster 2024-09-13T22:56:42.137659+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[29889]: cluster 2024-09-13T22:56:42.137779+0000 osd.7 (osd.7) 375 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[22840]: cluster 2024-09-13T22:56:40.872334+0000 mgr.y (mgr.14613) 707 : cluster [DBG] pgmap v826: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 2.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 10961/100521 objects misplaced (10.904%); 19 MiB/s, 298 objects/s recovering 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[22840]: cluster 2024-09-13T22:56:41.963288+0000 osd.5 (osd.5) 405 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[22840]: cluster 2024-09-13T22:56:41.965743+0000 osd.5 (osd.5) 406 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[22840]: cluster 2024-09-13T22:56:42.137659+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:42 smithi178 bash[22840]: cluster 2024-09-13T22:56:42.137779+0000 osd.7 (osd.7) 375 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:42.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:42.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:43.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:43.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:42 smithi192 bash[24608]: cluster 2024-09-13T22:56:40.872334+0000 mgr.y (mgr.14613) 707 : cluster [DBG] pgmap v826: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 2.0 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 898 op/s; 10961/100521 objects misplaced (10.904%); 19 MiB/s, 298 objects/s recovering 2024-09-13T22:56:43.042 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:42 smithi192 bash[24608]: cluster 2024-09-13T22:56:41.963288+0000 osd.5 (osd.5) 405 : cluster [DBG] 3.1c scrub starts 2024-09-13T22:56:43.042 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:42 smithi192 bash[24608]: cluster 2024-09-13T22:56:41.965743+0000 osd.5 (osd.5) 406 : cluster [DBG] 3.1c scrub ok 2024-09-13T22:56:43.042 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:42 smithi192 bash[24608]: cluster 2024-09-13T22:56:42.137659+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:43.042 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:42 smithi192 bash[24608]: cluster 2024-09-13T22:56:42.137779+0000 osd.7 (osd.7) 375 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:43.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:43.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:43.180 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:43.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:43.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:43.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:43.549 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:43.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:43.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:43.710 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:43.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:43.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:43.908 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:43 smithi178 bash[29889]: cluster 2024-09-13T22:56:42.805954+0000 mon.a (mon.0) 1581 : cluster [DBG] osdmap e375: 8 total, 8 up, 6 in 2024-09-13T22:56:43.908 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:43 smithi178 bash[29889]: cluster 2024-09-13T22:56:42.872850+0000 mgr.y (mgr.14613) 708 : cluster [DBG] pgmap v828: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 700 op/s; 10574/96399 objects misplaced (10.969%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:56:43.909 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:43 smithi178 bash[22840]: cluster 2024-09-13T22:56:42.805954+0000 mon.a (mon.0) 1581 : cluster [DBG] osdmap e375: 8 total, 8 up, 6 in 2024-09-13T22:56:43.909 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:43 smithi178 bash[22840]: cluster 2024-09-13T22:56:42.872850+0000 mgr.y (mgr.14613) 708 : cluster [DBG] pgmap v828: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 700 op/s; 10574/96399 objects misplaced (10.969%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:56:43.936 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:43.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:43.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:44.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:44.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:43 smithi192 bash[24608]: cluster 2024-09-13T22:56:42.805954+0000 mon.a (mon.0) 1581 : cluster [DBG] osdmap e375: 8 total, 8 up, 6 in 2024-09-13T22:56:44.078 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:43 smithi192 bash[24608]: cluster 2024-09-13T22:56:42.872850+0000 mgr.y (mgr.14613) 708 : cluster [DBG] pgmap v828: 158 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 700 op/s; 10574/96399 objects misplaced (10.969%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:56:44.093 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:44.198 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:44.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:44.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:44.360 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:44.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:44.489 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:44.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:44.616 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:44.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:44.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:44.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:44.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:44.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:44.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:44.981 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:44 smithi178 bash[22840]: cluster 2024-09-13T22:56:44.927218+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:45.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:45.150 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:45.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:44 smithi192 bash[24608]: cluster 2024-09-13T22:56:44.927218+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:45.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:45.281 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:45.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:45.406 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:45.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:45.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:45.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:44 smithi178 bash[29889]: cluster 2024-09-13T22:56:44.927218+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:56:45.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:45.663 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:45.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:45.796 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:45.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:45.924 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:45.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:46.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:46.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:46.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:46.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:56:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:56:46.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:45 smithi192 bash[24608]: cluster 2024-09-13T22:56:44.875303+0000 mgr.y (mgr.14613) 709 : cluster [DBG] pgmap v829: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 1.8 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 843 op/s; 7539/89106 objects misplaced (8.461%); 24 MiB/s, 388 objects/s recovering 2024-09-13T22:56:46.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:45 smithi192 bash[24608]: cluster 2024-09-13T22:56:44.947887+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:56:46.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:45 smithi192 bash[24608]: cluster 2024-09-13T22:56:44.954769+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:56:46.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:46.206 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:45 smithi178 bash[29889]: cluster 2024-09-13T22:56:44.875303+0000 mgr.y (mgr.14613) 709 : cluster [DBG] pgmap v829: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 1.8 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 843 op/s; 7539/89106 objects misplaced (8.461%); 24 MiB/s, 388 objects/s recovering 2024-09-13T22:56:46.206 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:45 smithi178 bash[29889]: cluster 2024-09-13T22:56:44.947887+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:56:46.206 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:45 smithi178 bash[29889]: cluster 2024-09-13T22:56:44.954769+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:56:46.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:45 smithi178 bash[22840]: cluster 2024-09-13T22:56:44.875303+0000 mgr.y (mgr.14613) 709 : cluster [DBG] pgmap v829: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 1.8 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 843 op/s; 7539/89106 objects misplaced (8.461%); 24 MiB/s, 388 objects/s recovering 2024-09-13T22:56:46.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:45 smithi178 bash[22840]: cluster 2024-09-13T22:56:44.947887+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.3 scrub starts 2024-09-13T22:56:46.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:45 smithi178 bash[22840]: cluster 2024-09-13T22:56:44.954769+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.3 scrub ok 2024-09-13T22:56:46.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:46.306 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:46.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:46.434 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:46.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.582 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:46.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:46.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:46.707 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:46.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:46.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:46.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:46.975 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:46.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:47.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:47.087 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:46 smithi178 bash[22840]: audit 2024-09-13T22:56:45.794729+0000 mgr.y (mgr.14613) 710 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:47.087 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:46 smithi178 bash[29889]: audit 2024-09-13T22:56:45.794729+0000 mgr.y (mgr.14613) 710 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:47.106 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:47.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:47.208 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:56:47.208 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:56:47.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:47.216 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:46 smithi192 bash[24608]: audit 2024-09-13T22:56:45.794729+0000 mgr.y (mgr.14613) 710 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:47.228 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:47.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:47.352 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:47.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:47.477 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:47.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:47.602 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:47.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:47.723 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:47.825 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:47.848 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:47.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:47.976 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:47.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:48.005 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T22:56:48.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:48.119 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:48.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:56:48.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:56:48.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:56:48.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:56:48.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:56:48.192 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:56:48.193 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:56:48.194 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:56:48.206 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:56:48.207 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:56:48.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[29889]: cluster 2024-09-13T22:56:46.875944+0000 mgr.y (mgr.14613) 711 : cluster [DBG] pgmap v830: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 1.7 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 7242/85986 objects misplaced (8.422%); 20 MiB/s, 326 objects/s recovering 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[29889]: cluster 2024-09-13T22:56:47.006550+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e376: 8 total, 8 up, 6 in 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[29889]: audit 2024-09-13T22:56:47.516025+0000 mon.c (mon.2) 71 : audit [INF] from='client.? v1:172.21.15.178:0/2426168051' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[29889]: audit 2024-09-13T22:56:47.516415+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25694 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[29889]: audit 2024-09-13T22:56:47.879217+0000 mon.a (mon.0) 1585 : audit [DBG] from='client.? v1:172.21.15.178:0/4138344774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[22840]: cluster 2024-09-13T22:56:46.875944+0000 mgr.y (mgr.14613) 711 : cluster [DBG] pgmap v830: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 1.7 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 7242/85986 objects misplaced (8.422%); 20 MiB/s, 326 objects/s recovering 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[22840]: cluster 2024-09-13T22:56:47.006550+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e376: 8 total, 8 up, 6 in 2024-09-13T22:56:48.241 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[22840]: audit 2024-09-13T22:56:47.516025+0000 mon.c (mon.2) 71 : audit [INF] from='client.? v1:172.21.15.178:0/2426168051' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:56:48.242 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[22840]: audit 2024-09-13T22:56:47.516415+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25694 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:56:48.242 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:48 smithi178 bash[22840]: audit 2024-09-13T22:56:47.879217+0000 mon.a (mon.0) 1585 : audit [DBG] from='client.? v1:172.21.15.178:0/4138344774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:56:48.251 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:48.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:47 smithi192 bash[24608]: cluster 2024-09-13T22:56:46.875944+0000 mgr.y (mgr.14613) 711 : cluster [DBG] pgmap v830: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 1.7 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 816 op/s; 7242/85986 objects misplaced (8.422%); 20 MiB/s, 326 objects/s recovering 2024-09-13T22:56:48.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:47 smithi192 bash[24608]: cluster 2024-09-13T22:56:47.006550+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e376: 8 total, 8 up, 6 in 2024-09-13T22:56:48.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:47 smithi192 bash[24608]: audit 2024-09-13T22:56:47.516025+0000 mon.c (mon.2) 71 : audit [INF] from='client.? v1:172.21.15.178:0/2426168051' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:56:48.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:47 smithi192 bash[24608]: audit 2024-09-13T22:56:47.516415+0000 mon.a (mon.0) 1584 : audit [INF] from='client.25694 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T22:56:48.254 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:47 smithi192 bash[24608]: audit 2024-09-13T22:56:47.879217+0000 mon.a (mon.0) 1585 : audit [DBG] from='client.? v1:172.21.15.178:0/4138344774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:56:48.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:48.380 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:48.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:48.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:48.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:48.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:48.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:48.765 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:48.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:48.889 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:48.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:48.993 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:49.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:49.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:49.131 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:49 smithi192 bash[24608]: audit 2024-09-13T22:56:48.002736+0000 mon.a (mon.0) 1586 : audit [INF] from='client.25694 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:56:49.131 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:49 smithi192 bash[24608]: cluster 2024-09-13T22:56:48.005761+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e377: 8 total, 8 up, 6 in 2024-09-13T22:56:49.145 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:49 smithi178 bash[29889]: audit 2024-09-13T22:56:48.002736+0000 mon.a (mon.0) 1586 : audit [INF] from='client.25694 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:56:49.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:49 smithi178 bash[29889]: cluster 2024-09-13T22:56:48.005761+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e377: 8 total, 8 up, 6 in 2024-09-13T22:56:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:49 smithi178 bash[22840]: audit 2024-09-13T22:56:48.002736+0000 mon.a (mon.0) 1586 : audit [INF] from='client.25694 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T22:56:49.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:49 smithi178 bash[22840]: cluster 2024-09-13T22:56:48.005761+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e377: 8 total, 8 up, 6 in 2024-09-13T22:56:49.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:49.288 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:49.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:49.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:49.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:49.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:49.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:49.693 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:49.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:49.827 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:49.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:49.960 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:49.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:50.065 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:50.075 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:56:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:56:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:56:50.090 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:50.122 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:56:50.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:50.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:50.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[22840]: cluster 2024-09-13T22:56:48.630582+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[22840]: cluster 2024-09-13T22:56:48.631911+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[22840]: cluster 2024-09-13T22:56:48.876674+0000 mgr.y (mgr.14613) 712 : cluster [DBG] pgmap v833: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.6 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 865 op/s; 5446/80658 objects misplaced (6.752%); 28 MiB/s, 451 objects/s recovering 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[22840]: audit 2024-09-13T22:56:49.245246+0000 mon.b (mon.1) 254 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[22840]: audit 2024-09-13T22:56:50.060462+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.178:0/1052909497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[22840]: audit 2024-09-13T22:56:50.060895+0000 mon.a (mon.0) 1588 : audit [INF] from='client.25706 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[29889]: cluster 2024-09-13T22:56:48.630582+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[29889]: cluster 2024-09-13T22:56:48.631911+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:56:50.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[29889]: cluster 2024-09-13T22:56:48.876674+0000 mgr.y (mgr.14613) 712 : cluster [DBG] pgmap v833: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.6 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 865 op/s; 5446/80658 objects misplaced (6.752%); 28 MiB/s, 451 objects/s recovering 2024-09-13T22:56:50.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[29889]: audit 2024-09-13T22:56:49.245246+0000 mon.b (mon.1) 254 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:50.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[29889]: audit 2024-09-13T22:56:50.060462+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.178:0/1052909497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:56:50.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:50 smithi178 bash[29889]: audit 2024-09-13T22:56:50.060895+0000 mon.a (mon.0) 1588 : audit [INF] from='client.25706 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:56:50.346 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:50.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:50 smithi192 bash[24608]: cluster 2024-09-13T22:56:48.630582+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:56:50.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:50 smithi192 bash[24608]: cluster 2024-09-13T22:56:48.631911+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:56:50.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:50 smithi192 bash[24608]: cluster 2024-09-13T22:56:48.876674+0000 mgr.y (mgr.14613) 712 : cluster [DBG] pgmap v833: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.6 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 865 op/s; 5446/80658 objects misplaced (6.752%); 28 MiB/s, 451 objects/s recovering 2024-09-13T22:56:50.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:50 smithi192 bash[24608]: audit 2024-09-13T22:56:49.245246+0000 mon.b (mon.1) 254 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:56:50.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:50 smithi192 bash[24608]: audit 2024-09-13T22:56:50.060462+0000 mon.c (mon.2) 72 : audit [INF] from='client.? v1:172.21.15.178:0/1052909497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:56:50.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:50 smithi192 bash[24608]: audit 2024-09-13T22:56:50.060895+0000 mon.a (mon.0) 1588 : audit [INF] from='client.25706 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:56:50.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:50.469 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:50.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:50.603 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:50.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:50.835 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:50.858 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:50.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:50.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:50.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:51.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:51.116 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:51.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:51.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:51 smithi178 bash[29889]: cluster 2024-09-13T22:56:50.108779+0000 mon.a (mon.0) 1589 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:56:51.236 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:51 smithi178 bash[29889]: audit 2024-09-13T22:56:50.117313+0000 mon.a (mon.0) 1590 : audit [INF] from='client.25706 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:56:51.237 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:51 smithi178 bash[29889]: cluster 2024-09-13T22:56:50.121155+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e378: 8 total, 8 up, 6 in 2024-09-13T22:56:51.237 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:51 smithi178 bash[22840]: cluster 2024-09-13T22:56:50.108779+0000 mon.a (mon.0) 1589 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:56:51.237 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:51 smithi178 bash[22840]: audit 2024-09-13T22:56:50.117313+0000 mon.a (mon.0) 1590 : audit [INF] from='client.25706 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:56:51.237 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:51 smithi178 bash[22840]: cluster 2024-09-13T22:56:50.121155+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e378: 8 total, 8 up, 6 in 2024-09-13T22:56:51.252 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:51.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:51.364 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:51 smithi192 bash[24608]: cluster 2024-09-13T22:56:50.108779+0000 mon.a (mon.0) 1589 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:56:51.364 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:51 smithi192 bash[24608]: audit 2024-09-13T22:56:50.117313+0000 mon.a (mon.0) 1590 : audit [INF] from='client.25706 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:56:51.364 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:51 smithi192 bash[24608]: cluster 2024-09-13T22:56:50.121155+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e378: 8 total, 8 up, 6 in 2024-09-13T22:56:51.377 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:51.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:51.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:51.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:51.634 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:51.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:51.762 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:51.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:51.891 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:51.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:51.994 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:52.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:52.125 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:52.156 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:52.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:52.270 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[29889]: cluster 2024-09-13T22:56:50.877340+0000 mgr.y (mgr.14613) 713 : cluster [DBG] pgmap v835: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.6 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 532 op/s; 4620/79536 objects misplaced (5.809%); 17 MiB/s, 275 objects/s recovering 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[29889]: cluster 2024-09-13T22:56:51.137284+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e379: 8 total, 8 up, 6 in 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[29889]: cluster 2024-09-13T22:56:51.608702+0000 osd.1 (osd.1) 350 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[29889]: cluster 2024-09-13T22:56:51.610143+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[22840]: cluster 2024-09-13T22:56:50.877340+0000 mgr.y (mgr.14613) 713 : cluster [DBG] pgmap v835: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.6 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 532 op/s; 4620/79536 objects misplaced (5.809%); 17 MiB/s, 275 objects/s recovering 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[22840]: cluster 2024-09-13T22:56:51.137284+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e379: 8 total, 8 up, 6 in 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[22840]: cluster 2024-09-13T22:56:51.608702+0000 osd.1 (osd.1) 350 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:56:52.271 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:52 smithi178 bash[22840]: cluster 2024-09-13T22:56:51.610143+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:56:52.294 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:52.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:52.405 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:52 smithi192 bash[24608]: cluster 2024-09-13T22:56:50.877340+0000 mgr.y (mgr.14613) 713 : cluster [DBG] pgmap v835: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.6 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 532 op/s; 4620/79536 objects misplaced (5.809%); 17 MiB/s, 275 objects/s recovering 2024-09-13T22:56:52.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:52 smithi192 bash[24608]: cluster 2024-09-13T22:56:51.137284+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e379: 8 total, 8 up, 6 in 2024-09-13T22:56:52.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:52 smithi192 bash[24608]: cluster 2024-09-13T22:56:51.608702+0000 osd.1 (osd.1) 350 : cluster [DBG] 2.11 scrub starts 2024-09-13T22:56:52.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:52 smithi192 bash[24608]: cluster 2024-09-13T22:56:51.610143+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.11 scrub ok 2024-09-13T22:56:52.418 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:52.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:52.548 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:52.652 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:52.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:52.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:52.817 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:52.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:52.954 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:52.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:53.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:53.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:53.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:53.241 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:53 smithi192 bash[24608]: cluster 2024-09-13T22:56:52.138587+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e380: 8 total, 8 up, 6 in 2024-09-13T22:56:53.258 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:53.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:53.398 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:53.402 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:53 smithi178 bash[22840]: cluster 2024-09-13T22:56:52.138587+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e380: 8 total, 8 up, 6 in 2024-09-13T22:56:53.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:53 smithi178 bash[29889]: cluster 2024-09-13T22:56:52.138587+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e380: 8 total, 8 up, 6 in 2024-09-13T22:56:53.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:53.527 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:53.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:53.657 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:53.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:53.788 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:53.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:53.912 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:53.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:54.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:54.037 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:54.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T22:56:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:54.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T22:56:54.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:54.242 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:54.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[29889]: cluster 2024-09-13T22:56:52.273375+0000 osd.7 (osd.7) 376 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[29889]: cluster 2024-09-13T22:56:52.280899+0000 osd.7 (osd.7) 377 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[29889]: cluster 2024-09-13T22:56:52.877802+0000 mgr.y (mgr.14613) 714 : cluster [DBG] pgmap v838: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.5 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 637 op/s; 4349/76680 objects misplaced (5.672%); 21 MiB/s, 339 objects/s recovering 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[29889]: cluster 2024-09-13T22:56:53.111481+0000 osd.7 (osd.7) 378 : cluster [DBG] 6.1c deep-scrub starts 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[29889]: cluster 2024-09-13T22:56:53.115994+0000 osd.7 (osd.7) 379 : cluster [DBG] 6.1c deep-scrub ok 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[22840]: cluster 2024-09-13T22:56:52.273375+0000 osd.7 (osd.7) 376 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:56:54.358 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[22840]: cluster 2024-09-13T22:56:52.280899+0000 osd.7 (osd.7) 377 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:56:54.359 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[22840]: cluster 2024-09-13T22:56:52.877802+0000 mgr.y (mgr.14613) 714 : cluster [DBG] pgmap v838: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.5 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 637 op/s; 4349/76680 objects misplaced (5.672%); 21 MiB/s, 339 objects/s recovering 2024-09-13T22:56:54.359 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[22840]: cluster 2024-09-13T22:56:53.111481+0000 osd.7 (osd.7) 378 : cluster [DBG] 6.1c deep-scrub starts 2024-09-13T22:56:54.359 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:54 smithi178 bash[22840]: cluster 2024-09-13T22:56:53.115994+0000 osd.7 (osd.7) 379 : cluster [DBG] 6.1c deep-scrub ok 2024-09-13T22:56:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.388 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:54.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:54.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:54 smithi192 bash[24608]: cluster 2024-09-13T22:56:52.273375+0000 osd.7 (osd.7) 376 : cluster [DBG] 3.11 scrub starts 2024-09-13T22:56:54.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:54 smithi192 bash[24608]: cluster 2024-09-13T22:56:52.280899+0000 osd.7 (osd.7) 377 : cluster [DBG] 3.11 scrub ok 2024-09-13T22:56:54.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:54 smithi192 bash[24608]: cluster 2024-09-13T22:56:52.877802+0000 mgr.y (mgr.14613) 714 : cluster [DBG] pgmap v838: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 1.5 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 637 op/s; 4349/76680 objects misplaced (5.672%); 21 MiB/s, 339 objects/s recovering 2024-09-13T22:56:54.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:54 smithi192 bash[24608]: cluster 2024-09-13T22:56:53.111481+0000 osd.7 (osd.7) 378 : cluster [DBG] 6.1c deep-scrub starts 2024-09-13T22:56:54.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:54 smithi192 bash[24608]: cluster 2024-09-13T22:56:53.115994+0000 osd.7 (osd.7) 379 : cluster [DBG] 6.1c deep-scrub ok 2024-09-13T22:56:54.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:54.527 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:54.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:54.669 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:54.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:54.799 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:54.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:54.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:54.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:55.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:55.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:55.169 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:55.194 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:55.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:55.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:55.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:55.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:55 smithi178 bash[29889]: cluster 2024-09-13T22:56:54.131306+0000 osd.7 (osd.7) 380 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:56:55.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:55 smithi178 bash[29889]: cluster 2024-09-13T22:56:54.137948+0000 osd.7 (osd.7) 381 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:56:55.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:55 smithi178 bash[22840]: cluster 2024-09-13T22:56:54.131306+0000 osd.7 (osd.7) 380 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:56:55.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:55 smithi178 bash[22840]: cluster 2024-09-13T22:56:54.137948+0000 osd.7 (osd.7) 381 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:56:55.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:55 smithi192 bash[24608]: cluster 2024-09-13T22:56:54.131306+0000 osd.7 (osd.7) 380 : cluster [DBG] 1.2 scrub starts 2024-09-13T22:56:55.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:55 smithi192 bash[24608]: cluster 2024-09-13T22:56:54.137948+0000 osd.7 (osd.7) 381 : cluster [DBG] 1.2 scrub ok 2024-09-13T22:56:55.447 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:55.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:55.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:55.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:56:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:56:55.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:55.703 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:55.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:55.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:56:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:56:55.832 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:55.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:55.967 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:55.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:56.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:56.096 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:56.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:56:56.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:56:56.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:56.212 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:54.624930+0000 osd.2 (osd.2) 267 : cluster [DBG] 5.b scrub starts 2024-09-13T22:56:56.212 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:54.626002+0000 osd.2 (osd.2) 268 : cluster [DBG] 5.b scrub ok 2024-09-13T22:56:56.212 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:54.879554+0000 mgr.y (mgr.14613) 715 : cluster [DBG] pgmap v839: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 582 op/s; 3527/70188 objects misplaced (5.025%); 12 MiB/s, 196 objects/s recovering 2024-09-13T22:56:56.213 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:55.135214+0000 osd.7 (osd.7) 382 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:56:56.213 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:55.145262+0000 osd.7 (osd.7) 383 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:56:56.213 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:55.735002+0000 osd.1 (osd.1) 352 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:56:56.213 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[22840]: cluster 2024-09-13T22:56:55.737259+0000 osd.1 (osd.1) 353 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:54.624930+0000 osd.2 (osd.2) 267 : cluster [DBG] 5.b scrub starts 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:54.626002+0000 osd.2 (osd.2) 268 : cluster [DBG] 5.b scrub ok 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:54.879554+0000 mgr.y (mgr.14613) 715 : cluster [DBG] pgmap v839: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 582 op/s; 3527/70188 objects misplaced (5.025%); 12 MiB/s, 196 objects/s recovering 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:55.135214+0000 osd.7 (osd.7) 382 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:55.145262+0000 osd.7 (osd.7) 383 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:55.735002+0000 osd.1 (osd.1) 352 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:56:56.214 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:56 smithi178 bash[29889]: cluster 2024-09-13T22:56:55.737259+0000 osd.1 (osd.1) 353 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:56:56.249 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:56.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:56.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:56.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:54.624930+0000 osd.2 (osd.2) 267 : cluster [DBG] 5.b scrub starts 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:54.626002+0000 osd.2 (osd.2) 268 : cluster [DBG] 5.b scrub ok 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:54.879554+0000 mgr.y (mgr.14613) 715 : cluster [DBG] pgmap v839: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.4 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 582 op/s; 3527/70188 objects misplaced (5.025%); 12 MiB/s, 196 objects/s recovering 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:55.135214+0000 osd.7 (osd.7) 382 : cluster [DBG] 5.10 scrub starts 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:55.145262+0000 osd.7 (osd.7) 383 : cluster [DBG] 5.10 scrub ok 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:55.735002+0000 osd.1 (osd.1) 352 : cluster [DBG] 6.1 scrub starts 2024-09-13T22:56:56.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:56 smithi192 bash[24608]: cluster 2024-09-13T22:56:55.737259+0000 osd.1 (osd.1) 353 : cluster [DBG] 6.1 scrub ok 2024-09-13T22:56:56.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:56.641 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:56.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:56.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:56.798 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:56.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:56.942 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:56.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:57.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:57.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:57.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:57.193 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:56:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:56:57.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:57.344 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:57.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:57.456 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[29889]: audit 2024-09-13T22:56:55.805409+0000 mgr.y (mgr.14613) 716 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:57.456 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.243387+0000 osd.0 (osd.0) 365 : cluster [DBG] 10.5 scrub starts 2024-09-13T22:56:57.457 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.535750+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:56:57.457 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.539203+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:56:57.457 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.750275+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:56:57.457 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.760152+0000 osd.1 (osd.1) 355 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:56:57.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[22840]: audit 2024-09-13T22:56:55.805409+0000 mgr.y (mgr.14613) 716 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:57.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.243387+0000 osd.0 (osd.0) 365 : cluster [DBG] 10.5 scrub starts 2024-09-13T22:56:57.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.535750+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:56:57.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.539203+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:56:57.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.750275+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:56:57.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:57 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.760152+0000 osd.1 (osd.1) 355 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:56:57.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:57.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:57.585 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:57 smithi192 bash[24608]: audit 2024-09-13T22:56:55.805409+0000 mgr.y (mgr.14613) 716 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:56:57.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:57 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.243387+0000 osd.0 (osd.0) 365 : cluster [DBG] 10.5 scrub starts 2024-09-13T22:56:57.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:57 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.535750+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.6 scrub starts 2024-09-13T22:56:57.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:57 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.539203+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.6 scrub ok 2024-09-13T22:56:57.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:57 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.750275+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.0 deep-scrub starts 2024-09-13T22:56:57.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:57 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.760152+0000 osd.1 (osd.1) 355 : cluster [DBG] 3.0 deep-scrub ok 2024-09-13T22:56:57.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:57.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:57.724 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:57.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:57.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:57.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:57.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:57.998 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:58.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:58.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:58.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:58.258 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:58.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:58.392 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:56:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:56:58.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:58 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.785766+0000 osd.0 (osd.0) 366 : cluster [DBG] 10.5 scrub ok 2024-09-13T22:56:58.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:58 smithi192 bash[24608]: cluster 2024-09-13T22:56:56.880172+0000 mgr.y (mgr.14613) 717 : cluster [DBG] pgmap v840: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 757 B/s rd, 0 B/s wr, 659 op/s; 3126/66171 objects misplaced (4.724%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T22:56:58.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:58 smithi192 bash[24608]: cluster 2024-09-13T22:56:57.104037+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.f scrub starts 2024-09-13T22:56:58.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:58 smithi192 bash[24608]: cluster 2024-09-13T22:56:57.107489+0000 osd.7 (osd.7) 385 : cluster [DBG] 3.f scrub ok 2024-09-13T22:56:58.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:58 smithi192 bash[24608]: cluster 2024-09-13T22:56:57.788036+0000 osd.1 (osd.1) 356 : cluster [DBG] 10.0 scrub starts 2024-09-13T22:56:58.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:58.506 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.785766+0000 osd.0 (osd.0) 366 : cluster [DBG] 10.5 scrub ok 2024-09-13T22:56:58.506 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[29889]: cluster 2024-09-13T22:56:56.880172+0000 mgr.y (mgr.14613) 717 : cluster [DBG] pgmap v840: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 757 B/s rd, 0 B/s wr, 659 op/s; 3126/66171 objects misplaced (4.724%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T22:56:58.506 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[29889]: cluster 2024-09-13T22:56:57.104037+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.f scrub starts 2024-09-13T22:56:58.506 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[29889]: cluster 2024-09-13T22:56:57.107489+0000 osd.7 (osd.7) 385 : cluster [DBG] 3.f scrub ok 2024-09-13T22:56:58.507 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[29889]: cluster 2024-09-13T22:56:57.788036+0000 osd.1 (osd.1) 356 : cluster [DBG] 10.0 scrub starts 2024-09-13T22:56:58.507 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.785766+0000 osd.0 (osd.0) 366 : cluster [DBG] 10.5 scrub ok 2024-09-13T22:56:58.507 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[22840]: cluster 2024-09-13T22:56:56.880172+0000 mgr.y (mgr.14613) 717 : cluster [DBG] pgmap v840: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 757 B/s rd, 0 B/s wr, 659 op/s; 3126/66171 objects misplaced (4.724%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T22:56:58.507 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[22840]: cluster 2024-09-13T22:56:57.104037+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.f scrub starts 2024-09-13T22:56:58.507 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[22840]: cluster 2024-09-13T22:56:57.107489+0000 osd.7 (osd.7) 385 : cluster [DBG] 3.f scrub ok 2024-09-13T22:56:58.507 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:58 smithi178 bash[22840]: cluster 2024-09-13T22:56:57.788036+0000 osd.1 (osd.1) 356 : cluster [DBG] 10.0 scrub starts 2024-09-13T22:56:58.521 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:58.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:58.662 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:58.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:58.786 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:58.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:58.915 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:58.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:58.967 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:58.967 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:56:59.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:59.043 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:59.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:59.166 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:56:59.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:59.295 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:59.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:59.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:57.668700+0000 osd.2 (osd.2) 269 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:56:59.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:57.675263+0000 osd.2 (osd.2) 270 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:56:59.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.121475+0000 osd.6 (osd.6) 306 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:59.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.121537+0000 osd.6 (osd.6) 307 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:59.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.203010+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e381: 8 total, 8 up, 6 in 2024-09-13T22:56:59.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.267811+0000 osd.1 (osd.1) 357 : cluster [DBG] 10.0 scrub ok 2024-09-13T22:56:59.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.477571+0000 osd.3 (osd.3) 351 : cluster [DBG] 6.f scrub starts 2024-09-13T22:56:59.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.481743+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.f scrub ok 2024-09-13T22:56:59.408 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:57.668700+0000 osd.2 (osd.2) 269 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:57.675263+0000 osd.2 (osd.2) 270 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.121475+0000 osd.6 (osd.6) 306 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.121537+0000 osd.6 (osd.6) 307 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.203010+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e381: 8 total, 8 up, 6 in 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.267811+0000 osd.1 (osd.1) 357 : cluster [DBG] 10.0 scrub ok 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.477571+0000 osd.3 (osd.3) 351 : cluster [DBG] 6.f scrub starts 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.481743+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.f scrub ok 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:57.668700+0000 osd.2 (osd.2) 269 : cluster [DBG] 2.10 scrub starts 2024-09-13T22:56:59.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:57.675263+0000 osd.2 (osd.2) 270 : cluster [DBG] 2.10 scrub ok 2024-09-13T22:56:59.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.121475+0000 osd.6 (osd.6) 306 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:56:59.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.121537+0000 osd.6 (osd.6) 307 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:56:59.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.203010+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e381: 8 total, 8 up, 6 in 2024-09-13T22:56:59.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.267811+0000 osd.1 (osd.1) 357 : cluster [DBG] 10.0 scrub ok 2024-09-13T22:56:59.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.477571+0000 osd.3 (osd.3) 351 : cluster [DBG] 6.f scrub starts 2024-09-13T22:56:59.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:56:59 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.481743+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.f scrub ok 2024-09-13T22:56:59.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:56:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:56:59.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:59.551 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:56:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:56:59.654 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:59.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:56:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:56:59.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:59.815 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:56:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:56:59.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:59.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:56:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:56:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:56:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:56:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:56:59.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:00.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:00.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:00.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:00.223 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:00.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:00.351 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:00.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:00.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:00 smithi178 bash[29889]: cluster 2024-09-13T22:56:58.880973+0000 mgr.y (mgr.14613) 718 : cluster [DBG] pgmap v842: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 843 op/s; 2790/57099 objects misplaced (4.886%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T22:57:00.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:00 smithi178 bash[29889]: cluster 2024-09-13T22:56:59.508124+0000 osd.3 (osd.3) 353 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:57:00.466 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:00 smithi178 bash[29889]: cluster 2024-09-13T22:56:59.523931+0000 osd.3 (osd.3) 354 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:57:00.466 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:00 smithi178 bash[22840]: cluster 2024-09-13T22:56:58.880973+0000 mgr.y (mgr.14613) 718 : cluster [DBG] pgmap v842: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 843 op/s; 2790/57099 objects misplaced (4.886%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T22:57:00.466 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:00 smithi178 bash[22840]: cluster 2024-09-13T22:56:59.508124+0000 osd.3 (osd.3) 353 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:57:00.466 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:00 smithi178 bash[22840]: cluster 2024-09-13T22:56:59.523931+0000 osd.3 (osd.3) 354 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:57:00.484 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:00.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:00.621 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:00.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:00 smithi192 bash[24608]: cluster 2024-09-13T22:56:58.880973+0000 mgr.y (mgr.14613) 718 : cluster [DBG] pgmap v842: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 843 op/s; 2790/57099 objects misplaced (4.886%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T22:57:00.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:00 smithi192 bash[24608]: cluster 2024-09-13T22:56:59.508124+0000 osd.3 (osd.3) 353 : cluster [DBG] 3.7 scrub starts 2024-09-13T22:57:00.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:00 smithi192 bash[24608]: cluster 2024-09-13T22:56:59.523931+0000 osd.3 (osd.3) 354 : cluster [DBG] 3.7 scrub ok 2024-09-13T22:57:00.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:00.754 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:00.860 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:00.892 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:00.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:00.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.004 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:01.007 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:01.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:01.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.154 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:01.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:01.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:01.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:01.411 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:01.514 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.549 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:01.654 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:01 smithi192 bash[24608]: cluster 2024-09-13T22:56:59.666414+0000 osd.2 (osd.2) 271 : cluster [DBG] 2.c scrub starts 2024-09-13T22:57:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:01 smithi192 bash[24608]: cluster 2024-09-13T22:56:59.672443+0000 osd.2 (osd.2) 272 : cluster [DBG] 2.c scrub ok 2024-09-13T22:57:01.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:01 smithi192 bash[24608]: cluster 2024-09-13T22:57:00.766307+0000 osd.1 (osd.1) 358 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:57:01.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:01 smithi192 bash[24608]: cluster 2024-09-13T22:57:00.773831+0000 osd.1 (osd.1) 359 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:57:01.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[29889]: cluster 2024-09-13T22:56:59.666414+0000 osd.2 (osd.2) 271 : cluster [DBG] 2.c scrub starts 2024-09-13T22:57:01.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[29889]: cluster 2024-09-13T22:56:59.672443+0000 osd.2 (osd.2) 272 : cluster [DBG] 2.c scrub ok 2024-09-13T22:57:01.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[29889]: cluster 2024-09-13T22:57:00.766307+0000 osd.1 (osd.1) 358 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:57:01.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[29889]: cluster 2024-09-13T22:57:00.773831+0000 osd.1 (osd.1) 359 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:57:01.665 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[22840]: cluster 2024-09-13T22:56:59.666414+0000 osd.2 (osd.2) 271 : cluster [DBG] 2.c scrub starts 2024-09-13T22:57:01.665 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[22840]: cluster 2024-09-13T22:56:59.672443+0000 osd.2 (osd.2) 272 : cluster [DBG] 2.c scrub ok 2024-09-13T22:57:01.665 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[22840]: cluster 2024-09-13T22:57:00.766307+0000 osd.1 (osd.1) 358 : cluster [DBG] 5.18 scrub starts 2024-09-13T22:57:01.665 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:01 smithi178 bash[22840]: cluster 2024-09-13T22:57:00.773831+0000 osd.1 (osd.1) 359 : cluster [DBG] 5.18 scrub ok 2024-09-13T22:57:01.681 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:01.784 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:01.814 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:01.920 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:01.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:02.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:02.081 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:02.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:02.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:02.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:02.338 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:02.444 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:02.474 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:02.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:02.589 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:02 smithi192 bash[24608]: cluster 2024-09-13T22:57:00.881540+0000 mgr.y (mgr.14613) 719 : cluster [DBG] pgmap v843: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 816 op/s; 2790/55287 objects misplaced (5.046%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T22:57:02.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:02 smithi192 bash[24608]: cluster 2024-09-13T22:57:01.012326+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:57:02.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:02 smithi192 bash[24608]: cluster 2024-09-13T22:57:01.014311+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:57:02.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:02 smithi192 bash[24608]: cluster 2024-09-13T22:57:01.678491+0000 osd.3 (osd.3) 355 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:02.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:02 smithi192 bash[24608]: cluster 2024-09-13T22:57:01.678549+0000 osd.3 (osd.3) 356 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:02.603 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:02.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:02.740 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[29889]: cluster 2024-09-13T22:57:00.881540+0000 mgr.y (mgr.14613) 719 : cluster [DBG] pgmap v843: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 816 op/s; 2790/55287 objects misplaced (5.046%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T22:57:02.740 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[29889]: cluster 2024-09-13T22:57:01.012326+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:57:02.740 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[29889]: cluster 2024-09-13T22:57:01.014311+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:57:02.740 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[29889]: cluster 2024-09-13T22:57:01.678491+0000 osd.3 (osd.3) 355 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:02.741 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[29889]: cluster 2024-09-13T22:57:01.678549+0000 osd.3 (osd.3) 356 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:02.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[22840]: cluster 2024-09-13T22:57:00.881540+0000 mgr.y (mgr.14613) 719 : cluster [DBG] pgmap v843: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.1 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 816 op/s; 2790/55287 objects misplaced (5.046%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T22:57:02.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[22840]: cluster 2024-09-13T22:57:01.012326+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.1a scrub starts 2024-09-13T22:57:02.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[22840]: cluster 2024-09-13T22:57:01.014311+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1a scrub ok 2024-09-13T22:57:02.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[22840]: cluster 2024-09-13T22:57:01.678491+0000 osd.3 (osd.3) 355 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:02.741 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:02 smithi178 bash[22840]: cluster 2024-09-13T22:57:01.678549+0000 osd.3 (osd.3) 356 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:02.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:02.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:02.877 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:02.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:02.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:03.005 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:03.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:03.140 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:03.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:03.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:03 smithi192 bash[24608]: cluster 2024-09-13T22:57:01.712208+0000 osd.2 (osd.2) 273 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:57:03.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:03 smithi192 bash[24608]: cluster 2024-09-13T22:57:01.727459+0000 osd.2 (osd.2) 274 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:57:03.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:03 smithi192 bash[24608]: cluster 2024-09-13T22:57:02.043222+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.e scrub starts 2024-09-13T22:57:03.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:03 smithi192 bash[24608]: cluster 2024-09-13T22:57:02.049262+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.e scrub ok 2024-09-13T22:57:03.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:03 smithi192 bash[24608]: cluster 2024-09-13T22:57:02.223407+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e382: 8 total, 8 up, 6 in 2024-09-13T22:57:03.266 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:03.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:03.392 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:03.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[29889]: cluster 2024-09-13T22:57:01.712208+0000 osd.2 (osd.2) 273 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:57:03.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[29889]: cluster 2024-09-13T22:57:01.727459+0000 osd.2 (osd.2) 274 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:57:03.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[29889]: cluster 2024-09-13T22:57:02.043222+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.e scrub starts 2024-09-13T22:57:03.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[29889]: cluster 2024-09-13T22:57:02.049262+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.e scrub ok 2024-09-13T22:57:03.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[29889]: cluster 2024-09-13T22:57:02.223407+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e382: 8 total, 8 up, 6 in 2024-09-13T22:57:03.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[22840]: cluster 2024-09-13T22:57:01.712208+0000 osd.2 (osd.2) 273 : cluster [DBG] 6.14 scrub starts 2024-09-13T22:57:03.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[22840]: cluster 2024-09-13T22:57:01.727459+0000 osd.2 (osd.2) 274 : cluster [DBG] 6.14 scrub ok 2024-09-13T22:57:03.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[22840]: cluster 2024-09-13T22:57:02.043222+0000 osd.5 (osd.5) 409 : cluster [DBG] 5.e scrub starts 2024-09-13T22:57:03.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[22840]: cluster 2024-09-13T22:57:02.049262+0000 osd.5 (osd.5) 410 : cluster [DBG] 5.e scrub ok 2024-09-13T22:57:03.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:03 smithi178 bash[22840]: cluster 2024-09-13T22:57:02.223407+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e382: 8 total, 8 up, 6 in 2024-09-13T22:57:03.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:03.522 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:03.629 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:03.661 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:03.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:03.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:03.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:03.920 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:03.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:03.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:04.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:04.049 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:04.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:04.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:04.284 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:04.304 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:04.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:04 smithi192 bash[24608]: cluster 2024-09-13T22:57:02.882150+0000 mgr.y (mgr.14613) 720 : cluster [DBG] pgmap v845: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 821 op/s; 2364/50490 objects misplaced (4.682%) 2024-09-13T22:57:04.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:04 smithi192 bash[24608]: cluster 2024-09-13T22:57:03.029936+0000 osd.5 (osd.5) 411 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:57:04.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:04 smithi192 bash[24608]: cluster 2024-09-13T22:57:03.031027+0000 osd.5 (osd.5) 412 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:57:04.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:04.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:04.544 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:04 smithi178 bash[22840]: cluster 2024-09-13T22:57:02.882150+0000 mgr.y (mgr.14613) 720 : cluster [DBG] pgmap v845: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 821 op/s; 2364/50490 objects misplaced (4.682%) 2024-09-13T22:57:04.544 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:04 smithi178 bash[22840]: cluster 2024-09-13T22:57:03.029936+0000 osd.5 (osd.5) 411 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:57:04.544 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:04 smithi178 bash[22840]: cluster 2024-09-13T22:57:03.031027+0000 osd.5 (osd.5) 412 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:57:04.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:04 smithi178 bash[29889]: cluster 2024-09-13T22:57:02.882150+0000 mgr.y (mgr.14613) 720 : cluster [DBG] pgmap v845: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 821 op/s; 2364/50490 objects misplaced (4.682%) 2024-09-13T22:57:04.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:04 smithi178 bash[29889]: cluster 2024-09-13T22:57:03.029936+0000 osd.5 (osd.5) 411 : cluster [DBG] 5.7 scrub starts 2024-09-13T22:57:04.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:04 smithi178 bash[29889]: cluster 2024-09-13T22:57:03.031027+0000 osd.5 (osd.5) 412 : cluster [DBG] 5.7 scrub ok 2024-09-13T22:57:04.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:04.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:04.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:04.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:04.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:04.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:04.942 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:04.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:05.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:05.080 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:05.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:05.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:05.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:05.325 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:05 smithi178 bash[29889]: audit 2024-09-13T22:57:04.263710+0000 mon.a (mon.0) 1596 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:05.325 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:05 smithi178 bash[29889]: audit 2024-09-13T22:57:04.265696+0000 mon.b (mon.1) 255 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:05.325 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:05 smithi178 bash[22840]: audit 2024-09-13T22:57:04.263710+0000 mon.a (mon.0) 1596 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:05.325 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:05 smithi178 bash[22840]: audit 2024-09-13T22:57:04.265696+0000 mon.b (mon.1) 255 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:05.345 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:05.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:05.473 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:05.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:05.587 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:05 smithi192 bash[24608]: audit 2024-09-13T22:57:04.263710+0000 mon.a (mon.0) 1596 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:05.587 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:05 smithi192 bash[24608]: audit 2024-09-13T22:57:04.265696+0000 mon.b (mon.1) 255 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:05.599 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:05.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:05.728 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:05.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:05.860 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:05.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:05.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:05.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:06.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:06.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:57:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:57:06.120 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:06.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:06.242 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:06.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:06.368 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:06.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:06.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:06.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[29889]: cluster 2024-09-13T22:57:04.448180+0000 osd.0 (osd.0) 367 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[29889]: cluster 2024-09-13T22:57:04.448215+0000 osd.0 (osd.0) 368 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[29889]: cluster 2024-09-13T22:57:04.883119+0000 mgr.y (mgr.14613) 721 : cluster [DBG] pgmap v846: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 825 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1867/40176 objects misplaced (4.647%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[29889]: cluster 2024-09-13T22:57:05.261949+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[29889]: cluster 2024-09-13T22:57:05.285375+0000 mon.a (mon.0) 1598 : cluster [DBG] osdmap e383: 8 total, 8 up, 6 in 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[22840]: cluster 2024-09-13T22:57:04.448180+0000 osd.0 (osd.0) 367 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[22840]: cluster 2024-09-13T22:57:04.448215+0000 osd.0 (osd.0) 368 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[22840]: cluster 2024-09-13T22:57:04.883119+0000 mgr.y (mgr.14613) 721 : cluster [DBG] pgmap v846: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 825 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1867/40176 objects misplaced (4.647%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T22:57:06.605 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[22840]: cluster 2024-09-13T22:57:05.261949+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:57:06.606 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:06 smithi178 bash[22840]: cluster 2024-09-13T22:57:05.285375+0000 mon.a (mon.0) 1598 : cluster [DBG] osdmap e383: 8 total, 8 up, 6 in 2024-09-13T22:57:06.617 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:06.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:06 smithi192 bash[24608]: cluster 2024-09-13T22:57:04.448180+0000 osd.0 (osd.0) 367 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:06.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:06 smithi192 bash[24608]: cluster 2024-09-13T22:57:04.448215+0000 osd.0 (osd.0) 368 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:06.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:06 smithi192 bash[24608]: cluster 2024-09-13T22:57:04.883119+0000 mgr.y (mgr.14613) 721 : cluster [DBG] pgmap v846: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 825 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1867/40176 objects misplaced (4.647%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T22:57:06.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:06 smithi192 bash[24608]: cluster 2024-09-13T22:57:05.261949+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T22:57:06.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:06 smithi192 bash[24608]: cluster 2024-09-13T22:57:05.285375+0000 mon.a (mon.0) 1598 : cluster [DBG] osdmap e383: 8 total, 8 up, 6 in 2024-09-13T22:57:06.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:06.738 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:06.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:06.869 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:06.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:06.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:07.004 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:07.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:07.134 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:07.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:07.265 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:07.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:07.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:05.708669+0000 osd.2 (osd.2) 275 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:57:07.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:05.736846+0000 osd.2 (osd.2) 276 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:57:07.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: audit 2024-09-13T22:57:05.815713+0000 mgr.y (mgr.14613) 722 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:07.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.282220+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e384: 8 total, 8 up, 6 in 2024-09-13T22:57:07.372 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.873626+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:57:07.401 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:07.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:07.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:05.708669+0000 osd.2 (osd.2) 275 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:57:07.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:05.736846+0000 osd.2 (osd.2) 276 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:57:07.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: audit 2024-09-13T22:57:05.815713+0000 mgr.y (mgr.14613) 722 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:07.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.282220+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e384: 8 total, 8 up, 6 in 2024-09-13T22:57:07.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.873626+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:57:07.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.876288+0000 osd.1 (osd.1) 361 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:57:07.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.959849+0000 osd.7 (osd.7) 388 : cluster [DBG] 5.d scrub starts 2024-09-13T22:57:07.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:07 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.962070+0000 osd.7 (osd.7) 389 : cluster [DBG] 5.d scrub ok 2024-09-13T22:57:07.528 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:07.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:07.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.876288+0000 osd.1 (osd.1) 361 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.959849+0000 osd.7 (osd.7) 388 : cluster [DBG] 5.d scrub starts 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.962070+0000 osd.7 (osd.7) 389 : cluster [DBG] 5.d scrub ok 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:05.708669+0000 osd.2 (osd.2) 275 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:05.736846+0000 osd.2 (osd.2) 276 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: audit 2024-09-13T22:57:05.815713+0000 mgr.y (mgr.14613) 722 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.282220+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e384: 8 total, 8 up, 6 in 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.873626+0000 osd.1 (osd.1) 360 : cluster [DBG] 2.4 scrub starts 2024-09-13T22:57:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.876288+0000 osd.1 (osd.1) 361 : cluster [DBG] 2.4 scrub ok 2024-09-13T22:57:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.959849+0000 osd.7 (osd.7) 388 : cluster [DBG] 5.d scrub starts 2024-09-13T22:57:07.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:07 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.962070+0000 osd.7 (osd.7) 389 : cluster [DBG] 5.d scrub ok 2024-09-13T22:57:07.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:07.782 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:07.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:07.911 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:07.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:08.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:08.041 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:08.144 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:08.167 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:08.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:08.300 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:08.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:08.430 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:08.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.413414+0000 osd.0 (osd.0) 369 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:57:08.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.415271+0000 osd.0 (osd.0) 370 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:57:08.457 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:06.884055+0000 mgr.y (mgr.14613) 723 : cluster [DBG] pgmap v849: 158 pgs: 1 peering, 1 active+remapped+backfilling, 156 active+clean; 741 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 602/36126 objects misplaced (1.666%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.278720+0000 mon.a (mon.0) 1600 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.278763+0000 mon.a (mon.0) 1601 : cluster [INF] Cluster is now healthy 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.297782+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e385: 8 total, 8 up, 6 in 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.529643+0000 osd.3 (osd.3) 357 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.530743+0000 osd.3 (osd.3) 358 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.879831+0000 osd.1 (osd.1) 362 : cluster [DBG] 2.d scrub starts 2024-09-13T22:57:08.458 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[22840]: cluster 2024-09-13T22:57:07.882153+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.d scrub ok 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.413414+0000 osd.0 (osd.0) 369 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.415271+0000 osd.0 (osd.0) 370 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:06.884055+0000 mgr.y (mgr.14613) 723 : cluster [DBG] pgmap v849: 158 pgs: 1 peering, 1 active+remapped+backfilling, 156 active+clean; 741 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 602/36126 objects misplaced (1.666%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.278720+0000 mon.a (mon.0) 1600 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.278763+0000 mon.a (mon.0) 1601 : cluster [INF] Cluster is now healthy 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.297782+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e385: 8 total, 8 up, 6 in 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.529643+0000 osd.3 (osd.3) 357 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.530743+0000 osd.3 (osd.3) 358 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.879831+0000 osd.1 (osd.1) 362 : cluster [DBG] 2.d scrub starts 2024-09-13T22:57:08.459 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:08 smithi178 bash[29889]: cluster 2024-09-13T22:57:07.882153+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.d scrub ok 2024-09-13T22:57:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:08.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.413414+0000 osd.0 (osd.0) 369 : cluster [DBG] 3.1f scrub starts 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.415271+0000 osd.0 (osd.0) 370 : cluster [DBG] 3.1f scrub ok 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:06.884055+0000 mgr.y (mgr.14613) 723 : cluster [DBG] pgmap v849: 158 pgs: 1 peering, 1 active+remapped+backfilling, 156 active+clean; 741 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 602/36126 objects misplaced (1.666%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.278720+0000 mon.a (mon.0) 1600 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.278763+0000 mon.a (mon.0) 1601 : cluster [INF] Cluster is now healthy 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.297782+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e385: 8 total, 8 up, 6 in 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.529643+0000 osd.3 (osd.3) 357 : cluster [DBG] 6.11 scrub starts 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.530743+0000 osd.3 (osd.3) 358 : cluster [DBG] 6.11 scrub ok 2024-09-13T22:57:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.879831+0000 osd.1 (osd.1) 362 : cluster [DBG] 2.d scrub starts 2024-09-13T22:57:08.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:08 smithi192 bash[24608]: cluster 2024-09-13T22:57:07.882153+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.d scrub ok 2024-09-13T22:57:08.583 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:08.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:08.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:08.710 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:08.815 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:08.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:08.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:08.982 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:08.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:09.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:09.109 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:09.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:09.235 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:09.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:09.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:09 smithi192 bash[24608]: cluster 2024-09-13T22:57:08.293671+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e386: 8 total, 8 up, 6 in 2024-09-13T22:57:09.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:09.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:09.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:09.509 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:09 smithi178 bash[22840]: cluster 2024-09-13T22:57:08.293671+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e386: 8 total, 8 up, 6 in 2024-09-13T22:57:09.510 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:09 smithi178 bash[29889]: cluster 2024-09-13T22:57:08.293671+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e386: 8 total, 8 up, 6 in 2024-09-13T22:57:09.523 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:57:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:57:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:57:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:57:09.524 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:57:09.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:09.628 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:09.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:09.758 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:09.797 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:57:09.797 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:57:09.855 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:57:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:57:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:57:09.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:09.865 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:57:09.865 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:57:09.881 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:09.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:09.905 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:57:09.905 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:57:09.927 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:57:09.928 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:57:09.944 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:57:09.945 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:57:09.961 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:57:09.961 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:57:09.977 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:57:09.977 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:57:09.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:09.995 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:57:09.996 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:57:10.004 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:10.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:10.134 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:10.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:10.262 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:10.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:10.385 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:08.884763+0000 mgr.y (mgr.14613) 724 : cluster [DBG] pgmap v852: 158 pgs: 1 peering, 157 active+clean; 650 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 19 MiB/s, 312 objects/s recovering 2024-09-13T22:57:10.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:09.113192+0000 osd.5 (osd.5) 413 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:57:10.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:09.114972+0000 osd.5 (osd.5) 414 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:57:10.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: audit 2024-09-13T22:57:09.744940+0000 mon.a (mon.0) 1604 : audit [DBG] from='client.? v1:172.21.15.178:0/1917952414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:10.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:09.959380+0000 osd.1 (osd.1) 364 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:57:10.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:09.960360+0000 osd.1 (osd.1) 365 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:57:10.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.038627+0000 osd.7 (osd.7) 390 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:57:10.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:10 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.040487+0000 osd.7 (osd.7) 391 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:57:10.436 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:08.884763+0000 mgr.y (mgr.14613) 724 : cluster [DBG] pgmap v852: 158 pgs: 1 peering, 157 active+clean; 650 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 19 MiB/s, 312 objects/s recovering 2024-09-13T22:57:10.436 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:09.113192+0000 osd.5 (osd.5) 413 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:57:10.436 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:09.114972+0000 osd.5 (osd.5) 414 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:57:10.436 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: audit 2024-09-13T22:57:09.744940+0000 mon.a (mon.0) 1604 : audit [DBG] from='client.? v1:172.21.15.178:0/1917952414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:10.436 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:09.959380+0000 osd.1 (osd.1) 364 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:57:10.436 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:09.960360+0000 osd.1 (osd.1) 365 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.038627+0000 osd.7 (osd.7) 390 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.040487+0000 osd.7 (osd.7) 391 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:08.884763+0000 mgr.y (mgr.14613) 724 : cluster [DBG] pgmap v852: 158 pgs: 1 peering, 157 active+clean; 650 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 19 MiB/s, 312 objects/s recovering 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:09.113192+0000 osd.5 (osd.5) 413 : cluster [DBG] 6.1a scrub starts 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:09.114972+0000 osd.5 (osd.5) 414 : cluster [DBG] 6.1a scrub ok 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: audit 2024-09-13T22:57:09.744940+0000 mon.a (mon.0) 1604 : audit [DBG] from='client.? v1:172.21.15.178:0/1917952414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:09.959380+0000 osd.1 (osd.1) 364 : cluster [DBG] 6.1d scrub starts 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:09.960360+0000 osd.1 (osd.1) 365 : cluster [DBG] 6.1d scrub ok 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.038627+0000 osd.7 (osd.7) 390 : cluster [DBG] 6.12 scrub starts 2024-09-13T22:57:10.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:10 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.040487+0000 osd.7 (osd.7) 391 : cluster [DBG] 6.12 scrub ok 2024-09-13T22:57:10.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:10.511 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:10.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:10.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:10.763 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:10.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:10.899 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:10.916 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:10.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:11.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:11.044 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:11.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:11.175 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:11.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:11.302 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:11.307 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:57:11.316 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T22:57:11.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:11.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: cluster 2024-09-13T22:57:09.477901+0000 osd.0 (osd.0) 371 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:57:11.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: cluster 2024-09-13T22:57:09.483691+0000 osd.0 (osd.0) 372 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:57:11.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.530749+0000 osd.3 (osd.3) 359 : cluster [DBG] 10.f scrub starts 2024-09-13T22:57:11.414 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: audit 2024-09-13T22:57:10.665903+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? v1:172.21.15.178:0/3809561255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: audit 2024-09-13T22:57:10.900387+0000 mon.c (mon.2) 73 : audit [INF] from='client.? v1:172.21.15.178:0/4016131944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: audit 2024-09-13T22:57:10.900642+0000 mon.a (mon.0) 1606 : audit [INF] from='client.25721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.937902+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.942672+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: cluster 2024-09-13T22:57:09.477901+0000 osd.0 (osd.0) 371 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: cluster 2024-09-13T22:57:09.483691+0000 osd.0 (osd.0) 372 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.530749+0000 osd.3 (osd.3) 359 : cluster [DBG] 10.f scrub starts 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: audit 2024-09-13T22:57:10.665903+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? v1:172.21.15.178:0/3809561255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: audit 2024-09-13T22:57:10.900387+0000 mon.c (mon.2) 73 : audit [INF] from='client.? v1:172.21.15.178:0/4016131944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:57:11.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: audit 2024-09-13T22:57:10.900642+0000 mon.a (mon.0) 1606 : audit [INF] from='client.25721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:57:11.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.937902+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:57:11.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:11 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.942672+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:57:11.424 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:11.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:11.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: cluster 2024-09-13T22:57:09.477901+0000 osd.0 (osd.0) 371 : cluster [DBG] 5.5 scrub starts 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: cluster 2024-09-13T22:57:09.483691+0000 osd.0 (osd.0) 372 : cluster [DBG] 5.5 scrub ok 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.530749+0000 osd.3 (osd.3) 359 : cluster [DBG] 10.f scrub starts 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: audit 2024-09-13T22:57:10.665903+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? v1:172.21.15.178:0/3809561255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: audit 2024-09-13T22:57:10.900387+0000 mon.c (mon.2) 73 : audit [INF] from='client.? v1:172.21.15.178:0/4016131944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: audit 2024-09-13T22:57:10.900642+0000 mon.a (mon.0) 1606 : audit [INF] from='client.25721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T22:57:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.937902+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.19 scrub starts 2024-09-13T22:57:11.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:11 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.942672+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.19 scrub ok 2024-09-13T22:57:11.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:11.698 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:11.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:11.825 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:11.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:11.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:11.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:12.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:12.080 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:12.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:12.205 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:12.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:12.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.885469+0000 mgr.y (mgr.14613) 725 : cluster [DBG] pgmap v853: 158 pgs: 1 peering, 157 active+clean; 631 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 912 B/s rd, 0 B/s wr, 554 op/s; 14 MiB/s, 222 objects/s recovering 2024-09-13T22:57:12.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: cluster 2024-09-13T22:57:10.969155+0000 osd.3 (osd.3) 360 : cluster [DBG] 10.f scrub ok 2024-09-13T22:57:12.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: cluster 2024-09-13T22:57:11.099916+0000 osd.5 (osd.5) 415 : cluster [DBG] 6.d scrub starts 2024-09-13T22:57:12.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: cluster 2024-09-13T22:57:11.104357+0000 osd.5 (osd.5) 416 : cluster [DBG] 6.d scrub ok 2024-09-13T22:57:12.311 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: cluster 2024-09-13T22:57:11.304552+0000 mon.a (mon.0) 1607 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:12.330 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:12.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.885469+0000 mgr.y (mgr.14613) 725 : cluster [DBG] pgmap v853: 158 pgs: 1 peering, 157 active+clean; 631 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 912 B/s rd, 0 B/s wr, 554 op/s; 14 MiB/s, 222 objects/s recovering 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: cluster 2024-09-13T22:57:10.969155+0000 osd.3 (osd.3) 360 : cluster [DBG] 10.f scrub ok 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: cluster 2024-09-13T22:57:11.099916+0000 osd.5 (osd.5) 415 : cluster [DBG] 6.d scrub starts 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: cluster 2024-09-13T22:57:11.104357+0000 osd.5 (osd.5) 416 : cluster [DBG] 6.d scrub ok 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: cluster 2024-09-13T22:57:11.304552+0000 mon.a (mon.0) 1607 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: audit 2024-09-13T22:57:11.308775+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? v1:172.21.15.178:0/3809561255' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: audit 2024-09-13T22:57:11.308832+0000 mon.a (mon.0) 1609 : audit [INF] from='client.25721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:57:12.445 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:12 smithi192 bash[24608]: cluster 2024-09-13T22:57:11.313078+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e387: 8 total, 8 up, 6 in 2024-09-13T22:57:12.460 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:12.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.885469+0000 mgr.y (mgr.14613) 725 : cluster [DBG] pgmap v853: 158 pgs: 1 peering, 157 active+clean; 631 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 912 B/s rd, 0 B/s wr, 554 op/s; 14 MiB/s, 222 objects/s recovering 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: cluster 2024-09-13T22:57:10.969155+0000 osd.3 (osd.3) 360 : cluster [DBG] 10.f scrub ok 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: cluster 2024-09-13T22:57:11.099916+0000 osd.5 (osd.5) 415 : cluster [DBG] 6.d scrub starts 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: cluster 2024-09-13T22:57:11.104357+0000 osd.5 (osd.5) 416 : cluster [DBG] 6.d scrub ok 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: cluster 2024-09-13T22:57:11.304552+0000 mon.a (mon.0) 1607 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: audit 2024-09-13T22:57:11.308775+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? v1:172.21.15.178:0/3809561255' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: audit 2024-09-13T22:57:11.308832+0000 mon.a (mon.0) 1609 : audit [INF] from='client.25721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[29889]: cluster 2024-09-13T22:57:11.313078+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e387: 8 total, 8 up, 6 in 2024-09-13T22:57:12.560 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: audit 2024-09-13T22:57:11.308775+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? v1:172.21.15.178:0/3809561255' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:57:12.561 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: audit 2024-09-13T22:57:11.308832+0000 mon.a (mon.0) 1609 : audit [INF] from='client.25721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T22:57:12.561 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:12 smithi178 bash[22840]: cluster 2024-09-13T22:57:11.313078+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e387: 8 total, 8 up, 6 in 2024-09-13T22:57:12.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:12.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:12.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:12.721 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:12.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:12.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:12.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:12.986 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:12.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:13.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:13.129 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:13.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T22:57:13.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:13.255 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:13.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:13.377 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:13.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:13.516 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:13.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:13.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:13.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:13.777 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:13.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:13.912 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:13.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:14.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:14.038 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:14.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:14.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:14.173 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:14.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:14.304 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:14.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:14.409 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:14.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:14 smithi178 bash[29889]: cluster 2024-09-13T22:57:12.886012+0000 mgr.y (mgr.14613) 726 : cluster [DBG] pgmap v855: 158 pgs: 158 active+clean; 570 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 455 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-13T22:57:14.418 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:14 smithi178 bash[22840]: cluster 2024-09-13T22:57:12.886012+0000 mgr.y (mgr.14613) 726 : cluster [DBG] pgmap v855: 158 pgs: 158 active+clean; 570 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 455 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-13T22:57:14.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:14.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:14.574 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:14.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:14 smithi192 bash[24608]: cluster 2024-09-13T22:57:12.886012+0000 mgr.y (mgr.14613) 726 : cluster [DBG] pgmap v855: 158 pgs: 158 active+clean; 570 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 455 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-13T22:57:14.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:14.723 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:14.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:14.851 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:14.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:14.981 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:14.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:15.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:15.113 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:15.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:15.245 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:15.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.386 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:15.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:15.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:15.510 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:15.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:15.641 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:15.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:15.771 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:15.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:15.920 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:15.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:15.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T22:57:16.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:16.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:57:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:57:16.048 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:16.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:16.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:16.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:16.333 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[22840]: cluster 2024-09-13T22:57:14.887463+0000 mgr.y (mgr.14613) 727 : cluster [DBG] pgmap v856: 158 pgs: 158 active+clean; 406 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 705 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[22840]: cluster 2024-09-13T22:57:14.936406+0000 osd.5 (osd.5) 417 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[22840]: cluster 2024-09-13T22:57:14.936541+0000 osd.5 (osd.5) 418 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[22840]: cluster 2024-09-13T22:57:15.323507+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e388: 8 total, 8 up, 6 in 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[29889]: cluster 2024-09-13T22:57:14.887463+0000 mgr.y (mgr.14613) 727 : cluster [DBG] pgmap v856: 158 pgs: 158 active+clean; 406 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 705 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[29889]: cluster 2024-09-13T22:57:14.936406+0000 osd.5 (osd.5) 417 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[29889]: cluster 2024-09-13T22:57:14.936541+0000 osd.5 (osd.5) 418 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:16.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:16 smithi178 bash[29889]: cluster 2024-09-13T22:57:15.323507+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e388: 8 total, 8 up, 6 in 2024-09-13T22:57:16.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:16.456 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:16.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:16.585 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:16.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:16 smithi192 bash[24608]: cluster 2024-09-13T22:57:14.887463+0000 mgr.y (mgr.14613) 727 : cluster [DBG] pgmap v856: 158 pgs: 158 active+clean; 406 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 705 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-09-13T22:57:16.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:16 smithi192 bash[24608]: cluster 2024-09-13T22:57:14.936406+0000 osd.5 (osd.5) 417 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:16.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:16 smithi192 bash[24608]: cluster 2024-09-13T22:57:14.936541+0000 osd.5 (osd.5) 418 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:16.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:16 smithi192 bash[24608]: cluster 2024-09-13T22:57:15.323507+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e388: 8 total, 8 up, 6 in 2024-09-13T22:57:16.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:16.715 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:16.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:16.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:16.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:16.970 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:16.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:17.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:17.094 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:17.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:57:17.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:17.219 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:17.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:17.353 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:17.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:17 smithi192 bash[24608]: audit 2024-09-13T22:57:15.825540+0000 mgr.y (mgr.14613) 728 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:17.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:17.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:17.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:17.615 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:17 smithi178 bash[29889]: audit 2024-09-13T22:57:15.825540+0000 mgr.y (mgr.14613) 728 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:17 smithi178 bash[22840]: audit 2024-09-13T22:57:15.825540+0000 mgr.y (mgr.14613) 728 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:17.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:17.748 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:17.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:17.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:57:17.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:17.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:17.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:17.999 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:18.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:18.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:18.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:18.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:18.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:18.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:18.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:18.525 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:18.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:18.639 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:18 smithi178 bash[29889]: cluster 2024-09-13T22:57:16.888137+0000 mgr.y (mgr.14613) 729 : cluster [DBG] pgmap v858: 158 pgs: 158 active+clean; 331 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 638 op/s 2024-09-13T22:57:18.639 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:18 smithi178 bash[22840]: cluster 2024-09-13T22:57:16.888137+0000 mgr.y (mgr.14613) 729 : cluster [DBG] pgmap v858: 158 pgs: 158 active+clean; 331 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 638 op/s 2024-09-13T22:57:18.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:18 smithi192 bash[24608]: cluster 2024-09-13T22:57:16.888137+0000 mgr.y (mgr.14613) 729 : cluster [DBG] pgmap v858: 158 pgs: 158 active+clean; 331 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 638 op/s 2024-09-13T22:57:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:18.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:18.789 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:18.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:18.924 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:18.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:18.955 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:19.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:19.052 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:19.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:19.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:19.284 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:19.317 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:19.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:19.464 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:19.489 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Removed 110733 objects 2024-09-13T22:57:19.489 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Clean up completed and total clean up time :136.561 2024-09-13T22:57:19.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:19.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:19.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:19.721 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:19.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:19.852 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:19.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:57:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:57:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:57:19.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:19.980 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:19.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:19.983 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:20.083 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:20.108 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:20.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:20.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:20.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:20.370 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:20 smithi178 bash[29889]: cluster 2024-09-13T22:57:18.888715+0000 mgr.y (mgr.14613) 730 : cluster [DBG] pgmap v859: 158 pgs: 158 active+clean; 143 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 976 op/s 2024-09-13T22:57:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:20 smithi178 bash[29889]: audit 2024-09-13T22:57:19.283516+0000 mon.a (mon.0) 1612 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:20 smithi178 bash[29889]: audit 2024-09-13T22:57:19.287184+0000 mon.b (mon.1) 256 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:20 smithi178 bash[22840]: cluster 2024-09-13T22:57:18.888715+0000 mgr.y (mgr.14613) 730 : cluster [DBG] pgmap v859: 158 pgs: 158 active+clean; 143 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 976 op/s 2024-09-13T22:57:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:20 smithi178 bash[22840]: audit 2024-09-13T22:57:19.283516+0000 mon.a (mon.0) 1612 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:20.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:20 smithi178 bash[22840]: audit 2024-09-13T22:57:19.287184+0000 mon.b (mon.1) 256 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:20.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:20.501 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:20.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:20.631 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:20 smithi192 bash[24608]: cluster 2024-09-13T22:57:18.888715+0000 mgr.y (mgr.14613) 730 : cluster [DBG] pgmap v859: 158 pgs: 158 active+clean; 143 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 976 op/s 2024-09-13T22:57:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:20 smithi192 bash[24608]: audit 2024-09-13T22:57:19.283516+0000 mon.a (mon.0) 1612 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:20 smithi192 bash[24608]: audit 2024-09-13T22:57:19.287184+0000 mon.b (mon.1) 256 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:20.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:20.759 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:20.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:20.890 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:20.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:20.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:21.021 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:21.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:21.151 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:21.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:21.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:21.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:21.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:21 smithi178 bash[29889]: cluster 2024-09-13T22:57:20.610500+0000 osd.2 (osd.2) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:21.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:21 smithi178 bash[29889]: cluster 2024-09-13T22:57:20.610612+0000 osd.2 (osd.2) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:21.397 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:21 smithi178 bash[22840]: cluster 2024-09-13T22:57:20.610500+0000 osd.2 (osd.2) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:21.398 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:21 smithi178 bash[22840]: cluster 2024-09-13T22:57:20.610612+0000 osd.2 (osd.2) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:21.413 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:21.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:21.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:21.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:21.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:21 smithi192 bash[24608]: cluster 2024-09-13T22:57:20.610500+0000 osd.2 (osd.2) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:21.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:21 smithi192 bash[24608]: cluster 2024-09-13T22:57:20.610612+0000 osd.2 (osd.2) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:21.669 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:21.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:21.801 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:21.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:21.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:21.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:22.030 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:22.030 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:22.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:22.061 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:22.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:22.192 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:22.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:22.323 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:22.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:22.454 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:22.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:22.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:22 smithi192 bash[24608]: cluster 2024-09-13T22:57:20.889283+0000 mgr.y (mgr.14613) 731 : cluster [DBG] pgmap v860: 158 pgs: 158 active+clean; 101 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 885 op/s 2024-09-13T22:57:22.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:22 smithi192 bash[24608]: cluster 2024-09-13T22:57:21.298524+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e389: 8 total, 8 up, 6 in 2024-09-13T22:57:22.570 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:22 smithi178 bash[29889]: cluster 2024-09-13T22:57:20.889283+0000 mgr.y (mgr.14613) 731 : cluster [DBG] pgmap v860: 158 pgs: 158 active+clean; 101 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 885 op/s 2024-09-13T22:57:22.570 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:22 smithi178 bash[29889]: cluster 2024-09-13T22:57:21.298524+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e389: 8 total, 8 up, 6 in 2024-09-13T22:57:22.570 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:22 smithi178 bash[22840]: cluster 2024-09-13T22:57:20.889283+0000 mgr.y (mgr.14613) 731 : cluster [DBG] pgmap v860: 158 pgs: 158 active+clean; 101 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 885 op/s 2024-09-13T22:57:22.570 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:22 smithi178 bash[22840]: cluster 2024-09-13T22:57:21.298524+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e389: 8 total, 8 up, 6 in 2024-09-13T22:57:22.585 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:22.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:22.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:22.846 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:22.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:22.978 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:22.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:23.081 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:23.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:23.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:23.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:23.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:23.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:23.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:23.510 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:23.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:23.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:23.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:23.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:23.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:23.968 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:23.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:24.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:24.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:24.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:24.228 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:24.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:24.361 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:24.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:24 smithi178 bash[22840]: cluster 2024-09-13T22:57:22.890173+0000 mgr.y (mgr.14613) 732 : cluster [DBG] pgmap v862: 158 pgs: 158 active+clean; 37 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 739 op/s 2024-09-13T22:57:24.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:24 smithi178 bash[29889]: cluster 2024-09-13T22:57:22.890173+0000 mgr.y (mgr.14613) 732 : cluster [DBG] pgmap v862: 158 pgs: 158 active+clean; 37 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 739 op/s 2024-09-13T22:57:24.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:24.491 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:24.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:24.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:24 smithi192 bash[24608]: cluster 2024-09-13T22:57:22.890173+0000 mgr.y (mgr.14613) 732 : cluster [DBG] pgmap v862: 158 pgs: 158 active+clean; 37 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 739 op/s 2024-09-13T22:57:24.685 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-13T22:57:24.685 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-13T22:57:24.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:24.758 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:24.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:24.889 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:24.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:24.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:25.017 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:25.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:25.149 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:25.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:25.282 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:25.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:25.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:25.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:25.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:25.563 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:57:25.563 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:57:25.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:25.657 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:25.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:25.786 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:25.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:25.917 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:25.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:26.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:26.038 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:26.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:26.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:57:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:57:26.159 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:26.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:26.285 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:26.304 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T22:57:26.304 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:57:26.349 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:26 smithi178 bash[22840]: cluster 2024-09-13T22:57:24.891049+0000 mgr.y (mgr.14613) 733 : cluster [DBG] pgmap v863: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 679 op/s 2024-09-13T22:57:26.349 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:26 smithi178 bash[22840]: audit 2024-09-13T22:57:25.745545+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? v1:172.21.15.178:0/2331205001' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:57:26.350 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:26 smithi178 bash[22840]: audit 2024-09-13T22:57:26.130379+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? v1:172.21.15.178:0/588367083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:26.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:26 smithi178 bash[29889]: cluster 2024-09-13T22:57:24.891049+0000 mgr.y (mgr.14613) 733 : cluster [DBG] pgmap v863: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 679 op/s 2024-09-13T22:57:26.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:26 smithi178 bash[29889]: audit 2024-09-13T22:57:25.745545+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? v1:172.21.15.178:0/2331205001' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:57:26.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:26 smithi178 bash[29889]: audit 2024-09-13T22:57:26.130379+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? v1:172.21.15.178:0/588367083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:26.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:26.407 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:26.425 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:57:26.426 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:57:26.442 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:57:26.442 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:57:26.457 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:57:26.458 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:57:26.474 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:57:26.474 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:57:26.495 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:57:26.495 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:57:26.510 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:26.515 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:57:26.515 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:57:26.529 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:26.536 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:57:26.537 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:57:26.557 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:57:26.558 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:57:26.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:26.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:26.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:26 smithi192 bash[24608]: cluster 2024-09-13T22:57:24.891049+0000 mgr.y (mgr.14613) 733 : cluster [DBG] pgmap v863: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 679 op/s 2024-09-13T22:57:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:26 smithi192 bash[24608]: audit 2024-09-13T22:57:25.745545+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? v1:172.21.15.178:0/2331205001' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T22:57:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:26 smithi192 bash[24608]: audit 2024-09-13T22:57:26.130379+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? v1:172.21.15.178:0/588367083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:26.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:26.781 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:26.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:26.909 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:26.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:27.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:27.040 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:27.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.168 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:27.271 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:27.297 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:27.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.428 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:27.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[22840]: audit 2024-09-13T22:57:25.833685+0000 mgr.y (mgr.14613) 734 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[22840]: cluster 2024-09-13T22:57:26.298427+0000 mon.a (mon.0) 1616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[22840]: audit 2024-09-13T22:57:26.305590+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? v1:172.21.15.178:0/2331205001' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[22840]: audit 2024-09-13T22:57:26.305662+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? v1:172.21.15.178:0/588367083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[22840]: cluster 2024-09-13T22:57:26.309334+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e390: 8 total, 8 up, 6 in 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[22840]: audit 2024-09-13T22:57:26.409419+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.178:0/560045084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[29889]: audit 2024-09-13T22:57:25.833685+0000 mgr.y (mgr.14613) 734 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[29889]: cluster 2024-09-13T22:57:26.298427+0000 mon.a (mon.0) 1616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[29889]: audit 2024-09-13T22:57:26.305590+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? v1:172.21.15.178:0/2331205001' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:57:27.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[29889]: audit 2024-09-13T22:57:26.305662+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? v1:172.21.15.178:0/588367083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:57:27.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[29889]: cluster 2024-09-13T22:57:26.309334+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e390: 8 total, 8 up, 6 in 2024-09-13T22:57:27.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:27 smithi178 bash[29889]: audit 2024-09-13T22:57:26.409419+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.178:0/560045084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:27.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.556 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.558 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:27 smithi192 bash[24608]: audit 2024-09-13T22:57:25.833685+0000 mgr.y (mgr.14613) 734 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:27.558 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:27 smithi192 bash[24608]: cluster 2024-09-13T22:57:26.298427+0000 mon.a (mon.0) 1616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:27.558 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:27 smithi192 bash[24608]: audit 2024-09-13T22:57:26.305590+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? v1:172.21.15.178:0/2331205001' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T22:57:27.558 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:27 smithi192 bash[24608]: audit 2024-09-13T22:57:26.305662+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? v1:172.21.15.178:0/588367083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:57:27.558 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:27 smithi192 bash[24608]: cluster 2024-09-13T22:57:26.309334+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e390: 8 total, 8 up, 6 in 2024-09-13T22:57:27.558 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:27 smithi192 bash[24608]: audit 2024-09-13T22:57:26.409419+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.178:0/560045084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:27.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:27.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:27.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:27.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:27.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:27.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:28.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:28.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:28.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:28.204 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:28.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:28.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:28 smithi178 bash[22840]: cluster 2024-09-13T22:57:26.891968+0000 mgr.y (mgr.14613) 735 : cluster [DBG] pgmap v865: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 286 op/s 2024-09-13T22:57:28.335 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:28.438 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:28.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:28.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:28.596 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:28.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:28 smithi192 bash[24608]: cluster 2024-09-13T22:57:26.891968+0000 mgr.y (mgr.14613) 735 : cluster [DBG] pgmap v865: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 286 op/s 2024-09-13T22:57:28.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:28 smithi178 bash[29889]: cluster 2024-09-13T22:57:26.891968+0000 mgr.y (mgr.14613) 735 : cluster [DBG] pgmap v865: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 286 op/s 2024-09-13T22:57:28.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:28.727 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:28.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:28.857 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:28.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T22:57:28.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T22:57:28.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:28.989 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:28.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:29.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:29.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:29.242 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:29.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:29.371 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:29.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:29.500 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:29.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:29.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:29.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:29.761 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:29.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:29.893 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:29.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:29.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:30.007 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:57:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:57:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:57:30.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:30.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:30.155 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:30.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:30.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:30.390 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:30.418 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:30.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:30.545 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:30.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:30.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:30 smithi192 bash[24608]: cluster 2024-09-13T22:57:28.893260+0000 mgr.y (mgr.14613) 736 : cluster [DBG] pgmap v866: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 202 op/s 2024-09-13T22:57:30.676 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:30.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:30 smithi178 bash[29889]: cluster 2024-09-13T22:57:28.893260+0000 mgr.y (mgr.14613) 736 : cluster [DBG] pgmap v866: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 202 op/s 2024-09-13T22:57:30.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:30 smithi178 bash[22840]: cluster 2024-09-13T22:57:28.893260+0000 mgr.y (mgr.14613) 736 : cluster [DBG] pgmap v866: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 202 op/s 2024-09-13T22:57:30.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:30.804 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:30.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:30.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:30.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:31.038 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:31.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:31.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:31.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:31.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:31.328 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:31.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:31.457 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:31.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:31.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:31.717 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:31.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:57:31.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:57:31.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:31.848 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:31.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:31.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:32.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:32.108 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:32.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:32.238 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:32.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:32.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:32.454 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:32.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:32.500 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:32.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:32 smithi178 bash[29889]: cluster 2024-09-13T22:57:30.894305+0000 mgr.y (mgr.14613) 737 : cluster [DBG] pgmap v867: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 169 op/s 2024-09-13T22:57:32.557 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:32 smithi178 bash[22840]: cluster 2024-09-13T22:57:30.894305+0000 mgr.y (mgr.14613) 737 : cluster [DBG] pgmap v867: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 169 op/s 2024-09-13T22:57:32.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:32.630 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:32.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:32 smithi192 bash[24608]: cluster 2024-09-13T22:57:30.894305+0000 mgr.y (mgr.14613) 737 : cluster [DBG] pgmap v867: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 169 op/s 2024-09-13T22:57:32.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:32.760 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:32.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:32.890 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:32.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:32.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:33.021 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:33.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:33.152 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:33.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:33.282 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:33.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:33.413 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:33.473 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:33.474 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:33.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:33.540 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:33.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:33.667 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:33.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:33.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:33.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:33.928 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:33.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:34.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:34.057 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:34.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:34.189 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:34.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:34.319 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:34.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:34.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:34 smithi178 bash[22840]: cluster 2024-09-13T22:57:32.894942+0000 mgr.y (mgr.14613) 738 : cluster [DBG] pgmap v868: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 58 op/s 2024-09-13T22:57:34.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:34 smithi178 bash[22840]: audit 2024-09-13T22:57:34.302845+0000 mon.b (mon.1) 257 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:34 smithi178 bash[29889]: cluster 2024-09-13T22:57:32.894942+0000 mgr.y (mgr.14613) 738 : cluster [DBG] pgmap v868: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 58 op/s 2024-09-13T22:57:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:34 smithi178 bash[29889]: audit 2024-09-13T22:57:34.302845+0000 mon.b (mon.1) 257 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:34.450 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:34.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:34.574 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:34 smithi192 bash[24608]: cluster 2024-09-13T22:57:32.894942+0000 mgr.y (mgr.14613) 738 : cluster [DBG] pgmap v868: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 58 op/s 2024-09-13T22:57:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:34 smithi192 bash[24608]: audit 2024-09-13T22:57:34.302845+0000 mon.b (mon.1) 257 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:34.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:34.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:34.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:34.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:34.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:34.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:34.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:35.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:35.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:35.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:35.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:35.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:35.354 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:35.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:35.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:35.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:35.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:35.608 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:35.710 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:35.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:35.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:35.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:35.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:35.993 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:35.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:36.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:36.124 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:36.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:57:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:57:36.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:36.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:36.352 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_3' removed 2024-09-13T22:57:36.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:36.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:36.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: cluster 2024-09-13T22:57:34.896433+0000 mgr.y (mgr.14613) 739 : cluster [DBG] pgmap v869: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:57:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.392756+0000 mon.b (mon.1) 258 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]: dispatch 2024-09-13T22:57:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.392958+0000 mon.a (mon.0) 1620 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.392994+0000 mon.b (mon.1) 259 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.393181+0000 mon.b (mon.1) 260 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.393360+0000 mon.b (mon.1) 261 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.393776+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.394177+0000 mon.a (mon.0) 1622 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.394562+0000 mon.a (mon.0) 1623 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.461523+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.178:0/1265048773' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:57:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:36 smithi192 bash[24608]: audit 2024-09-13T22:57:35.462023+0000 mon.a (mon.0) 1624 : audit [INF] from='client.25745 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:57:36.537 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:36.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:36.660 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: cluster 2024-09-13T22:57:34.896433+0000 mgr.y (mgr.14613) 739 : cluster [DBG] pgmap v869: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.392756+0000 mon.b (mon.1) 258 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.392958+0000 mon.a (mon.0) 1620 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.392994+0000 mon.b (mon.1) 259 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.393181+0000 mon.b (mon.1) 260 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.393360+0000 mon.b (mon.1) 261 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.393776+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.394177+0000 mon.a (mon.0) 1622 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.394562+0000 mon.a (mon.0) 1623 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-09-13T22:57:36.673 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.461523+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.178:0/1265048773' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[29889]: audit 2024-09-13T22:57:35.462023+0000 mon.a (mon.0) 1624 : audit [INF] from='client.25745 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: cluster 2024-09-13T22:57:34.896433+0000 mgr.y (mgr.14613) 739 : cluster [DBG] pgmap v869: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.392756+0000 mon.b (mon.1) 258 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.392958+0000 mon.a (mon.0) 1620 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.392994+0000 mon.b (mon.1) 259 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.393181+0000 mon.b (mon.1) 260 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.393360+0000 mon.b (mon.1) 261 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.393776+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.394177+0000 mon.a (mon.0) 1622 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.394562+0000 mon.a (mon.0) 1623 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]: dispatch 2024-09-13T22:57:36.674 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.461523+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.178:0/1265048773' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:57:36.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:36 smithi178 bash[22840]: audit 2024-09-13T22:57:35.462023+0000 mon.a (mon.0) 1624 : audit [INF] from='client.25745 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T22:57:36.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:36.792 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:36.848 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:57:36.848 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:57:36.848 INFO:teuthology.orchestra.run.smithi178.stdout: "num_ops": 0 2024-09-13T22:57:36.848 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:57:36.895 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:36.912 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:36.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:37.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:37.041 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:37.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:37.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:37.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:37.305 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:37.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: audit 2024-09-13T22:57:35.844966+0000 mgr.y (mgr.14613) 740 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:37.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: audit 2024-09-13T22:57:36.334630+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]': finished 2024-09-13T22:57:37.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: audit 2024-09-13T22:57:36.334800+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]': finished 2024-09-13T22:57:37.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: audit 2024-09-13T22:57:36.334948+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]': finished 2024-09-13T22:57:37.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: audit 2024-09-13T22:57:36.335109+0000 mon.a (mon.0) 1628 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-13T22:57:37.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: audit 2024-09-13T22:57:36.335251+0000 mon.a (mon.0) 1629 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:57:37.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:37 smithi192 bash[24608]: cluster 2024-09-13T22:57:36.342466+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e391: 8 total, 8 up, 6 in 2024-09-13T22:57:37.409 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:37.429 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:37.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:37.560 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:37.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: audit 2024-09-13T22:57:35.844966+0000 mgr.y (mgr.14613) 740 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: audit 2024-09-13T22:57:36.334630+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: audit 2024-09-13T22:57:36.334800+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: audit 2024-09-13T22:57:36.334948+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: audit 2024-09-13T22:57:36.335109+0000 mon.a (mon.0) 1628 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: audit 2024-09-13T22:57:36.335251+0000 mon.a (mon.0) 1629 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[29889]: cluster 2024-09-13T22:57:36.342466+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e391: 8 total, 8 up, 6 in 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: audit 2024-09-13T22:57:35.844966+0000 mgr.y (mgr.14613) 740 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: audit 2024-09-13T22:57:36.334630+0000 mon.a (mon.0) 1625 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 5]}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: audit 2024-09-13T22:57:36.334800+0000 mon.a (mon.0) 1626 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [0, 2]}]': finished 2024-09-13T22:57:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: audit 2024-09-13T22:57:36.334948+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [0, 5]}]': finished 2024-09-13T22:57:37.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: audit 2024-09-13T22:57:36.335109+0000 mon.a (mon.0) 1628 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-13T22:57:37.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: audit 2024-09-13T22:57:36.335251+0000 mon.a (mon.0) 1629 : audit [INF] from='client.25745 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T22:57:37.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:37 smithi178 bash[22840]: cluster 2024-09-13T22:57:36.342466+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e391: 8 total, 8 up, 6 in 2024-09-13T22:57:37.689 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:37.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:37.820 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:37.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:37.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:37.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:38.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:38.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:38.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:38.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:38.339 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:38.345 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T22:57:38.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:38.465 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:38.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:38.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:38 smithi192 bash[24608]: cluster 2024-09-13T22:57:36.896914+0000 mgr.y (mgr.14613) 741 : cluster [DBG] pgmap v871: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:57:38.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:38 smithi192 bash[24608]: cluster 2024-09-13T22:57:37.347200+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e392: 8 total, 8 up, 6 in 2024-09-13T22:57:38.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:38 smithi192 bash[24608]: audit 2024-09-13T22:57:37.653240+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.178:0/144662866' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:57:38.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:38 smithi192 bash[24608]: audit 2024-09-13T22:57:37.653785+0000 mon.a (mon.0) 1632 : audit [INF] from='client.25751 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:57:38.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[22840]: cluster 2024-09-13T22:57:36.896914+0000 mgr.y (mgr.14613) 741 : cluster [DBG] pgmap v871: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:57:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[22840]: cluster 2024-09-13T22:57:37.347200+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e392: 8 total, 8 up, 6 in 2024-09-13T22:57:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[22840]: audit 2024-09-13T22:57:37.653240+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.178:0/144662866' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:57:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[22840]: audit 2024-09-13T22:57:37.653785+0000 mon.a (mon.0) 1632 : audit [INF] from='client.25751 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:57:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[29889]: cluster 2024-09-13T22:57:36.896914+0000 mgr.y (mgr.14613) 741 : cluster [DBG] pgmap v871: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2024-09-13T22:57:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[29889]: cluster 2024-09-13T22:57:37.347200+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e392: 8 total, 8 up, 6 in 2024-09-13T22:57:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[29889]: audit 2024-09-13T22:57:37.653240+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.178:0/144662866' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:57:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:38 smithi178 bash[29889]: audit 2024-09-13T22:57:37.653785+0000 mon.a (mon.0) 1632 : audit [INF] from='client.25751 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T22:57:38.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:38.721 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T22:57:38.722 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T22:57:38.722 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-13T22:57:38.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-13T22:57:38.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:38.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T22:57:38.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:38.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:38.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:38.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:39.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:39.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:39.145 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:39.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:39.314 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:39.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:39 smithi192 bash[24608]: audit 2024-09-13T22:57:38.342772+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25751 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:57:39.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:39 smithi192 bash[24608]: cluster 2024-09-13T22:57:38.346307+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e393: 8 total, 8 up, 6 in 2024-09-13T22:57:39.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:39 smithi192 bash[24608]: audit 2024-09-13T22:57:38.888681+0000 mon.b (mon.1) 262 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:57:39.416 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:39.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[29889]: audit 2024-09-13T22:57:38.342772+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25751 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:57:39.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[29889]: cluster 2024-09-13T22:57:38.346307+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e393: 8 total, 8 up, 6 in 2024-09-13T22:57:39.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[29889]: audit 2024-09-13T22:57:38.888681+0000 mon.b (mon.1) 262 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:57:39.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[22840]: audit 2024-09-13T22:57:38.342772+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25751 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T22:57:39.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[22840]: cluster 2024-09-13T22:57:38.346307+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e393: 8 total, 8 up, 6 in 2024-09-13T22:57:39.426 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[22840]: audit 2024-09-13T22:57:38.888681+0000 mon.b (mon.1) 262 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:57:39.436 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:39.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:39.550 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T22:57:39.561 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:39.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:39.684 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:39.787 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:39.807 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:39.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:39.934 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:39.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:39.975 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:39.976 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:40.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:40.065 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:40.114 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:57:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:57:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:57:40.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:40.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:40.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:40.325 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:40.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:40.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[22840]: cluster 2024-09-13T22:57:38.897587+0000 mgr.y (mgr.14613) 742 : cluster [DBG] pgmap v874: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:57:40.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[22840]: audit 2024-09-13T22:57:39.564340+0000 mon.b (mon.1) 263 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:57:40.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[22840]: audit 2024-09-13T22:57:39.565309+0000 mon.b (mon.1) 264 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:57:40.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[22840]: audit 2024-09-13T22:57:39.570468+0000 mon.a (mon.0) 1635 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[22840]: cluster 2024-09-13T22:57:39.898806+0000 osd.1 (osd.1) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[22840]: cluster 2024-09-13T22:57:39.898959+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[29889]: cluster 2024-09-13T22:57:38.897587+0000 mgr.y (mgr.14613) 742 : cluster [DBG] pgmap v874: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[29889]: audit 2024-09-13T22:57:39.564340+0000 mon.b (mon.1) 263 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[29889]: audit 2024-09-13T22:57:39.565309+0000 mon.b (mon.1) 264 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[29889]: audit 2024-09-13T22:57:39.570468+0000 mon.a (mon.0) 1635 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[29889]: cluster 2024-09-13T22:57:39.898806+0000 osd.1 (osd.1) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:40.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:40 smithi178 bash[29889]: cluster 2024-09-13T22:57:39.898959+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:40.457 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:40.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:40.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:40 smithi192 bash[24608]: cluster 2024-09-13T22:57:38.897587+0000 mgr.y (mgr.14613) 742 : cluster [DBG] pgmap v874: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T22:57:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:40 smithi192 bash[24608]: audit 2024-09-13T22:57:39.564340+0000 mon.b (mon.1) 263 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:57:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:40 smithi192 bash[24608]: audit 2024-09-13T22:57:39.565309+0000 mon.b (mon.1) 264 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:57:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:40 smithi192 bash[24608]: audit 2024-09-13T22:57:39.570468+0000 mon.a (mon.0) 1635 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:57:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:40 smithi192 bash[24608]: cluster 2024-09-13T22:57:39.898806+0000 osd.1 (osd.1) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:40 smithi192 bash[24608]: cluster 2024-09-13T22:57:39.898959+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:40.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:40.712 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:40.815 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:40.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:40.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:40.973 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:40.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:41.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:41.103 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:41.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:41.227 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:41.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:41.358 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:41.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:41.486 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:41.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:41.615 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:41 smithi192 bash[24608]: cluster 2024-09-13T22:57:40.578365+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e394: 8 total, 8 up, 6 in 2024-09-13T22:57:41.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:41.728 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:41 smithi178 bash[29889]: cluster 2024-09-13T22:57:40.578365+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e394: 8 total, 8 up, 6 in 2024-09-13T22:57:41.729 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:41 smithi178 bash[22840]: cluster 2024-09-13T22:57:40.578365+0000 mon.a (mon.0) 1636 : cluster [DBG] osdmap e394: 8 total, 8 up, 6 in 2024-09-13T22:57:41.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:41.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:41.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:41.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:41.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.003 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:42.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.133 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:42.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.260 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:42.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:42.389 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:42.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:42.520 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:42.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:42.635 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:42 smithi192 bash[24608]: cluster 2024-09-13T22:57:40.898255+0000 mgr.y (mgr.14613) 743 : cluster [DBG] pgmap v876: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 120 B/s, 4 objects/s recovering 2024-09-13T22:57:42.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:42.747 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:57:42.748 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:57:42.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.766 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:57:42.767 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:57:42.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:42.785 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:57:42.785 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:57:42.802 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:57:42.802 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:57:42.821 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:57:42.822 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:57:42.840 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:57:42.840 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:57:42.860 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:57:42.861 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:57:42.881 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:57:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:57:42.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:42.892 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:42 smithi178 bash[29889]: cluster 2024-09-13T22:57:40.898255+0000 mgr.y (mgr.14613) 743 : cluster [DBG] pgmap v876: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 120 B/s, 4 objects/s recovering 2024-09-13T22:57:42.892 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:42 smithi178 bash[22840]: cluster 2024-09-13T22:57:40.898255+0000 mgr.y (mgr.14613) 743 : cluster [DBG] pgmap v876: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 120 B/s, 4 objects/s recovering 2024-09-13T22:57:42.906 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:42.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:42.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:43.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:43.036 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:43.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:43.165 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:43.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:43.297 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:43.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:43.427 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:43.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:43.557 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:43.589 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:57:43.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:43.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:43 smithi178 bash[22840]: audit 2024-09-13T22:57:42.729800+0000 mon.a (mon.0) 1637 : audit [DBG] from='client.? v1:172.21.15.178:0/1674071620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:43.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:43 smithi178 bash[22840]: audit 2024-09-13T22:57:42.956884+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? v1:172.21.15.178:0/545703095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:57:43.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:43 smithi178 bash[29889]: audit 2024-09-13T22:57:42.729800+0000 mon.a (mon.0) 1637 : audit [DBG] from='client.? v1:172.21.15.178:0/1674071620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:43.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:43 smithi178 bash[29889]: audit 2024-09-13T22:57:42.956884+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? v1:172.21.15.178:0/545703095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:57:43.687 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:43.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:43.801 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:43 smithi192 bash[24608]: audit 2024-09-13T22:57:42.729800+0000 mon.a (mon.0) 1637 : audit [DBG] from='client.? v1:172.21.15.178:0/1674071620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:43.801 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:43 smithi192 bash[24608]: audit 2024-09-13T22:57:42.956884+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? v1:172.21.15.178:0/545703095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:57:43.817 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:43.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:43.954 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:43.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:44.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:44.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:44.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:44.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:44.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:44.340 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:44.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:44.469 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:44.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:44.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:44.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:44.729 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:44.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:44.843 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:44 smithi192 bash[24608]: cluster 2024-09-13T22:57:42.898881+0000 mgr.y (mgr.14613) 744 : cluster [DBG] pgmap v877: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 91 B/s, 3 objects/s recovering 2024-09-13T22:57:44.843 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:44 smithi192 bash[24608]: cluster 2024-09-13T22:57:43.582082+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:44.844 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:44 smithi192 bash[24608]: audit 2024-09-13T22:57:43.589508+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? v1:172.21.15.178:0/545703095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:57:44.844 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:44 smithi192 bash[24608]: cluster 2024-09-13T22:57:43.598795+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e395: 8 total, 8 up, 6 in 2024-09-13T22:57:44.860 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[29889]: cluster 2024-09-13T22:57:42.898881+0000 mgr.y (mgr.14613) 744 : cluster [DBG] pgmap v877: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 91 B/s, 3 objects/s recovering 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[29889]: cluster 2024-09-13T22:57:43.582082+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[29889]: audit 2024-09-13T22:57:43.589508+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? v1:172.21.15.178:0/545703095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[29889]: cluster 2024-09-13T22:57:43.598795+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e395: 8 total, 8 up, 6 in 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[22840]: cluster 2024-09-13T22:57:42.898881+0000 mgr.y (mgr.14613) 744 : cluster [DBG] pgmap v877: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 91 B/s, 3 objects/s recovering 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[22840]: cluster 2024-09-13T22:57:43.582082+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[22840]: audit 2024-09-13T22:57:43.589508+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? v1:172.21.15.178:0/545703095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:57:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:44 smithi178 bash[22840]: cluster 2024-09-13T22:57:43.598795+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e395: 8 total, 8 up, 6 in 2024-09-13T22:57:44.936 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:44.936 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:44.936 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:44.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:44.982 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:44.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:45.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:45.113 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:45.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:45.242 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:45.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:45.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:45.375 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:45.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:45.504 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:45.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:45.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:45.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:45.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:45.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:57:45.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:45.894 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:45.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:45.907 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:57:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:57:45.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:46.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:46.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:46.150 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:46.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:46.277 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:46.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:46.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:46.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:46.539 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:46.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:57:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:46.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:46.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:46 smithi192 bash[24608]: cluster 2024-09-13T22:57:44.900288+0000 mgr.y (mgr.14613) 745 : cluster [DBG] pgmap v879: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 83 B/s, 2 objects/s recovering 2024-09-13T22:57:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:46 smithi192 bash[24608]: cluster 2024-09-13T22:57:45.562924+0000 osd.2 (osd.2) 279 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:57:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:46 smithi192 bash[24608]: cluster 2024-09-13T22:57:45.567546+0000 osd.2 (osd.2) 280 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:57:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:46 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.248855+0000 osd.0 (osd.0) 373 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:57:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:46 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.251405+0000 osd.0 (osd.0) 374 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:57:46.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:46.768 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:46.788 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:46.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[22840]: cluster 2024-09-13T22:57:44.900288+0000 mgr.y (mgr.14613) 745 : cluster [DBG] pgmap v879: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 83 B/s, 2 objects/s recovering 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[22840]: cluster 2024-09-13T22:57:45.562924+0000 osd.2 (osd.2) 279 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[22840]: cluster 2024-09-13T22:57:45.567546+0000 osd.2 (osd.2) 280 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.248855+0000 osd.0 (osd.0) 373 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.251405+0000 osd.0 (osd.0) 374 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[29889]: cluster 2024-09-13T22:57:44.900288+0000 mgr.y (mgr.14613) 745 : cluster [DBG] pgmap v879: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 83 B/s, 2 objects/s recovering 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[29889]: cluster 2024-09-13T22:57:45.562924+0000 osd.2 (osd.2) 279 : cluster [DBG] 3.3 scrub starts 2024-09-13T22:57:46.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[29889]: cluster 2024-09-13T22:57:45.567546+0000 osd.2 (osd.2) 280 : cluster [DBG] 3.3 scrub ok 2024-09-13T22:57:46.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.248855+0000 osd.0 (osd.0) 373 : cluster [DBG] 5.8 scrub starts 2024-09-13T22:57:46.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:46 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.251405+0000 osd.0 (osd.0) 374 : cluster [DBG] 5.8 scrub ok 2024-09-13T22:57:46.917 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:46.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:47.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:47.050 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:47.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:47.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:47.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:47.312 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:47.338 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:57:47.338 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:57:47.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:47.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:47.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:47.573 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:47.595 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T22:57:47.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:47.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: audit 2024-09-13T22:57:45.853216+0000 mgr.y (mgr.14613) 746 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:47.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.010050+0000 osd.5 (osd.5) 419 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:57:47.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.013966+0000 osd.5 (osd.5) 420 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:57:47.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.570398+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:57:47.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.572996+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:57:47.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: cluster 2024-09-13T22:57:47.257794+0000 osd.0 (osd.0) 375 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:57:47.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: cluster 2024-09-13T22:57:47.259082+0000 osd.0 (osd.0) 376 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:57:47.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:47 smithi192 bash[24608]: audit 2024-09-13T22:57:47.588309+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? v1:172.21.15.178:0/3905570842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:57:47.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:47.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:47.817 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: audit 2024-09-13T22:57:45.853216+0000 mgr.y (mgr.14613) 746 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.010050+0000 osd.5 (osd.5) 419 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.013966+0000 osd.5 (osd.5) 420 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.570398+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.572996+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: cluster 2024-09-13T22:57:47.257794+0000 osd.0 (osd.0) 375 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: cluster 2024-09-13T22:57:47.259082+0000 osd.0 (osd.0) 376 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[29889]: audit 2024-09-13T22:57:47.588309+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? v1:172.21.15.178:0/3905570842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: audit 2024-09-13T22:57:45.853216+0000 mgr.y (mgr.14613) 746 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.010050+0000 osd.5 (osd.5) 419 : cluster [DBG] 3.16 scrub starts 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.013966+0000 osd.5 (osd.5) 420 : cluster [DBG] 3.16 scrub ok 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.570398+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-09-13T22:57:47.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.572996+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-09-13T22:57:47.819 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: cluster 2024-09-13T22:57:47.257794+0000 osd.0 (osd.0) 375 : cluster [DBG] 2.1f scrub starts 2024-09-13T22:57:47.819 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: cluster 2024-09-13T22:57:47.259082+0000 osd.0 (osd.0) 376 : cluster [DBG] 2.1f scrub ok 2024-09-13T22:57:47.819 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:47 smithi178 bash[22840]: audit 2024-09-13T22:57:47.588309+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? v1:172.21.15.178:0/3905570842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T22:57:47.830 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:47.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:47.961 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:47.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:48.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:48.092 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:48.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:48.219 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:48.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:48.347 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:48.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:48.476 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:48.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:48.605 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:48.619 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_4' created 2024-09-13T22:57:48.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:48.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: cluster 2024-09-13T22:57:46.901152+0000 mgr.y (mgr.14613) 747 : cluster [DBG] pgmap v880: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-13T22:57:48.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: cluster 2024-09-13T22:57:47.488422+0000 osd.2 (osd.2) 281 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:57:48.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: cluster 2024-09-13T22:57:47.489922+0000 osd.2 (osd.2) 282 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:57:48.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: audit 2024-09-13T22:57:47.596421+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.178:0/3905570842' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:57:48.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: cluster 2024-09-13T22:57:47.600508+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e396: 8 total, 8 up, 6 in 2024-09-13T22:57:48.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: audit 2024-09-13T22:57:47.649967+0000 mon.c (mon.2) 77 : audit [INF] from='client.? v1:172.21.15.178:0/3734295574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-13T22:57:48.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:48 smithi192 bash[24608]: audit 2024-09-13T22:57:47.650324+0000 mon.a (mon.0) 1645 : audit [INF] from='client.25775 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-13T22:57:48.732 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:48.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:48.862 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: cluster 2024-09-13T22:57:46.901152+0000 mgr.y (mgr.14613) 747 : cluster [DBG] pgmap v880: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: cluster 2024-09-13T22:57:47.488422+0000 osd.2 (osd.2) 281 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: cluster 2024-09-13T22:57:47.489922+0000 osd.2 (osd.2) 282 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: audit 2024-09-13T22:57:47.596421+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.178:0/3905570842' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: cluster 2024-09-13T22:57:47.600508+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e396: 8 total, 8 up, 6 in 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: audit 2024-09-13T22:57:47.649967+0000 mon.c (mon.2) 77 : audit [INF] from='client.? v1:172.21.15.178:0/3734295574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[22840]: audit 2024-09-13T22:57:47.650324+0000 mon.a (mon.0) 1645 : audit [INF] from='client.25775 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-13T22:57:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: cluster 2024-09-13T22:57:46.901152+0000 mgr.y (mgr.14613) 747 : cluster [DBG] pgmap v880: 142 pgs: 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 54 B/s, 1 objects/s recovering 2024-09-13T22:57:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: cluster 2024-09-13T22:57:47.488422+0000 osd.2 (osd.2) 281 : cluster [DBG] 6.17 scrub starts 2024-09-13T22:57:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: cluster 2024-09-13T22:57:47.489922+0000 osd.2 (osd.2) 282 : cluster [DBG] 6.17 scrub ok 2024-09-13T22:57:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: audit 2024-09-13T22:57:47.596421+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.178:0/3905570842' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T22:57:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: cluster 2024-09-13T22:57:47.600508+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e396: 8 total, 8 up, 6 in 2024-09-13T22:57:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: audit 2024-09-13T22:57:47.649967+0000 mon.c (mon.2) 77 : audit [INF] from='client.? v1:172.21.15.178:0/3734295574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-13T22:57:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:48 smithi178 bash[29889]: audit 2024-09-13T22:57:47.650324+0000 mon.a (mon.0) 1645 : audit [INF] from='client.25775 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-13T22:57:48.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:48.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:48.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:49.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T22:57:49.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:49.124 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:49.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:49.252 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:49.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:49.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:49.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:49.512 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:49.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:49.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.140557+0000 osd.1 (osd.1) 370 : cluster [DBG] 3.a scrub starts 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.143710+0000 osd.1 (osd.1) 371 : cluster [DBG] 3.a scrub ok 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.271016+0000 osd.7 (osd.7) 392 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.271109+0000 osd.7 (osd.7) 393 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.504938+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.507754+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: audit 2024-09-13T22:57:48.613111+0000 mon.a (mon.0) 1646 : audit [INF] from='client.25775 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.620742+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e397: 8 total, 8 up, 6 in 2024-09-13T22:57:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:49 smithi192 bash[24608]: audit 2024-09-13T22:57:49.331589+0000 mon.b (mon.1) 265 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:49.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:49.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:49.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:49.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:49.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:49.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:57:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.140557+0000 osd.1 (osd.1) 370 : cluster [DBG] 3.a scrub starts 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.143710+0000 osd.1 (osd.1) 371 : cluster [DBG] 3.a scrub ok 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.271016+0000 osd.7 (osd.7) 392 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.271109+0000 osd.7 (osd.7) 393 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.504938+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.507754+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: audit 2024-09-13T22:57:48.613111+0000 mon.a (mon.0) 1646 : audit [INF] from='client.25775 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.620742+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e397: 8 total, 8 up, 6 in 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[29889]: audit 2024-09-13T22:57:49.331589+0000 mon.b (mon.1) 265 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:49.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.140557+0000 osd.1 (osd.1) 370 : cluster [DBG] 3.a scrub starts 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.143710+0000 osd.1 (osd.1) 371 : cluster [DBG] 3.a scrub ok 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.271016+0000 osd.7 (osd.7) 392 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.271109+0000 osd.7 (osd.7) 393 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.504938+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.1 scrub starts 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.507754+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.1 scrub ok 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: audit 2024-09-13T22:57:48.613111+0000 mon.a (mon.0) 1646 : audit [INF] from='client.25775 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.620742+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e397: 8 total, 8 up, 6 in 2024-09-13T22:57:49.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:49 smithi178 bash[22840]: audit 2024-09-13T22:57:49.331589+0000 mon.b (mon.1) 265 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:57:50.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:50.041 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:50.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:50.168 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:50.271 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:50.298 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:50.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:50.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:50.453 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T22:57:50.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:57:50.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:50.565 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:50.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:50.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:50.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-13T22:57:50.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:50.822 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:48.902486+0000 mgr.y (mgr.14613) 748 : cluster [DBG] pgmap v883: 158 pgs: 10 creating+peering, 6 unknown, 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:49.168424+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:49.170833+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:49.591560+0000 osd.3 (osd.3) 363 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:49.593213+0000 osd.3 (osd.3) 364 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:49.613402+0000 mon.a (mon.0) 1648 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:57:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:50 smithi192 bash[24608]: cluster 2024-09-13T22:57:49.629587+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e398: 8 total, 8 up, 6 in 2024-09-13T22:57:50.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:50.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:48.902486+0000 mgr.y (mgr.14613) 748 : cluster [DBG] pgmap v883: 158 pgs: 10 creating+peering, 6 unknown, 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T22:57:50.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:49.168424+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:57:50.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:49.170833+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:57:50.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:49.591560+0000 osd.3 (osd.3) 363 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:49.593213+0000 osd.3 (osd.3) 364 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:49.613402+0000 mon.a (mon.0) 1648 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[29889]: cluster 2024-09-13T22:57:49.629587+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e398: 8 total, 8 up, 6 in 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:48.902486+0000 mgr.y (mgr.14613) 748 : cluster [DBG] pgmap v883: 158 pgs: 10 creating+peering, 6 unknown, 142 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:49.168424+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.19 scrub starts 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:49.170833+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.19 scrub ok 2024-09-13T22:57:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:49.591560+0000 osd.3 (osd.3) 363 : cluster [DBG] 5.1 scrub starts 2024-09-13T22:57:50.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:49.593213+0000 osd.3 (osd.3) 364 : cluster [DBG] 5.1 scrub ok 2024-09-13T22:57:50.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:49.613402+0000 mon.a (mon.0) 1648 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T22:57:50.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:50 smithi178 bash[22840]: cluster 2024-09-13T22:57:49.629587+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e398: 8 total, 8 up, 6 in 2024-09-13T22:57:50.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:50.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:51.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:51.080 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:51.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:51.212 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:51.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:51.343 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:51.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:51.475 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:51.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:51.605 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:51.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:51.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:51.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:51.864 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.135240+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:57:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.137571+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:57:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.493757+0000 osd.2 (osd.2) 285 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:57:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.496696+0000 osd.2 (osd.2) 286 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:57:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.592555+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.b scrub starts 2024-09-13T22:57:51.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.594116+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.b scrub ok 2024-09-13T22:57:51.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.632424+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e399: 8 total, 8 up, 6 in 2024-09-13T22:57:51.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:51.202186+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:57:51.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:51.203540+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:57:51.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:51.207334+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:57:51.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:51 smithi192 bash[24608]: cluster 2024-09-13T22:57:51.208409+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:57:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.135240+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.137571+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.493757+0000 osd.2 (osd.2) 285 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.496696+0000 osd.2 (osd.2) 286 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.592555+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.b scrub starts 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.594116+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.b scrub ok 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.632424+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e399: 8 total, 8 up, 6 in 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:51.202186+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:51.203540+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:51.207334+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[29889]: cluster 2024-09-13T22:57:51.208409+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:57:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.135240+0000 osd.1 (osd.1) 374 : cluster [DBG] 2.9 scrub starts 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.137571+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub ok 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.493757+0000 osd.2 (osd.2) 285 : cluster [DBG] 3.9 scrub starts 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.496696+0000 osd.2 (osd.2) 286 : cluster [DBG] 3.9 scrub ok 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.592555+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.b scrub starts 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.594116+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.b scrub ok 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.632424+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e399: 8 total, 8 up, 6 in 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:51.202186+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.18 scrub starts 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:51.203540+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.18 scrub ok 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:51.207334+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.8 scrub starts 2024-09-13T22:57:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:51 smithi178 bash[22840]: cluster 2024-09-13T22:57:51.208409+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.8 scrub ok 2024-09-13T22:57:51.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:51.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:51.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:52.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:52.124 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:52.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:52.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:52.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:52.382 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:52.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:52.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:52.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:52.646 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:52.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:52.760 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:52 smithi178 bash[29889]: cluster 2024-09-13T22:57:50.903232+0000 mgr.y (mgr.14613) 749 : cluster [DBG] pgmap v886: 158 pgs: 14 creating+peering, 1 unknown, 143 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-13T22:57:52.760 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:52 smithi178 bash[29889]: cluster 2024-09-13T22:57:52.047840+0000 osd.1 (osd.1) 376 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:57:52.761 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:52 smithi178 bash[29889]: cluster 2024-09-13T22:57:52.050551+0000 osd.1 (osd.1) 377 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:57:52.761 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:52 smithi178 bash[22840]: cluster 2024-09-13T22:57:50.903232+0000 mgr.y (mgr.14613) 749 : cluster [DBG] pgmap v886: 158 pgs: 14 creating+peering, 1 unknown, 143 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-13T22:57:52.761 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:52 smithi178 bash[22840]: cluster 2024-09-13T22:57:52.047840+0000 osd.1 (osd.1) 376 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:57:52.761 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:52 smithi178 bash[22840]: cluster 2024-09-13T22:57:52.050551+0000 osd.1 (osd.1) 377 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:57:52.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:52.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:52.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:52 smithi192 bash[24608]: cluster 2024-09-13T22:57:50.903232+0000 mgr.y (mgr.14613) 749 : cluster [DBG] pgmap v886: 158 pgs: 14 creating+peering, 1 unknown, 143 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-13T22:57:52.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:52 smithi192 bash[24608]: cluster 2024-09-13T22:57:52.047840+0000 osd.1 (osd.1) 376 : cluster [DBG] 1.3 scrub starts 2024-09-13T22:57:52.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:52 smithi192 bash[24608]: cluster 2024-09-13T22:57:52.050551+0000 osd.1 (osd.1) 377 : cluster [DBG] 1.3 scrub ok 2024-09-13T22:57:52.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:52.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:53.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:53.035 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:53.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:53.165 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:53.222 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:53.223 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:53.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:53.297 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:53.400 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:53.427 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:53.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:53.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:53.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:53.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:53 smithi178 bash[22840]: cluster 2024-09-13T22:57:53.156553+0000 osd.0 (osd.0) 379 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:57:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:53 smithi178 bash[22840]: cluster 2024-09-13T22:57:53.159054+0000 osd.0 (osd.0) 380 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:57:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:53 smithi178 bash[29889]: cluster 2024-09-13T22:57:53.156553+0000 osd.0 (osd.0) 379 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:57:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:53 smithi178 bash[29889]: cluster 2024-09-13T22:57:53.159054+0000 osd.0 (osd.0) 380 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:57:53.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:53.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:53.818 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:53 smithi192 bash[24608]: cluster 2024-09-13T22:57:53.156553+0000 osd.0 (osd.0) 379 : cluster [DBG] 3.12 scrub starts 2024-09-13T22:57:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:53 smithi192 bash[24608]: cluster 2024-09-13T22:57:53.159054+0000 osd.0 (osd.0) 380 : cluster [DBG] 3.12 scrub ok 2024-09-13T22:57:53.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:53.949 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:53.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:54.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:54.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:54.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:54.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:54.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:54.340 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:54.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:54.469 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:54.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:54.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:54.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:54.729 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:54.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:52.904319+0000 mgr.y (mgr.14613) 750 : cluster [DBG] pgmap v887: 158 pgs: 14 creating+peering, 144 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:53.055465+0000 osd.5 (osd.5) 421 : cluster [DBG] 3.c scrub starts 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:53.059025+0000 osd.5 (osd.5) 422 : cluster [DBG] 3.c scrub ok 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:53.250779+0000 osd.7 (osd.7) 396 : cluster [DBG] 3.e scrub starts 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:53.254241+0000 osd.7 (osd.7) 397 : cluster [DBG] 3.e scrub ok 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.097245+0000 osd.1 (osd.1) 378 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.170704+0000 osd.0 (osd.0) 381 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.173828+0000 osd.0 (osd.0) 382 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:57:54.844 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:52.904319+0000 mgr.y (mgr.14613) 750 : cluster [DBG] pgmap v887: 158 pgs: 14 creating+peering, 144 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:53.055465+0000 osd.5 (osd.5) 421 : cluster [DBG] 3.c scrub starts 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:53.059025+0000 osd.5 (osd.5) 422 : cluster [DBG] 3.c scrub ok 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:53.250779+0000 osd.7 (osd.7) 396 : cluster [DBG] 3.e scrub starts 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:53.254241+0000 osd.7 (osd.7) 397 : cluster [DBG] 3.e scrub ok 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.097245+0000 osd.1 (osd.1) 378 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.170704+0000 osd.0 (osd.0) 381 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:57:54.845 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:54 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.173828+0000 osd.0 (osd.0) 382 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:57:54.859 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:52.904319+0000 mgr.y (mgr.14613) 750 : cluster [DBG] pgmap v887: 158 pgs: 14 creating+peering, 144 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:53.055465+0000 osd.5 (osd.5) 421 : cluster [DBG] 3.c scrub starts 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:53.059025+0000 osd.5 (osd.5) 422 : cluster [DBG] 3.c scrub ok 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:53.250779+0000 osd.7 (osd.7) 396 : cluster [DBG] 3.e scrub starts 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:53.254241+0000 osd.7 (osd.7) 397 : cluster [DBG] 3.e scrub ok 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.097245+0000 osd.1 (osd.1) 378 : cluster [DBG] 3.1a scrub starts 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.170704+0000 osd.0 (osd.0) 381 : cluster [DBG] 3.10 scrub starts 2024-09-13T22:57:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:54 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.173828+0000 osd.0 (osd.0) 382 : cluster [DBG] 3.10 scrub ok 2024-09-13T22:57:54.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:54.989 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:54.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:55.093 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:55.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:55.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:55.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:55.265 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:55.265 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:55.265 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:57:55.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:55.382 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:55.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:55.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:55.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:55.645 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:55.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:55.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:55.870 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.100962+0000 osd.1 (osd.1) 379 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.686338+0000 osd.3 (osd.3) 367 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.689185+0000 osd.3 (osd.3) 368 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[29889]: cluster 2024-09-13T22:57:55.220365+0000 osd.0 (osd.0) 383 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[29889]: cluster 2024-09-13T22:57:55.222311+0000 osd.0 (osd.0) 384 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.100962+0000 osd.1 (osd.1) 379 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.686338+0000 osd.3 (osd.3) 367 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.689185+0000 osd.3 (osd.3) 368 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[22840]: cluster 2024-09-13T22:57:55.220365+0000 osd.0 (osd.0) 383 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:55 smithi178 bash[22840]: cluster 2024-09-13T22:57:55.222311+0000 osd.0 (osd.0) 384 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:57:55.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:55.905 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:55.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:55.907 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:57:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:57:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:55 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.100962+0000 osd.1 (osd.1) 379 : cluster [DBG] 3.1a scrub ok 2024-09-13T22:57:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:55 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.686338+0000 osd.3 (osd.3) 367 : cluster [DBG] 2.14 scrub starts 2024-09-13T22:57:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:55 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.689185+0000 osd.3 (osd.3) 368 : cluster [DBG] 2.14 scrub ok 2024-09-13T22:57:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:55 smithi192 bash[24608]: cluster 2024-09-13T22:57:55.220365+0000 osd.0 (osd.0) 383 : cluster [DBG] 6.16 scrub starts 2024-09-13T22:57:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:55 smithi192 bash[24608]: cluster 2024-09-13T22:57:55.222311+0000 osd.0 (osd.0) 384 : cluster [DBG] 6.16 scrub ok 2024-09-13T22:57:56.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:56.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:56.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:56.163 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:56.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:56.293 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:56.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:56.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:56.526 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:56.554 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:56.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:56.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[22840]: cluster 2024-09-13T22:57:54.905508+0000 mgr.y (mgr.14613) 751 : cluster [DBG] pgmap v888: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-13T22:57:56.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[22840]: cluster 2024-09-13T22:57:55.676075+0000 osd.3 (osd.3) 369 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:57:56.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[22840]: cluster 2024-09-13T22:57:55.678030+0000 osd.3 (osd.3) 370 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:57:56.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[22840]: cluster 2024-09-13T22:57:56.256414+0000 osd.0 (osd.0) 385 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:57:56.660 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[22840]: cluster 2024-09-13T22:57:56.258773+0000 osd.0 (osd.0) 386 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:57:56.684 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:56.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:56.813 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:56 smithi192 bash[24608]: cluster 2024-09-13T22:57:54.905508+0000 mgr.y (mgr.14613) 751 : cluster [DBG] pgmap v888: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-13T22:57:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:56 smithi192 bash[24608]: cluster 2024-09-13T22:57:55.676075+0000 osd.3 (osd.3) 369 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:57:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:56 smithi192 bash[24608]: cluster 2024-09-13T22:57:55.678030+0000 osd.3 (osd.3) 370 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:57:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:56 smithi192 bash[24608]: cluster 2024-09-13T22:57:56.256414+0000 osd.0 (osd.0) 385 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:57:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:56 smithi192 bash[24608]: cluster 2024-09-13T22:57:56.258773+0000 osd.0 (osd.0) 386 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:57:56.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:56.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[29889]: cluster 2024-09-13T22:57:54.905508+0000 mgr.y (mgr.14613) 751 : cluster [DBG] pgmap v888: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-13T22:57:56.928 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[29889]: cluster 2024-09-13T22:57:55.676075+0000 osd.3 (osd.3) 369 : cluster [DBG] 4.0 scrub starts 2024-09-13T22:57:56.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[29889]: cluster 2024-09-13T22:57:55.678030+0000 osd.3 (osd.3) 370 : cluster [DBG] 4.0 scrub ok 2024-09-13T22:57:56.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[29889]: cluster 2024-09-13T22:57:56.256414+0000 osd.0 (osd.0) 385 : cluster [DBG] 6.0 scrub starts 2024-09-13T22:57:56.929 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:56 smithi178 bash[29889]: cluster 2024-09-13T22:57:56.258773+0000 osd.0 (osd.0) 386 : cluster [DBG] 6.0 scrub ok 2024-09-13T22:57:56.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:56.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:57.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:57.071 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:57.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:57.202 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:57.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:57.333 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:57.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:57.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:57.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:57.595 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:57.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:57.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:57 smithi192 bash[24608]: audit 2024-09-13T22:57:55.864541+0000 mgr.y (mgr.14613) 752 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:57.724 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:57:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:57:57.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:57.852 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:57.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:57 smithi178 bash[29889]: audit 2024-09-13T22:57:55.864541+0000 mgr.y (mgr.14613) 752 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:57.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:57 smithi178 bash[22840]: audit 2024-09-13T22:57:55.864541+0000 mgr.y (mgr.14613) 752 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:57:57.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:57.983 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:57.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:58.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:58.113 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:58.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:58.244 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:57:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:57:58.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:58.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:58.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:58.500 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:58.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:58.622 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:58.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:56.906418+0000 mgr.y (mgr.14613) 753 : cluster [DBG] pgmap v889: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-13T22:57:58.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:57.047284+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:57:58.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:57.050031+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:57.112814+0000 osd.1 (osd.1) 380 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:57.114942+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:58.247580+0000 osd.0 (osd.0) 387 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[22840]: cluster 2024-09-13T22:57:58.251931+0000 osd.0 (osd.0) 388 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:56.906418+0000 mgr.y (mgr.14613) 753 : cluster [DBG] pgmap v889: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:57.047284+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:57.050031+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:57.112814+0000 osd.1 (osd.1) 380 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:57.114942+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:58.247580+0000 osd.0 (osd.0) 387 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:57:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:58 smithi178 bash[29889]: cluster 2024-09-13T22:57:58.251931+0000 osd.0 (osd.0) 388 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:57:58.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:58.753 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:58.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:58.881 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:58.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:58.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:56.906418+0000 mgr.y (mgr.14613) 753 : cluster [DBG] pgmap v889: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:57.047284+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.1 scrub starts 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:57.050031+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub ok 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:57.112814+0000 osd.1 (osd.1) 380 : cluster [DBG] 5.1d scrub starts 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:57.114942+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.1d scrub ok 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:58.247580+0000 osd.0 (osd.0) 387 : cluster [DBG] 3.6 scrub starts 2024-09-13T22:57:58.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:58 smithi192 bash[24608]: cluster 2024-09-13T22:57:58.251931+0000 osd.0 (osd.0) 388 : cluster [DBG] 3.6 scrub ok 2024-09-13T22:57:59.012 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:57:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:57:59.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:59.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:59.168 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:57:59.169 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:57:59.186 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:57:59.186 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:57:59.204 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:57:59.204 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:57:59.221 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:57:59.221 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:57:59.242 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:57:59.242 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:57:59.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:59.263 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:57:59.263 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:57:59.264 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:57:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:57:59.284 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:57:59.284 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:57:59.305 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:57:59.305 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:57:59.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:59.390 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:57:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:57:59.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:59.519 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:57:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:57:59.622 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:59.650 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:59.673 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:57:59.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:59.762 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[29889]: audit 2024-09-13T22:57:58.779196+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.178:0/3927094615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:59.762 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[29889]: audit 2024-09-13T22:57:58.779778+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:59.762 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[29889]: audit 2024-09-13T22:57:59.151814+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.178:0/2542985601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:59.762 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[22840]: audit 2024-09-13T22:57:58.779196+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.178:0/3927094615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:59.762 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[22840]: audit 2024-09-13T22:57:58.779778+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:57:59.762 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[22840]: audit 2024-09-13T22:57:59.151814+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.178:0/2542985601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:57:59.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:57:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:57:59.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:59.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:57:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:57:59.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:57:59.917 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:57:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:57:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:58:00.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:00.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:00.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:00.148 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:59 smithi192 bash[24608]: audit 2024-09-13T22:57:58.779196+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.178:0/3927094615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:00.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:59 smithi192 bash[24608]: audit 2024-09-13T22:57:58.779778+0000 mon.a (mon.0) 1651 : audit [INF] from='client.25781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:00.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:57:59 smithi192 bash[24608]: audit 2024-09-13T22:57:59.151814+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.178:0/2542985601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:00.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:00.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:00.293 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:00.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:00.424 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:00.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:00.555 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:00.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:00.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:00.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:00.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[29889]: cluster 2024-09-13T22:57:58.907332+0000 mgr.y (mgr.14613) 754 : cluster [DBG] pgmap v890: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-13T22:58:00.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[29889]: cluster 2024-09-13T22:57:59.668090+0000 mon.a (mon.0) 1652 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:58:00.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[29889]: audit 2024-09-13T22:57:59.669578+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:58:00.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[29889]: cluster 2024-09-13T22:57:59.673206+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e400: 8 total, 8 up, 6 in 2024-09-13T22:58:00.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[29889]: cluster 2024-09-13T22:57:59.730075+0000 osd.3 (osd.3) 371 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:58:00.801 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[29889]: cluster 2024-09-13T22:57:59.732683+0000 osd.3 (osd.3) 372 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:58:00.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[22840]: cluster 2024-09-13T22:57:58.907332+0000 mgr.y (mgr.14613) 754 : cluster [DBG] pgmap v890: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-13T22:58:00.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[22840]: cluster 2024-09-13T22:57:59.668090+0000 mon.a (mon.0) 1652 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:58:00.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[22840]: audit 2024-09-13T22:57:59.669578+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:58:00.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[22840]: cluster 2024-09-13T22:57:59.673206+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e400: 8 total, 8 up, 6 in 2024-09-13T22:58:00.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[22840]: cluster 2024-09-13T22:57:59.730075+0000 osd.3 (osd.3) 371 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:58:00.802 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:00 smithi178 bash[22840]: cluster 2024-09-13T22:57:59.732683+0000 osd.3 (osd.3) 372 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:58:00.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:00.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:00.949 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:00.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:01.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:01.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:00 smithi192 bash[24608]: cluster 2024-09-13T22:57:58.907332+0000 mgr.y (mgr.14613) 754 : cluster [DBG] pgmap v890: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-13T22:58:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:00 smithi192 bash[24608]: cluster 2024-09-13T22:57:59.668090+0000 mon.a (mon.0) 1652 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:58:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:00 smithi192 bash[24608]: audit 2024-09-13T22:57:59.669578+0000 mon.a (mon.0) 1653 : audit [INF] from='client.25781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:58:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:00 smithi192 bash[24608]: cluster 2024-09-13T22:57:59.673206+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e400: 8 total, 8 up, 6 in 2024-09-13T22:58:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:00 smithi192 bash[24608]: cluster 2024-09-13T22:57:59.730075+0000 osd.3 (osd.3) 371 : cluster [DBG] 6.13 scrub starts 2024-09-13T22:58:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:00 smithi192 bash[24608]: cluster 2024-09-13T22:57:59.732683+0000 osd.3 (osd.3) 372 : cluster [DBG] 6.13 scrub ok 2024-09-13T22:58:01.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:01.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:01.313 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:01.339 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:01.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:01.467 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:01.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:01.598 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:01.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:01.728 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:01.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:01.858 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.017522+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.019951+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.108462+0000 osd.4 (osd.4) 274 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.108591+0000 osd.4 (osd.4) 275 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.145448+0000 osd.1 (osd.1) 382 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.147361+0000 osd.1 (osd.1) 383 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:58:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.694274+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e401: 8 total, 8 up, 6 in 2024-09-13T22:58:01.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:01.249179+0000 osd.0 (osd.0) 389 : cluster [DBG] 2.f scrub starts 2024-09-13T22:58:01.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:01 smithi192 bash[24608]: cluster 2024-09-13T22:58:01.252307+0000 osd.0 (osd.0) 390 : cluster [DBG] 2.f scrub ok 2024-09-13T22:58:01.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.017522+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.019951+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.108462+0000 osd.4 (osd.4) 274 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.108591+0000 osd.4 (osd.4) 275 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.145448+0000 osd.1 (osd.1) 382 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.147361+0000 osd.1 (osd.1) 383 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.694274+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e401: 8 total, 8 up, 6 in 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:01.249179+0000 osd.0 (osd.0) 389 : cluster [DBG] 2.f scrub starts 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[29889]: cluster 2024-09-13T22:58:01.252307+0000 osd.0 (osd.0) 390 : cluster [DBG] 2.f scrub ok 2024-09-13T22:58:01.973 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.017522+0000 osd.5 (osd.5) 425 : cluster [DBG] 2.16 scrub starts 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.019951+0000 osd.5 (osd.5) 426 : cluster [DBG] 2.16 scrub ok 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.108462+0000 osd.4 (osd.4) 274 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.108591+0000 osd.4 (osd.4) 275 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.145448+0000 osd.1 (osd.1) 382 : cluster [DBG] 2.15 scrub starts 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.147361+0000 osd.1 (osd.1) 383 : cluster [DBG] 2.15 scrub ok 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.694274+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e401: 8 total, 8 up, 6 in 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:01.249179+0000 osd.0 (osd.0) 389 : cluster [DBG] 2.f scrub starts 2024-09-13T22:58:01.974 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:01 smithi178 bash[22840]: cluster 2024-09-13T22:58:01.252307+0000 osd.0 (osd.0) 390 : cluster [DBG] 2.f scrub ok 2024-09-13T22:58:01.987 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:01.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:02.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:02.117 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:02.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:02.245 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:02.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:02.376 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:02.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:02.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:02.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:02.632 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:02.654 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 20, 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 600, 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:50544 9.4 9:292db84a:::benchmark_data_smithi178_96665_object50543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:02.919359+0000", 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 479.732080709, 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3797643270000002, 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.655 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 50544 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.919359+0000", 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.919359+0000", 2024-09-13T22:58:02.656 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.919360+0000", 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.44e-07 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.919380+0000", 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0117000000000001e-05 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.657 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.919386+0000", 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8159999999999999e-06 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.919391+0000", 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.7910000000000004e-06 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.932537+0000", 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013146038000000001 2024-09-13T22:58:02.658 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.932671+0000", 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013410599999999999 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,3", 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.932814+0000", 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000142686 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938439+0000", 2024-09-13T22:58:02.659 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0056249489999999997 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.979204+0000", 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.040765352999999997 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.979215+0000", 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1345000000000001e-05 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.660 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949060+0000", 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.96984498500000005 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949075+0000", 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5044e-05 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.661 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.298973+0000", 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.349897635 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.299124+0000", 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00015081799999999999 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:50568 9.4 9:2b2033eb:::benchmark_data_smithi178_96665_object50567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:02.938343+0000", 2024-09-13T22:58:02.662 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 479.713096785, 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3967619230000001, 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 50568 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.663 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938343+0000", 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938343+0000", 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938345+0000", 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9439999999999999e-06 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.664 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938388+0000", 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2815000000000003e-05 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938399+0000", 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.136e-05 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938407+0000", 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6769999999999995e-06 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.665 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.938467+0000", 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9506000000000003e-05 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.963792+0000", 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.025325737000000001 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,3", 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.963964+0000", 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00017193500000000001 2024-09-13T22:58:02.666 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.979248+0000", 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.015283346 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.979256+0000", 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9109999999999993e-06 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.667 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949116+0000", 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.96986030000000001 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949134+0000", 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7795000000000002e-05 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.299302+0000", 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3501687950000001 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for degraded object", 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.299891+0000", 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00058815600000000005 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.335005+0000", 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.035114021000000002 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.669 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.335105+0000", 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010062500000000001 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:50570 9.4 9:2c27af6c:::benchmark_data_smithi178_96665_object50569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:02.940788+0000", 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 479.71065225299998, 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3828401260000001, 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.670 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 50570 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.940788+0000", 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.940788+0000", 2024-09-13T22:58:02.671 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.940791+0000", 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.937e-06 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.940821+0000", 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.976e-05 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.672 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.940831+0000", 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0434999999999999e-05 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.940840+0000", 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3049999999999998e-06 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.979280+0000", 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.038439372999999999 2024-09-13T22:58:02.673 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:02.979286+0000", 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2219999999999999e-06 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949184+0000", 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.96989835400000002 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949200+0000", 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6147e-05 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.300035+0000", 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.35083452 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.300194+0000", 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00015872 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 3,4", 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.300383+0000", 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000189421 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.304608+0000", 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0042254099999999998 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.310863+0000", 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.006254374 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.321357+0000", 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010494033999999999 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.321434+0000", 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6743999999999996e-05 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.677 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.323628+0000", 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0021943700000000002 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.678 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:50619 9.4 9:297a7e19:::benchmark_data_smithi178_96665_object50618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:03.011712+0000", 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 479.639728338, 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3181140600000001, 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 50619 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011712+0000", 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011712+0000", 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011714+0000", 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8589999999999999e-06 2024-09-13T22:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011736+0000", 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2308999999999999e-05 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011747+0000", 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0594000000000001e-05 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.681 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011754+0000", 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4560000000000003e-06 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011809+0000", 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5114999999999997e-05 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.011818+0000", 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4479999999999994e-06 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.682 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949245+0000", 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.93742713 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949258+0000", 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2833999999999999e-05 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.302463+0000", 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3532056610000001 2024-09-13T22:58:02.683 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.302609+0000", 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000145675 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 3,4", 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.302788+0000", 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000179285 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.684 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.306579+0000", 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.003790931 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.312063+0000", 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0054840210000000004 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.323750+0000", 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011687272 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.685 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.323811+0000", 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0757000000000001e-05 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.329826+0000", 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0060147129999999997 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.686 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:50629 9.4 9:240a6a89:::benchmark_data_smithi178_96665_object50628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:03.021199+0000", 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 479.63024080700001, 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3111162699999999, 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 50629 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.687 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021199+0000", 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021199+0000", 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021201+0000", 2024-09-13T22:58:02.688 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8810000000000001e-06 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021224+0000", 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3283000000000001e-05 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021241+0000", 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6254000000000001e-05 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.689 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021249+0000", 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3159999999999997e-06 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021308+0000", 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9246999999999997e-05 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.021318+0000", 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0169999999999999e-05 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.690 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949281+0000", 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.92796220500000004 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:03.949288+0000", 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2439999999999996e-06 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.304647+0000", 2024-09-13T22:58:02.691 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3553594229999999 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.304843+0000", 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000195525 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 3,4", 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.305060+0000", 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000216861 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.692 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.306696+0000", 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0016365539999999999 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.321295+0000", 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014599187 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.329895+0000", 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0086000150000000008 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.693 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.329938+0000", 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2697000000000001e-05 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:05.332316+0000", 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0023774080000000001 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:81919 9.4 9:233b298f:::benchmark_data_smithi178_96665_object81918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:58:02.694 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:39.242746+0000", 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 443.40869411599999, 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0315692740000002, 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 81919 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.242746+0000", 2024-09-13T22:58:02.695 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.242746+0000", 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.242747+0000", 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4869999999999999e-06 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.696 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243206+0000", 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00045815800000000001 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243221+0000", 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5811000000000002e-05 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243233+0000", 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1712999999999999e-05 2024-09-13T22:58:02.697 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.251840+0000", 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0086066960000000005 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op must wait for map", 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.251862+0000", 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1642e-05 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.412170+0000", 2024-09-13T22:58:02.698 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.16030825800000001 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.412182+0000", 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2296999999999999e-05 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:40.237922+0000", 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.82573996000000005 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.699 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:40.237936+0000", 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4088e-05 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.274225+0000", 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0362893129999999 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.274315+0000", 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.9851000000000004e-05 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.700 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:81934 9.4 9:27316e4a:::benchmark_data_smithi178_96665_object81933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:39.243252+0000", 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 443.40818799599998, 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.042365642, 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 81934 2024-09-13T22:58:02.701 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243252+0000", 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243252+0000", 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.702 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243253+0000", 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3500000000000005e-07 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243273+0000", 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9774000000000001e-05 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.703 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243281+0000", 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0860000000000002e-06 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243287+0000", 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.728e-06 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.412207+0000", 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.16892005800000001 2024-09-13T22:58:02.704 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.412216+0000", 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3219999999999999e-06 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:40.237962+0000", 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.82574615200000001 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.705 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:40.237969+0000", 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.0010000000000004e-06 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.275012+0000", 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0370430820000001 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for degraded object", 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.275415+0000", 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00040241000000000002 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.706 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.285503+0000", 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010088740000000001 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.285618+0000", 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000114354 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:81935 9.4 9:25d68cf4:::benchmark_data_smithi178_96665_object81934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-09-13T22:58:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:39.243297+0000", 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 443.40814298399999, 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0478883830000001, 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 81935 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.708 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243297+0000", 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243297+0000", 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243298+0000", 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.8800000000000002e-07 2024-09-13T22:58:02.709 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243317+0000", 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9524000000000001e-05 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243327+0000", 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.234e-06 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.243332+0000", 2024-09-13T22:58:02.710 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.9840000000000001e-06 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.412242+0000", 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.168910219 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:39.412250+0000", 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2199999999999992e-06 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.711 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:40.237991+0000", 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.82574090899999997 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:40.237998+0000", 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2980000000000003e-06 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.275488+0000", 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0374894189999999 2024-09-13T22:58:02.712 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for degraded object", 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.275866+0000", 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000378291 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.291067+0000", 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.015200949 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:41.291186+0000", 2024-09-13T22:58:02.713 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011864800000000001 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:82303 9.16 9:6dcdbdf0:::benchmark_data_smithi178_96665_object82302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:42.242468+0000", 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 440.40897240700002, 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0853284759999999, 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.714 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 82303 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242468+0000", 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242468+0000", 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.715 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242469+0000", 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.47e-07 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242509+0000", 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0632e-05 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242516+0000", 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.0489999999999998e-06 2024-09-13T22:58:02.716 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242522+0000", 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.7389999999999996e-06 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242853+0000", 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00033113800000000003 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.717 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting_for_map not empty", 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242857+0000", 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2649999999999998e-06 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.249363+0000", 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0065058800000000003 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.249368+0000", 2024-09-13T22:58:02.718 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6e-06 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.317928+0000", 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.068560621 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.318091+0000", 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000162604 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.719 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 2,4,5", 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.318290+0000", 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00019859599999999999 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.320216+0000", 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0019266890000000001 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.320819+0000", 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000602696 2024-09-13T22:58:02.720 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.320919+0000", 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010018699999999999 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.327623+0000", 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0067034750000000004 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.721 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.327688+0000", 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.5559e-05 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.327796+0000", 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000107899 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:82309 9.12 9:4b1cdfd0:::benchmark_data_smithi178_96665_object82308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:42.242529+0000", 2024-09-13T22:58:02.722 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 440.40891085499999, 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0972642130000001, 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 82309 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242529+0000", 2024-09-13T22:58:02.723 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242529+0000", 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242530+0000", 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.8400000000000004e-07 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.724 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242568+0000", 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.8302e-05 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242575+0000", 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.579e-06 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242580+0000", 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3059999999999997e-06 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.725 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.244793+0000", 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0022131939999999999 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.244803+0000", 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4790000000000002e-06 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.329387+0000", 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.084584293 2024-09-13T22:58:02.726 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.329527+0000", 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013970399999999999 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,2,3,5", 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.329826+0000", 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000299186 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.727 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339477+0000", 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0096514429999999991 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339491+0000", 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3696000000000001e-05 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339533+0000", 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2027000000000003e-05 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.728 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339574+0000", 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0543000000000001e-05 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339613+0000", 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9749000000000002e-05 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339667+0000", 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3624e-05 2024-09-13T22:58:02.729 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339793+0000", 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012640399999999999 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:82312 9.12 9:4d3a859e:::benchmark_data_smithi178_96665_object82311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:42.242587+0000", 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 440.40885311900001, 2024-09-13T22:58:02.730 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1062968639999999, 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 82312 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242587+0000", 2024-09-13T22:58:02.731 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242587+0000", 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242588+0000", 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4900000000000004e-07 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.732 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242630+0000", 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2177e-05 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242637+0000", 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7780000000000002e-06 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242643+0000", 2024-09-13T22:58:02.733 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6239999999999999e-06 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.244826+0000", 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002183506 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.244832+0000", 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9680000000000003e-06 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.734 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.331843+0000", 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0870114040000001 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.331980+0000", 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013664699999999999 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,2,3,5", 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.332280+0000", 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00030009900000000001 2024-09-13T22:58:02.735 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339497+0000", 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0072170890000000003 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339835+0000", 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00033799500000000001 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.736 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.339875+0000", 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9563000000000001e-05 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.340794+0000", 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00091894899999999996 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.348683+0000", 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0078891529999999994 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.737 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.348748+0000", 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4675e-05 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.348884+0000", 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013628800000000001 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.738 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15408.0:82317 9.12 9:4a7982fc:::benchmark_data_smithi178_96665_object82316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:50:42.242650+0000", 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 440.408789877, 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.109866995, 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15408", 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/723173568", 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 82317 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.739 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242650+0000", 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242650+0000", 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242651+0000", 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.85e-07 2024-09-13T22:58:02.740 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242692+0000", 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0676999999999997e-05 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242699+0000", 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9210000000000003e-06 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.741 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.242706+0000", 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2570000000000001e-06 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.244854+0000", 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00214832 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:42.244860+0000", 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8599999999999998e-06 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.742 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.337750+0000", 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0928904189999999 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.337845+0000", 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4712999999999999e-05 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,2,3,5", 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.338109+0000", 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00026415800000000001 2024-09-13T22:58:02.743 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.340819+0000", 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0027099960000000001 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.340839+0000", 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0089e-05 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.744 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.344305+0000", 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0034658839999999998 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.346131+0000", 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001825758 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.352329+0000", 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0061982210000000003 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.745 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.352386+0000", 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6913999999999998e-05 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:50:43.352517+0000", 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000131023 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.746 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160571 10.b 10:d5b32143:::benchmark_data_smithi178_111521_object49835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.246099+0000", 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 123.405341263, 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1318281890000002, 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160571 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.747 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246099+0000", 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246099+0000", 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246102+0000", 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.8140000000000001e-06 2024-09-13T22:58:02.748 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246110+0000", 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4379999999999995e-06 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246123+0000", 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2533e-05 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246131+0000", 2024-09-13T22:58:02.749 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3389999999999993e-06 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246203+0000", 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1680000000000005e-05 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.246215+0000", 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2717e-05 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.750 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.207461+0000", 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.96124581499999995 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.207479+0000", 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8119999999999999e-05 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.353721+0000", 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1462419850000001 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.751 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.355472+0000", 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0017504409999999999 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.355692+0000", 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000219953 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.363328+0000", 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0076364969999999999 2024-09-13T22:58:02.752 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.373636+0000", 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010308207999999999 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.373700+0000", 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3678999999999998e-05 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.377772+0000", 2024-09-13T22:58:02.753 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0040715940000000004 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.377847+0000", 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5820000000000003e-05 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.377927+0000", 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9555999999999995e-05 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160587 10.0 10:0f88e70a:::benchmark_data_smithi178_111521_object49851:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:00.191753+0000", 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 122.459687346, 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.166676584, 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160587 2024-09-13T22:58:02.755 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191753+0000", 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191753+0000", 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.756 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191755+0000", 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.03e-06 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191760+0000", 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5260000000000001e-06 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191772+0000", 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1585e-05 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.757 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191780+0000", 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2239999999999995e-06 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191844+0000", 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3751000000000007e-05 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op must wait for map", 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191855+0000", 2024-09-13T22:58:02.758 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1368e-05 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.197503+0000", 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0056474250000000002 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.197514+0000", 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1018e-05 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.759 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.340383+0000", 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.142869709 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.340547+0000", 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016367700000000001 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,4,6,7", 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.340991+0000", 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00044380599999999999 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.352707+0000", 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011716374 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.352761+0000", 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3621e-05 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.353455+0000", 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00069394499999999996 2024-09-13T22:58:02.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358082+0000", 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0046267720000000003 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358236+0000", 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00015400400000000001 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358303+0000", 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7643000000000003e-05 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358429+0000", 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012610599999999999 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160598 10.0 10:02efaf00:::benchmark_data_smithi178_111521_object49862:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:00.191797+0000", 2024-09-13T22:58:02.763 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 122.459642961, 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1668679120000001, 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160598 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.764 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191797+0000", 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191797+0000", 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191798+0000", 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9100000000000003e-07 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191802+0000", 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.5880000000000002e-06 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191808+0000", 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7959999999999998e-06 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191814+0000", 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9780000000000002e-06 2024-09-13T22:58:02.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191909+0000", 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4858999999999994e-05 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting_for_map not empty", 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.191917+0000", 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5490000000000001e-06 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.197543+0000", 2024-09-13T22:58:02.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0056262880000000001 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.197553+0000", 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6989999999999998e-06 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.350310+0000", 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1527569090000001 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.350442+0000", 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013277800000000001 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,4,6,7", 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.350581+0000", 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013910300000000001 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.353001+0000", 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0024195589999999999 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.353503+0000", 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00050170300000000004 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.353581+0000", 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8214999999999996e-05 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358158+0000", 2024-09-13T22:58:02.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0045767469999999999 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358479+0000", 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000321744 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358538+0000", 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8820999999999998e-05 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.358665+0000", 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000126785 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:177093 10.b 10:d885f522:::benchmark_data_smithi178_111521_object66357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:21.216687+0000", 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 101.434753273, 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0883204120000001, 2024-09-13T22:58:02.772 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 177093 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.216687+0000", 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.216687+0000", 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.216688+0000", 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4099999999999998e-07 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.216691+0000", 2024-09-13T22:58:02.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.1870000000000001e-06 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.216696+0000", 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6629999999999998e-06 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.216701+0000", 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.011e-06 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.223513+0000", 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0068118199999999997 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.223655+0000", 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014166 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.223799+0000", 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014366699999999999 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.776 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.233715+0000", 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0099164509999999997 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:21.233722+0000", 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2570000000000001e-06 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.276348+0000", 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.042625876 2024-09-13T22:58:02.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for missing object", 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.276485+0000", 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013708599999999999 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.304942+0000", 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.028456364000000001 2024-09-13T22:58:02.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.305007+0000", 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.5629000000000005e-05 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:196831 10.0 10:01463fbb:::benchmark_data_smithi178_111521_object86095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:50.122273+0000", 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 72.529167473000001, 2024-09-13T22:58:02.779 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.093920402, 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 196831 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122273+0000", 2024-09-13T22:58:02.780 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122273+0000", 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122274+0000", 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7260000000000001e-06 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.781 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122279+0000", 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.3239999999999996e-06 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122287+0000", 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4149999999999999e-06 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122292+0000", 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.2480000000000004e-06 2024-09-13T22:58:02.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122351+0000", 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8513e-05 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122482+0000", 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013077299999999999 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,4,6,7", 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.122603+0000", 2024-09-13T22:58:02.783 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000121283 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.125021+0000", 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002418374 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.132423+0000", 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0074014950000000001 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.784 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.132434+0000", 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1199999999999999e-05 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.216075+0000", 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.083640588 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.216193+0000", 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011846299999999999 2024-09-13T22:58:02.785 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:196836 10.0 10:0443411a:::benchmark_data_smithi178_111521_object86100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:50.130152+0000", 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 72.521288554999998, 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1049027149999999, 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.786 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 196836 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.130152+0000", 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.130152+0000", 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.787 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.130153+0000", 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8360000000000001e-06 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.130158+0000", 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0599999999999998e-06 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.130169+0000", 2024-09-13T22:58:02.788 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0358e-05 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.130177+0000", 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7209999999999994e-06 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.132459+0000", 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002282699 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.789 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.132467+0000", 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6480000000000003e-06 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.217037+0000", 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0845702989999999 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.217364+0000", 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00032683799999999998 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.790 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,7", 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.217646+0000", 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00028188600000000002 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.223315+0000", 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0056688950000000002 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.225166+0000", 2024-09-13T22:58:02.791 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0018515770000000001 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.234949+0000", 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0097823849999999993 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.235010+0000", 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1437000000000001e-05 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.792 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.235054+0000", 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.4076000000000003e-05 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:196851 10.0 10:04126e6b:::benchmark_data_smithi178_111521_object86115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:50.151340+0000", 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 72.500100046, 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.092698811, 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.793 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 196851 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151340+0000", 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.794 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151340+0000", 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151342+0000", 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.012e-06 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151349+0000", 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6259999999999997e-06 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.795 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151373+0000", 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3853999999999999e-05 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151382+0000", 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7669999999999998e-06 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151441+0000", 2024-09-13T22:58:02.796 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8619999999999998e-05 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.151450+0000", 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3570000000000004e-06 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.220925+0000", 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.069475191 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.797 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.221117+0000", 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00019115899999999999 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,7", 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.221318+0000", 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00020105200000000001 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.234775+0000", 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.013457363 2024-09-13T22:58:02.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.235130+0000", 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00035493200000000002 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.243905+0000", 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0087747999999999993 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.799 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.243974+0000", 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.8918999999999995e-05 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.244039+0000", 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.5159000000000004e-05 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:196869 10.0 10:099084de:::benchmark_data_smithi178_111521_object86133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:50.166013+0000", 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 72.485427534999999, 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0856732250000001, 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 196869 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.801 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166013+0000", 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166013+0000", 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166014+0000", 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8589999999999999e-06 2024-09-13T22:58:02.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166019+0000", 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.177e-06 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166028+0000", 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.0909999999999999e-06 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:58:02.803 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166033+0000", 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8130000000000003e-06 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166087+0000", 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.4218999999999997e-05 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:50.166097+0000", 2024-09-13T22:58:02.804 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0742e-05 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.225286+0000", 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0591888519999999 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.225610+0000", 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00032370899999999999 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,7", 2024-09-13T22:58:02.805 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.225929+0000", 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00031926600000000002 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.235076+0000", 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0091471750000000004 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.244175+0000", 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0090983920000000003 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.806 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.251553+0000", 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0073784469999999998 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.251626+0000", 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2455999999999999e-05 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:51.251686+0000", 2024-09-13T22:58:02.807 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0027e-05 2024-09-13T22:58:02.808 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.808 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.808 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.808 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:58:02.808 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:58:02.808 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:58:02.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:02.817 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:02 smithi178 bash[22840]: cluster 2024-09-13T22:58:00.907781+0000 mgr.y (mgr.14613) 755 : cluster [DBG] pgmap v893: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-13T22:58:02.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:02 smithi178 bash[22840]: cluster 2024-09-13T22:58:02.219153+0000 osd.7 (osd.7) 398 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:58:02.818 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:02 smithi178 bash[22840]: cluster 2024-09-13T22:58:02.221644+0000 osd.7 (osd.7) 399 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:58:02.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:02 smithi178 bash[29889]: cluster 2024-09-13T22:58:00.907781+0000 mgr.y (mgr.14613) 755 : cluster [DBG] pgmap v893: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-13T22:58:02.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:02 smithi178 bash[29889]: cluster 2024-09-13T22:58:02.219153+0000 osd.7 (osd.7) 398 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:58:02.818 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:02 smithi178 bash[29889]: cluster 2024-09-13T22:58:02.221644+0000 osd.7 (osd.7) 399 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:58:02.832 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:02.837 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:58:02.838 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:58:02.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:02.954 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:02.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:03.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:03.082 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:03.122 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-13T22:58:03.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:02 smithi192 bash[24608]: cluster 2024-09-13T22:58:00.907781+0000 mgr.y (mgr.14613) 755 : cluster [DBG] pgmap v893: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-13T22:58:03.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:02 smithi192 bash[24608]: cluster 2024-09-13T22:58:02.219153+0000 osd.7 (osd.7) 398 : cluster [DBG] 1.9 scrub starts 2024-09-13T22:58:03.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:02 smithi192 bash[24608]: cluster 2024-09-13T22:58:02.221644+0000 osd.7 (osd.7) 399 : cluster [DBG] 1.9 scrub ok 2024-09-13T22:58:03.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:03.203 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:03.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:03.313 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:58:03.335 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:03.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:03.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:03.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:03.589 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:03.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:03.715 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:03.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:03.828 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[22840]: cluster 2024-09-13T22:58:02.748105+0000 osd.3 (osd.3) 373 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[22840]: cluster 2024-09-13T22:58:02.750199+0000 osd.3 (osd.3) 374 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[22840]: audit 2024-09-13T22:58:02.879305+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.178:0/1376471549' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[22840]: audit 2024-09-13T22:58:02.879607+0000 mon.a (mon.0) 1656 : audit [INF] from='client.25799 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[22840]: audit 2024-09-13T22:58:03.120096+0000 mon.a (mon.0) 1657 : audit [INF] from='client.25799 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[22840]: cluster 2024-09-13T22:58:03.123056+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e402: 8 total, 8 up, 6 in 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[29889]: cluster 2024-09-13T22:58:02.748105+0000 osd.3 (osd.3) 373 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[29889]: cluster 2024-09-13T22:58:02.750199+0000 osd.3 (osd.3) 374 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:58:03.829 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[29889]: audit 2024-09-13T22:58:02.879305+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.178:0/1376471549' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:58:03.830 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[29889]: audit 2024-09-13T22:58:02.879607+0000 mon.a (mon.0) 1656 : audit [INF] from='client.25799 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:58:03.830 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[29889]: audit 2024-09-13T22:58:03.120096+0000 mon.a (mon.0) 1657 : audit [INF] from='client.25799 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:58:03.830 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:03 smithi178 bash[29889]: cluster 2024-09-13T22:58:03.123056+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e402: 8 total, 8 up, 6 in 2024-09-13T22:58:03.842 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:03.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:03.971 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:03.972 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:03 smithi192 bash[24608]: cluster 2024-09-13T22:58:02.748105+0000 osd.3 (osd.3) 373 : cluster [DBG] 5.0 scrub starts 2024-09-13T22:58:03.972 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:03 smithi192 bash[24608]: cluster 2024-09-13T22:58:02.750199+0000 osd.3 (osd.3) 374 : cluster [DBG] 5.0 scrub ok 2024-09-13T22:58:03.972 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:03 smithi192 bash[24608]: audit 2024-09-13T22:58:02.879305+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.178:0/1376471549' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:58:03.972 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:03 smithi192 bash[24608]: audit 2024-09-13T22:58:02.879607+0000 mon.a (mon.0) 1656 : audit [INF] from='client.25799 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T22:58:03.972 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:03 smithi192 bash[24608]: audit 2024-09-13T22:58:03.120096+0000 mon.a (mon.0) 1657 : audit [INF] from='client.25799 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T22:58:03.972 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:03 smithi192 bash[24608]: cluster 2024-09-13T22:58:03.123056+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e402: 8 total, 8 up, 6 in 2024-09-13T22:58:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:03.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:04.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:04.099 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:04.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:04.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:04.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:58:04.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:04.360 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:04.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:04.484 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:04.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T22:58:04.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:04.613 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:04.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:04.744 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:04.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:04.876 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:04.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: audit 2024-09-13T22:58:02.819282+0000 mgr.y (mgr.14613) 756 : audit [DBG] from='client.15894 v1:172.21.15.178:0/2026648120' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: cluster 2024-09-13T22:58:02.908394+0000 mgr.y (mgr.14613) 757 : cluster [DBG] pgmap v894: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: cluster 2024-09-13T22:58:03.018266+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.a scrub starts 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: cluster 2024-09-13T22:58:03.020197+0000 osd.5 (osd.5) 428 : cluster [DBG] 6.a scrub ok 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: cluster 2024-09-13T22:58:03.161504+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: cluster 2024-09-13T22:58:03.162916+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:58:04.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.240303+0000 osd.7 (osd.7) 400 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: audit 2024-09-13T22:58:04.349946+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[29889]: audit 2024-09-13T22:58:04.354676+0000 mon.b (mon.1) 266 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: audit 2024-09-13T22:58:02.819282+0000 mgr.y (mgr.14613) 756 : audit [DBG] from='client.15894 v1:172.21.15.178:0/2026648120' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: cluster 2024-09-13T22:58:02.908394+0000 mgr.y (mgr.14613) 757 : cluster [DBG] pgmap v894: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: cluster 2024-09-13T22:58:03.018266+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.a scrub starts 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: cluster 2024-09-13T22:58:03.020197+0000 osd.5 (osd.5) 428 : cluster [DBG] 6.a scrub ok 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: cluster 2024-09-13T22:58:03.161504+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: cluster 2024-09-13T22:58:03.162916+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.240303+0000 osd.7 (osd.7) 400 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:58:04.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: audit 2024-09-13T22:58:04.349946+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:04.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:04 smithi178 bash[22840]: audit 2024-09-13T22:58:04.354676+0000 mon.b (mon.1) 266 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:04.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:05.005 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:05.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:05.120 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: audit 2024-09-13T22:58:02.819282+0000 mgr.y (mgr.14613) 756 : audit [DBG] from='client.15894 v1:172.21.15.178:0/2026648120' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:58:05.120 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: cluster 2024-09-13T22:58:02.908394+0000 mgr.y (mgr.14613) 757 : cluster [DBG] pgmap v894: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T22:58:05.120 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: cluster 2024-09-13T22:58:03.018266+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.a scrub starts 2024-09-13T22:58:05.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: cluster 2024-09-13T22:58:03.020197+0000 osd.5 (osd.5) 428 : cluster [DBG] 6.a scrub ok 2024-09-13T22:58:05.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: cluster 2024-09-13T22:58:03.161504+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.12 scrub starts 2024-09-13T22:58:05.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: cluster 2024-09-13T22:58:03.162916+0000 osd.1 (osd.1) 385 : cluster [DBG] 5.12 scrub ok 2024-09-13T22:58:05.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.240303+0000 osd.7 (osd.7) 400 : cluster [DBG] 2.7 scrub starts 2024-09-13T22:58:05.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: audit 2024-09-13T22:58:04.349946+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:05.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:04 smithi192 bash[24608]: audit 2024-09-13T22:58:04.354676+0000 mon.b (mon.1) 266 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:05.136 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:05.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:05.265 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:05.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:05.396 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:05.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:05.529 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:05.570 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":893,"stamp":"2024-09-13T22:58:00.907538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2784,"num_read_kb":2536,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":7524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2617,"ondisk_log_size":2617,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":3685780,"kb_used_data":7092,"kb_used_omap":9,"kb_used_meta":3678582,"kb_avail":558785132,"statfs":{"total":575970213888,"available":572195975168,"internally_reserved":0,"allocated":7262208,"data_stored":4557741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":3766868645},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":286,"num_read_kb":286,"num_write":184,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003948"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":707,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630929+0000","last_change":"2024-09-13T22:56:00.229464+0000","last_active":"2024-09-13T22:57:50.630929+0000","last_peered":"2024-09-13T22:57:50.630929+0000","last_clean":"2024-09-13T22:57:50.630929+0000","last_became_active":"2024-09-13T22:56:00.229118+0000","last_became_peered":"2024-09-13T22:56:00.229118+0000","last_unstale":"2024-09-13T22:57:50.630929+0000","last_undegraded":"2024-09-13T22:57:50.630929+0000","last_fullsized":"2024-09-13T22:57:50.630929+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:21.378509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:54:21.378509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":769,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.203265+0000","last_change":"2024-09-13T22:57:47.259212+0000","last_active":"2024-09-13T22:57:51.203265+0000","last_peered":"2024-09-13T22:57:51.203265+0000","last_clean":"2024-09-13T22:57:51.203265+0000","last_became_active":"2024-09-13T22:56:00.216437+0000","last_became_peered":"2024-09-13T22:56:00.216437+0000","last_unstale":"2024-09-13T22:57:51.203265+0000","last_undegraded":"2024-09-13T22:57:51.203265+0000","last_fullsized":"2024-09-13T22:57:51.203265+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.259089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:22.232806+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.259089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.198025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":795,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630630+0000","last_change":"2024-09-13T22:57:46.573120+0000","last_active":"2024-09-13T22:57:50.630630+0000","last_peered":"2024-09-13T22:57:50.630630+0000","last_clean":"2024-09-13T22:57:50.630630+0000","last_became_active":"2024-09-13T22:56:00.218204+0000","last_became_peered":"2024-09-13T22:56:00.218204+0000","last_unstale":"2024-09-13T22:57:50.630630+0000","last_undegraded":"2024-09-13T22:57:50.630630+0000","last_fullsized":"2024-09-13T22:57:50.630630+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'25","last_scrub_stamp":"2024-09-13T22:57:46.573001+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.573001+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:09.136946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":226,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144375+0000","last_change":"2024-09-13T22:57:00.773936+0000","last_active":"2024-09-13T22:57:51.144375+0000","last_peered":"2024-09-13T22:57:51.144375+0000","last_clean":"2024-09-13T22:57:51.144375+0000","last_became_active":"2024-09-13T22:56:00.235200+0000","last_became_peered":"2024-09-13T22:56:00.235200+0000","last_unstale":"2024-09-13T22:57:51.144375+0000","last_undegraded":"2024-09-13T22:57:51.144375+0000","last_fullsized":"2024-09-13T22:57:51.144375+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:00.773837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:57:00.773837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:12.925262+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":723,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630731+0000","last_change":"2024-09-13T22:56:28.054932+0000","last_active":"2024-09-13T22:57:50.630731+0000","last_peered":"2024-09-13T22:57:50.630731+0000","last_clean":"2024-09-13T22:57:50.630731+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:57:50.630731+0000","last_undegraded":"2024-09-13T22:57:50.630731+0000","last_fullsized":"2024-09-13T22:57:50.630731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:28.051743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.051743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":783,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.203128+0000","last_change":"2024-09-13T22:57:06.415408+0000","last_active":"2024-09-13T22:57:51.203128+0000","last_peered":"2024-09-13T22:57:51.203128+0000","last_clean":"2024-09-13T22:57:51.203128+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:57:51.203128+0000","last_undegraded":"2024-09-13T22:57:51.203128+0000","last_fullsized":"2024-09-13T22:57:51.203128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:57:06.415278+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.415278+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:09.310918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":721,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144365+0000","last_change":"2024-09-13T22:57:49.171117+0000","last_active":"2024-09-13T22:57:51.144365+0000","last_peered":"2024-09-13T22:57:51.144365+0000","last_clean":"2024-09-13T22:57:51.144365+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:57:51.144365+0000","last_undegraded":"2024-09-13T22:57:51.144365+0000","last_fullsized":"2024-09-13T22:57:51.144365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.170845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.170845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.075808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":238,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089826+0000","last_change":"2024-09-13T22:57:09.115091+0000","last_active":"2024-09-13T22:57:51.089826+0000","last_peered":"2024-09-13T22:57:51.089826+0000","last_clean":"2024-09-13T22:57:51.089826+0000","last_became_active":"2024-09-13T22:55:38.143783+0000","last_became_peered":"2024-09-13T22:55:38.143783+0000","last_unstale":"2024-09-13T22:57:51.089826+0000","last_undegraded":"2024-09-13T22:57:51.089826+0000","last_fullsized":"2024-09-13T22:57:51.089826+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.114979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.114979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:34.393471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":727,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207207+0000","last_change":"2024-09-13T22:56:00.226676+0000","last_active":"2024-09-13T22:57:51.207207+0000","last_peered":"2024-09-13T22:57:51.207207+0000","last_clean":"2024-09-13T22:57:51.207207+0000","last_became_active":"2024-09-13T22:56:00.225672+0000","last_became_peered":"2024-09-13T22:56:00.225672+0000","last_unstale":"2024-09-13T22:57:51.207207+0000","last_undegraded":"2024-09-13T22:57:51.207207+0000","last_fullsized":"2024-09-13T22:57:51.207207+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:20.805535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:20.805535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":889,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090241+0000","last_change":"2024-09-13T22:56:41.965853+0000","last_active":"2024-09-13T22:57:51.090241+0000","last_peered":"2024-09-13T22:57:51.090241+0000","last_clean":"2024-09-13T22:57:51.090241+0000","last_became_active":"2024-09-13T22:56:00.233191+0000","last_became_peered":"2024-09-13T22:56:00.233191+0000","last_unstale":"2024-09-13T22:57:51.090241+0000","last_undegraded":"2024-09-13T22:57:51.090241+0000","last_fullsized":"2024-09-13T22:57:51.090241+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:56:41.965747+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:56:41.965747+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:57:59.906939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":723,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204439+0000","last_change":"2024-09-13T22:55:38.153026+0000","last_active":"2024-09-13T22:57:51.204439+0000","last_peered":"2024-09-13T22:57:51.204439+0000","last_clean":"2024-09-13T22:57:51.204439+0000","last_became_active":"2024-09-13T22:55:38.151599+0000","last_became_peered":"2024-09-13T22:55:38.151599+0000","last_unstale":"2024-09-13T22:57:51.204439+0000","last_undegraded":"2024-09-13T22:57:51.204439+0000","last_fullsized":"2024-09-13T22:57:51.204439+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:21.721055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:55:21.721055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":717,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089880+0000","last_change":"2024-09-13T22:55:26.349094+0000","last_active":"2024-09-13T22:57:51.089880+0000","last_peered":"2024-09-13T22:57:51.089880+0000","last_clean":"2024-09-13T22:57:51.089880+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:57:51.089880+0000","last_undegraded":"2024-09-13T22:57:51.089880+0000","last_fullsized":"2024-09-13T22:57:51.089880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:26.348995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:55:26.348995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":721,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207012+0000","last_change":"2024-09-13T22:54:29.857171+0000","last_active":"2024-09-13T22:57:51.207012+0000","last_peered":"2024-09-13T22:57:51.207012+0000","last_clean":"2024-09-13T22:57:51.207012+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:57:51.207012+0000","last_undegraded":"2024-09-13T22:57:51.207012+0000","last_fullsized":"2024-09-13T22:57:51.207012+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:29.857063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:29.857063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"397'24","reported_seq":275,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.205640+0000","last_change":"2024-09-13T22:56:00.249252+0000","last_active":"2024-09-13T22:57:51.205640+0000","last_peered":"2024-09-13T22:57:51.205640+0000","last_clean":"2024-09-13T22:57:51.205640+0000","last_became_active":"2024-09-13T22:56:00.228121+0000","last_became_peered":"2024-09-13T22:56:00.228121+0000","last_unstale":"2024-09-13T22:57:51.205640+0000","last_undegraded":"2024-09-13T22:57:51.205640+0000","last_fullsized":"2024-09-13T22:57:51.205640+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:54:31.842940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:31.842940+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":721,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090247+0000","last_change":"2024-09-13T22:55:11.307718+0000","last_active":"2024-09-13T22:57:51.090247+0000","last_peered":"2024-09-13T22:57:51.090247+0000","last_clean":"2024-09-13T22:57:51.090247+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:57:51.090247+0000","last_undegraded":"2024-09-13T22:57:51.090247+0000","last_fullsized":"2024-09-13T22:57:51.090247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:11.307535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T22:55:11.307535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":726,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.203601+0000","last_change":"2024-09-13T22:57:51.203622+0000","last_active":"2024-09-13T22:57:51.203601+0000","last_peered":"2024-09-13T22:57:51.203601+0000","last_clean":"2024-09-13T22:57:51.203601+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:57:51.203601+0000","last_undegraded":"2024-09-13T22:57:51.203601+0000","last_fullsized":"2024-09-13T22:57:51.203601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.203544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.203544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:56.844866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631067+0000","last_change":"2024-09-13T22:57:49.649946+0000","last_active":"2024-09-13T22:57:50.631067+0000","last_peered":"2024-09-13T22:57:50.631067+0000","last_clean":"2024-09-13T22:57:50.631067+0000","last_became_active":"2024-09-13T22:57:49.649466+0000","last_became_peered":"2024-09-13T22:57:49.649466+0000","last_unstale":"2024-09-13T22:57:50.631067+0000","last_undegraded":"2024-09-13T22:57:50.631067+0000","last_fullsized":"2024-09-13T22:57:50.631067+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.791245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":783,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089224+0000","last_change":"2024-09-13T22:56:37.051234+0000","last_active":"2024-09-13T22:57:51.089224+0000","last_peered":"2024-09-13T22:57:51.089224+0000","last_clean":"2024-09-13T22:57:51.089224+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:57:51.089224+0000","last_undegraded":"2024-09-13T22:57:51.089224+0000","last_fullsized":"2024-09-13T22:57:51.089224+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:37.051143+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:37.051143+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":598,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144525+0000","last_change":"2024-09-13T22:57:48.143805+0000","last_active":"2024-09-13T22:57:51.144525+0000","last_peered":"2024-09-13T22:57:51.144525+0000","last_clean":"2024-09-13T22:57:51.144525+0000","last_became_active":"2024-09-13T22:56:00.227932+0000","last_became_peered":"2024-09-13T22:56:00.227932+0000","last_unstale":"2024-09-13T22:57:51.144525+0000","last_undegraded":"2024-09-13T22:57:51.144525+0000","last_fullsized":"2024-09-13T22:57:51.144525+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:57:48.143715+0000","last_deep_scrub":"228'44","last_deep_scrub_stamp":"2024-09-13T22:54:34.302937+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.143715+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:54.753954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":732,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.205548+0000","last_change":"2024-09-13T22:55:38.144148+0000","last_active":"2024-09-13T22:57:51.205548+0000","last_peered":"2024-09-13T22:57:51.205548+0000","last_clean":"2024-09-13T22:57:51.205548+0000","last_became_active":"2024-09-13T22:55:38.140796+0000","last_became_peered":"2024-09-13T22:55:38.140796+0000","last_unstale":"2024-09-13T22:57:51.205548+0000","last_undegraded":"2024-09-13T22:57:51.205548+0000","last_fullsized":"2024-09-13T22:57:51.205548+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:28.840664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:54:28.840664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":745,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145262+0000","last_change":"2024-09-13T22:56:00.215756+0000","last_active":"2024-09-13T22:57:51.145262+0000","last_peered":"2024-09-13T22:57:51.145262+0000","last_clean":"2024-09-13T22:57:51.145262+0000","last_became_active":"2024-09-13T22:56:00.215333+0000","last_became_peered":"2024-09-13T22:56:00.215333+0000","last_unstale":"2024-09-13T22:57:51.145262+0000","last_undegraded":"2024-09-13T22:57:51.145262+0000","last_fullsized":"2024-09-13T22:57:51.145262+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:29.486543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.486543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":649,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630218+0000","last_change":"2024-09-13T22:56:58.481848+0000","last_active":"2024-09-13T22:57:50.630218+0000","last_peered":"2024-09-13T22:57:50.630218+0000","last_clean":"2024-09-13T22:57:50.630218+0000","last_became_active":"2024-09-13T22:56:00.222668+0000","last_became_peered":"2024-09-13T22:56:00.222668+0000","last_unstale":"2024-09-13T22:57:50.630218+0000","last_undegraded":"2024-09-13T22:57:50.630218+0000","last_fullsized":"2024-09-13T22:57:50.630218+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:58.481749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:58.481749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:15.836773+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204681+0000","last_change":"2024-09-13T22:57:49.652606+0000","last_active":"2024-09-13T22:57:51.204681+0000","last_peered":"2024-09-13T22:57:51.204681+0000","last_clean":"2024-09-13T22:57:51.204681+0000","last_became_active":"2024-09-13T22:57:49.651025+0000","last_became_peered":"2024-09-13T22:57:49.651025+0000","last_unstale":"2024-09-13T22:57:51.204681+0000","last_undegraded":"2024-09-13T22:57:51.204681+0000","last_fullsized":"2024-09-13T22:57:51.204681+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:49.861752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":992,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.203939+0000","last_change":"2024-09-13T22:54:10.860962+0000","last_active":"2024-09-13T22:57:51.203939+0000","last_peered":"2024-09-13T22:57:51.203939+0000","last_clean":"2024-09-13T22:57:51.203939+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:57:51.203939+0000","last_undegraded":"2024-09-13T22:57:51.203939+0000","last_fullsized":"2024-09-13T22:57:51.203939+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:54:10.860597+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:54:10.860597+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":804,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631547+0000","last_change":"2024-09-13T22:55:38.114946+0000","last_active":"2024-09-13T22:57:50.631547+0000","last_peered":"2024-09-13T22:57:50.631547+0000","last_clean":"2024-09-13T22:57:50.631547+0000","last_became_active":"2024-09-13T22:55:38.095029+0000","last_became_peered":"2024-09-13T22:55:38.095029+0000","last_unstale":"2024-09-13T22:57:50.631547+0000","last_undegraded":"2024-09-13T22:57:50.631547+0000","last_fullsized":"2024-09-13T22:57:50.631547+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:55:13.915384+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:55:13.915384+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":730,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144764+0000","last_change":"2024-09-13T22:55:35.535365+0000","last_active":"2024-09-13T22:57:51.144764+0000","last_peered":"2024-09-13T22:57:51.144764+0000","last_clean":"2024-09-13T22:57:51.144764+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:57:51.144764+0000","last_undegraded":"2024-09-13T22:57:51.144764+0000","last_fullsized":"2024-09-13T22:57:51.144764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:35.535254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:35.535254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":439,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.748618+0000","last_change":"2024-09-13T22:57:06.962170+0000","last_active":"2024-09-13T22:57:51.748618+0000","last_peered":"2024-09-13T22:57:51.748618+0000","last_clean":"2024-09-13T22:57:51.748618+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T22:57:51.748618+0000","last_undegraded":"2024-09-13T22:57:51.748618+0000","last_fullsized":"2024-09-13T22:57:51.748618+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'12","last_scrub_stamp":"2024-09-13T22:57:06.962075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.962075+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:32.148372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":231,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145810+0000","last_change":"2024-09-13T22:55:38.103986+0000","last_active":"2024-09-13T22:57:51.145810+0000","last_peered":"2024-09-13T22:57:51.145810+0000","last_clean":"2024-09-13T22:57:51.145810+0000","last_became_active":"2024-09-13T22:55:38.103505+0000","last_became_peered":"2024-09-13T22:55:38.103505+0000","last_unstale":"2024-09-13T22:57:51.145810+0000","last_undegraded":"2024-09-13T22:57:51.145810+0000","last_fullsized":"2024-09-13T22:57:51.145810+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:37.084444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:37.084444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144704+0000","last_change":"2024-09-13T22:57:49.644230+0000","last_active":"2024-09-13T22:57:51.144704+0000","last_peered":"2024-09-13T22:57:51.144704+0000","last_clean":"2024-09-13T22:57:51.144704+0000","last_became_active":"2024-09-13T22:57:49.643953+0000","last_became_peered":"2024-09-13T22:57:49.643953+0000","last_unstale":"2024-09-13T22:57:51.144704+0000","last_undegraded":"2024-09-13T22:57:51.144704+0000","last_fullsized":"2024-09-13T22:57:51.144704+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:55.703071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":792,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089712+0000","last_change":"2024-09-13T22:56:03.248749+0000","last_active":"2024-09-13T22:57:51.089712+0000","last_peered":"2024-09-13T22:57:51.089712+0000","last_clean":"2024-09-13T22:57:51.089712+0000","last_became_active":"2024-09-13T22:56:03.248435+0000","last_became_peered":"2024-09-13T22:56:03.248435+0000","last_unstale":"2024-09-13T22:57:51.089712+0000","last_undegraded":"2024-09-13T22:57:51.089712+0000","last_fullsized":"2024-09-13T22:57:51.089712+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:55:30.307495+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:55:30.307495+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"397'39","reported_seq":779,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.628770+0000","last_change":"2024-09-13T22:55:29.816311+0000","last_active":"2024-09-13T22:57:50.628770+0000","last_peered":"2024-09-13T22:57:50.628770+0000","last_clean":"2024-09-13T22:57:50.628770+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:57:50.628770+0000","last_undegraded":"2024-09-13T22:57:50.628770+0000","last_fullsized":"2024-09-13T22:57:50.628770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:55:29.816212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.816212+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":738,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144672+0000","last_change":"2024-09-13T22:57:50.137862+0000","last_active":"2024-09-13T22:57:51.144672+0000","last_peered":"2024-09-13T22:57:51.144672+0000","last_clean":"2024-09-13T22:57:51.144672+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:57:51.144672+0000","last_undegraded":"2024-09-13T22:57:51.144672+0000","last_fullsized":"2024-09-13T22:57:51.144672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.137581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.137581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:02.936942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"351'15","reported_seq":401,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:53.690329+0000","last_change":"2024-09-13T22:57:02.049588+0000","last_active":"2024-09-13T22:57:53.690329+0000","last_peered":"2024-09-13T22:57:53.690329+0000","last_clean":"2024-09-13T22:57:53.690329+0000","last_became_active":"2024-09-13T22:55:38.106061+0000","last_became_peered":"2024-09-13T22:55:38.106061+0000","last_unstale":"2024-09-13T22:57:53.690329+0000","last_undegraded":"2024-09-13T22:57:53.690329+0000","last_fullsized":"2024-09-13T22:57:53.690329+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'15","last_scrub_stamp":"2024-09-13T22:57:02.049279+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:57:02.049279+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:10.722870+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":717,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090900+0000","last_change":"2024-09-13T22:57:11.104485+0000","last_active":"2024-09-13T22:57:51.090900+0000","last_peered":"2024-09-13T22:57:51.090900+0000","last_clean":"2024-09-13T22:57:51.090900+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:57:51.090900+0000","last_undegraded":"2024-09-13T22:57:51.090900+0000","last_fullsized":"2024-09-13T22:57:51.090900+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:11.104365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:57:11.104365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:29.195915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.627832+0000","last_change":"2024-09-13T22:57:49.643143+0000","last_active":"2024-09-13T22:57:50.627832+0000","last_peered":"2024-09-13T22:57:50.627832+0000","last_clean":"2024-09-13T22:57:50.627832+0000","last_became_active":"2024-09-13T22:57:49.642516+0000","last_became_peered":"2024-09-13T22:57:49.642516+0000","last_unstale":"2024-09-13T22:57:50.627832+0000","last_undegraded":"2024-09-13T22:57:50.627832+0000","last_fullsized":"2024-09-13T22:57:50.627832+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.725279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":331,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.483264+0000","last_change":"2024-09-13T22:57:50.496802+0000","last_active":"2024-09-13T22:57:51.483264+0000","last_peered":"2024-09-13T22:57:51.483264+0000","last_clean":"2024-09-13T22:57:51.483264+0000","last_became_active":"2024-09-13T22:55:38.124814+0000","last_became_peered":"2024-09-13T22:55:38.124814+0000","last_unstale":"2024-09-13T22:57:51.483264+0000","last_undegraded":"2024-09-13T22:57:51.483264+0000","last_fullsized":"2024-09-13T22:57:51.483264+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T22:57:50.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.496702+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.914857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":722,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.208464+0000","last_change":"2024-09-13T22:57:51.208483+0000","last_active":"2024-09-13T22:57:51.208464+0000","last_peered":"2024-09-13T22:57:51.208464+0000","last_clean":"2024-09-13T22:57:51.208464+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:57:51.208464+0000","last_undegraded":"2024-09-13T22:57:51.208464+0000","last_fullsized":"2024-09-13T22:57:51.208464+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.208413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.208413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.011784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":543,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.206982+0000","last_change":"2024-09-13T22:56:00.228231+0000","last_active":"2024-09-13T22:57:51.206982+0000","last_peered":"2024-09-13T22:57:51.206982+0000","last_clean":"2024-09-13T22:57:51.206982+0000","last_became_active":"2024-09-13T22:56:00.227885+0000","last_became_peered":"2024-09-13T22:56:00.227885+0000","last_unstale":"2024-09-13T22:57:51.206982+0000","last_undegraded":"2024-09-13T22:57:51.206982+0000","last_fullsized":"2024-09-13T22:57:51.206982+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.848113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.848113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":707,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.627916+0000","last_change":"2024-09-13T22:56:00.232863+0000","last_active":"2024-09-13T22:57:50.627916+0000","last_peered":"2024-09-13T22:57:50.627916+0000","last_clean":"2024-09-13T22:57:50.627916+0000","last_became_active":"2024-09-13T22:56:00.232543+0000","last_became_peered":"2024-09-13T22:56:00.232543+0000","last_unstale":"2024-09-13T22:57:50.627916+0000","last_undegraded":"2024-09-13T22:57:50.627916+0000","last_fullsized":"2024-09-13T22:57:50.627916+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:26.426360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:54:26.426360+0000","last_clean_scrub_stamp":"2024-09-13T22:54:26.426360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145839+0000","last_change":"2024-09-13T22:57:49.642973+0000","last_active":"2024-09-13T22:57:51.145839+0000","last_peered":"2024-09-13T22:57:51.145839+0000","last_clean":"2024-09-13T22:57:51.145839+0000","last_became_active":"2024-09-13T22:57:49.642612+0000","last_became_peered":"2024-09-13T22:57:49.642612+0000","last_unstale":"2024-09-13T22:57:51.145839+0000","last_undegraded":"2024-09-13T22:57:51.145839+0000","last_fullsized":"2024-09-13T22:57:51.145839+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:02.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":766,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.627816+0000","last_change":"2024-09-13T22:54:37.280042+0000","last_active":"2024-09-13T22:57:50.627816+0000","last_peered":"2024-09-13T22:57:50.627816+0000","last_clean":"2024-09-13T22:57:50.627816+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:57:50.627816+0000","last_undegraded":"2024-09-13T22:57:50.627816+0000","last_fullsized":"2024-09-13T22:57:50.627816+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:54:37.279940+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T22:54:37.279940+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":841,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202959+0000","last_change":"2024-09-13T22:55:38.167819+0000","last_active":"2024-09-13T22:57:51.202959+0000","last_peered":"2024-09-13T22:57:51.202959+0000","last_clean":"2024-09-13T22:57:51.202959+0000","last_became_active":"2024-09-13T22:55:38.135796+0000","last_became_peered":"2024-09-13T22:55:38.135796+0000","last_unstale":"2024-09-13T22:57:51.202959+0000","last_undegraded":"2024-09-13T22:57:51.202959+0000","last_fullsized":"2024-09-13T22:57:51.202959+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:54:34.778831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:34.778831+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207204+0000","last_change":"2024-09-13T22:56:00.226625+0000","last_active":"2024-09-13T22:57:51.207204+0000","last_peered":"2024-09-13T22:57:51.207204+0000","last_clean":"2024-09-13T22:57:51.207204+0000","last_became_active":"2024-09-13T22:56:00.226303+0000","last_became_peered":"2024-09-13T22:56:00.226303+0000","last_unstale":"2024-09-13T22:57:51.207204+0000","last_undegraded":"2024-09-13T22:57:51.207204+0000","last_fullsized":"2024-09-13T22:57:51.207204+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:18.188211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:18.188211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":540,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089953+0000","last_change":"2024-09-13T22:56:00.248252+0000","last_active":"2024-09-13T22:57:51.089953+0000","last_peered":"2024-09-13T22:57:51.089953+0000","last_clean":"2024-09-13T22:57:51.089953+0000","last_became_active":"2024-09-13T22:56:00.227907+0000","last_became_peered":"2024-09-13T22:56:00.227907+0000","last_unstale":"2024-09-13T22:57:51.089953+0000","last_undegraded":"2024-09-13T22:57:51.089953+0000","last_fullsized":"2024-09-13T22:57:51.089953+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:55:13.390955+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:55:13.390955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"351'14","reported_seq":993,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:53.713918+0000","last_change":"2024-09-13T22:55:38.102330+0000","last_active":"2024-09-13T22:57:53.713918+0000","last_peered":"2024-09-13T22:57:53.713918+0000","last_clean":"2024-09-13T22:57:53.713918+0000","last_became_active":"2024-09-13T22:55:38.087590+0000","last_became_peered":"2024-09-13T22:55:38.087590+0000","last_unstale":"2024-09-13T22:57:53.713918+0000","last_undegraded":"2024-09-13T22:57:53.713918+0000","last_fullsized":"2024-09-13T22:57:53.713918+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'13","last_scrub_stamp":"2024-09-13T22:54:27.486851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:54:27.486851+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":708,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204930+0000","last_change":"2024-09-13T22:56:00.228424+0000","last_active":"2024-09-13T22:57:51.204930+0000","last_peered":"2024-09-13T22:57:51.204930+0000","last_clean":"2024-09-13T22:57:51.204930+0000","last_became_active":"2024-09-13T22:56:00.228068+0000","last_became_peered":"2024-09-13T22:56:00.228068+0000","last_unstale":"2024-09-13T22:57:51.204930+0000","last_undegraded":"2024-09-13T22:57:51.204930+0000","last_fullsized":"2024-09-13T22:57:51.204930+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:24.785859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:54:24.785859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.206991+0000","last_change":"2024-09-13T22:57:49.650986+0000","last_active":"2024-09-13T22:57:51.206991+0000","last_peered":"2024-09-13T22:57:51.206991+0000","last_clean":"2024-09-13T22:57:51.206991+0000","last_became_active":"2024-09-13T22:57:49.650245+0000","last_became_peered":"2024-09-13T22:57:49.650245+0000","last_unstale":"2024-09-13T22:57:51.206991+0000","last_undegraded":"2024-09-13T22:57:51.206991+0000","last_fullsized":"2024-09-13T22:57:51.206991+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:53.919096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":583,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482477+0000","last_change":"2024-09-13T22:57:05.736962+0000","last_active":"2024-09-13T22:57:51.482477+0000","last_peered":"2024-09-13T22:57:51.482477+0000","last_clean":"2024-09-13T22:57:51.482477+0000","last_became_active":"2024-09-13T22:56:03.248140+0000","last_became_peered":"2024-09-13T22:56:03.248140+0000","last_unstale":"2024-09-13T22:57:51.482477+0000","last_undegraded":"2024-09-13T22:57:51.482477+0000","last_fullsized":"2024-09-13T22:57:51.482477+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:05.736855+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:57:05.736855+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:13.835173+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":762,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630737+0000","last_change":"2024-09-13T22:56:59.524045+0000","last_active":"2024-09-13T22:57:50.630737+0000","last_peered":"2024-09-13T22:57:50.630737+0000","last_clean":"2024-09-13T22:57:50.630737+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:57:50.630737+0000","last_undegraded":"2024-09-13T22:57:50.630737+0000","last_fullsized":"2024-09-13T22:57:50.630737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:56:59.523938+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:56:59.523938+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:19.627246+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":772,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145992+0000","last_change":"2024-09-13T22:56:00.242087+0000","last_active":"2024-09-13T22:57:51.145992+0000","last_peered":"2024-09-13T22:57:51.145992+0000","last_clean":"2024-09-13T22:57:51.145992+0000","last_became_active":"2024-09-13T22:56:00.222510+0000","last_became_peered":"2024-09-13T22:56:00.222510+0000","last_unstale":"2024-09-13T22:57:51.145992+0000","last_undegraded":"2024-09-13T22:57:51.145992+0000","last_fullsized":"2024-09-13T22:57:51.145992+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:55:26.471638+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:55:26.471638+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":1922,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:57.684390+0000","last_change":"2024-09-13T22:57:55.678130+0000","last_active":"2024-09-13T22:57:57.684390+0000","last_peered":"2024-09-13T22:57:57.684390+0000","last_clean":"2024-09-13T22:57:57.684390+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:57:57.684390+0000","last_undegraded":"2024-09-13T22:57:57.684390+0000","last_fullsized":"2024-09-13T22:57:57.684390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T22:57:55.678034+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.678034+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.773483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1192,"num_read_kb":1187,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":238,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.629019+0000","last_change":"2024-09-13T22:57:49.593322+0000","last_active":"2024-09-13T22:57:50.629019+0000","last_peered":"2024-09-13T22:57:50.629019+0000","last_clean":"2024-09-13T22:57:50.629019+0000","last_became_active":"2024-09-13T22:55:38.124054+0000","last_became_peered":"2024-09-13T22:55:38.124054+0000","last_unstale":"2024-09-13T22:57:50.629019+0000","last_undegraded":"2024-09-13T22:57:50.629019+0000","last_fullsized":"2024-09-13T22:57:50.629019+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.593224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.593224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.823607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":229,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631376+0000","last_change":"2024-09-13T22:56:00.234182+0000","last_active":"2024-09-13T22:57:50.631376+0000","last_peered":"2024-09-13T22:57:50.631376+0000","last_clean":"2024-09-13T22:57:50.631376+0000","last_became_active":"2024-09-13T22:56:00.231167+0000","last_became_peered":"2024-09-13T22:56:00.231167+0000","last_unstale":"2024-09-13T22:57:50.631376+0000","last_undegraded":"2024-09-13T22:57:50.631376+0000","last_fullsized":"2024-09-13T22:57:50.631376+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.153155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.153155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145441+0000","last_change":"2024-09-13T22:57:49.647079+0000","last_active":"2024-09-13T22:57:51.145441+0000","last_peered":"2024-09-13T22:57:51.145441+0000","last_clean":"2024-09-13T22:57:51.145441+0000","last_became_active":"2024-09-13T22:57:49.646766+0000","last_became_peered":"2024-09-13T22:57:49.646766+0000","last_unstale":"2024-09-13T22:57:51.145441+0000","last_undegraded":"2024-09-13T22:57:51.145441+0000","last_fullsized":"2024-09-13T22:57:51.145441+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:04.428671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":416,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202600+0000","last_change":"2024-09-13T22:56:28.769649+0000","last_active":"2024-09-13T22:57:51.202600+0000","last_peered":"2024-09-13T22:57:51.202600+0000","last_clean":"2024-09-13T22:57:51.202600+0000","last_became_active":"2024-09-13T22:56:03.263189+0000","last_became_peered":"2024-09-13T22:56:03.263189+0000","last_unstale":"2024-09-13T22:57:51.202600+0000","last_undegraded":"2024-09-13T22:57:51.202600+0000","last_fullsized":"2024-09-13T22:57:51.202600+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:28.769564+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.769564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:57:56.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1066,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.747545+0000","last_change":"2024-09-13T22:55:36.588155+0000","last_active":"2024-09-13T22:57:51.747545+0000","last_peered":"2024-09-13T22:57:51.747545+0000","last_clean":"2024-09-13T22:57:51.747545+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:57:51.747545+0000","last_undegraded":"2024-09-13T22:57:51.747545+0000","last_fullsized":"2024-09-13T22:57:51.747545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'52","last_scrub_stamp":"2024-09-13T22:55:36.587996+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:55:36.587996+0000","last_clean_scrub_stamp":"2024-09-13T22:55:36.587996+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":732,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207287+0000","last_change":"2024-09-13T22:55:38.146043+0000","last_active":"2024-09-13T22:57:51.207287+0000","last_peered":"2024-09-13T22:57:51.207287+0000","last_clean":"2024-09-13T22:57:51.207287+0000","last_became_active":"2024-09-13T22:55:38.145203+0000","last_became_peered":"2024-09-13T22:55:38.145203+0000","last_unstale":"2024-09-13T22:57:51.207287+0000","last_undegraded":"2024-09-13T22:57:51.207287+0000","last_fullsized":"2024-09-13T22:57:51.207287+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:19.987765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:19.987765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":94,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.201802+0000","last_change":"2024-09-13T22:56:00.226914+0000","last_active":"2024-09-13T22:57:51.201802+0000","last_peered":"2024-09-13T22:57:51.201802+0000","last_clean":"2024-09-13T22:57:51.201802+0000","last_became_active":"2024-09-13T22:56:00.226492+0000","last_became_peered":"2024-09-13T22:56:00.226492+0000","last_unstale":"2024-09-13T22:57:51.201802+0000","last_undegraded":"2024-09-13T22:57:51.201802+0000","last_fullsized":"2024-09-13T22:57:51.201802+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:08.441565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:55:08.441565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":724,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144227+0000","last_change":"2024-09-13T22:56:55.737374+0000","last_active":"2024-09-13T22:57:51.144227+0000","last_peered":"2024-09-13T22:57:51.144227+0000","last_clean":"2024-09-13T22:57:51.144227+0000","last_became_active":"2024-09-13T22:56:00.235117+0000","last_became_peered":"2024-09-13T22:56:00.235117+0000","last_unstale":"2024-09-13T22:57:51.144227+0000","last_undegraded":"2024-09-13T22:57:51.144227+0000","last_fullsized":"2024-09-13T22:57:51.144227+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:55.737268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:55.737268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:22.264525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482793+0000","last_change":"2024-09-13T22:57:49.641514+0000","last_active":"2024-09-13T22:57:51.482793+0000","last_peered":"2024-09-13T22:57:51.482793+0000","last_clean":"2024-09-13T22:57:51.482793+0000","last_became_active":"2024-09-13T22:57:49.641164+0000","last_became_peered":"2024-09-13T22:57:49.641164+0000","last_unstale":"2024-09-13T22:57:51.482793+0000","last_undegraded":"2024-09-13T22:57:51.482793+0000","last_fullsized":"2024-09-13T22:57:51.482793+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.259155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":419,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630108+0000","last_change":"2024-09-13T22:56:30.161262+0000","last_active":"2024-09-13T22:57:50.630108+0000","last_peered":"2024-09-13T22:57:50.630108+0000","last_clean":"2024-09-13T22:57:50.630108+0000","last_became_active":"2024-09-13T22:56:03.238687+0000","last_became_peered":"2024-09-13T22:56:03.238687+0000","last_unstale":"2024-09-13T22:57:50.630108+0000","last_undegraded":"2024-09-13T22:57:50.630108+0000","last_fullsized":"2024-09-13T22:57:50.630108+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:30.160971+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:21.295886+0000","last_clean_scrub_stamp":"2024-09-13T22:56:30.160971+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:57:53.678321+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1036,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.748720+0000","last_change":"2024-09-13T22:55:14.331379+0000","last_active":"2024-09-13T22:57:51.748720+0000","last_peered":"2024-09-13T22:57:51.748720+0000","last_clean":"2024-09-13T22:57:51.748720+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:57:51.748720+0000","last_undegraded":"2024-09-13T22:57:51.748720+0000","last_fullsized":"2024-09-13T22:57:51.748720+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'36","last_scrub_stamp":"2024-09-13T22:55:14.331261+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:55:14.331261+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":747,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145574+0000","last_change":"2024-09-13T22:57:06.876493+0000","last_active":"2024-09-13T22:57:51.145574+0000","last_peered":"2024-09-13T22:57:51.145574+0000","last_clean":"2024-09-13T22:57:51.145574+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:57:51.145574+0000","last_undegraded":"2024-09-13T22:57:51.145574+0000","last_fullsized":"2024-09-13T22:57:51.145574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T22:57:06.876301+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T22:54:26.175576+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.876301+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:14.626954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":766,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144646+0000","last_change":"2024-09-13T22:55:38.117618+0000","last_active":"2024-09-13T22:57:51.144646+0000","last_peered":"2024-09-13T22:57:51.144646+0000","last_clean":"2024-09-13T22:57:51.144646+0000","last_became_active":"2024-09-13T22:55:38.114687+0000","last_became_peered":"2024-09-13T22:55:38.114687+0000","last_unstale":"2024-09-13T22:57:51.144646+0000","last_undegraded":"2024-09-13T22:57:51.144646+0000","last_fullsized":"2024-09-13T22:57:51.144646+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T22:55:21.433994+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T22:55:21.433994+0000","last_clean_scrub_stamp":"2024-09-13T22:55:21.433994+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"358'13","reported_seq":996,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.201955+0000","last_change":"2024-09-13T22:56:00.230244+0000","last_active":"2024-09-13T22:57:51.201955+0000","last_peered":"2024-09-13T22:57:51.201955+0000","last_clean":"2024-09-13T22:57:51.201955+0000","last_became_active":"2024-09-13T22:56:00.218154+0000","last_became_peered":"2024-09-13T22:56:00.218154+0000","last_unstale":"2024-09-13T22:57:51.201955+0000","last_undegraded":"2024-09-13T22:57:51.201955+0000","last_fullsized":"2024-09-13T22:57:51.201955+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'12","last_scrub_stamp":"2024-09-13T22:55:11.848208+0000","last_deep_scrub":"100'12","last_deep_scrub_stamp":"2024-09-13T22:39:01.655949+0000","last_clean_scrub_stamp":"2024-09-13T22:55:11.848208+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":728,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:56.259013+0000","last_change":"2024-09-13T22:57:56.259013+0000","last_active":"2024-09-13T22:57:56.259013+0000","last_peered":"2024-09-13T22:57:56.259013+0000","last_clean":"2024-09-13T22:57:56.259013+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:57:56.259013+0000","last_undegraded":"2024-09-13T22:57:56.259013+0000","last_fullsized":"2024-09-13T22:57:56.259013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:56.258783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:56.258783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.527327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631089+0000","last_change":"2024-09-13T22:57:49.647535+0000","last_active":"2024-09-13T22:57:50.631089+0000","last_peered":"2024-09-13T22:57:50.631089+0000","last_clean":"2024-09-13T22:57:50.631089+0000","last_became_active":"2024-09-13T22:57:49.646701+0000","last_became_peered":"2024-09-13T22:57:49.646701+0000","last_unstale":"2024-09-13T22:57:50.631089+0000","last_undegraded":"2024-09-13T22:57:50.631089+0000","last_fullsized":"2024-09-13T22:57:50.631089+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:14.537853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":783,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:57.050213+0000","last_change":"2024-09-13T22:57:57.050275+0000","last_active":"2024-09-13T22:57:57.050213+0000","last_peered":"2024-09-13T22:57:57.050213+0000","last_clean":"2024-09-13T22:57:57.050213+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:57:57.050213+0000","last_undegraded":"2024-09-13T22:57:57.050213+0000","last_fullsized":"2024-09-13T22:57:57.050213+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:57.050042+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.050042+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.549861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"397'39","reported_seq":66,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.481972+0000","last_change":"2024-09-13T22:57:45.567779+0000","last_active":"2024-09-13T22:57:51.481972+0000","last_peered":"2024-09-13T22:57:51.481972+0000","last_clean":"2024-09-13T22:57:51.481972+0000","last_became_active":"2024-09-13T22:57:37.355011+0000","last_became_peered":"2024-09-13T22:57:37.355011+0000","last_unstale":"2024-09-13T22:57:51.481972+0000","last_undegraded":"2024-09-13T22:57:51.481972+0000","last_fullsized":"2024-09-13T22:57:51.481972+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'29","last_scrub_stamp":"2024-09-13T22:57:45.567559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:45.567559+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:01.804961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":744,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090708+0000","last_change":"2024-09-13T22:56:00.242569+0000","last_active":"2024-09-13T22:57:51.090708+0000","last_peered":"2024-09-13T22:57:51.090708+0000","last_clean":"2024-09-13T22:57:51.090708+0000","last_became_active":"2024-09-13T22:56:00.242041+0000","last_became_peered":"2024-09-13T22:56:00.242041+0000","last_unstale":"2024-09-13T22:57:51.090708+0000","last_undegraded":"2024-09-13T22:57:51.090708+0000","last_fullsized":"2024-09-13T22:57:51.090708+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:29.340956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.340956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":755,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202488+0000","last_change":"2024-09-13T22:57:09.483802+0000","last_active":"2024-09-13T22:57:51.202488+0000","last_peered":"2024-09-13T22:57:51.202488+0000","last_clean":"2024-09-13T22:57:51.202488+0000","last_became_active":"2024-09-13T22:56:00.216504+0000","last_became_peered":"2024-09-13T22:56:00.216504+0000","last_unstale":"2024-09-13T22:57:51.202488+0000","last_undegraded":"2024-09-13T22:57:51.202488+0000","last_fullsized":"2024-09-13T22:57:51.202488+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.483697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.483697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:14.950895+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":734,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630416+0000","last_change":"2024-09-13T22:56:56.539298+0000","last_active":"2024-09-13T22:57:50.630416+0000","last_peered":"2024-09-13T22:57:50.630416+0000","last_clean":"2024-09-13T22:57:50.630416+0000","last_became_active":"2024-09-13T22:56:00.232669+0000","last_became_peered":"2024-09-13T22:56:00.232669+0000","last_unstale":"2024-09-13T22:57:50.630416+0000","last_undegraded":"2024-09-13T22:57:50.630416+0000","last_fullsized":"2024-09-13T22:57:50.630416+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:56.539208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.539208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:14.370938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.627428+0000","last_change":"2024-09-13T22:57:49.643169+0000","last_active":"2024-09-13T22:57:50.627428+0000","last_peered":"2024-09-13T22:57:50.627428+0000","last_clean":"2024-09-13T22:57:50.627428+0000","last_became_active":"2024-09-13T22:57:49.642830+0000","last_became_peered":"2024-09-13T22:57:49.642830+0000","last_unstale":"2024-09-13T22:57:50.627428+0000","last_undegraded":"2024-09-13T22:57:50.627428+0000","last_fullsized":"2024-09-13T22:57:50.627428+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:57.983033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1027,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204719+0000","last_change":"2024-09-13T22:56:54.138074+0000","last_active":"2024-09-13T22:57:51.204719+0000","last_peered":"2024-09-13T22:57:51.204719+0000","last_clean":"2024-09-13T22:57:51.204719+0000","last_became_active":"2024-09-13T22:55:41.103393+0000","last_became_peered":"2024-09-13T22:55:41.103393+0000","last_unstale":"2024-09-13T22:57:51.204719+0000","last_undegraded":"2024-09-13T22:57:51.204719+0000","last_fullsized":"2024-09-13T22:57:51.204719+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:54.137952+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:54.137952+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:14.326641+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":61,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089495+0000","last_change":"2024-09-13T22:57:37.368591+0000","last_active":"2024-09-13T22:57:51.089495+0000","last_peered":"2024-09-13T22:57:51.089495+0000","last_clean":"2024-09-13T22:57:51.089495+0000","last_became_active":"2024-09-13T22:57:37.355384+0000","last_became_peered":"2024-09-13T22:57:37.355384+0000","last_unstale":"2024-09-13T22:57:51.089495+0000","last_undegraded":"2024-09-13T22:57:51.089495+0000","last_fullsized":"2024-09-13T22:57:51.089495+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.760160+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:01.113542+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":779,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.483258+0000","last_change":"2024-09-13T22:57:48.508060+0000","last_active":"2024-09-13T22:57:51.483258+0000","last_peered":"2024-09-13T22:57:51.483258+0000","last_clean":"2024-09-13T22:57:51.483258+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:57:51.483258+0000","last_undegraded":"2024-09-13T22:57:51.483258+0000","last_fullsized":"2024-09-13T22:57:51.483258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:57:48.507763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.507763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.560730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":755,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482523+0000","last_change":"2024-09-13T22:56:48.632053+0000","last_active":"2024-09-13T22:57:51.482523+0000","last_peered":"2024-09-13T22:57:51.482523+0000","last_clean":"2024-09-13T22:57:51.482523+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:57:51.482523+0000","last_undegraded":"2024-09-13T22:57:51.482523+0000","last_fullsized":"2024-09-13T22:57:51.482523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:48.631915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:56:48.631915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:15.306680+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":716,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.205792+0000","last_change":"2024-09-13T22:56:00.227065+0000","last_active":"2024-09-13T22:57:51.205792+0000","last_peered":"2024-09-13T22:57:51.205792+0000","last_clean":"2024-09-13T22:57:51.205792+0000","last_became_active":"2024-09-13T22:56:00.226279+0000","last_became_peered":"2024-09-13T22:56:00.226279+0000","last_unstale":"2024-09-13T22:57:51.205792+0000","last_undegraded":"2024-09-13T22:57:51.205792+0000","last_fullsized":"2024-09-13T22:57:51.205792+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:18.683688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:18.683688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.629467+0000","last_change":"2024-09-13T22:57:49.647142+0000","last_active":"2024-09-13T22:57:50.629467+0000","last_peered":"2024-09-13T22:57:50.629467+0000","last_clean":"2024-09-13T22:57:50.629467+0000","last_became_active":"2024-09-13T22:57:49.646772+0000","last_became_peered":"2024-09-13T22:57:49.646772+0000","last_unstale":"2024-09-13T22:57:50.629467+0000","last_undegraded":"2024-09-13T22:57:50.629467+0000","last_fullsized":"2024-09-13T22:57:50.629467+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:57.612517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":818,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:52.050712+0000","last_change":"2024-09-13T22:57:52.050773+0000","last_active":"2024-09-13T22:57:52.050712+0000","last_peered":"2024-09-13T22:57:52.050712+0000","last_clean":"2024-09-13T22:57:52.050712+0000","last_became_active":"2024-09-13T22:56:03.239906+0000","last_became_peered":"2024-09-13T22:56:03.239906+0000","last_unstale":"2024-09-13T22:57:52.050712+0000","last_undegraded":"2024-09-13T22:57:52.050712+0000","last_fullsized":"2024-09-13T22:57:52.050712+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:52.050564+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T22:57:52.050564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.906269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"397'42","reported_seq":864,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.201983+0000","last_change":"2024-09-13T22:55:38.127806+0000","last_active":"2024-09-13T22:57:51.201983+0000","last_peered":"2024-09-13T22:57:51.201983+0000","last_clean":"2024-09-13T22:57:51.201983+0000","last_became_active":"2024-09-13T22:55:38.098695+0000","last_became_peered":"2024-09-13T22:55:38.098695+0000","last_unstale":"2024-09-13T22:57:51.201983+0000","last_undegraded":"2024-09-13T22:57:51.201983+0000","last_fullsized":"2024-09-13T22:57:51.201983+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'28","last_scrub_stamp":"2024-09-13T22:55:34.128466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:34.128466+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":497,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204165+0000","last_change":"2024-09-13T22:55:11.632270+0000","last_active":"2024-09-13T22:57:51.204165+0000","last_peered":"2024-09-13T22:57:51.204165+0000","last_clean":"2024-09-13T22:57:51.204165+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:57:51.204165+0000","last_undegraded":"2024-09-13T22:57:51.204165+0000","last_fullsized":"2024-09-13T22:57:51.204165+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:11.632162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:55:11.632162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":730,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090927+0000","last_change":"2024-09-13T22:57:03.031136+0000","last_active":"2024-09-13T22:57:51.090927+0000","last_peered":"2024-09-13T22:57:51.090927+0000","last_clean":"2024-09-13T22:57:51.090927+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:57:51.090927+0000","last_undegraded":"2024-09-13T22:57:51.090927+0000","last_fullsized":"2024-09-13T22:57:51.090927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:03.031031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:57:03.031031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:20.803600+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":718,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145158+0000","last_change":"2024-09-13T22:55:31.460012+0000","last_active":"2024-09-13T22:57:51.145158+0000","last_peered":"2024-09-13T22:57:51.145158+0000","last_clean":"2024-09-13T22:57:51.145158+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:57:51.145158+0000","last_undegraded":"2024-09-13T22:57:51.145158+0000","last_fullsized":"2024-09-13T22:57:51.145158+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.459912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:18.659096+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.459912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631515+0000","last_change":"2024-09-13T22:57:49.647406+0000","last_active":"2024-09-13T22:57:50.631515+0000","last_peered":"2024-09-13T22:57:50.631515+0000","last_clean":"2024-09-13T22:57:50.631515+0000","last_became_active":"2024-09-13T22:57:49.646955+0000","last_became_peered":"2024-09-13T22:57:49.646955+0000","last_unstale":"2024-09-13T22:57:50.631515+0000","last_undegraded":"2024-09-13T22:57:50.631515+0000","last_fullsized":"2024-09-13T22:57:50.631515+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.811814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":773,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631502+0000","last_change":"2024-09-13T22:56:00.253284+0000","last_active":"2024-09-13T22:57:50.631502+0000","last_peered":"2024-09-13T22:57:50.631502+0000","last_clean":"2024-09-13T22:57:50.631502+0000","last_became_active":"2024-09-13T22:56:00.226771+0000","last_became_peered":"2024-09-13T22:56:00.226771+0000","last_unstale":"2024-09-13T22:57:50.631502+0000","last_undegraded":"2024-09-13T22:57:50.631502+0000","last_fullsized":"2024-09-13T22:57:50.631502+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:54:18.363744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:18.363744+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":738,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090425+0000","last_change":"2024-09-13T22:56:44.954891+0000","last_active":"2024-09-13T22:57:51.090425+0000","last_peered":"2024-09-13T22:57:51.090425+0000","last_clean":"2024-09-13T22:57:51.090425+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:57:51.090425+0000","last_undegraded":"2024-09-13T22:57:51.090425+0000","last_fullsized":"2024-09-13T22:57:51.090425+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:44.954773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:44.954773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:02.539606+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1002,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204716+0000","last_change":"2024-09-13T22:56:03.243249+0000","last_active":"2024-09-13T22:57:51.204716+0000","last_peered":"2024-09-13T22:57:51.204716+0000","last_clean":"2024-09-13T22:57:51.204716+0000","last_became_active":"2024-09-13T22:56:03.242893+0000","last_became_peered":"2024-09-13T22:56:03.242893+0000","last_unstale":"2024-09-13T22:57:51.204716+0000","last_undegraded":"2024-09-13T22:57:51.204716+0000","last_fullsized":"2024-09-13T22:57:51.204716+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:55:26.889228+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:55:26.889228+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":714,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207111+0000","last_change":"2024-09-13T22:55:32.846798+0000","last_active":"2024-09-13T22:57:51.207111+0000","last_peered":"2024-09-13T22:57:51.207111+0000","last_clean":"2024-09-13T22:57:51.207111+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:57:51.207111+0000","last_undegraded":"2024-09-13T22:57:51.207111+0000","last_fullsized":"2024-09-13T22:57:51.207111+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:32.846649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:55:32.846649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":744,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090867+0000","last_change":"2024-09-13T22:56:33.129911+0000","last_active":"2024-09-13T22:57:51.090867+0000","last_peered":"2024-09-13T22:57:51.090867+0000","last_clean":"2024-09-13T22:57:51.090867+0000","last_became_active":"2024-09-13T22:55:38.101749+0000","last_became_peered":"2024-09-13T22:55:38.101749+0000","last_unstale":"2024-09-13T22:57:51.090867+0000","last_undegraded":"2024-09-13T22:57:51.090867+0000","last_fullsized":"2024-09-13T22:57:51.090867+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:33.129825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:33.129825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207069+0000","last_change":"2024-09-13T22:57:49.653049+0000","last_active":"2024-09-13T22:57:51.207069+0000","last_peered":"2024-09-13T22:57:51.207069+0000","last_clean":"2024-09-13T22:57:51.207069+0000","last_became_active":"2024-09-13T22:57:49.651751+0000","last_became_peered":"2024-09-13T22:57:49.651751+0000","last_unstale":"2024-09-13T22:57:51.207069+0000","last_undegraded":"2024-09-13T22:57:51.207069+0000","last_fullsized":"2024-09-13T22:57:51.207069+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:54.330507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":804,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207199+0000","last_change":"2024-09-13T22:56:00.266974+0000","last_active":"2024-09-13T22:57:51.207199+0000","last_peered":"2024-09-13T22:57:51.207199+0000","last_clean":"2024-09-13T22:57:51.207199+0000","last_became_active":"2024-09-13T22:56:00.229726+0000","last_became_peered":"2024-09-13T22:56:00.229726+0000","last_unstale":"2024-09-13T22:57:51.207199+0000","last_undegraded":"2024-09-13T22:57:51.207199+0000","last_fullsized":"2024-09-13T22:57:51.207199+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'31","last_scrub_stamp":"2024-09-13T22:54:15.863951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:15.863951+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":763,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482842+0000","last_change":"2024-09-13T22:56:59.672579+0000","last_active":"2024-09-13T22:57:51.482842+0000","last_peered":"2024-09-13T22:57:51.482842+0000","last_clean":"2024-09-13T22:57:51.482842+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:57:51.482842+0000","last_undegraded":"2024-09-13T22:57:51.482842+0000","last_fullsized":"2024-09-13T22:57:51.482842+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:59.672450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:59.672450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:10.194224+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":746,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482496+0000","last_change":"2024-09-13T22:56:54.626081+0000","last_active":"2024-09-13T22:57:51.482496+0000","last_peered":"2024-09-13T22:57:51.482496+0000","last_clean":"2024-09-13T22:57:51.482496+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:57:51.482496+0000","last_undegraded":"2024-09-13T22:57:51.482496+0000","last_fullsized":"2024-09-13T22:57:51.482496+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:54.626006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:56:54.626006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:03.640619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":702,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204602+0000","last_change":"2024-09-13T22:54:25.819955+0000","last_active":"2024-09-13T22:57:51.204602+0000","last_peered":"2024-09-13T22:57:51.204602+0000","last_clean":"2024-09-13T22:57:51.204602+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:57:51.204602+0000","last_undegraded":"2024-09-13T22:57:51.204602+0000","last_fullsized":"2024-09-13T22:57:51.204602+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:25.819823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:54:25.819823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.205399+0000","last_change":"2024-09-13T22:57:49.652798+0000","last_active":"2024-09-13T22:57:51.205399+0000","last_peered":"2024-09-13T22:57:51.205399+0000","last_clean":"2024-09-13T22:57:51.205399+0000","last_became_active":"2024-09-13T22:57:49.651792+0000","last_became_peered":"2024-09-13T22:57:49.651792+0000","last_unstale":"2024-09-13T22:57:51.205399+0000","last_undegraded":"2024-09-13T22:57:51.205399+0000","last_fullsized":"2024-09-13T22:57:51.205399+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"397'30","reported_seq":797,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:53.059246+0000","last_change":"2024-09-13T22:57:53.059246+0000","last_active":"2024-09-13T22:57:53.059246+0000","last_peered":"2024-09-13T22:57:53.059246+0000","last_clean":"2024-09-13T22:57:53.059246+0000","last_became_active":"2024-09-13T22:56:00.241217+0000","last_became_peered":"2024-09-13T22:56:00.241217+0000","last_unstale":"2024-09-13T22:57:53.059246+0000","last_undegraded":"2024-09-13T22:57:53.059246+0000","last_fullsized":"2024-09-13T22:57:53.059246+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:57:53.059034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.059034+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.454055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":756,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145636+0000","last_change":"2024-09-13T22:57:07.882465+0000","last_active":"2024-09-13T22:57:51.145636+0000","last_peered":"2024-09-13T22:57:51.145636+0000","last_clean":"2024-09-13T22:57:51.145636+0000","last_became_active":"2024-09-13T22:55:38.093381+0000","last_became_peered":"2024-09-13T22:55:38.093381+0000","last_unstale":"2024-09-13T22:57:51.145636+0000","last_undegraded":"2024-09-13T22:57:51.145636+0000","last_fullsized":"2024-09-13T22:57:51.145636+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.882167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.882167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:32.285343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":451,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631268+0000","last_change":"2024-09-13T22:55:38.098852+0000","last_active":"2024-09-13T22:57:50.631268+0000","last_peered":"2024-09-13T22:57:50.631268+0000","last_clean":"2024-09-13T22:57:50.631268+0000","last_became_active":"2024-09-13T22:55:38.098515+0000","last_became_peered":"2024-09-13T22:55:38.098515+0000","last_unstale":"2024-09-13T22:57:50.631268+0000","last_undegraded":"2024-09-13T22:57:50.631268+0000","last_fullsized":"2024-09-13T22:57:50.631268+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:15.398485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:15.398485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":727,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.201755+0000","last_change":"2024-09-13T22:55:14.852608+0000","last_active":"2024-09-13T22:57:51.201755+0000","last_peered":"2024-09-13T22:57:51.201755+0000","last_clean":"2024-09-13T22:57:51.201755+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:57:51.201755+0000","last_undegraded":"2024-09-13T22:57:51.201755+0000","last_fullsized":"2024-09-13T22:57:51.201755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:14.852498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:55:14.852498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090704+0000","last_change":"2024-09-13T22:57:49.646353+0000","last_active":"2024-09-13T22:57:51.090704+0000","last_peered":"2024-09-13T22:57:51.090704+0000","last_clean":"2024-09-13T22:57:51.090704+0000","last_became_active":"2024-09-13T22:57:49.645497+0000","last_became_peered":"2024-09-13T22:57:49.645497+0000","last_unstale":"2024-09-13T22:57:51.090704+0000","last_undegraded":"2024-09-13T22:57:51.090704+0000","last_fullsized":"2024-09-13T22:57:51.090704+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.252571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":852,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207198+0000","last_change":"2024-09-13T22:56:57.107636+0000","last_active":"2024-09-13T22:57:51.207198+0000","last_peered":"2024-09-13T22:57:51.207198+0000","last_clean":"2024-09-13T22:57:51.207198+0000","last_became_active":"2024-09-13T22:55:38.151658+0000","last_became_peered":"2024-09-13T22:55:38.151658+0000","last_unstale":"2024-09-13T22:57:51.207198+0000","last_undegraded":"2024-09-13T22:57:51.207198+0000","last_fullsized":"2024-09-13T22:57:51.207198+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:56:57.107495+0000","last_deep_scrub":"228'34","last_deep_scrub_stamp":"2024-09-13T22:53:27.660042+0000","last_clean_scrub_stamp":"2024-09-13T22:56:57.107495+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:24.614766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":312,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630822+0000","last_change":"2024-09-13T22:55:19.900066+0000","last_active":"2024-09-13T22:57:50.630822+0000","last_peered":"2024-09-13T22:57:50.630822+0000","last_clean":"2024-09-13T22:57:50.630822+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T22:57:50.630822+0000","last_undegraded":"2024-09-13T22:57:50.630822+0000","last_fullsized":"2024-09-13T22:57:50.630822+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:19.899953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:32.054441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:19.899953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"358'15","reported_seq":1011,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.748713+0000","last_change":"2024-09-13T22:56:00.261590+0000","last_active":"2024-09-13T22:57:51.748713+0000","last_peered":"2024-09-13T22:57:51.748713+0000","last_clean":"2024-09-13T22:57:51.748713+0000","last_became_active":"2024-09-13T22:56:00.231283+0000","last_became_peered":"2024-09-13T22:56:00.231283+0000","last_unstale":"2024-09-13T22:57:51.748713+0000","last_undegraded":"2024-09-13T22:57:51.748713+0000","last_fullsized":"2024-09-13T22:57:51.748713+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'13","last_scrub_stamp":"2024-09-13T22:54:27.850029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:54:27.850029+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":724,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089913+0000","last_change":"2024-09-13T22:56:00.225538+0000","last_active":"2024-09-13T22:57:51.089913+0000","last_peered":"2024-09-13T22:57:51.089913+0000","last_clean":"2024-09-13T22:57:51.089913+0000","last_became_active":"2024-09-13T22:56:00.225061+0000","last_became_peered":"2024-09-13T22:56:00.225061+0000","last_unstale":"2024-09-13T22:57:51.089913+0000","last_undegraded":"2024-09-13T22:57:51.089913+0000","last_fullsized":"2024-09-13T22:57:51.089913+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:30.580766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T22:54:30.580766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202464+0000","last_change":"2024-09-13T22:57:49.644003+0000","last_active":"2024-09-13T22:57:51.202464+0000","last_peered":"2024-09-13T22:57:51.202464+0000","last_clean":"2024-09-13T22:57:51.202464+0000","last_became_active":"2024-09-13T22:57:49.643715+0000","last_became_peered":"2024-09-13T22:57:49.643715+0000","last_unstale":"2024-09-13T22:57:51.202464+0000","last_undegraded":"2024-09-13T22:57:51.202464+0000","last_fullsized":"2024-09-13T22:57:51.202464+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:52.087280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"397'27","reported_seq":823,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:53.254550+0000","last_change":"2024-09-13T22:57:53.254550+0000","last_active":"2024-09-13T22:57:53.254550+0000","last_peered":"2024-09-13T22:57:53.254550+0000","last_clean":"2024-09-13T22:57:53.254550+0000","last_became_active":"2024-09-13T22:55:38.151005+0000","last_became_peered":"2024-09-13T22:55:38.151005+0000","last_unstale":"2024-09-13T22:57:53.254550+0000","last_undegraded":"2024-09-13T22:57:53.254550+0000","last_fullsized":"2024-09-13T22:57:53.254550+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T22:57:53.254254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.254254+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:54.913543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":355,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.203271+0000","last_change":"2024-09-13T22:55:38.159081+0000","last_active":"2024-09-13T22:57:51.203271+0000","last_peered":"2024-09-13T22:57:51.203271+0000","last_clean":"2024-09-13T22:57:51.203271+0000","last_became_active":"2024-09-13T22:55:38.136788+0000","last_became_peered":"2024-09-13T22:55:38.136788+0000","last_unstale":"2024-09-13T22:57:51.203271+0000","last_undegraded":"2024-09-13T22:57:51.203271+0000","last_fullsized":"2024-09-13T22:57:51.203271+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:54:35.115922+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:54:35.115922+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":321,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.201558+0000","last_change":"2024-09-13T22:57:46.251728+0000","last_active":"2024-09-13T22:57:51.201558+0000","last_peered":"2024-09-13T22:57:51.201558+0000","last_clean":"2024-09-13T22:57:51.201558+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T22:57:51.201558+0000","last_undegraded":"2024-09-13T22:57:51.201558+0000","last_fullsized":"2024-09-13T22:57:51.201558+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:46.251423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.251423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:08.732783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":717,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631168+0000","last_change":"2024-09-13T22:57:50.594198+0000","last_active":"2024-09-13T22:57:50.631168+0000","last_peered":"2024-09-13T22:57:50.631168+0000","last_clean":"2024-09-13T22:57:50.631168+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:57:50.631168+0000","last_undegraded":"2024-09-13T22:57:50.631168+0000","last_fullsized":"2024-09-13T22:57:50.631168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.594120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.594120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:56.833221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":837,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204510+0000","last_change":"2024-09-13T22:56:52.281067+0000","last_active":"2024-09-13T22:57:51.204510+0000","last_peered":"2024-09-13T22:57:51.204510+0000","last_clean":"2024-09-13T22:57:51.204510+0000","last_became_active":"2024-09-13T22:56:00.226680+0000","last_became_peered":"2024-09-13T22:56:00.226680+0000","last_unstale":"2024-09-13T22:57:51.204510+0000","last_undegraded":"2024-09-13T22:57:51.204510+0000","last_fullsized":"2024-09-13T22:57:51.204510+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:56:52.280906+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:53:21.566725+0000","last_clean_scrub_stamp":"2024-09-13T22:56:52.280906+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:03.703786+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":762,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.481904+0000","last_change":"2024-09-13T22:56:57.677342+0000","last_active":"2024-09-13T22:57:51.481904+0000","last_peered":"2024-09-13T22:57:51.481904+0000","last_clean":"2024-09-13T22:57:51.481904+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:57:51.481904+0000","last_undegraded":"2024-09-13T22:57:51.481904+0000","last_fullsized":"2024-09-13T22:57:51.481904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:57.675268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:56:57.675268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:15.714564+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":713,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630636+0000","last_change":"2024-09-13T22:55:34.729429+0000","last_active":"2024-09-13T22:57:50.630636+0000","last_peered":"2024-09-13T22:57:50.630636+0000","last_clean":"2024-09-13T22:57:50.630636+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:57:50.630636+0000","last_undegraded":"2024-09-13T22:57:50.630636+0000","last_fullsized":"2024-09-13T22:57:50.630636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:34.729322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:34.729322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":776,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482300+0000","last_change":"2024-09-13T22:57:01.727570+0000","last_active":"2024-09-13T22:57:51.482300+0000","last_peered":"2024-09-13T22:57:51.482300+0000","last_clean":"2024-09-13T22:57:51.482300+0000","last_became_active":"2024-09-13T22:55:38.126360+0000","last_became_peered":"2024-09-13T22:55:38.126360+0000","last_unstale":"2024-09-13T22:57:51.482300+0000","last_undegraded":"2024-09-13T22:57:51.482300+0000","last_fullsized":"2024-09-13T22:57:51.482300+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:01.727464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:57:01.727464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:29.484184+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":507,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:54.173974+0000","last_change":"2024-09-13T22:57:54.174037+0000","last_active":"2024-09-13T22:57:54.173974+0000","last_peered":"2024-09-13T22:57:54.173974+0000","last_clean":"2024-09-13T22:57:54.173974+0000","last_became_active":"2024-09-13T22:56:00.212121+0000","last_became_peered":"2024-09-13T22:56:00.212121+0000","last_unstale":"2024-09-13T22:57:54.173974+0000","last_undegraded":"2024-09-13T22:57:54.173974+0000","last_fullsized":"2024-09-13T22:57:54.173974+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T22:57:54.173839+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.173839+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:13.804584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":759,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144559+0000","last_change":"2024-09-13T22:56:51.610256+0000","last_active":"2024-09-13T22:57:51.144559+0000","last_peered":"2024-09-13T22:57:51.144559+0000","last_clean":"2024-09-13T22:57:51.144559+0000","last_became_active":"2024-09-13T22:56:00.227777+0000","last_became_peered":"2024-09-13T22:56:00.227777+0000","last_unstale":"2024-09-13T22:57:51.144559+0000","last_undegraded":"2024-09-13T22:57:51.144559+0000","last_fullsized":"2024-09-13T22:57:51.144559+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:51.610149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T22:56:51.610149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:00.537564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":721,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090547+0000","last_change":"2024-09-13T22:54:35.464811+0000","last_active":"2024-09-13T22:57:51.090547+0000","last_peered":"2024-09-13T22:57:51.090547+0000","last_clean":"2024-09-13T22:57:51.090547+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:57:51.090547+0000","last_undegraded":"2024-09-13T22:57:51.090547+0000","last_fullsized":"2024-09-13T22:57:51.090547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:35.464730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:54:35.464730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":725,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.205963+0000","last_change":"2024-09-13T22:56:00.235503+0000","last_active":"2024-09-13T22:57:51.205963+0000","last_peered":"2024-09-13T22:57:51.205963+0000","last_clean":"2024-09-13T22:57:51.205963+0000","last_became_active":"2024-09-13T22:56:00.234954+0000","last_became_peered":"2024-09-13T22:56:00.234954+0000","last_unstale":"2024-09-13T22:57:51.205963+0000","last_undegraded":"2024-09-13T22:57:51.205963+0000","last_fullsized":"2024-09-13T22:57:51.205963+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:09.848281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:54:09.848281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":832,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.205285+0000","last_change":"2024-09-13T22:56:00.243298+0000","last_active":"2024-09-13T22:57:51.205285+0000","last_peered":"2024-09-13T22:57:51.205285+0000","last_clean":"2024-09-13T22:57:51.205285+0000","last_became_active":"2024-09-13T22:56:00.207375+0000","last_became_peered":"2024-09-13T22:56:00.207375+0000","last_unstale":"2024-09-13T22:57:51.205285+0000","last_undegraded":"2024-09-13T22:57:51.205285+0000","last_fullsized":"2024-09-13T22:57:51.205285+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:55:30.790120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:30.790120+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":740,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090795+0000","last_change":"2024-09-13T22:56:35.136404+0000","last_active":"2024-09-13T22:57:51.090795+0000","last_peered":"2024-09-13T22:57:51.090795+0000","last_clean":"2024-09-13T22:57:51.090795+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:57:51.090795+0000","last_undegraded":"2024-09-13T22:57:51.090795+0000","last_fullsized":"2024-09-13T22:57:51.090795+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:35.136329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:56:35.136329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":987,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.748745+0000","last_change":"2024-09-13T22:56:36.091025+0000","last_active":"2024-09-13T22:57:51.748745+0000","last_peered":"2024-09-13T22:57:51.748745+0000","last_clean":"2024-09-13T22:57:51.748745+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:57:51.748745+0000","last_undegraded":"2024-09-13T22:57:51.748745+0000","last_fullsized":"2024-09-13T22:57:51.748745+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T22:56:36.090908+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T22:53:18.805172+0000","last_clean_scrub_stamp":"2024-09-13T22:56:36.090908+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:00.344195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":726,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:55.222380+0000","last_change":"2024-09-13T22:57:55.222402+0000","last_active":"2024-09-13T22:57:55.222380+0000","last_peered":"2024-09-13T22:57:55.222380+0000","last_clean":"2024-09-13T22:57:55.222380+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:57:55.222380+0000","last_undegraded":"2024-09-13T22:57:55.222380+0000","last_fullsized":"2024-09-13T22:57:55.222380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:55.222315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.222315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.342997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":768,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:53.159125+0000","last_change":"2024-09-13T22:57:53.159155+0000","last_active":"2024-09-13T22:57:53.159125+0000","last_peered":"2024-09-13T22:57:53.159125+0000","last_clean":"2024-09-13T22:57:53.159125+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:57:53.159125+0000","last_undegraded":"2024-09-13T22:57:53.159125+0000","last_fullsized":"2024-09-13T22:57:53.159125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T22:57:53.159058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.159058+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.582286+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":756,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202222+0000","last_change":"2024-09-13T22:55:38.138681+0000","last_active":"2024-09-13T22:57:51.202222+0000","last_peered":"2024-09-13T22:57:51.202222+0000","last_clean":"2024-09-13T22:57:51.202222+0000","last_became_active":"2024-09-13T22:55:38.138035+0000","last_became_peered":"2024-09-13T22:55:38.138035+0000","last_unstale":"2024-09-13T22:57:51.202222+0000","last_undegraded":"2024-09-13T22:57:51.202222+0000","last_fullsized":"2024-09-13T22:57:51.202222+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:36.129393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:36.129393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":967,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.747567+0000","last_change":"2024-09-13T22:54:25.412280+0000","last_active":"2024-09-13T22:57:51.747567+0000","last_peered":"2024-09-13T22:57:51.747567+0000","last_clean":"2024-09-13T22:57:51.747567+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:57:51.747567+0000","last_undegraded":"2024-09-13T22:57:51.747567+0000","last_fullsized":"2024-09-13T22:57:51.747567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:54:25.412167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:54:25.412167+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":246,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.482907+0000","last_change":"2024-09-13T22:57:47.490067+0000","last_active":"2024-09-13T22:57:51.482907+0000","last_peered":"2024-09-13T22:57:51.482907+0000","last_clean":"2024-09-13T22:57:51.482907+0000","last_became_active":"2024-09-13T22:55:38.098486+0000","last_became_peered":"2024-09-13T22:55:38.098486+0000","last_unstale":"2024-09-13T22:57:51.482907+0000","last_undegraded":"2024-09-13T22:57:51.482907+0000","last_fullsized":"2024-09-13T22:57:51.482907+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.489929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.489929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:49.045088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":799,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.206159+0000","last_change":"2024-09-13T22:55:38.169420+0000","last_active":"2024-09-13T22:57:51.206159+0000","last_peered":"2024-09-13T22:57:51.206159+0000","last_clean":"2024-09-13T22:57:51.206159+0000","last_became_active":"2024-09-13T22:55:38.144744+0000","last_became_peered":"2024-09-13T22:55:38.144744+0000","last_unstale":"2024-09-13T22:57:51.206159+0000","last_undegraded":"2024-09-13T22:57:51.206159+0000","last_fullsized":"2024-09-13T22:57:51.206159+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:55:28.849442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:28.849442+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":103,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:54.689335+0000","last_change":"2024-09-13T22:57:54.689394+0000","last_active":"2024-09-13T22:57:54.689335+0000","last_peered":"2024-09-13T22:57:54.689335+0000","last_clean":"2024-09-13T22:57:54.689335+0000","last_became_active":"2024-09-13T22:56:00.232875+0000","last_became_peered":"2024-09-13T22:56:00.232875+0000","last_unstale":"2024-09-13T22:57:54.689335+0000","last_undegraded":"2024-09-13T22:57:54.689335+0000","last_fullsized":"2024-09-13T22:57:54.689335+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:54.689197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.689197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:05.878279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":713,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631309+0000","last_change":"2024-09-13T22:55:27.845977+0000","last_active":"2024-09-13T22:57:50.631309+0000","last_peered":"2024-09-13T22:57:50.631309+0000","last_clean":"2024-09-13T22:57:50.631309+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:57:50.631309+0000","last_undegraded":"2024-09-13T22:57:50.631309+0000","last_fullsized":"2024-09-13T22:57:50.631309+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:27.845519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:55:27.845519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":718,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.201670+0000","last_change":"2024-09-13T22:55:10.922994+0000","last_active":"2024-09-13T22:57:51.201670+0000","last_peered":"2024-09-13T22:57:51.201670+0000","last_clean":"2024-09-13T22:57:51.201670+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:57:51.201670+0000","last_undegraded":"2024-09-13T22:57:51.201670+0000","last_fullsized":"2024-09-13T22:57:51.201670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:10.922916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:10.922916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":337,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207035+0000","last_change":"2024-09-13T22:56:00.240679+0000","last_active":"2024-09-13T22:57:51.207035+0000","last_peered":"2024-09-13T22:57:51.207035+0000","last_clean":"2024-09-13T22:57:51.207035+0000","last_became_active":"2024-09-13T22:56:00.207460+0000","last_became_peered":"2024-09-13T22:56:00.207460+0000","last_unstale":"2024-09-13T22:57:51.207035+0000","last_undegraded":"2024-09-13T22:57:51.207035+0000","last_fullsized":"2024-09-13T22:57:51.207035+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:54:30.095316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:30.095316+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":729,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145561+0000","last_change":"2024-09-13T22:55:24.440576+0000","last_active":"2024-09-13T22:57:51.145561+0000","last_peered":"2024-09-13T22:57:51.145561+0000","last_clean":"2024-09-13T22:57:51.145561+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:57:51.145561+0000","last_undegraded":"2024-09-13T22:57:51.145561+0000","last_fullsized":"2024-09-13T22:57:51.145561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T22:55:24.439970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":721,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.144393+0000","last_change":"2024-09-13T22:55:15.318055+0000","last_active":"2024-09-13T22:57:51.144393+0000","last_peered":"2024-09-13T22:57:51.144393+0000","last_clean":"2024-09-13T22:57:51.144393+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:57:51.144393+0000","last_undegraded":"2024-09-13T22:57:51.144393+0000","last_fullsized":"2024-09-13T22:57:51.144393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:15.317834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:55:15.317834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":709,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.627936+0000","last_change":"2024-09-13T22:57:07.530820+0000","last_active":"2024-09-13T22:57:50.627936+0000","last_peered":"2024-09-13T22:57:50.627936+0000","last_clean":"2024-09-13T22:57:50.627936+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:57:50.627936+0000","last_undegraded":"2024-09-13T22:57:50.627936+0000","last_fullsized":"2024-09-13T22:57:50.627936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.530747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.530747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:17.131005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"397'18","reported_seq":761,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.203195+0000","last_change":"2024-09-13T22:55:29.003407+0000","last_active":"2024-09-13T22:57:51.203195+0000","last_peered":"2024-09-13T22:57:51.203195+0000","last_clean":"2024-09-13T22:57:51.203195+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:57:51.203195+0000","last_undegraded":"2024-09-13T22:57:51.203195+0000","last_fullsized":"2024-09-13T22:57:51.203195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12","last_scrub_stamp":"2024-09-13T22:55:29.003276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.003276+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":735,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090881+0000","last_change":"2024-09-13T22:56:00.240165+0000","last_active":"2024-09-13T22:57:51.090881+0000","last_peered":"2024-09-13T22:57:51.090881+0000","last_clean":"2024-09-13T22:57:51.090881+0000","last_became_active":"2024-09-13T22:56:00.239806+0000","last_became_peered":"2024-09-13T22:56:00.239806+0000","last_unstale":"2024-09-13T22:57:51.090881+0000","last_undegraded":"2024-09-13T22:57:51.090881+0000","last_fullsized":"2024-09-13T22:57:51.090881+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:29.591215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:54:29.591215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":743,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204737+0000","last_change":"2024-09-13T22:56:00.236969+0000","last_active":"2024-09-13T22:57:51.204737+0000","last_peered":"2024-09-13T22:57:51.204737+0000","last_clean":"2024-09-13T22:57:51.204737+0000","last_became_active":"2024-09-13T22:56:00.235156+0000","last_became_peered":"2024-09-13T22:56:00.235156+0000","last_unstale":"2024-09-13T22:57:51.204737+0000","last_undegraded":"2024-09-13T22:57:51.204737+0000","last_fullsized":"2024-09-13T22:57:51.204737+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:22.255559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:54:22.255559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":727,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204286+0000","last_change":"2024-09-13T22:57:10.040687+0000","last_active":"2024-09-13T22:57:51.204286+0000","last_peered":"2024-09-13T22:57:51.204286+0000","last_clean":"2024-09-13T22:57:51.204286+0000","last_became_active":"2024-09-13T22:55:38.135406+0000","last_became_peered":"2024-09-13T22:55:38.135406+0000","last_unstale":"2024-09-13T22:57:51.204286+0000","last_undegraded":"2024-09-13T22:57:51.204286+0000","last_fullsized":"2024-09-13T22:57:51.204286+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:10.040498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:10.040498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:10.046652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":767,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090707+0000","last_change":"2024-09-13T22:57:46.014205+0000","last_active":"2024-09-13T22:57:51.090707+0000","last_peered":"2024-09-13T22:57:51.090707+0000","last_clean":"2024-09-13T22:57:51.090707+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:57:51.090707+0000","last_undegraded":"2024-09-13T22:57:51.090707+0000","last_fullsized":"2024-09-13T22:57:51.090707+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:57:46.013979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.013979+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:55.078753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":95,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.091105+0000","last_change":"2024-09-13T22:56:00.242840+0000","last_active":"2024-09-13T22:57:51.091105+0000","last_peered":"2024-09-13T22:57:51.091105+0000","last_clean":"2024-09-13T22:57:51.091105+0000","last_became_active":"2024-09-13T22:56:00.242421+0000","last_became_peered":"2024-09-13T22:56:00.242421+0000","last_unstale":"2024-09-13T22:57:51.091105+0000","last_undegraded":"2024-09-13T22:57:51.091105+0000","last_fullsized":"2024-09-13T22:57:51.091105+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.579369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.579369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":737,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204251+0000","last_change":"2024-09-13T22:56:55.145345+0000","last_active":"2024-09-13T22:57:51.204251+0000","last_peered":"2024-09-13T22:57:51.204251+0000","last_clean":"2024-09-13T22:57:51.204251+0000","last_became_active":"2024-09-13T22:56:00.213590+0000","last_became_peered":"2024-09-13T22:56:00.213590+0000","last_unstale":"2024-09-13T22:57:51.204251+0000","last_undegraded":"2024-09-13T22:57:51.204251+0000","last_fullsized":"2024-09-13T22:57:51.204251+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:55.145267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:56:55.145267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:04.466787+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":715,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631402+0000","last_change":"2024-09-13T22:56:00.218467+0000","last_active":"2024-09-13T22:57:50.631402+0000","last_peered":"2024-09-13T22:57:50.631402+0000","last_clean":"2024-09-13T22:57:50.631402+0000","last_became_active":"2024-09-13T22:56:00.218040+0000","last_became_peered":"2024-09-13T22:56:00.218040+0000","last_unstale":"2024-09-13T22:57:50.631402+0000","last_undegraded":"2024-09-13T22:57:50.631402+0000","last_fullsized":"2024-09-13T22:57:50.631402+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:23.408905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:54:23.408905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":712,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.207160+0000","last_change":"2024-09-13T22:56:53.116102+0000","last_active":"2024-09-13T22:57:51.207160+0000","last_peered":"2024-09-13T22:57:51.207160+0000","last_clean":"2024-09-13T22:57:51.207160+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:57:51.207160+0000","last_undegraded":"2024-09-13T22:57:51.207160+0000","last_fullsized":"2024-09-13T22:57:51.207160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_clean_scrub_stamp":"2024-09-13T22:56:53.115999+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:19.659179+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":59,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.481821+0000","last_change":"2024-09-13T22:57:37.368671+0000","last_active":"2024-09-13T22:57:51.481821+0000","last_peered":"2024-09-13T22:57:51.481821+0000","last_clean":"2024-09-13T22:57:51.481821+0000","last_became_active":"2024-09-13T22:57:37.355449+0000","last_became_peered":"2024-09-13T22:57:37.355449+0000","last_unstale":"2024-09-13T22:57:51.481821+0000","last_undegraded":"2024-09-13T22:57:51.481821+0000","last_fullsized":"2024-09-13T22:57:51.481821+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:57:10.942679+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:57:10.942679+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:39.719181+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":761,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.089702+0000","last_change":"2024-09-13T22:56:34.086608+0000","last_active":"2024-09-13T22:57:51.089702+0000","last_peered":"2024-09-13T22:57:51.089702+0000","last_clean":"2024-09-13T22:57:51.089702+0000","last_became_active":"2024-09-13T22:56:00.242209+0000","last_became_peered":"2024-09-13T22:56:00.242209+0000","last_unstale":"2024-09-13T22:57:51.089702+0000","last_undegraded":"2024-09-13T22:57:51.089702+0000","last_fullsized":"2024-09-13T22:57:51.089702+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:34.086426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:56:34.086426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:02.708507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"358'16","reported_seq":458,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.748766+0000","last_change":"2024-09-13T22:56:00.255155+0000","last_active":"2024-09-13T22:57:51.748766+0000","last_peered":"2024-09-13T22:57:51.748766+0000","last_clean":"2024-09-13T22:57:51.748766+0000","last_became_active":"2024-09-13T22:56:00.229080+0000","last_became_peered":"2024-09-13T22:56:00.229080+0000","last_unstale":"2024-09-13T22:57:51.748766+0000","last_undegraded":"2024-09-13T22:57:51.748766+0000","last_fullsized":"2024-09-13T22:57:51.748766+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'14","last_scrub_stamp":"2024-09-13T22:55:25.529898+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T22:55:25.529898+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":743,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.145354+0000","last_change":"2024-09-13T22:57:09.960429+0000","last_active":"2024-09-13T22:57:51.145354+0000","last_peered":"2024-09-13T22:57:51.145354+0000","last_clean":"2024-09-13T22:57:51.145354+0000","last_became_active":"2024-09-13T22:55:38.096392+0000","last_became_peered":"2024-09-13T22:55:38.096392+0000","last_unstale":"2024-09-13T22:57:51.145354+0000","last_undegraded":"2024-09-13T22:57:51.145354+0000","last_fullsized":"2024-09-13T22:57:51.145354+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.960363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.960363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:25.840517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":767,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.628294+0000","last_change":"2024-09-13T22:57:37.364148+0000","last_active":"2024-09-13T22:57:50.628294+0000","last_peered":"2024-09-13T22:57:50.628294+0000","last_clean":"2024-09-13T22:57:50.628294+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T22:57:50.628294+0000","last_undegraded":"2024-09-13T22:57:50.628294+0000","last_fullsized":"2024-09-13T22:57:50.628294+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:55:35.746045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:35.746045+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":724,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630148+0000","last_change":"2024-09-13T22:56:00.224528+0000","last_active":"2024-09-13T22:57:50.630148+0000","last_peered":"2024-09-13T22:57:50.630148+0000","last_clean":"2024-09-13T22:57:50.630148+0000","last_became_active":"2024-09-13T22:56:00.213377+0000","last_became_peered":"2024-09-13T22:56:00.213377+0000","last_unstale":"2024-09-13T22:57:50.630148+0000","last_undegraded":"2024-09-13T22:57:50.630148+0000","last_fullsized":"2024-09-13T22:57:50.630148+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:54:35.340609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:35.340609+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":732,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202770+0000","last_change":"2024-09-13T22:55:32.033215+0000","last_active":"2024-09-13T22:57:51.202770+0000","last_peered":"2024-09-13T22:57:51.202770+0000","last_clean":"2024-09-13T22:57:51.202770+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:57:51.202770+0000","last_undegraded":"2024-09-13T22:57:51.202770+0000","last_fullsized":"2024-09-13T22:57:51.202770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:32.033129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:55:32.033129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":225,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.090433+0000","last_change":"2024-09-13T22:56:00.230642+0000","last_active":"2024-09-13T22:57:51.090433+0000","last_peered":"2024-09-13T22:57:51.090433+0000","last_clean":"2024-09-13T22:57:51.090433+0000","last_became_active":"2024-09-13T22:56:00.230261+0000","last_became_peered":"2024-09-13T22:56:00.230261+0000","last_unstale":"2024-09-13T22:57:51.090433+0000","last_undegraded":"2024-09-13T22:57:51.090433+0000","last_fullsized":"2024-09-13T22:57:51.090433+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:12.370205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:12.370205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":759,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.204930+0000","last_change":"2024-09-13T22:57:01.014387+0000","last_active":"2024-09-13T22:57:51.204930+0000","last_peered":"2024-09-13T22:57:51.204930+0000","last_clean":"2024-09-13T22:57:51.204930+0000","last_became_active":"2024-09-13T22:56:00.228879+0000","last_became_peered":"2024-09-13T22:56:00.228879+0000","last_unstale":"2024-09-13T22:57:51.204930+0000","last_undegraded":"2024-09-13T22:57:51.204930+0000","last_fullsized":"2024-09-13T22:57:51.204930+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:01.014315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:01.014315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:06.124639+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":788,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:51.202084+0000","last_change":"2024-09-13T22:55:38.163465+0000","last_active":"2024-09-13T22:57:51.202084+0000","last_peered":"2024-09-13T22:57:51.202084+0000","last_clean":"2024-09-13T22:57:51.202084+0000","last_became_active":"2024-09-13T22:55:38.138369+0000","last_became_peered":"2024-09-13T22:55:38.138369+0000","last_unstale":"2024-09-13T22:57:51.202084+0000","last_undegraded":"2024-09-13T22:57:51.202084+0000","last_fullsized":"2024-09-13T22:57:51.202084+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'9","last_scrub_stamp":"2024-09-13T22:55:18.819420+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:55:18.819420+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":747,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:57.115102+0000","last_change":"2024-09-13T22:57:57.115157+0000","last_active":"2024-09-13T22:57:57.115102+0000","last_peered":"2024-09-13T22:57:57.115102+0000","last_clean":"2024-09-13T22:57:57.115102+0000","last_became_active":"2024-09-13T22:55:38.094235+0000","last_became_peered":"2024-09-13T22:55:38.094235+0000","last_unstale":"2024-09-13T22:57:57.115102+0000","last_undegraded":"2024-09-13T22:57:57.115102+0000","last_fullsized":"2024-09-13T22:57:57.115102+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:57.114953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:17.313617+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.114953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.976193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":721,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630399+0000","last_change":"2024-09-13T22:56:00.255194+0000","last_active":"2024-09-13T22:57:50.630399+0000","last_peered":"2024-09-13T22:57:50.630399+0000","last_clean":"2024-09-13T22:57:50.630399+0000","last_became_active":"2024-09-13T22:56:00.230406+0000","last_became_peered":"2024-09-13T22:56:00.230406+0000","last_unstale":"2024-09-13T22:57:50.630399+0000","last_undegraded":"2024-09-13T22:57:50.630399+0000","last_fullsized":"2024-09-13T22:57:50.630399+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:55:10.933585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:10.933585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":727,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.631714+0000","last_change":"2024-09-13T22:56:00.231974+0000","last_active":"2024-09-13T22:57:50.631714+0000","last_peered":"2024-09-13T22:57:50.631714+0000","last_clean":"2024-09-13T22:57:50.631714+0000","last_became_active":"2024-09-13T22:56:00.231645+0000","last_became_peered":"2024-09-13T22:56:00.231645+0000","last_unstale":"2024-09-13T22:57:50.631714+0000","last_undegraded":"2024-09-13T22:57:50.631714+0000","last_fullsized":"2024-09-13T22:57:50.631714+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:24.878070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:24.878070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":574,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:54.101116+0000","last_change":"2024-09-13T22:57:54.101195+0000","last_active":"2024-09-13T22:57:54.101116+0000","last_peered":"2024-09-13T22:57:54.101116+0000","last_clean":"2024-09-13T22:57:54.101116+0000","last_became_active":"2024-09-13T22:55:38.099550+0000","last_became_peered":"2024-09-13T22:55:38.099550+0000","last_unstale":"2024-09-13T22:57:54.101116+0000","last_undegraded":"2024-09-13T22:57:54.101116+0000","last_fullsized":"2024-09-13T22:57:54.101116+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T22:57:54.100973+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.100973+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.423659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":223,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T22:57:50.630089+0000","last_change":"2024-09-13T22:55:38.133820+0000","last_active":"2024-09-13T22:57:50.630089+0000","last_peered":"2024-09-13T22:57:50.630089+0000","last_clean":"2024-09-13T22:57:50.630089+0000","last_became_active":"2024-09-13T22:55:38.133425+0000","last_became_peered":"2024-09-13T22:55:38.133425+0000","last_unstale":"2024-09-13T22:57:50.630089+0000","last_undegraded":"2024-09-13T22:57:50.630089+0000","last_fullsized":"2024-09-13T22:57:50.630089+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:28.200985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:55:28.200985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1192,"num_read_kb":1187,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1555,"num_read_kb":1312,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":7040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":14296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1055,"ondisk_log_size":1055,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":414,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809084,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627988,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93117164,"statfs":{"total":95995035648,"available":95351975936,"internally_reserved":0,"allocated":1249280,"data_stored":793017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":641792458},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":244813136135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528594,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596300,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148852,"statfs":{"total":95995035648,"available":95384424448,"internally_reserved":0,"allocated":1503232,"data_stored":1053987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084582,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704584,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040568,"statfs":{"total":95995035648,"available":95273541632,"internally_reserved":0,"allocated":1499136,"data_stored":1054362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":719976903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378930,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93283836,"statfs":{"total":95995035648,"available":95522648064,"internally_reserved":0,"allocated":1167360,"data_stored":726665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575164,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623512,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121640,"statfs":{"total":95995035648,"available":95356559360,"internally_reserved":0,"allocated":925696,"data_stored":464669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073072,"statfs":{"total":95995035648,"available":95306825728,"internally_reserved":0,"allocated":917504,"data_stored":465041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:58:05.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:58:05.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:05.660 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:05.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:05.790 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:05.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:05.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:58:05.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.050126+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:58:05.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.051692+0000 osd.5 (osd.5) 430 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:58:05.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.161052+0000 osd.1 (osd.1) 386 : cluster [DBG] 6.e scrub starts 2024-09-13T22:58:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.163163+0000 osd.1 (osd.1) 387 : cluster [DBG] 6.e scrub ok 2024-09-13T22:58:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.243046+0000 osd.7 (osd.7) 401 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:58:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:05.205561+0000 osd.0 (osd.0) 391 : cluster [DBG] 5.3 deep-scrub starts 2024-09-13T22:58:05.906 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:05 smithi192 bash[24608]: cluster 2024-09-13T22:58:05.208543+0000 osd.0 (osd.0) 392 : cluster [DBG] 5.3 deep-scrub ok 2024-09-13T22:58:05.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:05.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:05.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.050126+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.051692+0000 osd.5 (osd.5) 430 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.161052+0000 osd.1 (osd.1) 386 : cluster [DBG] 6.e scrub starts 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.163163+0000 osd.1 (osd.1) 387 : cluster [DBG] 6.e scrub ok 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.243046+0000 osd.7 (osd.7) 401 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:05.205561+0000 osd.0 (osd.0) 391 : cluster [DBG] 5.3 deep-scrub starts 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[22840]: cluster 2024-09-13T22:58:05.208543+0000 osd.0 (osd.0) 392 : cluster [DBG] 5.3 deep-scrub ok 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.050126+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.16 scrub starts 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.051692+0000 osd.5 (osd.5) 430 : cluster [DBG] 5.16 scrub ok 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.161052+0000 osd.1 (osd.1) 386 : cluster [DBG] 6.e scrub starts 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.163163+0000 osd.1 (osd.1) 387 : cluster [DBG] 6.e scrub ok 2024-09-13T22:58:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.243046+0000 osd.7 (osd.7) 401 : cluster [DBG] 2.7 scrub ok 2024-09-13T22:58:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:05.205561+0000 osd.0 (osd.0) 391 : cluster [DBG] 5.3 deep-scrub starts 2024-09-13T22:58:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:05 smithi178 bash[29889]: cluster 2024-09-13T22:58:05.208543+0000 osd.0 (osd.0) 392 : cluster [DBG] 5.3 deep-scrub ok 2024-09-13T22:58:06.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:06.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:06.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:06.184 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:06.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:06.314 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:06.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.444 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:06.547 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.562 DEBUG:teuthology.orchestra.run.smithi178:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-13T22:58:06.576 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.578 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T22:58:06.578 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T22:58:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:06.680 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.708 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:hints = 1 2024-09-13T22:58:06.708 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T22:58:06.708 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_128208 2024-09-13T22:58:06.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:58:06.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-13T22:58:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:06.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.851 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:06.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:06.981 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:06.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:06 smithi192 bash[24608]: cluster 2024-09-13T22:58:04.909909+0000 mgr.y (mgr.14613) 758 : cluster [DBG] pgmap v896: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:58:06.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:06 smithi192 bash[24608]: cluster 2024-09-13T22:58:05.709062+0000 mon.a (mon.0) 1660 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:58:06.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:06 smithi192 bash[24608]: cluster 2024-09-13T22:58:05.709112+0000 mon.a (mon.0) 1661 : cluster [INF] Cluster is now healthy 2024-09-13T22:58:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:06.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:07.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:07.106 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:07.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:06 smithi178 bash[29889]: cluster 2024-09-13T22:58:04.909909+0000 mgr.y (mgr.14613) 758 : cluster [DBG] pgmap v896: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:58:07.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:06 smithi178 bash[29889]: cluster 2024-09-13T22:58:05.709062+0000 mon.a (mon.0) 1660 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:58:07.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:06 smithi178 bash[29889]: cluster 2024-09-13T22:58:05.709112+0000 mon.a (mon.0) 1661 : cluster [INF] Cluster is now healthy 2024-09-13T22:58:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:06 smithi178 bash[22840]: cluster 2024-09-13T22:58:04.909909+0000 mgr.y (mgr.14613) 758 : cluster [DBG] pgmap v896: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T22:58:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:06 smithi178 bash[22840]: cluster 2024-09-13T22:58:05.709062+0000 mon.a (mon.0) 1660 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T22:58:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:06 smithi178 bash[22840]: cluster 2024-09-13T22:58:05.709112+0000 mon.a (mon.0) 1661 : cluster [INF] Cluster is now healthy 2024-09-13T22:58:07.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:07.233 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:07.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:07.361 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:07.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:07.489 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:07.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:07.618 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:07.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 1 16 2542 2526 157.866 157.875 0.00693897 0.00630833 2024-09-13T22:58:07.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:07.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:07.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:07.878 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:07.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:07.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:07.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:07 smithi192 bash[24608]: audit 2024-09-13T22:58:05.866094+0000 mgr.y (mgr.14613) 759 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:07.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:07 smithi192 bash[24608]: cluster 2024-09-13T22:58:06.131827+0000 osd.1 (osd.1) 388 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:58:07.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:07 smithi192 bash[24608]: cluster 2024-09-13T22:58:06.133988+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:58:07.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:07 smithi192 bash[24608]: cluster 2024-09-13T22:58:07.240658+0000 osd.7 (osd.7) 402 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:58:07.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:07 smithi192 bash[24608]: cluster 2024-09-13T22:58:07.403484+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-09-13T22:58:07.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:07 smithi192 bash[24608]: cluster 2024-09-13T22:58:07.404716+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-09-13T22:58:08.006 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:08.088 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:08.090 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:08.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:08.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[22840]: audit 2024-09-13T22:58:05.866094+0000 mgr.y (mgr.14613) 759 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:08.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[22840]: cluster 2024-09-13T22:58:06.131827+0000 osd.1 (osd.1) 388 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:58:08.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[22840]: cluster 2024-09-13T22:58:06.133988+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:58:08.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[22840]: cluster 2024-09-13T22:58:07.240658+0000 osd.7 (osd.7) 402 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[22840]: cluster 2024-09-13T22:58:07.403484+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[22840]: cluster 2024-09-13T22:58:07.404716+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[29889]: audit 2024-09-13T22:58:05.866094+0000 mgr.y (mgr.14613) 759 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[29889]: cluster 2024-09-13T22:58:06.131827+0000 osd.1 (osd.1) 388 : cluster [DBG] 4.2 scrub starts 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[29889]: cluster 2024-09-13T22:58:06.133988+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.2 scrub ok 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[29889]: cluster 2024-09-13T22:58:07.240658+0000 osd.7 (osd.7) 402 : cluster [DBG] 2.5 scrub starts 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[29889]: cluster 2024-09-13T22:58:07.403484+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-09-13T22:58:08.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:07 smithi178 bash[29889]: cluster 2024-09-13T22:58:07.404716+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-09-13T22:58:08.152 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:08.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:08.283 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:08.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:08.411 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:08.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:08.540 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:08.643 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:08.665 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:08.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 2 16 4935 4919 153.708 149.562 0.00403099 0.00649116 2024-09-13T22:58:08.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:06.910705+0000 mgr.y (mgr.14613) 760 : cluster [DBG] pgmap v897: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 707 B/s rd, 0 op/s 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:07.105020+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:07.109063+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:07.245365+0000 osd.7 (osd.7) 403 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:08.023661+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:08.025252+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:08.128134+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e403: 8 total, 8 up, 6 in 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:08.198943+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:58:08.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:08 smithi192 bash[24608]: cluster 2024-09-13T22:58:08.200981+0000 osd.7 (osd.7) 405 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:58:08.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:08.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:08.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:08.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:09.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:06.910705+0000 mgr.y (mgr.14613) 760 : cluster [DBG] pgmap v897: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 707 B/s rd, 0 op/s 2024-09-13T22:58:09.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:07.105020+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:58:09.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:07.109063+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:58:09.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:07.245365+0000 osd.7 (osd.7) 403 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:58:09.007 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:08.023661+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:08.025252+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:08.128134+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e403: 8 total, 8 up, 6 in 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:08.198943+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[29889]: cluster 2024-09-13T22:58:08.200981+0000 osd.7 (osd.7) 405 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:06.910705+0000 mgr.y (mgr.14613) 760 : cluster [DBG] pgmap v897: 158 pgs: 158 active+clean; 583 KiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 707 B/s rd, 0 op/s 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:07.105020+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub starts 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:07.109063+0000 osd.1 (osd.1) 391 : cluster [DBG] 6.4 scrub ok 2024-09-13T22:58:09.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:07.245365+0000 osd.7 (osd.7) 403 : cluster [DBG] 2.5 scrub ok 2024-09-13T22:58:09.009 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:08.023661+0000 osd.5 (osd.5) 431 : cluster [DBG] 6.1e scrub starts 2024-09-13T22:58:09.009 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:08.025252+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.1e scrub ok 2024-09-13T22:58:09.009 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:08.128134+0000 mon.a (mon.0) 1662 : cluster [DBG] osdmap e403: 8 total, 8 up, 6 in 2024-09-13T22:58:09.009 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:08.198943+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.11 scrub starts 2024-09-13T22:58:09.009 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:08 smithi178 bash[22840]: cluster 2024-09-13T22:58:08.200981+0000 osd.7 (osd.7) 405 : cluster [DBG] 5.11 scrub ok 2024-09-13T22:58:09.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:09.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:09.107 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:09.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:09.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:09.280 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:09.303 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:09.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:09.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:09.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:09.562 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:09.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:09.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:09.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 3 16 6941 6925 144.259 125.375 0.00865702 0.00691877 2024-09-13T22:58:09.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:09.821 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:09 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.014400+0000 osd.5 (osd.5) 433 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:58:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:09 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.016213+0000 osd.5 (osd.5) 434 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:58:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:09 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.050855+0000 osd.1 (osd.1) 392 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:58:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:09 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.054424+0000 osd.1 (osd.1) 393 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:58:09.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:09.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:58:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:58:09.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.014400+0000 osd.5 (osd.5) 433 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:58:09.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.016213+0000 osd.5 (osd.5) 434 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:58:09.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.050855+0000 osd.1 (osd.1) 392 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:58:09.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.054424+0000 osd.1 (osd.1) 393 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:58:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.014400+0000 osd.5 (osd.5) 433 : cluster [DBG] 5.1b scrub starts 2024-09-13T22:58:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.016213+0000 osd.5 (osd.5) 434 : cluster [DBG] 5.1b scrub ok 2024-09-13T22:58:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.050855+0000 osd.1 (osd.1) 392 : cluster [DBG] 3.4 scrub starts 2024-09-13T22:58:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:09 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.054424+0000 osd.1 (osd.1) 393 : cluster [DBG] 3.4 scrub ok 2024-09-13T22:58:09.949 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:09.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:10.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:10.079 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:10.140 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:10.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:10.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:10.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:10.336 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:10.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:10.465 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:10.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:10.596 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:10.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:10.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 4 16 8880 8864 138.488 121.188 0.00521415 0.00715715 2024-09-13T22:58:10.726 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:10.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:10.856 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:08.911341+0000 mgr.y (mgr.14613) 761 : cluster [DBG] pgmap v899: 158 pgs: 158 active+clean; 187 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 374 op/s 2024-09-13T22:58:10.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.753812+0000 osd.3 (osd.3) 375 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:58:10.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.777646+0000 osd.3 (osd.3) 376 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:58:10.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.993144+0000 osd.5 (osd.5) 435 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:58:10.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:09.995441+0000 osd.5 (osd.5) 436 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:10.095338+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.c deep-scrub starts 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:10.306564+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[29889]: cluster 2024-09-13T22:58:10.310664+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:08.911341+0000 mgr.y (mgr.14613) 761 : cluster [DBG] pgmap v899: 158 pgs: 158 active+clean; 187 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 374 op/s 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.753812+0000 osd.3 (osd.3) 375 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.777646+0000 osd.3 (osd.3) 376 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.993144+0000 osd.5 (osd.5) 435 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:09.995441+0000 osd.5 (osd.5) 436 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:10.095338+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.c deep-scrub starts 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:10.306564+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:58:10.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:10 smithi178 bash[22840]: cluster 2024-09-13T22:58:10.310664+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:58:10.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:08.911341+0000 mgr.y (mgr.14613) 761 : cluster [DBG] pgmap v899: 158 pgs: 158 active+clean; 187 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 374 op/s 2024-09-13T22:58:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.753812+0000 osd.3 (osd.3) 375 : cluster [DBG] 3.2 scrub starts 2024-09-13T22:58:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.777646+0000 osd.3 (osd.3) 376 : cluster [DBG] 3.2 scrub ok 2024-09-13T22:58:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.993144+0000 osd.5 (osd.5) 435 : cluster [DBG] 3.5 scrub starts 2024-09-13T22:58:10.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:09.995441+0000 osd.5 (osd.5) 436 : cluster [DBG] 3.5 scrub ok 2024-09-13T22:58:10.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:10.095338+0000 osd.1 (osd.1) 394 : cluster [DBG] 5.c deep-scrub starts 2024-09-13T22:58:10.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:10.306564+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.2 scrub starts 2024-09-13T22:58:10.970 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:10 smithi192 bash[24608]: cluster 2024-09-13T22:58:10.310664+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.2 scrub ok 2024-09-13T22:58:10.983 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:10.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:11.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:11.111 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:11.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:11.253 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:11.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:11.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:11.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:11.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:11.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:11.637 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:11.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 5 16 10653 10637 132.95 110.812 0.00584353 0.00751322 2024-09-13T22:58:11.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:11.768 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:11.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:11.895 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:11.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:11.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:11 smithi178 bash[29889]: cluster 2024-09-13T22:58:10.123457+0000 osd.1 (osd.1) 395 : cluster [DBG] 5.c deep-scrub ok 2024-09-13T22:58:11.961 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:11 smithi178 bash[29889]: cluster 2024-09-13T22:58:11.041453+0000 osd.5 (osd.5) 437 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:58:11.962 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:11 smithi178 bash[29889]: cluster 2024-09-13T22:58:11.043735+0000 osd.5 (osd.5) 438 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:58:11.962 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:11 smithi178 bash[22840]: cluster 2024-09-13T22:58:10.123457+0000 osd.1 (osd.1) 395 : cluster [DBG] 5.c deep-scrub ok 2024-09-13T22:58:11.962 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:11 smithi178 bash[22840]: cluster 2024-09-13T22:58:11.041453+0000 osd.5 (osd.5) 437 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:58:11.962 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:11 smithi178 bash[22840]: cluster 2024-09-13T22:58:11.043735+0000 osd.5 (osd.5) 438 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:58:11.998 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:12.016 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:12.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:12.129 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:11 smithi192 bash[24608]: cluster 2024-09-13T22:58:10.123457+0000 osd.1 (osd.1) 395 : cluster [DBG] 5.c deep-scrub ok 2024-09-13T22:58:12.129 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:11 smithi192 bash[24608]: cluster 2024-09-13T22:58:11.041453+0000 osd.5 (osd.5) 437 : cluster [DBG] 4.1 scrub starts 2024-09-13T22:58:12.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:11 smithi192 bash[24608]: cluster 2024-09-13T22:58:11.043735+0000 osd.5 (osd.5) 438 : cluster [DBG] 4.1 scrub ok 2024-09-13T22:58:12.143 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:12.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:12.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:12.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:12.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:12.531 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:12.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:12.655 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:12.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 6 16 12215 12199 127.061 97.625 0.00532279 0.00786132 2024-09-13T22:58:12.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:12.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:12.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:10.912237+0000 mgr.y (mgr.14613) 762 : cluster [DBG] pgmap v900: 158 pgs: 158 active+clean; 308 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 616 op/s 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.163849+0000 osd.5 (osd.5) 439 : cluster [DBG] 1.a scrub starts 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.165907+0000 osd.5 (osd.5) 440 : cluster [DBG] 1.a scrub ok 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.347785+0000 osd.0 (osd.0) 395 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.352668+0000 osd.0 (osd.0) 396 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.451627+0000 osd.2 (osd.2) 289 : cluster [DBG] 2.c scrub starts 2024-09-13T22:58:12.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:12 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.453743+0000 osd.2 (osd.2) 290 : cluster [DBG] 2.c scrub ok 2024-09-13T22:58:12.910 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:12.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:10.912237+0000 mgr.y (mgr.14613) 762 : cluster [DBG] pgmap v900: 158 pgs: 158 active+clean; 308 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 616 op/s 2024-09-13T22:58:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.163849+0000 osd.5 (osd.5) 439 : cluster [DBG] 1.a scrub starts 2024-09-13T22:58:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.165907+0000 osd.5 (osd.5) 440 : cluster [DBG] 1.a scrub ok 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.347785+0000 osd.0 (osd.0) 395 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.352668+0000 osd.0 (osd.0) 396 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.451627+0000 osd.2 (osd.2) 289 : cluster [DBG] 2.c scrub starts 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.453743+0000 osd.2 (osd.2) 290 : cluster [DBG] 2.c scrub ok 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:10.912237+0000 mgr.y (mgr.14613) 762 : cluster [DBG] pgmap v900: 158 pgs: 158 active+clean; 308 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 616 op/s 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.163849+0000 osd.5 (osd.5) 439 : cluster [DBG] 1.a scrub starts 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.165907+0000 osd.5 (osd.5) 440 : cluster [DBG] 1.a scrub ok 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.347785+0000 osd.0 (osd.0) 395 : cluster [DBG] 3.1b scrub starts 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.352668+0000 osd.0 (osd.0) 396 : cluster [DBG] 3.1b scrub ok 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.451627+0000 osd.2 (osd.2) 289 : cluster [DBG] 2.c scrub starts 2024-09-13T22:58:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:12 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.453743+0000 osd.2 (osd.2) 290 : cluster [DBG] 2.c scrub ok 2024-09-13T22:58:13.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:13.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:13.166 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:13.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:13.294 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:13.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:13.428 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:13.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:13.559 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:13.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:13.700 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:13.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 7 16 13957 13941 124.462 108.875 0.00501767 0.00802533 2024-09-13T22:58:13.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:13.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:13.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:13.962 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:13.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.712282+0000 osd.3 (osd.3) 377 : cluster [DBG] 6.c deep-scrub starts 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.718264+0000 osd.3 (osd.3) 378 : cluster [DBG] 6.c deep-scrub ok 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.127665+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e404: 8 total, 8 up, 6 in 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.131708+0000 osd.5 (osd.5) 441 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.134256+0000 osd.5 (osd.5) 442 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.325001+0000 osd.0 (osd.0) 397 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:58:14.065 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:13 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.326186+0000 osd.0 (osd.0) 398 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:58:14.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:14.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.712282+0000 osd.3 (osd.3) 377 : cluster [DBG] 6.c deep-scrub starts 2024-09-13T22:58:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.718264+0000 osd.3 (osd.3) 378 : cluster [DBG] 6.c deep-scrub ok 2024-09-13T22:58:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.127665+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e404: 8 total, 8 up, 6 in 2024-09-13T22:58:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.131708+0000 osd.5 (osd.5) 441 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:58:14.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.134256+0000 osd.5 (osd.5) 442 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:58:14.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.325001+0000 osd.0 (osd.0) 397 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:58:14.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:13 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.326186+0000 osd.0 (osd.0) 398 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:58:14.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:14.222 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:14.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:14.349 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.712282+0000 osd.3 (osd.3) 377 : cluster [DBG] 6.c deep-scrub starts 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.718264+0000 osd.3 (osd.3) 378 : cluster [DBG] 6.c deep-scrub ok 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.127665+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e404: 8 total, 8 up, 6 in 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.131708+0000 osd.5 (osd.5) 441 : cluster [DBG] 2.17 scrub starts 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.134256+0000 osd.5 (osd.5) 442 : cluster [DBG] 2.17 scrub ok 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.325001+0000 osd.0 (osd.0) 397 : cluster [DBG] 6.10 scrub starts 2024-09-13T22:58:14.354 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.326186+0000 osd.0 (osd.0) 398 : cluster [DBG] 6.10 scrub ok 2024-09-13T22:58:14.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:14.478 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:14.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:14.604 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:14.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:14.709 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 8 16 15414 15398 120.286 91.0625 0.00581946 0.00829685 2024-09-13T22:58:14.728 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:14.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:14.855 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:14.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:14.995 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:14.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:15.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:15.110 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:12.912860+0000 mgr.y (mgr.14613) 763 : cluster [DBG] pgmap v901: 158 pgs: 158 active+clean; 347 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 35 MiB/s wr, 566 op/s 2024-09-13T22:58:15.110 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.264833+0000 osd.7 (osd.7) 406 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:13.267272+0000 osd.7 (osd.7) 407 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.161274+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.162717+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.330572+0000 osd.0 (osd.0) 399 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.333480+0000 osd.0 (osd.0) 400 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.487479+0000 osd.2 (osd.2) 291 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:58:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:14 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.492064+0000 osd.2 (osd.2) 292 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:58:15.126 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:15.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:12.912860+0000 mgr.y (mgr.14613) 763 : cluster [DBG] pgmap v901: 158 pgs: 158 active+clean; 347 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 35 MiB/s wr, 566 op/s 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.264833+0000 osd.7 (osd.7) 406 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:13.267272+0000 osd.7 (osd.7) 407 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.161274+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.162717+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.330572+0000 osd.0 (osd.0) 399 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.333480+0000 osd.0 (osd.0) 400 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.487479+0000 osd.2 (osd.2) 291 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.492064+0000 osd.2 (osd.2) 292 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:58:15.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:12.912860+0000 mgr.y (mgr.14613) 763 : cluster [DBG] pgmap v901: 158 pgs: 158 active+clean; 347 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 35 MiB/s wr, 566 op/s 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.264833+0000 osd.7 (osd.7) 406 : cluster [DBG] 6.3 scrub starts 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:13.267272+0000 osd.7 (osd.7) 407 : cluster [DBG] 6.3 scrub ok 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.161274+0000 osd.5 (osd.5) 443 : cluster [DBG] 6.19 scrub starts 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.162717+0000 osd.5 (osd.5) 444 : cluster [DBG] 6.19 scrub ok 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.330572+0000 osd.0 (osd.0) 399 : cluster [DBG] 3.17 scrub starts 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.333480+0000 osd.0 (osd.0) 400 : cluster [DBG] 3.17 scrub ok 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.487479+0000 osd.2 (osd.2) 291 : cluster [DBG] 1.5 scrub starts 2024-09-13T22:58:15.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:14 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.492064+0000 osd.2 (osd.2) 292 : cluster [DBG] 1.5 scrub ok 2024-09-13T22:58:15.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:15.255 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:15.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:15.385 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:15.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:15.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:15.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:15.635 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:15.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 9 16 16820 16804 116.684 87.875 0.0114257 0.00856124 2024-09-13T22:58:15.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:15.756 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:15.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:15.891 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:15.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:15.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.314049+0000 osd.7 (osd.7) 408 : cluster [DBG] 3.d scrub starts 2024-09-13T22:58:15.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.318889+0000 osd.7 (osd.7) 409 : cluster [DBG] 3.d scrub ok 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.750010+0000 osd.3 (osd.3) 379 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.754422+0000 osd.3 (osd.3) 380 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.062476+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.074731+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.297653+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.305290+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.503778+0000 osd.2 (osd.2) 293 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.505975+0000 osd.2 (osd.2) 294 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.314049+0000 osd.7 (osd.7) 408 : cluster [DBG] 3.d scrub starts 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.318889+0000 osd.7 (osd.7) 409 : cluster [DBG] 3.d scrub ok 2024-09-13T22:58:15.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.750010+0000 osd.3 (osd.3) 379 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.754422+0000 osd.3 (osd.3) 380 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.062476+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.074731+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.297653+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.305290+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.503778+0000 osd.2 (osd.2) 293 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:58:15.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:15 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.505975+0000 osd.2 (osd.2) 294 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:58:15.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:16.019 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:16.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:16.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.314049+0000 osd.7 (osd.7) 408 : cluster [DBG] 3.d scrub starts 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.318889+0000 osd.7 (osd.7) 409 : cluster [DBG] 3.d scrub ok 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.750010+0000 osd.3 (osd.3) 379 : cluster [DBG] 5.14 scrub starts 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.754422+0000 osd.3 (osd.3) 380 : cluster [DBG] 5.14 scrub ok 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.062476+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.074731+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-09-13T22:58:16.132 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.297653+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.1 scrub starts 2024-09-13T22:58:16.133 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.305290+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.1 scrub ok 2024-09-13T22:58:16.133 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.503778+0000 osd.2 (osd.2) 293 : cluster [DBG] 5.6 scrub starts 2024-09-13T22:58:16.133 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:15 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.505975+0000 osd.2 (osd.2) 294 : cluster [DBG] 5.6 scrub ok 2024-09-13T22:58:16.146 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:16.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:16.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:16.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:16.399 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:16.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:16.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:16.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:16.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:16.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 10 16 18182 18166 113.527 85.125 0.00166789 0.00875723 2024-09-13T22:58:16.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:16.788 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:16.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:16.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:16 smithi192 bash[24608]: cluster 2024-09-13T22:58:14.913811+0000 mgr.y (mgr.14613) 764 : cluster [DBG] pgmap v903: 158 pgs: 158 active+clean; 873 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-13T22:58:16.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:16 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.348758+0000 osd.7 (osd.7) 410 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:58:16.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:16 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.349697+0000 osd.7 (osd.7) 411 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:58:16.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:16.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:17.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:17.042 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:16 smithi178 bash[22840]: cluster 2024-09-13T22:58:14.913811+0000 mgr.y (mgr.14613) 764 : cluster [DBG] pgmap v903: 158 pgs: 158 active+clean; 873 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-13T22:58:17.042 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:16 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.348758+0000 osd.7 (osd.7) 410 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:58:17.042 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:16 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.349697+0000 osd.7 (osd.7) 411 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:58:17.043 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:16 smithi178 bash[29889]: cluster 2024-09-13T22:58:14.913811+0000 mgr.y (mgr.14613) 764 : cluster [DBG] pgmap v903: 158 pgs: 158 active+clean; 873 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-13T22:58:17.043 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:16 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.348758+0000 osd.7 (osd.7) 410 : cluster [DBG] 6.15 scrub starts 2024-09-13T22:58:17.043 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:16 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.349697+0000 osd.7 (osd.7) 411 : cluster [DBG] 6.15 scrub ok 2024-09-13T22:58:17.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:17.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:17.178 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:17.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:17.308 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:17.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:17.438 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:17.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:17.576 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:17.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:17.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:17.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 11 15 18761 18746 106.502 36.25 0.00664443 0.00938073 2024-09-13T22:58:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:17.812 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:17.823 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.820875+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:58:17.823 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:15.827324+0000 osd.3 (osd.3) 382 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: audit 2024-09-13T22:58:15.873975+0000 mgr.y (mgr.14613) 765 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:16.341088+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.b scrub starts 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:16.344430+0000 osd.7 (osd.7) 413 : cluster [DBG] 2.b scrub ok 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:16.378032+0000 osd.0 (osd.0) 403 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:16.383907+0000 osd.0 (osd.0) 404 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:17.056160+0000 osd.5 (osd.5) 445 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[29889]: cluster 2024-09-13T22:58:17.058441+0000 osd.5 (osd.5) 446 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.820875+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:15.827324+0000 osd.3 (osd.3) 382 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:58:17.824 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: audit 2024-09-13T22:58:15.873975+0000 mgr.y (mgr.14613) 765 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:17.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:16.341088+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.b scrub starts 2024-09-13T22:58:17.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:16.344430+0000 osd.7 (osd.7) 413 : cluster [DBG] 2.b scrub ok 2024-09-13T22:58:17.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:16.378032+0000 osd.0 (osd.0) 403 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:58:17.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:16.383907+0000 osd.0 (osd.0) 404 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:58:17.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:17.056160+0000 osd.5 (osd.5) 445 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:58:17.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:17 smithi178 bash[22840]: cluster 2024-09-13T22:58:17.058441+0000 osd.5 (osd.5) 446 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:58:17.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:17.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:17.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.820875+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.1b scrub starts 2024-09-13T22:58:17.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:15.827324+0000 osd.3 (osd.3) 382 : cluster [DBG] 6.1b scrub ok 2024-09-13T22:58:17.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: audit 2024-09-13T22:58:15.873975+0000 mgr.y (mgr.14613) 765 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:17.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:16.341088+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.b scrub starts 2024-09-13T22:58:17.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:16.344430+0000 osd.7 (osd.7) 413 : cluster [DBG] 2.b scrub ok 2024-09-13T22:58:17.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:16.378032+0000 osd.0 (osd.0) 403 : cluster [DBG] 6.9 scrub starts 2024-09-13T22:58:17.957 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:16.383907+0000 osd.0 (osd.0) 404 : cluster [DBG] 6.9 scrub ok 2024-09-13T22:58:17.957 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:17.056160+0000 osd.5 (osd.5) 445 : cluster [DBG] 2.2 scrub starts 2024-09-13T22:58:17.957 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:17 smithi192 bash[24608]: cluster 2024-09-13T22:58:17.058441+0000 osd.5 (osd.5) 446 : cluster [DBG] 2.2 scrub ok 2024-09-13T22:58:17.971 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:17.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:18.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:18.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:18.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:18.227 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:18.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:18.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:18.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:18.487 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:18.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:18.614 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:18.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 12 16 20387 20371 106.089 101.562 0.00894498 0.00941887 2024-09-13T22:58:18.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:18.746 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:18.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:18.874 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:18.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:18.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:18.987 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:18 smithi178 bash[29889]: cluster 2024-09-13T22:58:16.914952+0000 mgr.y (mgr.14613) 766 : cluster [DBG] pgmap v904: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 902 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-13T22:58:18.987 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:18 smithi178 bash[29889]: cluster 2024-09-13T22:58:18.127543+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e405: 8 total, 8 up, 6 in 2024-09-13T22:58:18.988 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:18 smithi178 bash[22840]: cluster 2024-09-13T22:58:16.914952+0000 mgr.y (mgr.14613) 766 : cluster [DBG] pgmap v904: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 902 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-13T22:58:18.988 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:18 smithi178 bash[22840]: cluster 2024-09-13T22:58:18.127543+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e405: 8 total, 8 up, 6 in 2024-09-13T22:58:19.008 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:19.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:19.122 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:18 smithi192 bash[24608]: cluster 2024-09-13T22:58:16.914952+0000 mgr.y (mgr.14613) 766 : cluster [DBG] pgmap v904: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 902 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-13T22:58:19.122 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:18 smithi192 bash[24608]: cluster 2024-09-13T22:58:18.127543+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e405: 8 total, 8 up, 6 in 2024-09-13T22:58:19.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:19.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:19.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:19.294 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:58:19.294 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:58:19.342 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:58:19.343 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:58:19.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:19.398 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:19.402 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:58:19.402 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:58:19.413 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:58:19.413 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:58:19.437 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:58:19.438 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:58:19.489 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:58:19.490 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:58:19.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:19.525 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:58:19.525 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:58:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.532 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:19.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:19.568 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:58:19.568 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:58:19.600 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:58:19.600 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:58:19.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:19.666 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:19.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 13 16 21696 21680 104.221 81.8125 0.0157215 0.00958658 2024-09-13T22:58:19.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:58:19.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:19.771 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:58:19.804 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T22:58:19.835 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:19.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:19.963 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:19.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:20.042 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:58:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:58:20.042 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[29889]: audit 2024-09-13T22:58:19.218475+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.178:0/2296007894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:20.042 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[29889]: audit 2024-09-13T22:58:19.317744+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? v1:172.21.15.178:0/3635595246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:58:20.042 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[29889]: audit 2024-09-13T22:58:19.373569+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:20.042 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[29889]: audit 2024-09-13T22:58:19.375915+0000 mon.b (mon.1) 267 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:20.042 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[22840]: audit 2024-09-13T22:58:19.218475+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.178:0/2296007894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:20.043 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[22840]: audit 2024-09-13T22:58:19.317744+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? v1:172.21.15.178:0/3635595246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:58:20.043 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[22840]: audit 2024-09-13T22:58:19.373569+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:20.043 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:19 smithi178 bash[22840]: audit 2024-09-13T22:58:19.375915+0000 mon.b (mon.1) 267 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:20.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:20.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:19 smithi192 bash[24608]: audit 2024-09-13T22:58:19.218475+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.178:0/2296007894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:19 smithi192 bash[24608]: audit 2024-09-13T22:58:19.317744+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? v1:172.21.15.178:0/3635595246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T22:58:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:19 smithi192 bash[24608]: audit 2024-09-13T22:58:19.373569+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:19 smithi192 bash[24608]: audit 2024-09-13T22:58:19.375915+0000 mon.b (mon.1) 267 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:20.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:20.225 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:20.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:20.360 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:20.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:20.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:20.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:20.619 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:20.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 14 16 23334 23318 104.089 102.375 0.00915159 0.0095996 2024-09-13T22:58:20.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:20.745 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:20.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:20.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:18.829808+0000 osd.3 (osd.3) 383 : cluster [DBG] 1.4 scrub starts 2024-09-13T22:58:20.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:18.833188+0000 osd.3 (osd.3) 384 : cluster [DBG] 1.4 scrub ok 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:18.915851+0000 mgr.y (mgr.14613) 767 : cluster [DBG] pgmap v906: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:19.356183+0000 osd.7 (osd.7) 414 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:19.363452+0000 osd.7 (osd.7) 415 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: audit 2024-09-13T22:58:19.746236+0000 mgr.y (mgr.14613) 768 : audit [DBG] from='client.25826 v1:172.21.15.178:0/627110244' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:19.787920+0000 mon.a (mon.0) 1667 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: audit 2024-09-13T22:58:19.804673+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.178:0/3635595246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:20 smithi192 bash[24608]: cluster 2024-09-13T22:58:19.821009+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e406: 8 total, 8 up, 6 in 2024-09-13T22:58:20.870 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:20.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:20.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:20.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:21.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:21.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:21.197 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:18.829808+0000 osd.3 (osd.3) 383 : cluster [DBG] 1.4 scrub starts 2024-09-13T22:58:21.197 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:18.833188+0000 osd.3 (osd.3) 384 : cluster [DBG] 1.4 scrub ok 2024-09-13T22:58:21.197 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:18.915851+0000 mgr.y (mgr.14613) 767 : cluster [DBG] pgmap v906: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-13T22:58:21.197 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:19.356183+0000 osd.7 (osd.7) 414 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:58:21.197 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:19.363452+0000 osd.7 (osd.7) 415 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: audit 2024-09-13T22:58:19.746236+0000 mgr.y (mgr.14613) 768 : audit [DBG] from='client.25826 v1:172.21.15.178:0/627110244' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:19.787920+0000 mon.a (mon.0) 1667 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: audit 2024-09-13T22:58:19.804673+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.178:0/3635595246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[22840]: cluster 2024-09-13T22:58:19.821009+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e406: 8 total, 8 up, 6 in 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:18.829808+0000 osd.3 (osd.3) 383 : cluster [DBG] 1.4 scrub starts 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:18.833188+0000 osd.3 (osd.3) 384 : cluster [DBG] 1.4 scrub ok 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:18.915851+0000 mgr.y (mgr.14613) 767 : cluster [DBG] pgmap v906: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:19.356183+0000 osd.7 (osd.7) 414 : cluster [DBG] 3.1d scrub starts 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:19.363452+0000 osd.7 (osd.7) 415 : cluster [DBG] 3.1d scrub ok 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: audit 2024-09-13T22:58:19.746236+0000 mgr.y (mgr.14613) 768 : audit [DBG] from='client.25826 v1:172.21.15.178:0/627110244' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:58:21.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:19.787920+0000 mon.a (mon.0) 1667 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:58:21.199 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: audit 2024-09-13T22:58:19.804673+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.178:0/3635595246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T22:58:21.199 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:20 smithi178 bash[29889]: cluster 2024-09-13T22:58:19.821009+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e406: 8 total, 8 up, 6 in 2024-09-13T22:58:21.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:21.250 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:21.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:21.384 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:21.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:21.510 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:21.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:21.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:21.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 15 16 24846 24830 103.449 94.5 0.0133788 0.00965958 2024-09-13T22:58:21.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:21.772 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:21.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:21.884 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:21 smithi192 bash[24608]: cluster 2024-09-13T22:58:20.916286+0000 mgr.y (mgr.14613) 769 : cluster [DBG] pgmap v908: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.2 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-13T22:58:21.900 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:21.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:22.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.033 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:21 smithi178 bash[29889]: cluster 2024-09-13T22:58:20.916286+0000 mgr.y (mgr.14613) 769 : cluster [DBG] pgmap v908: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.2 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-13T22:58:22.039 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:21 smithi178 bash[22840]: cluster 2024-09-13T22:58:20.916286+0000 mgr.y (mgr.14613) 769 : cluster [DBG] pgmap v908: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.2 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-13T22:58:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:22.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.170 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:22.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.296 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:22.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:22.424 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:22.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:22.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:22.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:22.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:22.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:22.710 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 16 16 26416 26400 103.116 98.125 0.00434932 0.00969013 2024-09-13T22:58:22.796 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:22.824 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:22.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:22.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:23.065 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:23.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:23.235 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:23.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:23.362 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:23.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:23.486 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:23.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:23.614 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:23.711 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 17 16 27455 27439 100.87 64.9375 0.00732882 0.00981341 2024-09-13T22:58:23.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:23.745 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:23.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:23.876 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:23.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:23.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:24.010 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:24.113 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:24.143 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:24.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:23 smithi192 bash[24608]: cluster 2024-09-13T22:58:22.916994+0000 mgr.y (mgr.14613) 770 : cluster [DBG] pgmap v909: 158 pgs: 158 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 511 B/s rd, 53 MiB/s wr, 853 op/s 2024-09-13T22:58:24.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:24.256 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:23 smithi178 bash[29889]: cluster 2024-09-13T22:58:22.916994+0000 mgr.y (mgr.14613) 770 : cluster [DBG] pgmap v909: 158 pgs: 158 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 511 B/s rd, 53 MiB/s wr, 853 op/s 2024-09-13T22:58:24.257 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:23 smithi178 bash[22840]: cluster 2024-09-13T22:58:22.916994+0000 mgr.y (mgr.14613) 770 : cluster [DBG] pgmap v909: 158 pgs: 158 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 511 B/s rd, 53 MiB/s wr, 853 op/s 2024-09-13T22:58:24.280 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:24.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:24.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:24.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:24.531 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:24.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:24.662 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:24.711 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 18 16 28096 28080 97.4911 40.0625 0.00220397 0.0101055 2024-09-13T22:58:24.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:24.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:24.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T22:58:24.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:24.926 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:24.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:25.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:25.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:25.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.308 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:25.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:25.537 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:25.566 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:25.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:25.699 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:25.711 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 19 15 29262 29247 96.1983 72.9375 0.00456017 0.0103873 2024-09-13T22:58:25.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:25.829 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:25.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.956 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:25.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:26.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:26.075 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":906,"stamp":"2024-09-13T22:58:18.915220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1211177499,"num_objects":18671,"num_object_clones":0,"num_object_copies":56013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18671,"num_whiteouts":0,"num_read":2803,"num_read_kb":2555,"num_write":19480,"num_write_kb":1182253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":7524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21089,"ondisk_log_size":21089,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":7149952,"kb_used_data":3358812,"kb_used_omap":9,"kb_used_meta":3791030,"kb_avail":555320960,"statfs":{"total":575970213888,"available":568648663040,"internally_reserved":0,"allocated":3439423488,"data_stored":3436561725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":3882015397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,2,0,0,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":888012800,"num_objects":13550,"num_object_clones":0,"num_object_copies":40650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13550,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":13550,"num_write_kb":867200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003443"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":727,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157556+0000","last_change":"2024-09-13T22:58:15.827424+0000","last_active":"2024-09-13T22:58:18.157556+0000","last_peered":"2024-09-13T22:58:18.157556+0000","last_clean":"2024-09-13T22:58:18.157556+0000","last_became_active":"2024-09-13T22:56:00.229118+0000","last_became_peered":"2024-09-13T22:56:00.229118+0000","last_unstale":"2024-09-13T22:58:18.157556+0000","last_undegraded":"2024-09-13T22:58:18.157556+0000","last_fullsized":"2024-09-13T22:58:18.157556+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.827330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.827330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.236496+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":779,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129556+0000","last_change":"2024-09-13T22:57:47.259212+0000","last_active":"2024-09-13T22:58:13.129556+0000","last_peered":"2024-09-13T22:58:13.129556+0000","last_clean":"2024-09-13T22:58:13.129556+0000","last_became_active":"2024-09-13T22:56:00.216437+0000","last_became_peered":"2024-09-13T22:56:00.216437+0000","last_unstale":"2024-09-13T22:58:13.129556+0000","last_undegraded":"2024-09-13T22:58:13.129556+0000","last_fullsized":"2024-09-13T22:58:13.129556+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.259089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:22.232806+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.259089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.198025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":807,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158148+0000","last_change":"2024-09-13T22:57:46.573120+0000","last_active":"2024-09-13T22:58:18.158148+0000","last_peered":"2024-09-13T22:58:18.158148+0000","last_clean":"2024-09-13T22:58:18.158148+0000","last_became_active":"2024-09-13T22:56:00.218204+0000","last_became_peered":"2024-09-13T22:56:00.218204+0000","last_unstale":"2024-09-13T22:58:18.158148+0000","last_undegraded":"2024-09-13T22:58:18.158148+0000","last_fullsized":"2024-09-13T22:58:18.158148+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'25","last_scrub_stamp":"2024-09-13T22:57:46.573001+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.573001+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:09.136946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":238,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.143305+0000","last_change":"2024-09-13T22:57:00.773936+0000","last_active":"2024-09-13T22:58:18.143305+0000","last_peered":"2024-09-13T22:58:18.143305+0000","last_clean":"2024-09-13T22:58:18.143305+0000","last_became_active":"2024-09-13T22:56:00.235200+0000","last_became_peered":"2024-09-13T22:56:00.235200+0000","last_unstale":"2024-09-13T22:58:18.143305+0000","last_undegraded":"2024-09-13T22:58:18.143305+0000","last_fullsized":"2024-09-13T22:58:18.143305+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:00.773837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:57:00.773837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":735,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158181+0000","last_change":"2024-09-13T22:56:28.054932+0000","last_active":"2024-09-13T22:58:18.158181+0000","last_peered":"2024-09-13T22:58:18.158181+0000","last_clean":"2024-09-13T22:58:18.158181+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:58:18.158181+0000","last_undegraded":"2024-09-13T22:58:18.158181+0000","last_fullsized":"2024-09-13T22:58:18.158181+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:28.051743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.051743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":793,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129268+0000","last_change":"2024-09-13T22:57:06.415408+0000","last_active":"2024-09-13T22:58:13.129268+0000","last_peered":"2024-09-13T22:58:13.129268+0000","last_clean":"2024-09-13T22:58:13.129268+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:58:13.129268+0000","last_undegraded":"2024-09-13T22:58:13.129268+0000","last_fullsized":"2024-09-13T22:58:13.129268+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:57:06.415278+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.415278+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":733,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.143289+0000","last_change":"2024-09-13T22:57:49.171117+0000","last_active":"2024-09-13T22:58:18.143289+0000","last_peered":"2024-09-13T22:58:18.143289+0000","last_clean":"2024-09-13T22:58:18.143289+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:58:18.143289+0000","last_undegraded":"2024-09-13T22:58:18.143289+0000","last_fullsized":"2024-09-13T22:58:18.143289+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.170845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.170845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.075808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":251,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159272+0000","last_change":"2024-09-13T22:57:09.115091+0000","last_active":"2024-09-13T22:58:18.159272+0000","last_peered":"2024-09-13T22:58:18.159272+0000","last_clean":"2024-09-13T22:58:18.159272+0000","last_became_active":"2024-09-13T22:55:38.143783+0000","last_became_peered":"2024-09-13T22:55:38.143783+0000","last_unstale":"2024-09-13T22:58:18.159272+0000","last_undegraded":"2024-09-13T22:58:18.159272+0000","last_fullsized":"2024-09-13T22:58:18.159272+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.114979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.114979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:34.393471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":737,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137576+0000","last_change":"2024-09-13T22:56:00.226676+0000","last_active":"2024-09-13T22:58:13.137576+0000","last_peered":"2024-09-13T22:58:13.137576+0000","last_clean":"2024-09-13T22:58:13.137576+0000","last_became_active":"2024-09-13T22:56:00.225672+0000","last_became_peered":"2024-09-13T22:56:00.225672+0000","last_unstale":"2024-09-13T22:58:13.137576+0000","last_undegraded":"2024-09-13T22:58:13.137576+0000","last_fullsized":"2024-09-13T22:58:13.137576+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:20.805535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:20.805535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":901,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160069+0000","last_change":"2024-09-13T22:56:41.965853+0000","last_active":"2024-09-13T22:58:18.160069+0000","last_peered":"2024-09-13T22:58:18.160069+0000","last_clean":"2024-09-13T22:58:18.160069+0000","last_became_active":"2024-09-13T22:56:00.233191+0000","last_became_peered":"2024-09-13T22:56:00.233191+0000","last_unstale":"2024-09-13T22:58:18.160069+0000","last_undegraded":"2024-09-13T22:58:18.160069+0000","last_fullsized":"2024-09-13T22:58:18.160069+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:56:41.965747+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:56:41.965747+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":733,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135873+0000","last_change":"2024-09-13T22:55:38.153026+0000","last_active":"2024-09-13T22:58:13.135873+0000","last_peered":"2024-09-13T22:58:13.135873+0000","last_clean":"2024-09-13T22:58:13.135873+0000","last_became_active":"2024-09-13T22:55:38.151599+0000","last_became_peered":"2024-09-13T22:55:38.151599+0000","last_unstale":"2024-09-13T22:58:13.135873+0000","last_undegraded":"2024-09-13T22:58:13.135873+0000","last_fullsized":"2024-09-13T22:58:13.135873+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:21.721055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:55:21.721055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":737,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159926+0000","last_change":"2024-09-13T22:58:14.162918+0000","last_active":"2024-09-13T22:58:18.159926+0000","last_peered":"2024-09-13T22:58:18.159926+0000","last_clean":"2024-09-13T22:58:18.159926+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:58:18.159926+0000","last_undegraded":"2024-09-13T22:58:18.159926+0000","last_fullsized":"2024-09-13T22:58:18.159926+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:14.162730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.162730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:17.036097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":731,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137071+0000","last_change":"2024-09-13T22:54:29.857171+0000","last_active":"2024-09-13T22:58:13.137071+0000","last_peered":"2024-09-13T22:58:13.137071+0000","last_clean":"2024-09-13T22:58:13.137071+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:58:13.137071+0000","last_undegraded":"2024-09-13T22:58:13.137071+0000","last_fullsized":"2024-09-13T22:58:13.137071+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:29.857063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:29.857063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"397'24","reported_seq":285,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136815+0000","last_change":"2024-09-13T22:56:00.249252+0000","last_active":"2024-09-13T22:58:13.136815+0000","last_peered":"2024-09-13T22:58:13.136815+0000","last_clean":"2024-09-13T22:58:13.136815+0000","last_became_active":"2024-09-13T22:56:00.228121+0000","last_became_peered":"2024-09-13T22:56:00.228121+0000","last_unstale":"2024-09-13T22:58:13.136815+0000","last_undegraded":"2024-09-13T22:58:13.136815+0000","last_fullsized":"2024-09-13T22:58:13.136815+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'18","last_scrub_stamp":"2024-09-13T22:54:31.842940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:31.842940+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":741,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159882+0000","last_change":"2024-09-13T22:58:09.016326+0000","last_active":"2024-09-13T22:58:18.159882+0000","last_peered":"2024-09-13T22:58:18.159882+0000","last_clean":"2024-09-13T22:58:18.159882+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:58:18.159882+0000","last_undegraded":"2024-09-13T22:58:18.159882+0000","last_fullsized":"2024-09-13T22:58:18.159882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:09.016219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.016219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.276915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":736,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.130017+0000","last_change":"2024-09-13T22:57:51.203622+0000","last_active":"2024-09-13T22:58:13.130017+0000","last_peered":"2024-09-13T22:58:13.130017+0000","last_clean":"2024-09-13T22:58:13.130017+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:58:13.130017+0000","last_undegraded":"2024-09-13T22:58:13.130017+0000","last_fullsized":"2024-09-13T22:58:13.130017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.203544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.203544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:56.844866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"405'1326","reported_seq":1352,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.711273+0000","last_change":"2024-09-13T22:57:49.649946+0000","last_active":"2024-09-13T22:58:18.711273+0000","last_peered":"2024-09-13T22:58:18.711273+0000","last_clean":"2024-09-13T22:58:18.711273+0000","last_became_active":"2024-09-13T22:57:49.649466+0000","last_became_peered":"2024-09-13T22:57:49.649466+0000","last_unstale":"2024-09-13T22:58:18.711273+0000","last_undegraded":"2024-09-13T22:58:18.711273+0000","last_fullsized":"2024-09-13T22:58:18.711273+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1326,"log_dups_size":0,"ondisk_log_size":1326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":84864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":795,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159718+0000","last_change":"2024-09-13T22:56:37.051234+0000","last_active":"2024-09-13T22:58:18.159718+0000","last_peered":"2024-09-13T22:58:18.159718+0000","last_clean":"2024-09-13T22:58:18.159718+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:58:18.159718+0000","last_undegraded":"2024-09-13T22:58:18.159718+0000","last_fullsized":"2024-09-13T22:58:18.159718+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:37.051143+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:37.051143+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":611,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.141235+0000","last_change":"2024-09-13T22:57:48.143805+0000","last_active":"2024-09-13T22:58:18.141235+0000","last_peered":"2024-09-13T22:58:18.141235+0000","last_clean":"2024-09-13T22:58:18.141235+0000","last_became_active":"2024-09-13T22:56:00.227932+0000","last_became_peered":"2024-09-13T22:56:00.227932+0000","last_unstale":"2024-09-13T22:58:18.141235+0000","last_undegraded":"2024-09-13T22:58:18.141235+0000","last_fullsized":"2024-09-13T22:58:18.141235+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:57:48.143715+0000","last_deep_scrub":"228'44","last_deep_scrub_stamp":"2024-09-13T22:54:34.302937+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.143715+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:54.753954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":742,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136772+0000","last_change":"2024-09-13T22:55:38.144148+0000","last_active":"2024-09-13T22:58:13.136772+0000","last_peered":"2024-09-13T22:58:13.136772+0000","last_clean":"2024-09-13T22:58:13.136772+0000","last_became_active":"2024-09-13T22:55:38.140796+0000","last_became_peered":"2024-09-13T22:55:38.140796+0000","last_unstale":"2024-09-13T22:58:13.136772+0000","last_undegraded":"2024-09-13T22:58:13.136772+0000","last_fullsized":"2024-09-13T22:58:13.136772+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:28.840664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:54:28.840664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":766,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146841+0000","last_change":"2024-09-13T22:58:10.123536+0000","last_active":"2024-09-13T22:58:18.146841+0000","last_peered":"2024-09-13T22:58:18.146841+0000","last_clean":"2024-09-13T22:58:18.146841+0000","last_became_active":"2024-09-13T22:56:00.215333+0000","last_became_peered":"2024-09-13T22:56:00.215333+0000","last_unstale":"2024-09-13T22:58:18.146841+0000","last_undegraded":"2024-09-13T22:58:18.146841+0000","last_fullsized":"2024-09-13T22:58:18.146841+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.123464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:27.721026+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":661,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157835+0000","last_change":"2024-09-13T22:56:58.481848+0000","last_active":"2024-09-13T22:58:18.157835+0000","last_peered":"2024-09-13T22:58:18.157835+0000","last_clean":"2024-09-13T22:58:18.157835+0000","last_became_active":"2024-09-13T22:56:00.222668+0000","last_became_peered":"2024-09-13T22:56:00.222668+0000","last_unstale":"2024-09-13T22:58:18.157835+0000","last_undegraded":"2024-09-13T22:58:18.157835+0000","last_fullsized":"2024-09-13T22:58:18.157835+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:58.481749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:58.481749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"404'900","reported_seq":924,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.034574+0000","last_change":"2024-09-13T22:57:49.652606+0000","last_active":"2024-09-13T22:58:14.034574+0000","last_peered":"2024-09-13T22:58:14.034574+0000","last_clean":"2024-09-13T22:58:14.034574+0000","last_became_active":"2024-09-13T22:57:49.651025+0000","last_became_peered":"2024-09-13T22:57:49.651025+0000","last_unstale":"2024-09-13T22:58:14.034574+0000","last_undegraded":"2024-09-13T22:58:14.034574+0000","last_fullsized":"2024-09-13T22:58:14.034574+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:02.853478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58982400,"num_objects":900,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":900,"num_write_kb":57600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":1010,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136578+0000","last_change":"2024-09-13T22:58:02.221879+0000","last_active":"2024-09-13T22:58:13.136578+0000","last_peered":"2024-09-13T22:58:13.136578+0000","last_clean":"2024-09-13T22:58:13.136578+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:58:13.136578+0000","last_undegraded":"2024-09-13T22:58:13.136578+0000","last_fullsized":"2024-09-13T22:58:13.136578+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:02.221654+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.221654+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:20.958768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":816,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158260+0000","last_change":"2024-09-13T22:55:38.114946+0000","last_active":"2024-09-13T22:58:18.158260+0000","last_peered":"2024-09-13T22:58:18.158260+0000","last_clean":"2024-09-13T22:58:18.158260+0000","last_became_active":"2024-09-13T22:55:38.095029+0000","last_became_peered":"2024-09-13T22:55:38.095029+0000","last_unstale":"2024-09-13T22:58:18.158260+0000","last_undegraded":"2024-09-13T22:58:18.158260+0000","last_fullsized":"2024-09-13T22:58:18.158260+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:55:13.915384+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:55:13.915384+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":742,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.141382+0000","last_change":"2024-09-13T22:55:35.535365+0000","last_active":"2024-09-13T22:58:18.141382+0000","last_peered":"2024-09-13T22:58:18.141382+0000","last_clean":"2024-09-13T22:58:18.141382+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:58:18.141382+0000","last_undegraded":"2024-09-13T22:58:18.141382+0000","last_fullsized":"2024-09-13T22:58:18.141382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:35.535254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:35.535254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":452,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137176+0000","last_change":"2024-09-13T22:57:06.962170+0000","last_active":"2024-09-13T22:58:13.137176+0000","last_peered":"2024-09-13T22:58:13.137176+0000","last_clean":"2024-09-13T22:58:13.137176+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T22:58:13.137176+0000","last_undegraded":"2024-09-13T22:58:13.137176+0000","last_fullsized":"2024-09-13T22:58:13.137176+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'12","last_scrub_stamp":"2024-09-13T22:57:06.962075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.962075+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:32.148372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":251,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146011+0000","last_change":"2024-09-13T22:58:04.163362+0000","last_active":"2024-09-13T22:58:18.146011+0000","last_peered":"2024-09-13T22:58:18.146011+0000","last_clean":"2024-09-13T22:58:18.146011+0000","last_became_active":"2024-09-13T22:55:38.103505+0000","last_became_peered":"2024-09-13T22:55:38.103505+0000","last_unstale":"2024-09-13T22:58:18.146011+0000","last_undegraded":"2024-09-13T22:58:18.146011+0000","last_fullsized":"2024-09-13T22:58:18.146011+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.163173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.163173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.621623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"405'1267","reported_seq":1293,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.622220+0000","last_change":"2024-09-13T22:57:49.644230+0000","last_active":"2024-09-13T22:58:18.622220+0000","last_peered":"2024-09-13T22:58:18.622220+0000","last_clean":"2024-09-13T22:58:18.622220+0000","last_became_active":"2024-09-13T22:57:49.643953+0000","last_became_peered":"2024-09-13T22:57:49.643953+0000","last_unstale":"2024-09-13T22:58:18.622220+0000","last_undegraded":"2024-09-13T22:58:18.622220+0000","last_fullsized":"2024-09-13T22:58:18.622220+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:17.126537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83034112,"num_objects":1267,"num_object_clones":0,"num_object_copies":3801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1267,"num_write_kb":81088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":812,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159577+0000","last_change":"2024-09-13T22:58:12.166007+0000","last_active":"2024-09-13T22:58:18.159577+0000","last_peered":"2024-09-13T22:58:18.159577+0000","last_clean":"2024-09-13T22:58:18.159577+0000","last_became_active":"2024-09-13T22:56:03.248435+0000","last_became_peered":"2024-09-13T22:56:03.248435+0000","last_unstale":"2024-09-13T22:58:18.159577+0000","last_undegraded":"2024-09-13T22:58:18.159577+0000","last_fullsized":"2024-09-13T22:58:18.159577+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:12.165912+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.165912+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.645916+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"397'39","reported_seq":791,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.153642+0000","last_change":"2024-09-13T22:55:29.816311+0000","last_active":"2024-09-13T22:58:18.153642+0000","last_peered":"2024-09-13T22:58:18.153642+0000","last_clean":"2024-09-13T22:58:18.153642+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:58:18.153642+0000","last_undegraded":"2024-09-13T22:58:18.153642+0000","last_fullsized":"2024-09-13T22:58:18.153642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:55:29.816212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.816212+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":750,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.143543+0000","last_change":"2024-09-13T22:57:50.137862+0000","last_active":"2024-09-13T22:58:18.143543+0000","last_peered":"2024-09-13T22:58:18.143543+0000","last_clean":"2024-09-13T22:58:18.143543+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:58:18.143543+0000","last_undegraded":"2024-09-13T22:58:18.143543+0000","last_fullsized":"2024-09-13T22:58:18.143543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.137581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.137581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:02.936942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"351'15","reported_seq":417,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159327+0000","last_change":"2024-09-13T22:57:02.049588+0000","last_active":"2024-09-13T22:58:18.159327+0000","last_peered":"2024-09-13T22:58:18.159327+0000","last_clean":"2024-09-13T22:58:18.159327+0000","last_became_active":"2024-09-13T22:55:38.106061+0000","last_became_peered":"2024-09-13T22:55:38.106061+0000","last_unstale":"2024-09-13T22:58:18.159327+0000","last_undegraded":"2024-09-13T22:58:18.159327+0000","last_fullsized":"2024-09-13T22:58:18.159327+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'15","last_scrub_stamp":"2024-09-13T22:57:02.049279+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:57:02.049279+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":730,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159986+0000","last_change":"2024-09-13T22:57:11.104485+0000","last_active":"2024-09-13T22:58:18.159986+0000","last_peered":"2024-09-13T22:58:18.159986+0000","last_clean":"2024-09-13T22:58:18.159986+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:58:18.159986+0000","last_undegraded":"2024-09-13T22:58:18.159986+0000","last_fullsized":"2024-09-13T22:58:18.159986+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:11.104365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:57:11.104365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:29.195915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"405'1219","reported_seq":1245,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.694969+0000","last_change":"2024-09-13T22:57:49.643143+0000","last_active":"2024-09-13T22:58:18.694969+0000","last_peered":"2024-09-13T22:58:18.694969+0000","last_clean":"2024-09-13T22:58:18.694969+0000","last_became_active":"2024-09-13T22:57:49.642516+0000","last_became_peered":"2024-09-13T22:57:49.642516+0000","last_unstale":"2024-09-13T22:58:18.694969+0000","last_undegraded":"2024-09-13T22:58:18.694969+0000","last_fullsized":"2024-09-13T22:58:18.694969+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1219,"log_dups_size":0,"ondisk_log_size":1219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.266040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79888384,"num_objects":1219,"num_object_clones":0,"num_object_copies":3657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":78016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":341,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136294+0000","last_change":"2024-09-13T22:57:50.496802+0000","last_active":"2024-09-13T22:58:13.136294+0000","last_peered":"2024-09-13T22:58:13.136294+0000","last_clean":"2024-09-13T22:58:13.136294+0000","last_became_active":"2024-09-13T22:55:38.124814+0000","last_became_peered":"2024-09-13T22:55:38.124814+0000","last_unstale":"2024-09-13T22:58:13.136294+0000","last_undegraded":"2024-09-13T22:58:13.136294+0000","last_fullsized":"2024-09-13T22:58:13.136294+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T22:57:50.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.496702+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.914857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":732,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135971+0000","last_change":"2024-09-13T22:57:51.208483+0000","last_active":"2024-09-13T22:58:13.135971+0000","last_peered":"2024-09-13T22:58:13.135971+0000","last_clean":"2024-09-13T22:58:13.135971+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:58:13.135971+0000","last_undegraded":"2024-09-13T22:58:13.135971+0000","last_fullsized":"2024-09-13T22:58:13.135971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.208413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.208413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.011784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":553,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137006+0000","last_change":"2024-09-13T22:56:00.228231+0000","last_active":"2024-09-13T22:58:13.137006+0000","last_peered":"2024-09-13T22:58:13.137006+0000","last_clean":"2024-09-13T22:58:13.137006+0000","last_became_active":"2024-09-13T22:56:00.227885+0000","last_became_peered":"2024-09-13T22:56:00.227885+0000","last_unstale":"2024-09-13T22:58:13.137006+0000","last_undegraded":"2024-09-13T22:58:13.137006+0000","last_fullsized":"2024-09-13T22:58:13.137006+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.848113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.848113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":728,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157647+0000","last_change":"2024-09-13T22:58:12.718373+0000","last_active":"2024-09-13T22:58:18.157647+0000","last_peered":"2024-09-13T22:58:18.157647+0000","last_clean":"2024-09-13T22:58:18.157647+0000","last_became_active":"2024-09-13T22:56:00.232543+0000","last_became_peered":"2024-09-13T22:56:00.232543+0000","last_unstale":"2024-09-13T22:58:18.157647+0000","last_undegraded":"2024-09-13T22:58:18.157647+0000","last_fullsized":"2024-09-13T22:58:18.157647+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.718269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:39.511928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"405'1237","reported_seq":1263,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.623666+0000","last_change":"2024-09-13T22:57:49.642973+0000","last_active":"2024-09-13T22:58:18.623666+0000","last_peered":"2024-09-13T22:58:18.623666+0000","last_clean":"2024-09-13T22:58:18.623666+0000","last_became_active":"2024-09-13T22:57:49.642612+0000","last_became_peered":"2024-09-13T22:57:49.642612+0000","last_unstale":"2024-09-13T22:58:18.623666+0000","last_undegraded":"2024-09-13T22:58:18.623666+0000","last_fullsized":"2024-09-13T22:58:18.623666+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1237,"log_dups_size":0,"ondisk_log_size":1237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:51.155256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81068032,"num_objects":1237,"num_object_clones":0,"num_object_copies":3711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1237,"num_write_kb":79168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":778,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157517+0000","last_change":"2024-09-13T22:54:37.280042+0000","last_active":"2024-09-13T22:58:18.157517+0000","last_peered":"2024-09-13T22:58:18.157517+0000","last_clean":"2024-09-13T22:58:18.157517+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:58:18.157517+0000","last_undegraded":"2024-09-13T22:58:18.157517+0000","last_fullsized":"2024-09-13T22:58:18.157517+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:54:37.279940+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T22:54:37.279940+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":859,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129492+0000","last_change":"2024-09-13T22:57:58.252181+0000","last_active":"2024-09-13T22:58:13.129492+0000","last_peered":"2024-09-13T22:58:13.129492+0000","last_clean":"2024-09-13T22:58:13.129492+0000","last_became_active":"2024-09-13T22:55:38.135796+0000","last_became_peered":"2024-09-13T22:55:38.135796+0000","last_unstale":"2024-09-13T22:58:13.129492+0000","last_undegraded":"2024-09-13T22:58:13.129492+0000","last_fullsized":"2024-09-13T22:58:13.129492+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'36","last_scrub_stamp":"2024-09-13T22:57:58.251941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:58.251941+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:05.244239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":114,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137524+0000","last_change":"2024-09-13T22:58:04.243287+0000","last_active":"2024-09-13T22:58:13.137524+0000","last_peered":"2024-09-13T22:58:13.137524+0000","last_clean":"2024-09-13T22:58:13.137524+0000","last_became_active":"2024-09-13T22:56:00.226303+0000","last_became_peered":"2024-09-13T22:56:00.226303+0000","last_unstale":"2024-09-13T22:58:13.137524+0000","last_undegraded":"2024-09-13T22:58:13.137524+0000","last_fullsized":"2024-09-13T22:58:13.137524+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.243067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.243067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:20.856718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":560,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160032+0000","last_change":"2024-09-13T22:58:11.043833+0000","last_active":"2024-09-13T22:58:18.160032+0000","last_peered":"2024-09-13T22:58:18.160032+0000","last_clean":"2024-09-13T22:58:18.160032+0000","last_became_active":"2024-09-13T22:56:00.227907+0000","last_became_peered":"2024-09-13T22:56:00.227907+0000","last_unstale":"2024-09-13T22:58:18.160032+0000","last_undegraded":"2024-09-13T22:58:18.160032+0000","last_fullsized":"2024-09-13T22:58:18.160032+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:58:11.043740+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:58:11.043740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:35.377337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"351'14","reported_seq":1017,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.153155+0000","last_change":"2024-09-13T22:58:02.750352+0000","last_active":"2024-09-13T22:58:18.153155+0000","last_peered":"2024-09-13T22:58:18.153155+0000","last_clean":"2024-09-13T22:58:18.153155+0000","last_became_active":"2024-09-13T22:55:38.087590+0000","last_became_peered":"2024-09-13T22:55:38.087590+0000","last_unstale":"2024-09-13T22:58:18.153155+0000","last_undegraded":"2024-09-13T22:58:18.153155+0000","last_fullsized":"2024-09-13T22:58:18.153155+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'14","last_scrub_stamp":"2024-09-13T22:58:02.750208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.750208+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:22.642062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":726,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.267345+0000","last_change":"2024-09-13T22:58:13.267389+0000","last_active":"2024-09-13T22:58:13.267345+0000","last_peered":"2024-09-13T22:58:13.267345+0000","last_clean":"2024-09-13T22:58:13.267345+0000","last_became_active":"2024-09-13T22:56:00.228068+0000","last_became_peered":"2024-09-13T22:56:00.228068+0000","last_unstale":"2024-09-13T22:58:13.267345+0000","last_undegraded":"2024-09-13T22:58:13.267345+0000","last_fullsized":"2024-09-13T22:58:13.267345+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.267276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.267276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.127021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"404'942","reported_seq":966,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.021832+0000","last_change":"2024-09-13T22:57:49.650986+0000","last_active":"2024-09-13T22:58:14.021832+0000","last_peered":"2024-09-13T22:58:14.021832+0000","last_clean":"2024-09-13T22:58:14.021832+0000","last_became_active":"2024-09-13T22:57:49.650245+0000","last_became_peered":"2024-09-13T22:57:49.650245+0000","last_unstale":"2024-09-13T22:58:14.021832+0000","last_undegraded":"2024-09-13T22:58:14.021832+0000","last_fullsized":"2024-09-13T22:58:14.021832+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.292803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61734912,"num_objects":942,"num_object_clones":0,"num_object_copies":2826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":942,"num_write_kb":60288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":601,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.492139+0000","last_change":"2024-09-13T22:58:14.492179+0000","last_active":"2024-09-13T22:58:14.492139+0000","last_peered":"2024-09-13T22:58:14.492139+0000","last_clean":"2024-09-13T22:58:14.492139+0000","last_became_active":"2024-09-13T22:56:03.248140+0000","last_became_peered":"2024-09-13T22:56:03.248140+0000","last_unstale":"2024-09-13T22:58:14.492139+0000","last_undegraded":"2024-09-13T22:58:14.492139+0000","last_fullsized":"2024-09-13T22:58:14.492139+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:14.492071+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.492071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.554601+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":774,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158207+0000","last_change":"2024-09-13T22:56:59.524045+0000","last_active":"2024-09-13T22:58:18.158207+0000","last_peered":"2024-09-13T22:58:18.158207+0000","last_clean":"2024-09-13T22:58:18.158207+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:58:18.158207+0000","last_undegraded":"2024-09-13T22:58:18.158207+0000","last_fullsized":"2024-09-13T22:58:18.158207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:56:59.523938+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:56:59.523938+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:19.627246+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":792,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146201+0000","last_change":"2024-09-13T22:58:15.075066+0000","last_active":"2024-09-13T22:58:18.146201+0000","last_peered":"2024-09-13T22:58:18.146201+0000","last_clean":"2024-09-13T22:58:18.146201+0000","last_became_active":"2024-09-13T22:56:00.222510+0000","last_became_peered":"2024-09-13T22:56:00.222510+0000","last_unstale":"2024-09-13T22:58:18.146201+0000","last_undegraded":"2024-09-13T22:58:18.146201+0000","last_fullsized":"2024-09-13T22:58:18.146201+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:58:15.074737+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.074737+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:33.868674+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":1953,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.153274+0000","last_change":"2024-09-13T22:57:55.678130+0000","last_active":"2024-09-13T22:58:18.153274+0000","last_peered":"2024-09-13T22:58:18.153274+0000","last_clean":"2024-09-13T22:58:18.153274+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:58:18.153274+0000","last_undegraded":"2024-09-13T22:58:18.153274+0000","last_fullsized":"2024-09-13T22:58:18.153274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T22:57:55.678034+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.678034+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.773483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1211,"num_read_kb":1206,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":250,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.156084+0000","last_change":"2024-09-13T22:57:49.593322+0000","last_active":"2024-09-13T22:58:18.156084+0000","last_peered":"2024-09-13T22:58:18.156084+0000","last_clean":"2024-09-13T22:58:18.156084+0000","last_became_active":"2024-09-13T22:55:38.124054+0000","last_became_peered":"2024-09-13T22:55:38.124054+0000","last_unstale":"2024-09-13T22:58:18.156084+0000","last_undegraded":"2024-09-13T22:58:18.156084+0000","last_fullsized":"2024-09-13T22:58:18.156084+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.593224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.593224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.823607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":241,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158030+0000","last_change":"2024-09-13T22:56:00.234182+0000","last_active":"2024-09-13T22:58:18.158030+0000","last_peered":"2024-09-13T22:58:18.158030+0000","last_clean":"2024-09-13T22:58:18.158030+0000","last_became_active":"2024-09-13T22:56:00.231167+0000","last_became_peered":"2024-09-13T22:56:00.231167+0000","last_unstale":"2024-09-13T22:58:18.158030+0000","last_undegraded":"2024-09-13T22:58:18.158030+0000","last_fullsized":"2024-09-13T22:58:18.158030+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.153155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.153155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"405'1314","reported_seq":1340,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.622822+0000","last_change":"2024-09-13T22:57:49.647079+0000","last_active":"2024-09-13T22:58:18.622822+0000","last_peered":"2024-09-13T22:58:18.622822+0000","last_clean":"2024-09-13T22:58:18.622822+0000","last_became_active":"2024-09-13T22:57:49.646766+0000","last_became_peered":"2024-09-13T22:57:49.646766+0000","last_unstale":"2024-09-13T22:58:18.622822+0000","last_undegraded":"2024-09-13T22:58:18.622822+0000","last_fullsized":"2024-09-13T22:58:18.622822+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.932769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86114304,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1314,"num_write_kb":84096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":427,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129236+0000","last_change":"2024-09-13T22:56:28.769649+0000","last_active":"2024-09-13T22:58:13.129236+0000","last_peered":"2024-09-13T22:58:13.129236+0000","last_clean":"2024-09-13T22:58:13.129236+0000","last_became_active":"2024-09-13T22:56:03.263189+0000","last_became_peered":"2024-09-13T22:56:03.263189+0000","last_unstale":"2024-09-13T22:58:13.129236+0000","last_undegraded":"2024-09-13T22:58:13.129236+0000","last_fullsized":"2024-09-13T22:58:13.129236+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:28.769564+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.769564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1090,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.147350+0000","last_change":"2024-09-13T22:58:09.054534+0000","last_active":"2024-09-13T22:58:18.147350+0000","last_peered":"2024-09-13T22:58:18.147350+0000","last_clean":"2024-09-13T22:58:18.147350+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:58:18.147350+0000","last_undegraded":"2024-09-13T22:58:18.147350+0000","last_fullsized":"2024-09-13T22:58:18.147350+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'64","last_scrub_stamp":"2024-09-13T22:58:09.054428+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:55:36.587996+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.054428+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.603201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":750,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137668+0000","last_change":"2024-09-13T22:58:07.245450+0000","last_active":"2024-09-13T22:58:13.137668+0000","last_peered":"2024-09-13T22:58:13.137668+0000","last_clean":"2024-09-13T22:58:13.137668+0000","last_became_active":"2024-09-13T22:55:38.145203+0000","last_became_peered":"2024-09-13T22:55:38.145203+0000","last_unstale":"2024-09-13T22:58:13.137668+0000","last_undegraded":"2024-09-13T22:58:13.137668+0000","last_fullsized":"2024-09-13T22:58:13.137668+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.245370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.245370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.129975+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":113,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.131940+0000","last_change":"2024-09-13T22:58:10.310774+0000","last_active":"2024-09-13T22:58:13.131940+0000","last_peered":"2024-09-13T22:58:13.131940+0000","last_clean":"2024-09-13T22:58:13.131940+0000","last_became_active":"2024-09-13T22:56:00.226492+0000","last_became_peered":"2024-09-13T22:56:00.226492+0000","last_unstale":"2024-09-13T22:58:13.131940+0000","last_undegraded":"2024-09-13T22:58:13.131940+0000","last_fullsized":"2024-09-13T22:58:13.131940+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.310671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.310671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:31.221358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":736,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.141602+0000","last_change":"2024-09-13T22:56:55.737374+0000","last_active":"2024-09-13T22:58:18.141602+0000","last_peered":"2024-09-13T22:58:18.141602+0000","last_clean":"2024-09-13T22:58:18.141602+0000","last_became_active":"2024-09-13T22:56:00.235117+0000","last_became_peered":"2024-09-13T22:56:00.235117+0000","last_unstale":"2024-09-13T22:58:18.141602+0000","last_undegraded":"2024-09-13T22:58:18.141602+0000","last_fullsized":"2024-09-13T22:58:18.141602+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:55.737268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:55.737268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:22.264525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"404'1037","reported_seq":1061,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.698137+0000","last_change":"2024-09-13T22:57:49.641514+0000","last_active":"2024-09-13T22:58:14.698137+0000","last_peered":"2024-09-13T22:58:14.698137+0000","last_clean":"2024-09-13T22:58:14.698137+0000","last_became_active":"2024-09-13T22:57:49.641164+0000","last_became_peered":"2024-09-13T22:57:49.641164+0000","last_unstale":"2024-09-13T22:58:14.698137+0000","last_undegraded":"2024-09-13T22:58:14.698137+0000","last_fullsized":"2024-09-13T22:58:14.698137+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.914497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67960832,"num_objects":1037,"num_object_clones":0,"num_object_copies":3111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1037,"num_write_kb":66368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":432,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157718+0000","last_change":"2024-09-13T22:56:30.161262+0000","last_active":"2024-09-13T22:58:18.157718+0000","last_peered":"2024-09-13T22:58:18.157718+0000","last_clean":"2024-09-13T22:58:18.157718+0000","last_became_active":"2024-09-13T22:56:03.238687+0000","last_became_peered":"2024-09-13T22:56:03.238687+0000","last_unstale":"2024-09-13T22:58:18.157718+0000","last_undegraded":"2024-09-13T22:58:18.157718+0000","last_fullsized":"2024-09-13T22:58:18.157718+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:30.160971+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:21.295886+0000","last_clean_scrub_stamp":"2024-09-13T22:56:30.160971+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1060,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160538+0000","last_change":"2024-09-13T22:58:09.995562+0000","last_active":"2024-09-13T22:58:18.160538+0000","last_peered":"2024-09-13T22:58:18.160538+0000","last_clean":"2024-09-13T22:58:18.160538+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:58:18.160538+0000","last_undegraded":"2024-09-13T22:58:18.160538+0000","last_fullsized":"2024-09-13T22:58:18.160538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'48","last_scrub_stamp":"2024-09-13T22:58:09.995445+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.995445+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:28.734156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":759,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.147450+0000","last_change":"2024-09-13T22:57:06.876493+0000","last_active":"2024-09-13T22:58:18.147450+0000","last_peered":"2024-09-13T22:58:18.147450+0000","last_clean":"2024-09-13T22:58:18.147450+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:58:18.147450+0000","last_undegraded":"2024-09-13T22:58:18.147450+0000","last_fullsized":"2024-09-13T22:58:18.147450+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T22:57:06.876301+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T22:54:26.175576+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.876301+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":786,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.141263+0000","last_change":"2024-09-13T22:58:06.134146+0000","last_active":"2024-09-13T22:58:18.141263+0000","last_peered":"2024-09-13T22:58:18.141263+0000","last_clean":"2024-09-13T22:58:18.141263+0000","last_became_active":"2024-09-13T22:55:38.114687+0000","last_became_peered":"2024-09-13T22:55:38.114687+0000","last_unstale":"2024-09-13T22:58:18.141263+0000","last_undegraded":"2024-09-13T22:58:18.141263+0000","last_fullsized":"2024-09-13T22:58:18.141263+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T22:58:06.133998+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T22:55:21.433994+0000","last_clean_scrub_stamp":"2024-09-13T22:58:06.133998+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:31.928770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"358'13","reported_seq":1018,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129763+0000","last_change":"2024-09-13T22:58:05.208773+0000","last_active":"2024-09-13T22:58:13.129763+0000","last_peered":"2024-09-13T22:58:13.129763+0000","last_clean":"2024-09-13T22:58:13.129763+0000","last_became_active":"2024-09-13T22:56:00.218154+0000","last_became_peered":"2024-09-13T22:56:00.218154+0000","last_unstale":"2024-09-13T22:58:13.129763+0000","last_undegraded":"2024-09-13T22:58:13.129763+0000","last_fullsized":"2024-09-13T22:58:13.129763+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'13","last_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_deep_scrub":"358'13","last_deep_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_clean_scrub_stamp":"2024-09-13T22:58:05.208554+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:30.039011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":738,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129871+0000","last_change":"2024-09-13T22:57:56.259013+0000","last_active":"2024-09-13T22:58:13.129871+0000","last_peered":"2024-09-13T22:58:13.129871+0000","last_clean":"2024-09-13T22:58:13.129871+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:58:13.129871+0000","last_undegraded":"2024-09-13T22:58:13.129871+0000","last_fullsized":"2024-09-13T22:58:13.129871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:56.258783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:56.258783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.527327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"405'1238","reported_seq":1264,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.706613+0000","last_change":"2024-09-13T22:57:49.647535+0000","last_active":"2024-09-13T22:58:18.706613+0000","last_peered":"2024-09-13T22:58:18.706613+0000","last_clean":"2024-09-13T22:58:18.706613+0000","last_became_active":"2024-09-13T22:57:49.646701+0000","last_became_peered":"2024-09-13T22:57:49.646701+0000","last_unstale":"2024-09-13T22:58:18.706613+0000","last_undegraded":"2024-09-13T22:58:18.706613+0000","last_fullsized":"2024-09-13T22:58:18.706613+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:08.172073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81133568,"num_objects":1238,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":795,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159547+0000","last_change":"2024-09-13T22:57:57.050275+0000","last_active":"2024-09-13T22:58:18.159547+0000","last_peered":"2024-09-13T22:58:18.159547+0000","last_clean":"2024-09-13T22:58:18.159547+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:58:18.159547+0000","last_undegraded":"2024-09-13T22:58:18.159547+0000","last_fullsized":"2024-09-13T22:58:18.159547+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:57.050042+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.050042+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.549861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"397'39","reported_seq":76,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.134637+0000","last_change":"2024-09-13T22:57:45.567779+0000","last_active":"2024-09-13T22:58:13.134637+0000","last_peered":"2024-09-13T22:58:13.134637+0000","last_clean":"2024-09-13T22:58:13.134637+0000","last_became_active":"2024-09-13T22:57:37.355011+0000","last_became_peered":"2024-09-13T22:57:37.355011+0000","last_unstale":"2024-09-13T22:58:13.134637+0000","last_undegraded":"2024-09-13T22:58:13.134637+0000","last_fullsized":"2024-09-13T22:58:13.134637+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'29","last_scrub_stamp":"2024-09-13T22:57:45.567559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:45.567559+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:01.804961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":764,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160249+0000","last_change":"2024-09-13T22:58:17.058759+0000","last_active":"2024-09-13T22:58:18.160249+0000","last_peered":"2024-09-13T22:58:18.160249+0000","last_clean":"2024-09-13T22:58:18.160249+0000","last_became_active":"2024-09-13T22:56:00.242041+0000","last_became_peered":"2024-09-13T22:56:00.242041+0000","last_unstale":"2024-09-13T22:58:18.160249+0000","last_undegraded":"2024-09-13T22:58:18.160249+0000","last_fullsized":"2024-09-13T22:58:18.160249+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:17.058474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:58:17.058474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:37.342539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":765,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129492+0000","last_change":"2024-09-13T22:57:09.483802+0000","last_active":"2024-09-13T22:58:13.129492+0000","last_peered":"2024-09-13T22:58:13.129492+0000","last_clean":"2024-09-13T22:58:13.129492+0000","last_became_active":"2024-09-13T22:56:00.216504+0000","last_became_peered":"2024-09-13T22:56:00.216504+0000","last_unstale":"2024-09-13T22:58:13.129492+0000","last_undegraded":"2024-09-13T22:58:13.129492+0000","last_fullsized":"2024-09-13T22:58:13.129492+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.483697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.483697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:14.950895+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":746,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157859+0000","last_change":"2024-09-13T22:56:56.539298+0000","last_active":"2024-09-13T22:58:18.157859+0000","last_peered":"2024-09-13T22:58:18.157859+0000","last_clean":"2024-09-13T22:58:18.157859+0000","last_became_active":"2024-09-13T22:56:00.232669+0000","last_became_peered":"2024-09-13T22:56:00.232669+0000","last_unstale":"2024-09-13T22:58:18.157859+0000","last_undegraded":"2024-09-13T22:58:18.157859+0000","last_fullsized":"2024-09-13T22:58:18.157859+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:56.539208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.539208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"405'1240","reported_seq":1265,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.719466+0000","last_change":"2024-09-13T22:57:49.643169+0000","last_active":"2024-09-13T22:58:18.719466+0000","last_peered":"2024-09-13T22:58:18.719466+0000","last_clean":"2024-09-13T22:58:18.719466+0000","last_became_active":"2024-09-13T22:57:49.642830+0000","last_became_peered":"2024-09-13T22:57:49.642830+0000","last_unstale":"2024-09-13T22:58:18.719466+0000","last_undegraded":"2024-09-13T22:58:18.719466+0000","last_fullsized":"2024-09-13T22:58:18.719466+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:01.866781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":79360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1037,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136115+0000","last_change":"2024-09-13T22:56:54.138074+0000","last_active":"2024-09-13T22:58:13.136115+0000","last_peered":"2024-09-13T22:58:13.136115+0000","last_clean":"2024-09-13T22:58:13.136115+0000","last_became_active":"2024-09-13T22:55:41.103393+0000","last_became_peered":"2024-09-13T22:55:41.103393+0000","last_unstale":"2024-09-13T22:58:13.136115+0000","last_undegraded":"2024-09-13T22:58:13.136115+0000","last_fullsized":"2024-09-13T22:58:13.136115+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:54.137952+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:54.137952+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:14.326641+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":73,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159538+0000","last_change":"2024-09-13T22:57:37.368591+0000","last_active":"2024-09-13T22:58:18.159538+0000","last_peered":"2024-09-13T22:58:18.159538+0000","last_clean":"2024-09-13T22:58:18.159538+0000","last_became_active":"2024-09-13T22:57:37.355384+0000","last_became_peered":"2024-09-13T22:57:37.355384+0000","last_unstale":"2024-09-13T22:58:18.159538+0000","last_undegraded":"2024-09-13T22:58:18.159538+0000","last_fullsized":"2024-09-13T22:58:18.159538+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.760160+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":789,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136387+0000","last_change":"2024-09-13T22:57:48.508060+0000","last_active":"2024-09-13T22:58:13.136387+0000","last_peered":"2024-09-13T22:58:13.136387+0000","last_clean":"2024-09-13T22:58:13.136387+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:58:13.136387+0000","last_undegraded":"2024-09-13T22:58:13.136387+0000","last_fullsized":"2024-09-13T22:58:13.136387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:57:48.507763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.507763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.560730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":765,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.134942+0000","last_change":"2024-09-13T22:56:48.632053+0000","last_active":"2024-09-13T22:58:13.134942+0000","last_peered":"2024-09-13T22:58:13.134942+0000","last_clean":"2024-09-13T22:58:13.134942+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:58:13.134942+0000","last_undegraded":"2024-09-13T22:58:13.134942+0000","last_fullsized":"2024-09-13T22:58:13.134942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:48.631915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:56:48.631915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:15.306680+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":726,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137081+0000","last_change":"2024-09-13T22:56:00.227065+0000","last_active":"2024-09-13T22:58:13.137081+0000","last_peered":"2024-09-13T22:58:13.137081+0000","last_clean":"2024-09-13T22:58:13.137081+0000","last_became_active":"2024-09-13T22:56:00.226279+0000","last_became_peered":"2024-09-13T22:56:00.226279+0000","last_unstale":"2024-09-13T22:58:13.137081+0000","last_undegraded":"2024-09-13T22:58:13.137081+0000","last_fullsized":"2024-09-13T22:58:13.137081+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:18.683688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:18.683688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"405'1303","reported_seq":1329,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.725232+0000","last_change":"2024-09-13T22:57:49.647142+0000","last_active":"2024-09-13T22:58:18.725232+0000","last_peered":"2024-09-13T22:58:18.725232+0000","last_clean":"2024-09-13T22:58:18.725232+0000","last_became_active":"2024-09-13T22:57:49.646772+0000","last_became_peered":"2024-09-13T22:57:49.646772+0000","last_unstale":"2024-09-13T22:58:18.725232+0000","last_undegraded":"2024-09-13T22:58:18.725232+0000","last_fullsized":"2024-09-13T22:58:18.725232+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1303,"log_dups_size":0,"ondisk_log_size":1303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.716827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85393408,"num_objects":1303,"num_object_clones":0,"num_object_copies":3909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1303,"num_write_kb":83392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":830,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.147433+0000","last_change":"2024-09-13T22:57:52.050773+0000","last_active":"2024-09-13T22:58:18.147433+0000","last_peered":"2024-09-13T22:58:18.147433+0000","last_clean":"2024-09-13T22:58:18.147433+0000","last_became_active":"2024-09-13T22:56:03.239906+0000","last_became_peered":"2024-09-13T22:56:03.239906+0000","last_unstale":"2024-09-13T22:58:18.147433+0000","last_undegraded":"2024-09-13T22:58:18.147433+0000","last_fullsized":"2024-09-13T22:58:18.147433+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:52.050564+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T22:57:52.050564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.906269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"397'42","reported_seq":882,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:15.305583+0000","last_change":"2024-09-13T22:58:15.305788+0000","last_active":"2024-09-13T22:58:15.305583+0000","last_peered":"2024-09-13T22:58:15.305583+0000","last_clean":"2024-09-13T22:58:15.305583+0000","last_became_active":"2024-09-13T22:55:38.098695+0000","last_became_peered":"2024-09-13T22:55:38.098695+0000","last_unstale":"2024-09-13T22:58:15.305583+0000","last_undegraded":"2024-09-13T22:58:15.305583+0000","last_fullsized":"2024-09-13T22:58:15.305583+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-09-13T22:58:15.305297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.305297+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:37.735805+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":507,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135614+0000","last_change":"2024-09-13T22:55:11.632270+0000","last_active":"2024-09-13T22:58:13.135614+0000","last_peered":"2024-09-13T22:58:13.135614+0000","last_clean":"2024-09-13T22:58:13.135614+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:58:13.135614+0000","last_undegraded":"2024-09-13T22:58:13.135614+0000","last_fullsized":"2024-09-13T22:58:13.135614+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:11.632162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:55:11.632162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":742,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160747+0000","last_change":"2024-09-13T22:57:03.031136+0000","last_active":"2024-09-13T22:58:18.160747+0000","last_peered":"2024-09-13T22:58:18.160747+0000","last_clean":"2024-09-13T22:58:18.160747+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:58:18.160747+0000","last_undegraded":"2024-09-13T22:58:18.160747+0000","last_fullsized":"2024-09-13T22:58:18.160747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:03.031031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:57:03.031031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:20.803600+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":738,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146750+0000","last_change":"2024-09-13T22:58:07.109173+0000","last_active":"2024-09-13T22:58:18.146750+0000","last_peered":"2024-09-13T22:58:18.146750+0000","last_clean":"2024-09-13T22:58:18.146750+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:58:18.146750+0000","last_undegraded":"2024-09-13T22:58:18.146750+0000","last_fullsized":"2024-09-13T22:58:18.146750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.109070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:18.659096+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.109070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.174273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"405'1277","reported_seq":1303,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.721652+0000","last_change":"2024-09-13T22:57:49.647406+0000","last_active":"2024-09-13T22:58:18.721652+0000","last_peered":"2024-09-13T22:58:18.721652+0000","last_clean":"2024-09-13T22:58:18.721652+0000","last_became_active":"2024-09-13T22:57:49.646955+0000","last_became_peered":"2024-09-13T22:57:49.646955+0000","last_unstale":"2024-09-13T22:58:18.721652+0000","last_undegraded":"2024-09-13T22:58:18.721652+0000","last_fullsized":"2024-09-13T22:58:18.721652+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:10.733352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83689472,"num_objects":1277,"num_object_clones":0,"num_object_copies":3831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1277,"num_write_kb":81728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":793,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158306+0000","last_change":"2024-09-13T22:58:09.777789+0000","last_active":"2024-09-13T22:58:18.158306+0000","last_peered":"2024-09-13T22:58:18.158306+0000","last_clean":"2024-09-13T22:58:18.158306+0000","last_became_active":"2024-09-13T22:56:00.226771+0000","last_became_peered":"2024-09-13T22:56:00.226771+0000","last_unstale":"2024-09-13T22:58:18.158306+0000","last_undegraded":"2024-09-13T22:58:18.158306+0000","last_fullsized":"2024-09-13T22:58:18.158306+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:58:09.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.777653+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.138424+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":750,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159999+0000","last_change":"2024-09-13T22:56:44.954891+0000","last_active":"2024-09-13T22:58:18.159999+0000","last_peered":"2024-09-13T22:58:18.159999+0000","last_clean":"2024-09-13T22:58:18.159999+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:58:18.159999+0000","last_undegraded":"2024-09-13T22:58:18.159999+0000","last_fullsized":"2024-09-13T22:58:18.159999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:44.954773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:44.954773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1012,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136080+0000","last_change":"2024-09-13T22:56:03.243249+0000","last_active":"2024-09-13T22:58:13.136080+0000","last_peered":"2024-09-13T22:58:13.136080+0000","last_clean":"2024-09-13T22:58:13.136080+0000","last_became_active":"2024-09-13T22:56:03.242893+0000","last_became_peered":"2024-09-13T22:56:03.242893+0000","last_unstale":"2024-09-13T22:58:13.136080+0000","last_undegraded":"2024-09-13T22:58:13.136080+0000","last_fullsized":"2024-09-13T22:58:13.136080+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:55:26.889228+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:55:26.889228+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":724,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137180+0000","last_change":"2024-09-13T22:55:32.846798+0000","last_active":"2024-09-13T22:58:13.137180+0000","last_peered":"2024-09-13T22:58:13.137180+0000","last_clean":"2024-09-13T22:58:13.137180+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:58:13.137180+0000","last_undegraded":"2024-09-13T22:58:13.137180+0000","last_fullsized":"2024-09-13T22:58:13.137180+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:32.846649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:55:32.846649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":756,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160750+0000","last_change":"2024-09-13T22:56:33.129911+0000","last_active":"2024-09-13T22:58:18.160750+0000","last_peered":"2024-09-13T22:58:18.160750+0000","last_clean":"2024-09-13T22:58:18.160750+0000","last_became_active":"2024-09-13T22:55:38.101749+0000","last_became_peered":"2024-09-13T22:55:38.101749+0000","last_unstale":"2024-09-13T22:58:18.160750+0000","last_undegraded":"2024-09-13T22:58:18.160750+0000","last_fullsized":"2024-09-13T22:58:18.160750+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:33.129825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:33.129825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"404'889","reported_seq":913,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.002781+0000","last_change":"2024-09-13T22:57:49.653049+0000","last_active":"2024-09-13T22:58:14.002781+0000","last_peered":"2024-09-13T22:58:14.002781+0000","last_clean":"2024-09-13T22:58:14.002781+0000","last_became_active":"2024-09-13T22:57:49.651751+0000","last_became_peered":"2024-09-13T22:57:49.651751+0000","last_unstale":"2024-09-13T22:58:14.002781+0000","last_undegraded":"2024-09-13T22:58:14.002781+0000","last_fullsized":"2024-09-13T22:58:14.002781+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.832279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58261504,"num_objects":889,"num_object_clones":0,"num_object_copies":2667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":889,"num_write_kb":56896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":814,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137440+0000","last_change":"2024-09-13T22:56:00.266974+0000","last_active":"2024-09-13T22:58:13.137440+0000","last_peered":"2024-09-13T22:58:13.137440+0000","last_clean":"2024-09-13T22:58:13.137440+0000","last_became_active":"2024-09-13T22:56:00.229726+0000","last_became_peered":"2024-09-13T22:56:00.229726+0000","last_unstale":"2024-09-13T22:58:13.137440+0000","last_undegraded":"2024-09-13T22:58:13.137440+0000","last_fullsized":"2024-09-13T22:58:13.137440+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'31","last_scrub_stamp":"2024-09-13T22:54:15.863951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:15.863951+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":781,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137059+0000","last_change":"2024-09-13T22:58:12.453817+0000","last_active":"2024-09-13T22:58:13.137059+0000","last_peered":"2024-09-13T22:58:13.137059+0000","last_clean":"2024-09-13T22:58:13.137059+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:58:13.137059+0000","last_undegraded":"2024-09-13T22:58:13.137059+0000","last_fullsized":"2024-09-13T22:58:13.137059+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:12.453747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.453747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.100029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":765,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135136+0000","last_change":"2024-09-13T22:58:07.404829+0000","last_active":"2024-09-13T22:58:13.135136+0000","last_peered":"2024-09-13T22:58:13.135136+0000","last_clean":"2024-09-13T22:58:13.135136+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:58:13.135136+0000","last_undegraded":"2024-09-13T22:58:13.135136+0000","last_fullsized":"2024-09-13T22:58:13.135136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.404723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.404723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:35.558468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":712,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136019+0000","last_change":"2024-09-13T22:54:25.819955+0000","last_active":"2024-09-13T22:58:13.136019+0000","last_peered":"2024-09-13T22:58:13.136019+0000","last_clean":"2024-09-13T22:58:13.136019+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:58:13.136019+0000","last_undegraded":"2024-09-13T22:58:13.136019+0000","last_fullsized":"2024-09-13T22:58:13.136019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:25.819823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:54:25.819823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"404'910","reported_seq":934,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.023950+0000","last_change":"2024-09-13T22:57:49.652798+0000","last_active":"2024-09-13T22:58:14.023950+0000","last_peered":"2024-09-13T22:58:14.023950+0000","last_clean":"2024-09-13T22:58:14.023950+0000","last_became_active":"2024-09-13T22:57:49.651792+0000","last_became_peered":"2024-09-13T22:57:49.651792+0000","last_unstale":"2024-09-13T22:58:14.023950+0000","last_undegraded":"2024-09-13T22:58:14.023950+0000","last_fullsized":"2024-09-13T22:58:14.023950+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:14.044409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":58240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"397'30","reported_seq":809,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160197+0000","last_change":"2024-09-13T22:57:53.059246+0000","last_active":"2024-09-13T22:58:18.160197+0000","last_peered":"2024-09-13T22:58:18.160197+0000","last_clean":"2024-09-13T22:58:18.160197+0000","last_became_active":"2024-09-13T22:56:00.241217+0000","last_became_peered":"2024-09-13T22:56:00.241217+0000","last_unstale":"2024-09-13T22:58:18.160197+0000","last_undegraded":"2024-09-13T22:58:18.160197+0000","last_fullsized":"2024-09-13T22:58:18.160197+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:57:53.059034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.059034+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.454055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":768,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146527+0000","last_change":"2024-09-13T22:57:07.882465+0000","last_active":"2024-09-13T22:58:18.146527+0000","last_peered":"2024-09-13T22:58:18.146527+0000","last_clean":"2024-09-13T22:58:18.146527+0000","last_became_active":"2024-09-13T22:55:38.093381+0000","last_became_peered":"2024-09-13T22:55:38.093381+0000","last_unstale":"2024-09-13T22:58:18.146527+0000","last_undegraded":"2024-09-13T22:58:18.146527+0000","last_fullsized":"2024-09-13T22:58:18.146527+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.882167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.882167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:32.285343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":463,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157928+0000","last_change":"2024-09-13T22:55:38.098852+0000","last_active":"2024-09-13T22:58:18.157928+0000","last_peered":"2024-09-13T22:58:18.157928+0000","last_clean":"2024-09-13T22:58:18.157928+0000","last_became_active":"2024-09-13T22:55:38.098515+0000","last_became_peered":"2024-09-13T22:55:38.098515+0000","last_unstale":"2024-09-13T22:58:18.157928+0000","last_undegraded":"2024-09-13T22:58:18.157928+0000","last_fullsized":"2024-09-13T22:58:18.157928+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:15.398485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:15.398485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":738,"reported_epoch":404,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T22:58:13.130199+0000","last_change":"2024-09-13T22:58:16.342222+0000","last_active":"2024-09-13T22:58:13.130199+0000","last_peered":"2024-09-13T22:58:13.130199+0000","last_clean":"2024-09-13T22:58:13.130199+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:58:13.130199+0000","last_undegraded":"2024-09-13T22:58:13.130199+0000","last_fullsized":"2024-09-13T22:58:13.130199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:14.852498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:55:14.852498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"405'1290","reported_seq":1316,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.614006+0000","last_change":"2024-09-13T22:57:49.646353+0000","last_active":"2024-09-13T22:58:18.614006+0000","last_peered":"2024-09-13T22:58:18.614006+0000","last_clean":"2024-09-13T22:58:18.614006+0000","last_became_active":"2024-09-13T22:57:49.645497+0000","last_became_peered":"2024-09-13T22:57:49.645497+0000","last_unstale":"2024-09-13T22:58:18.614006+0000","last_undegraded":"2024-09-13T22:58:18.614006+0000","last_fullsized":"2024-09-13T22:58:18.614006+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.493617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":3870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":82560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":862,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137394+0000","last_change":"2024-09-13T22:56:57.107636+0000","last_active":"2024-09-13T22:58:13.137394+0000","last_peered":"2024-09-13T22:58:13.137394+0000","last_clean":"2024-09-13T22:58:13.137394+0000","last_became_active":"2024-09-13T22:55:38.151658+0000","last_became_peered":"2024-09-13T22:55:38.151658+0000","last_unstale":"2024-09-13T22:58:13.137394+0000","last_undegraded":"2024-09-13T22:58:13.137394+0000","last_fullsized":"2024-09-13T22:58:13.137394+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:56:57.107495+0000","last_deep_scrub":"228'34","last_deep_scrub_stamp":"2024-09-13T22:53:27.660042+0000","last_clean_scrub_stamp":"2024-09-13T22:56:57.107495+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:24.614766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":324,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158262+0000","last_change":"2024-09-13T22:55:19.900066+0000","last_active":"2024-09-13T22:58:18.158262+0000","last_peered":"2024-09-13T22:58:18.158262+0000","last_clean":"2024-09-13T22:58:18.158262+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T22:58:18.158262+0000","last_undegraded":"2024-09-13T22:58:18.158262+0000","last_fullsized":"2024-09-13T22:58:18.158262+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:19.899953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:32.054441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:19.899953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"358'15","reported_seq":1024,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136195+0000","last_change":"2024-09-13T22:56:00.261590+0000","last_active":"2024-09-13T22:58:13.136195+0000","last_peered":"2024-09-13T22:58:13.136195+0000","last_clean":"2024-09-13T22:58:13.136195+0000","last_became_active":"2024-09-13T22:56:00.231283+0000","last_became_peered":"2024-09-13T22:56:00.231283+0000","last_unstale":"2024-09-13T22:58:13.136195+0000","last_undegraded":"2024-09-13T22:58:13.136195+0000","last_fullsized":"2024-09-13T22:58:13.136195+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'13","last_scrub_stamp":"2024-09-13T22:54:27.850029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:54:27.850029+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":744,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159709+0000","last_change":"2024-09-13T22:58:03.020495+0000","last_active":"2024-09-13T22:58:18.159709+0000","last_peered":"2024-09-13T22:58:18.159709+0000","last_clean":"2024-09-13T22:58:18.159709+0000","last_became_active":"2024-09-13T22:56:00.225061+0000","last_became_peered":"2024-09-13T22:56:00.225061+0000","last_unstale":"2024-09-13T22:58:18.159709+0000","last_undegraded":"2024-09-13T22:58:18.159709+0000","last_fullsized":"2024-09-13T22:58:18.159709+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:03.020207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T22:58:03.020207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:28.642564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"404'1083","reported_seq":1106,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:16.366475+0000","last_change":"2024-09-13T22:57:49.644003+0000","last_active":"2024-09-13T22:58:16.366475+0000","last_peered":"2024-09-13T22:58:16.366475+0000","last_clean":"2024-09-13T22:58:16.366475+0000","last_became_active":"2024-09-13T22:57:49.643715+0000","last_became_peered":"2024-09-13T22:57:49.643715+0000","last_unstale":"2024-09-13T22:58:16.366475+0000","last_undegraded":"2024-09-13T22:58:16.366475+0000","last_fullsized":"2024-09-13T22:58:16.366475+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":1083,"log_dups_size":0,"ondisk_log_size":1083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.750653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70975488,"num_objects":1083,"num_object_clones":0,"num_object_copies":3249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":69312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"397'27","reported_seq":833,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137335+0000","last_change":"2024-09-13T22:57:53.254550+0000","last_active":"2024-09-13T22:58:13.137335+0000","last_peered":"2024-09-13T22:58:13.137335+0000","last_clean":"2024-09-13T22:58:13.137335+0000","last_became_active":"2024-09-13T22:55:38.151005+0000","last_became_peered":"2024-09-13T22:55:38.151005+0000","last_unstale":"2024-09-13T22:58:13.137335+0000","last_undegraded":"2024-09-13T22:58:13.137335+0000","last_fullsized":"2024-09-13T22:58:13.137335+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T22:57:53.254254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.254254+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:54.913543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":373,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129484+0000","last_change":"2024-09-13T22:58:01.252621+0000","last_active":"2024-09-13T22:58:13.129484+0000","last_peered":"2024-09-13T22:58:13.129484+0000","last_clean":"2024-09-13T22:58:13.129484+0000","last_became_active":"2024-09-13T22:55:38.136788+0000","last_became_peered":"2024-09-13T22:55:38.136788+0000","last_unstale":"2024-09-13T22:58:13.129484+0000","last_undegraded":"2024-09-13T22:58:13.129484+0000","last_fullsized":"2024-09-13T22:58:13.129484+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:58:01.252317+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:58:01.252317+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.966114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":331,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.130107+0000","last_change":"2024-09-13T22:57:46.251728+0000","last_active":"2024-09-13T22:58:13.130107+0000","last_peered":"2024-09-13T22:58:13.130107+0000","last_clean":"2024-09-13T22:58:13.130107+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T22:58:13.130107+0000","last_undegraded":"2024-09-13T22:58:13.130107+0000","last_fullsized":"2024-09-13T22:58:13.130107+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:46.251423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.251423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:08.732783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":730,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157739+0000","last_change":"2024-09-13T22:57:50.594198+0000","last_active":"2024-09-13T22:58:18.157739+0000","last_peered":"2024-09-13T22:58:18.157739+0000","last_clean":"2024-09-13T22:58:18.157739+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:58:18.157739+0000","last_undegraded":"2024-09-13T22:58:18.157739+0000","last_fullsized":"2024-09-13T22:58:18.157739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.594120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.594120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:56.833221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":847,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135992+0000","last_change":"2024-09-13T22:56:52.281067+0000","last_active":"2024-09-13T22:58:13.135992+0000","last_peered":"2024-09-13T22:58:13.135992+0000","last_clean":"2024-09-13T22:58:13.135992+0000","last_became_active":"2024-09-13T22:56:00.226680+0000","last_became_peered":"2024-09-13T22:56:00.226680+0000","last_unstale":"2024-09-13T22:58:13.135992+0000","last_undegraded":"2024-09-13T22:58:13.135992+0000","last_fullsized":"2024-09-13T22:58:13.135992+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:56:52.280906+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:53:21.566725+0000","last_clean_scrub_stamp":"2024-09-13T22:56:52.280906+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":772,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.134769+0000","last_change":"2024-09-13T22:56:57.677342+0000","last_active":"2024-09-13T22:58:13.134769+0000","last_peered":"2024-09-13T22:58:13.134769+0000","last_clean":"2024-09-13T22:58:13.134769+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:58:13.134769+0000","last_undegraded":"2024-09-13T22:58:13.134769+0000","last_fullsized":"2024-09-13T22:58:13.134769+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:57.675268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:56:57.675268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:15.714564+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":725,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158358+0000","last_change":"2024-09-13T22:55:34.729429+0000","last_active":"2024-09-13T22:58:18.158358+0000","last_peered":"2024-09-13T22:58:18.158358+0000","last_clean":"2024-09-13T22:58:18.158358+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:58:18.158358+0000","last_undegraded":"2024-09-13T22:58:18.158358+0000","last_fullsized":"2024-09-13T22:58:18.158358+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:34.729322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:34.729322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":786,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136674+0000","last_change":"2024-09-13T22:57:01.727570+0000","last_active":"2024-09-13T22:58:13.136674+0000","last_peered":"2024-09-13T22:58:13.136674+0000","last_clean":"2024-09-13T22:58:13.136674+0000","last_became_active":"2024-09-13T22:55:38.126360+0000","last_became_peered":"2024-09-13T22:55:38.126360+0000","last_unstale":"2024-09-13T22:58:13.136674+0000","last_undegraded":"2024-09-13T22:58:13.136674+0000","last_fullsized":"2024-09-13T22:58:13.136674+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:01.727464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:57:01.727464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:29.484184+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":517,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129817+0000","last_change":"2024-09-13T22:57:54.174037+0000","last_active":"2024-09-13T22:58:13.129817+0000","last_peered":"2024-09-13T22:58:13.129817+0000","last_clean":"2024-09-13T22:58:13.129817+0000","last_became_active":"2024-09-13T22:56:00.212121+0000","last_became_peered":"2024-09-13T22:56:00.212121+0000","last_unstale":"2024-09-13T22:58:13.129817+0000","last_undegraded":"2024-09-13T22:58:13.129817+0000","last_fullsized":"2024-09-13T22:58:13.129817+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T22:57:54.173839+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.173839+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:13.804584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":771,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.143457+0000","last_change":"2024-09-13T22:56:51.610256+0000","last_active":"2024-09-13T22:58:18.143457+0000","last_peered":"2024-09-13T22:58:18.143457+0000","last_clean":"2024-09-13T22:58:18.143457+0000","last_became_active":"2024-09-13T22:56:00.227777+0000","last_became_peered":"2024-09-13T22:56:00.227777+0000","last_unstale":"2024-09-13T22:58:18.143457+0000","last_undegraded":"2024-09-13T22:58:18.143457+0000","last_fullsized":"2024-09-13T22:58:18.143457+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:51.610149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T22:56:51.610149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":741,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160596+0000","last_change":"2024-09-13T22:58:04.051815+0000","last_active":"2024-09-13T22:58:18.160596+0000","last_peered":"2024-09-13T22:58:18.160596+0000","last_clean":"2024-09-13T22:58:18.160596+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:58:18.160596+0000","last_undegraded":"2024-09-13T22:58:18.160596+0000","last_fullsized":"2024-09-13T22:58:18.160596+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.051696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.051696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.541486+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":735,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136992+0000","last_change":"2024-09-13T22:56:00.235503+0000","last_active":"2024-09-13T22:58:13.136992+0000","last_peered":"2024-09-13T22:58:13.136992+0000","last_clean":"2024-09-13T22:58:13.136992+0000","last_became_active":"2024-09-13T22:56:00.234954+0000","last_became_peered":"2024-09-13T22:56:00.234954+0000","last_unstale":"2024-09-13T22:58:13.136992+0000","last_undegraded":"2024-09-13T22:58:13.136992+0000","last_fullsized":"2024-09-13T22:58:13.136992+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:09.848281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:54:09.848281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":842,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136620+0000","last_change":"2024-09-13T22:56:00.243298+0000","last_active":"2024-09-13T22:58:13.136620+0000","last_peered":"2024-09-13T22:58:13.136620+0000","last_clean":"2024-09-13T22:58:13.136620+0000","last_became_active":"2024-09-13T22:56:00.207375+0000","last_became_peered":"2024-09-13T22:56:00.207375+0000","last_unstale":"2024-09-13T22:58:13.136620+0000","last_undegraded":"2024-09-13T22:58:13.136620+0000","last_fullsized":"2024-09-13T22:58:13.136620+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:55:30.790120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:30.790120+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":752,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160116+0000","last_change":"2024-09-13T22:56:35.136404+0000","last_active":"2024-09-13T22:58:18.160116+0000","last_peered":"2024-09-13T22:58:18.160116+0000","last_clean":"2024-09-13T22:58:18.160116+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:58:18.160116+0000","last_undegraded":"2024-09-13T22:58:18.160116+0000","last_fullsized":"2024-09-13T22:58:18.160116+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:35.136329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:56:35.136329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":1003,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160160+0000","last_change":"2024-09-13T22:56:36.091025+0000","last_active":"2024-09-13T22:58:18.160160+0000","last_peered":"2024-09-13T22:58:18.160160+0000","last_clean":"2024-09-13T22:58:18.160160+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:58:18.160160+0000","last_undegraded":"2024-09-13T22:58:18.160160+0000","last_fullsized":"2024-09-13T22:58:18.160160+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T22:56:36.090908+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T22:53:18.805172+0000","last_clean_scrub_stamp":"2024-09-13T22:56:36.090908+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":736,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129290+0000","last_change":"2024-09-13T22:57:55.222402+0000","last_active":"2024-09-13T22:58:13.129290+0000","last_peered":"2024-09-13T22:58:13.129290+0000","last_clean":"2024-09-13T22:58:13.129290+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:58:13.129290+0000","last_undegraded":"2024-09-13T22:58:13.129290+0000","last_fullsized":"2024-09-13T22:58:13.129290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:55.222315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.222315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.342997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":778,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.132045+0000","last_change":"2024-09-13T22:57:53.159155+0000","last_active":"2024-09-13T22:58:13.132045+0000","last_peered":"2024-09-13T22:58:13.132045+0000","last_clean":"2024-09-13T22:58:13.132045+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:58:13.132045+0000","last_undegraded":"2024-09-13T22:58:13.132045+0000","last_fullsized":"2024-09-13T22:58:13.132045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T22:57:53.159058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.159058+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.582286+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":766,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.130083+0000","last_change":"2024-09-13T22:55:38.138681+0000","last_active":"2024-09-13T22:58:13.130083+0000","last_peered":"2024-09-13T22:58:13.130083+0000","last_clean":"2024-09-13T22:58:13.130083+0000","last_became_active":"2024-09-13T22:55:38.138035+0000","last_became_peered":"2024-09-13T22:55:38.138035+0000","last_unstale":"2024-09-13T22:58:13.130083+0000","last_undegraded":"2024-09-13T22:58:13.130083+0000","last_fullsized":"2024-09-13T22:58:13.130083+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:36.129393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:36.129393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":991,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157843+0000","last_change":"2024-09-13T22:58:14.754552+0000","last_active":"2024-09-13T22:58:18.157843+0000","last_peered":"2024-09-13T22:58:18.157843+0000","last_clean":"2024-09-13T22:58:18.157843+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:58:18.157843+0000","last_undegraded":"2024-09-13T22:58:18.157843+0000","last_fullsized":"2024-09-13T22:58:18.157843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:58:14.754430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.754430+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.336001+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":257,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135254+0000","last_change":"2024-09-13T22:57:47.490067+0000","last_active":"2024-09-13T22:58:13.135254+0000","last_peered":"2024-09-13T22:58:13.135254+0000","last_clean":"2024-09-13T22:58:13.135254+0000","last_became_active":"2024-09-13T22:55:38.098486+0000","last_became_peered":"2024-09-13T22:55:38.098486+0000","last_unstale":"2024-09-13T22:58:13.135254+0000","last_undegraded":"2024-09-13T22:58:13.135254+0000","last_fullsized":"2024-09-13T22:58:13.135254+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.489929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.489929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:49.045088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":809,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137220+0000","last_change":"2024-09-13T22:55:38.169420+0000","last_active":"2024-09-13T22:58:13.137220+0000","last_peered":"2024-09-13T22:58:13.137220+0000","last_clean":"2024-09-13T22:58:13.137220+0000","last_became_active":"2024-09-13T22:55:38.144744+0000","last_became_peered":"2024-09-13T22:55:38.144744+0000","last_unstale":"2024-09-13T22:58:13.137220+0000","last_undegraded":"2024-09-13T22:58:13.137220+0000","last_fullsized":"2024-09-13T22:58:13.137220+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:55:28.849442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:28.849442+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":115,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158040+0000","last_change":"2024-09-13T22:57:54.689394+0000","last_active":"2024-09-13T22:58:18.158040+0000","last_peered":"2024-09-13T22:58:18.158040+0000","last_clean":"2024-09-13T22:58:18.158040+0000","last_became_active":"2024-09-13T22:56:00.232875+0000","last_became_peered":"2024-09-13T22:56:00.232875+0000","last_unstale":"2024-09-13T22:58:18.158040+0000","last_undegraded":"2024-09-13T22:58:18.158040+0000","last_fullsized":"2024-09-13T22:58:18.158040+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:54.689197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.689197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:05.878279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":725,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157982+0000","last_change":"2024-09-13T22:55:27.845977+0000","last_active":"2024-09-13T22:58:18.157982+0000","last_peered":"2024-09-13T22:58:18.157982+0000","last_clean":"2024-09-13T22:58:18.157982+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:58:18.157982+0000","last_undegraded":"2024-09-13T22:58:18.157982+0000","last_fullsized":"2024-09-13T22:58:18.157982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:27.845519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:55:27.845519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":736,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.326285+0000","last_change":"2024-09-13T22:58:13.326329+0000","last_active":"2024-09-13T22:58:13.326285+0000","last_peered":"2024-09-13T22:58:13.326285+0000","last_clean":"2024-09-13T22:58:13.326285+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:58:13.326285+0000","last_undegraded":"2024-09-13T22:58:13.326285+0000","last_fullsized":"2024-09-13T22:58:13.326285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.326193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.326193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:43.161929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":347,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136967+0000","last_change":"2024-09-13T22:56:00.240679+0000","last_active":"2024-09-13T22:58:13.136967+0000","last_peered":"2024-09-13T22:58:13.136967+0000","last_clean":"2024-09-13T22:58:13.136967+0000","last_became_active":"2024-09-13T22:56:00.207460+0000","last_became_peered":"2024-09-13T22:56:00.207460+0000","last_unstale":"2024-09-13T22:58:13.136967+0000","last_undegraded":"2024-09-13T22:58:13.136967+0000","last_fullsized":"2024-09-13T22:58:13.136967+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:54:30.095316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:30.095316+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":749,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146448+0000","last_change":"2024-09-13T22:58:00.147479+0000","last_active":"2024-09-13T22:58:18.146448+0000","last_peered":"2024-09-13T22:58:18.146448+0000","last_clean":"2024-09-13T22:58:18.146448+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:58:18.146448+0000","last_undegraded":"2024-09-13T22:58:18.146448+0000","last_fullsized":"2024-09-13T22:58:18.146448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:00.147366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T22:58:00.147366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.200027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":741,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.141111+0000","last_change":"2024-09-13T22:58:03.162993+0000","last_active":"2024-09-13T22:58:18.141111+0000","last_peered":"2024-09-13T22:58:18.141111+0000","last_clean":"2024-09-13T22:58:18.141111+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:58:18.141111+0000","last_undegraded":"2024-09-13T22:58:18.141111+0000","last_fullsized":"2024-09-13T22:58:18.141111+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:03.162920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:58:03.162920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:22.631813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":721,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.156034+0000","last_change":"2024-09-13T22:57:07.530820+0000","last_active":"2024-09-13T22:58:18.156034+0000","last_peered":"2024-09-13T22:58:18.156034+0000","last_clean":"2024-09-13T22:58:18.156034+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:58:18.156034+0000","last_undegraded":"2024-09-13T22:58:18.156034+0000","last_fullsized":"2024-09-13T22:58:18.156034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.530747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.530747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"397'18","reported_seq":779,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:14.333638+0000","last_change":"2024-09-13T22:58:14.333753+0000","last_active":"2024-09-13T22:58:14.333638+0000","last_peered":"2024-09-13T22:58:14.333638+0000","last_clean":"2024-09-13T22:58:14.333638+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:58:14.333638+0000","last_undegraded":"2024-09-13T22:58:14.333638+0000","last_fullsized":"2024-09-13T22:58:14.333638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'18","last_scrub_stamp":"2024-09-13T22:58:14.333490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.333490+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:39.855178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":755,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160796+0000","last_change":"2024-09-13T22:58:00.020090+0000","last_active":"2024-09-13T22:58:18.160796+0000","last_peered":"2024-09-13T22:58:18.160796+0000","last_clean":"2024-09-13T22:58:18.160796+0000","last_became_active":"2024-09-13T22:56:00.239806+0000","last_became_peered":"2024-09-13T22:56:00.239806+0000","last_unstale":"2024-09-13T22:58:18.160796+0000","last_undegraded":"2024-09-13T22:58:18.160796+0000","last_fullsized":"2024-09-13T22:58:18.160796+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:00.019958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:00.019958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:10.648382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":761,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136357+0000","last_change":"2024-09-13T22:58:08.201088+0000","last_active":"2024-09-13T22:58:13.136357+0000","last_peered":"2024-09-13T22:58:13.136357+0000","last_clean":"2024-09-13T22:58:13.136357+0000","last_became_active":"2024-09-13T22:56:00.235156+0000","last_became_peered":"2024-09-13T22:56:00.235156+0000","last_unstale":"2024-09-13T22:58:13.136357+0000","last_undegraded":"2024-09-13T22:58:13.136357+0000","last_fullsized":"2024-09-13T22:58:13.136357+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.200989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.200989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:33.903639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":737,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135851+0000","last_change":"2024-09-13T22:57:10.040687+0000","last_active":"2024-09-13T22:58:13.135851+0000","last_peered":"2024-09-13T22:58:13.135851+0000","last_clean":"2024-09-13T22:58:13.135851+0000","last_became_active":"2024-09-13T22:55:38.135406+0000","last_became_peered":"2024-09-13T22:55:38.135406+0000","last_unstale":"2024-09-13T22:58:13.135851+0000","last_undegraded":"2024-09-13T22:58:13.135851+0000","last_fullsized":"2024-09-13T22:58:13.135851+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:10.040498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:10.040498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":779,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160703+0000","last_change":"2024-09-13T22:57:46.014205+0000","last_active":"2024-09-13T22:58:18.160703+0000","last_peered":"2024-09-13T22:58:18.160703+0000","last_clean":"2024-09-13T22:58:18.160703+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:58:18.160703+0000","last_undegraded":"2024-09-13T22:58:18.160703+0000","last_fullsized":"2024-09-13T22:58:18.160703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:57:46.013979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.013979+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:55.078753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":115,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160860+0000","last_change":"2024-09-13T22:58:13.134377+0000","last_active":"2024-09-13T22:58:18.160860+0000","last_peered":"2024-09-13T22:58:18.160860+0000","last_clean":"2024-09-13T22:58:18.160860+0000","last_became_active":"2024-09-13T22:56:00.242421+0000","last_became_peered":"2024-09-13T22:56:00.242421+0000","last_unstale":"2024-09-13T22:58:18.160860+0000","last_undegraded":"2024-09-13T22:58:18.160860+0000","last_fullsized":"2024-09-13T22:58:18.160860+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.134261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.134261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.326448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":747,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135710+0000","last_change":"2024-09-13T22:56:55.145345+0000","last_active":"2024-09-13T22:58:13.135710+0000","last_peered":"2024-09-13T22:58:13.135710+0000","last_clean":"2024-09-13T22:58:13.135710+0000","last_became_active":"2024-09-13T22:56:00.213590+0000","last_became_peered":"2024-09-13T22:56:00.213590+0000","last_unstale":"2024-09-13T22:58:13.135710+0000","last_undegraded":"2024-09-13T22:58:13.135710+0000","last_fullsized":"2024-09-13T22:58:13.135710+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:55.145267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:56:55.145267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":735,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158161+0000","last_change":"2024-09-13T22:57:59.732773+0000","last_active":"2024-09-13T22:58:18.158161+0000","last_peered":"2024-09-13T22:58:18.158161+0000","last_clean":"2024-09-13T22:58:18.158161+0000","last_became_active":"2024-09-13T22:56:00.218040+0000","last_became_peered":"2024-09-13T22:56:00.218040+0000","last_unstale":"2024-09-13T22:58:18.158161+0000","last_undegraded":"2024-09-13T22:58:18.158161+0000","last_fullsized":"2024-09-13T22:58:18.158161+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:59.732688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:57:59.732688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:18.019873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":722,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137195+0000","last_change":"2024-09-13T22:56:53.116102+0000","last_active":"2024-09-13T22:58:13.137195+0000","last_peered":"2024-09-13T22:58:13.137195+0000","last_clean":"2024-09-13T22:58:13.137195+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:58:13.137195+0000","last_undegraded":"2024-09-13T22:58:13.137195+0000","last_fullsized":"2024-09-13T22:58:13.137195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_clean_scrub_stamp":"2024-09-13T22:56:53.115999+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:19.659179+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.135936+0000","last_change":"2024-09-13T22:57:37.368671+0000","last_active":"2024-09-13T22:58:13.135936+0000","last_peered":"2024-09-13T22:58:13.135936+0000","last_clean":"2024-09-13T22:58:13.135936+0000","last_became_active":"2024-09-13T22:57:37.355449+0000","last_became_peered":"2024-09-13T22:57:37.355449+0000","last_unstale":"2024-09-13T22:58:13.135936+0000","last_undegraded":"2024-09-13T22:58:13.135936+0000","last_fullsized":"2024-09-13T22:58:13.135936+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:57:10.942679+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:57:10.942679+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:39.719181+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":773,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.159936+0000","last_change":"2024-09-13T22:56:34.086608+0000","last_active":"2024-09-13T22:58:18.159936+0000","last_peered":"2024-09-13T22:58:18.159936+0000","last_clean":"2024-09-13T22:58:18.159936+0000","last_became_active":"2024-09-13T22:56:00.242209+0000","last_became_peered":"2024-09-13T22:56:00.242209+0000","last_unstale":"2024-09-13T22:58:18.159936+0000","last_undegraded":"2024-09-13T22:58:18.159936+0000","last_fullsized":"2024-09-13T22:58:18.159936+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:34.086426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:56:34.086426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"358'16","reported_seq":472,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.137566+0000","last_change":"2024-09-13T22:56:00.255155+0000","last_active":"2024-09-13T22:58:13.137566+0000","last_peered":"2024-09-13T22:58:13.137566+0000","last_clean":"2024-09-13T22:58:13.137566+0000","last_became_active":"2024-09-13T22:56:00.229080+0000","last_became_peered":"2024-09-13T22:56:00.229080+0000","last_unstale":"2024-09-13T22:58:13.137566+0000","last_undegraded":"2024-09-13T22:58:13.137566+0000","last_fullsized":"2024-09-13T22:58:13.137566+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'14","last_scrub_stamp":"2024-09-13T22:55:25.529898+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T22:55:25.529898+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":755,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146099+0000","last_change":"2024-09-13T22:57:09.960429+0000","last_active":"2024-09-13T22:58:18.146099+0000","last_peered":"2024-09-13T22:58:18.146099+0000","last_clean":"2024-09-13T22:58:18.146099+0000","last_became_active":"2024-09-13T22:55:38.096392+0000","last_became_peered":"2024-09-13T22:55:38.096392+0000","last_unstale":"2024-09-13T22:58:18.146099+0000","last_undegraded":"2024-09-13T22:58:18.146099+0000","last_fullsized":"2024-09-13T22:58:18.146099+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.960363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.960363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:25.840517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":779,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.153464+0000","last_change":"2024-09-13T22:57:37.364148+0000","last_active":"2024-09-13T22:58:18.153464+0000","last_peered":"2024-09-13T22:58:18.153464+0000","last_clean":"2024-09-13T22:58:18.153464+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T22:58:18.153464+0000","last_undegraded":"2024-09-13T22:58:18.153464+0000","last_fullsized":"2024-09-13T22:58:18.153464+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:55:35.746045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:35.746045+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":736,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.156262+0000","last_change":"2024-09-13T22:56:00.224528+0000","last_active":"2024-09-13T22:58:18.156262+0000","last_peered":"2024-09-13T22:58:18.156262+0000","last_clean":"2024-09-13T22:58:18.156262+0000","last_became_active":"2024-09-13T22:56:00.213377+0000","last_became_peered":"2024-09-13T22:56:00.213377+0000","last_unstale":"2024-09-13T22:58:18.156262+0000","last_undegraded":"2024-09-13T22:58:18.156262+0000","last_fullsized":"2024-09-13T22:58:18.156262+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:54:35.340609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:35.340609+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":742,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129349+0000","last_change":"2024-09-13T22:55:32.033215+0000","last_active":"2024-09-13T22:58:13.129349+0000","last_peered":"2024-09-13T22:58:13.129349+0000","last_clean":"2024-09-13T22:58:13.129349+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:58:13.129349+0000","last_undegraded":"2024-09-13T22:58:13.129349+0000","last_fullsized":"2024-09-13T22:58:13.129349+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:32.033129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:55:32.033129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":246,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.160474+0000","last_change":"2024-09-13T22:58:08.025375+0000","last_active":"2024-09-13T22:58:18.160474+0000","last_peered":"2024-09-13T22:58:18.160474+0000","last_clean":"2024-09-13T22:58:18.160474+0000","last_became_active":"2024-09-13T22:56:00.230261+0000","last_became_peered":"2024-09-13T22:56:00.230261+0000","last_unstale":"2024-09-13T22:58:18.160474+0000","last_undegraded":"2024-09-13T22:58:18.160474+0000","last_fullsized":"2024-09-13T22:58:18.160474+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.025258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.025258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:32.564711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":769,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.136327+0000","last_change":"2024-09-13T22:57:01.014387+0000","last_active":"2024-09-13T22:58:13.136327+0000","last_peered":"2024-09-13T22:58:13.136327+0000","last_clean":"2024-09-13T22:58:13.136327+0000","last_became_active":"2024-09-13T22:56:00.228879+0000","last_became_peered":"2024-09-13T22:56:00.228879+0000","last_unstale":"2024-09-13T22:58:13.136327+0000","last_undegraded":"2024-09-13T22:58:13.136327+0000","last_fullsized":"2024-09-13T22:58:13.136327+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:01.014315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:01.014315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":806,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T22:58:13.129999+0000","last_change":"2024-09-13T22:58:12.352800+0000","last_active":"2024-09-13T22:58:13.129999+0000","last_peered":"2024-09-13T22:58:13.129999+0000","last_clean":"2024-09-13T22:58:13.129999+0000","last_became_active":"2024-09-13T22:55:38.138369+0000","last_became_peered":"2024-09-13T22:55:38.138369+0000","last_unstale":"2024-09-13T22:58:13.129999+0000","last_undegraded":"2024-09-13T22:58:13.129999+0000","last_fullsized":"2024-09-13T22:58:13.129999+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'11","last_scrub_stamp":"2024-09-13T22:58:12.352675+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.352675+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:37.199068+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":759,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.146366+0000","last_change":"2024-09-13T22:57:57.115157+0000","last_active":"2024-09-13T22:58:18.146366+0000","last_peered":"2024-09-13T22:58:18.146366+0000","last_clean":"2024-09-13T22:58:18.146366+0000","last_became_active":"2024-09-13T22:55:38.094235+0000","last_became_peered":"2024-09-13T22:55:38.094235+0000","last_unstale":"2024-09-13T22:58:18.146366+0000","last_undegraded":"2024-09-13T22:58:18.146366+0000","last_fullsized":"2024-09-13T22:58:18.146366+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:57.114953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:17.313617+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.114953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.976193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":733,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157945+0000","last_change":"2024-09-13T22:56:00.255194+0000","last_active":"2024-09-13T22:58:18.157945+0000","last_peered":"2024-09-13T22:58:18.157945+0000","last_clean":"2024-09-13T22:58:18.157945+0000","last_became_active":"2024-09-13T22:56:00.230406+0000","last_became_peered":"2024-09-13T22:56:00.230406+0000","last_unstale":"2024-09-13T22:58:18.157945+0000","last_undegraded":"2024-09-13T22:58:18.157945+0000","last_fullsized":"2024-09-13T22:58:18.157945+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:55:10.933585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:10.933585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":739,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.158366+0000","last_change":"2024-09-13T22:56:00.231974+0000","last_active":"2024-09-13T22:58:18.158366+0000","last_peered":"2024-09-13T22:58:18.158366+0000","last_clean":"2024-09-13T22:58:18.158366+0000","last_became_active":"2024-09-13T22:56:00.231645+0000","last_became_peered":"2024-09-13T22:56:00.231645+0000","last_unstale":"2024-09-13T22:58:18.158366+0000","last_undegraded":"2024-09-13T22:58:18.158366+0000","last_fullsized":"2024-09-13T22:58:18.158366+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:24.878070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:24.878070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":587,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.141150+0000","last_change":"2024-09-13T22:57:54.101195+0000","last_active":"2024-09-13T22:58:18.141150+0000","last_peered":"2024-09-13T22:58:18.141150+0000","last_clean":"2024-09-13T22:58:18.141150+0000","last_became_active":"2024-09-13T22:55:38.099550+0000","last_became_peered":"2024-09-13T22:55:38.099550+0000","last_unstale":"2024-09-13T22:58:18.141150+0000","last_undegraded":"2024-09-13T22:58:18.141150+0000","last_fullsized":"2024-09-13T22:58:18.141150+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T22:57:54.100973+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.100973+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.423659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":236,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T22:58:18.157745+0000","last_change":"2024-09-13T22:55:38.133820+0000","last_active":"2024-09-13T22:58:18.157745+0000","last_peered":"2024-09-13T22:58:18.157745+0000","last_clean":"2024-09-13T22:58:18.157745+0000","last_became_active":"2024-09-13T22:55:38.133425+0000","last_became_peered":"2024-09-13T22:55:38.133425+0000","last_unstale":"2024-09-13T22:58:18.157745+0000","last_undegraded":"2024-09-13T22:58:18.157745+0000","last_fullsized":"2024-09-13T22:58:18.157745+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:28.200985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:55:28.200985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1210580992,"num_objects":18472,"num_object_clones":0,"num_object_copies":55416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18472,"num_write_kb":1182208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3604938752,"data_stored":3604938752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18472,"ondisk_log_size":18472,"up":48,"acting":48,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1211,"num_read_kb":1206,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1555,"num_read_kb":1312,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":7040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":14296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1055,"ondisk_log_size":1055,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":414,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809087,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000596,"kb_used_data":369664,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92744556,"statfs":{"total":95995035648,"available":94970425344,"internally_reserved":0,"allocated":378535936,"data_stored":378053457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":244813136138,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528598,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058632,"kb_used_data":467960,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92686520,"statfs":{"total":95995035648,"available":94910996480,"internally_reserved":0,"allocated":479191040,"data_stored":478715579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084586,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1340740,"kb_used_data":601524,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92404412,"statfs":{"total":95995035648,"available":94622117888,"internally_reserved":0,"allocated":615960576,"data_stored":615489586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":756939207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378933,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889280,"kb_used_data":424944,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92855872,"statfs":{"total":95995035648,"available":95084412928,"internally_reserved":0,"allocated":435142656,"data_stored":434675745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575168,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1597588,"kb_used_data":934724,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92147564,"statfs":{"total":95995035648,"available":94359105536,"internally_reserved":0,"allocated":957157376,"data_stored":956670133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1263116,"kb_used_data":559996,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92482036,"statfs":{"total":95995035648,"available":94701604864,"internally_reserved":0,"allocated":573435904,"data_stored":572957225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":573636608,"data_stored":573636608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":999948288,"data_stored":999948288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":446824448,"data_stored":446824448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":664862720,"data_stored":664862720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":501612544,"data_stored":501612544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":418054144,"data_stored":418054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:58:26.082 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T22:58:26.082 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:58:26.082 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-13T22:58:26.083 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd in 6 2024-09-13T22:58:26.147 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:26.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:25 smithi192 bash[24608]: cluster 2024-09-13T22:58:24.917848+0000 mgr.y (mgr.14613) 771 : cluster [DBG] pgmap v910: 158 pgs: 158 active+clean; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-13T22:58:26.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:58:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:58:26.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:26.259 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:25 smithi178 bash[29889]: cluster 2024-09-13T22:58:24.917848+0000 mgr.y (mgr.14613) 771 : cluster [DBG] pgmap v910: 158 pgs: 158 active+clean; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-13T22:58:26.259 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:25 smithi178 bash[22840]: cluster 2024-09-13T22:58:24.917848+0000 mgr.y (mgr.14613) 771 : cluster [DBG] pgmap v910: 158 pgs: 158 active+clean; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-13T22:58:26.275 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:26.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:26.402 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:26.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:26.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:26.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:26.654 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:26.711 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:58:26.712801+0000 min lat: 0.00117945 max lat: 0.461558 avg lat: 0.0104195 2024-09-13T22:58:26.711 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:58:26.711 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 20 16 30713 30697 95.9193 90.625 0.0176652 0.0104195 2024-09-13T22:58:26.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:26.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:26.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:26.915 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:26.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:27.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:27.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:27.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:27.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:26 smithi178 bash[29889]: audit 2024-09-13T22:58:25.881899+0000 mgr.y (mgr.14613) 772 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:27.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:26 smithi178 bash[22840]: audit 2024-09-13T22:58:25.881899+0000 mgr.y (mgr.14613) 772 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:27.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:27.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:26 smithi192 bash[24608]: audit 2024-09-13T22:58:25.881899+0000 mgr.y (mgr.14613) 772 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:27.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:27.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:27.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.562 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:27.666 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 21 16 32202 32186 95.7634 93.0625 0.0109167 0.0104319 2024-09-13T22:58:27.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:27.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:27.852 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:27.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:27.981 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:27.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:28.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:58:28.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T22:58:28.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:28.099 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:27 smithi192 bash[24608]: cluster 2024-09-13T22:58:26.918405+0000 mgr.y (mgr.14613) 773 : cluster [DBG] pgmap v911: 158 pgs: 158 active+clean; 1.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T22:58:28.113 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:28.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:28.238 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:28.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:28.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:28.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:27 smithi178 bash[22840]: cluster 2024-09-13T22:58:26.918405+0000 mgr.y (mgr.14613) 773 : cluster [DBG] pgmap v911: 158 pgs: 158 active+clean; 1.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T22:58:28.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:27 smithi178 bash[29889]: cluster 2024-09-13T22:58:26.918405+0000 mgr.y (mgr.14613) 773 : cluster [DBG] pgmap v911: 158 pgs: 158 active+clean; 1.7 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-13T22:58:28.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:28.501 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:28.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:28.628 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:28.715 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 22 16 33527 33511 95.1745 82.8125 0.015099 0.0104991 2024-09-13T22:58:28.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:28.753 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:28.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:28.880 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:28.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:28.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:29.011 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:29.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:29 smithi178 bash[29889]: cluster 2024-09-13T22:58:27.559397+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:29 smithi178 bash[29889]: cluster 2024-09-13T22:58:27.559467+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:29 smithi178 bash[29889]: cluster 2024-09-13T22:58:28.007774+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e407: 8 total, 8 up, 6 in 2024-09-13T22:58:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:29 smithi178 bash[29889]: cluster 2024-09-13T22:58:28.135260+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e408: 8 total, 8 up, 6 in 2024-09-13T22:58:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:28 smithi178 bash[22840]: cluster 2024-09-13T22:58:27.559397+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:28 smithi178 bash[22840]: cluster 2024-09-13T22:58:27.559467+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:28 smithi178 bash[22840]: cluster 2024-09-13T22:58:28.007774+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e407: 8 total, 8 up, 6 in 2024-09-13T22:58:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:28 smithi178 bash[22840]: cluster 2024-09-13T22:58:28.135260+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e408: 8 total, 8 up, 6 in 2024-09-13T22:58:29.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:29.270 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:29.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:29.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:29 smithi192 bash[24608]: cluster 2024-09-13T22:58:27.559397+0000 osd.6 (osd.6) 308 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:29.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:29 smithi192 bash[24608]: cluster 2024-09-13T22:58:27.559467+0000 osd.6 (osd.6) 309 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:29.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:29 smithi192 bash[24608]: cluster 2024-09-13T22:58:28.007774+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e407: 8 total, 8 up, 6 in 2024-09-13T22:58:29.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:29 smithi192 bash[24608]: cluster 2024-09-13T22:58:28.135260+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e408: 8 total, 8 up, 6 in 2024-09-13T22:58:29.400 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:29.503 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:29.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:29.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:29.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:29.715 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 23 16 34960 34944 94.9302 89.5625 0.00525162 0.0105273 2024-09-13T22:58:29.717 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:29.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:29.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:29.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:29.932 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:29.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:30.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:30.046 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:58:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:58:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:58:30.059 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:30.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:30.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:30 smithi192 bash[24608]: cluster 2024-09-13T22:58:28.919139+0000 mgr.y (mgr.14613) 774 : cluster [DBG] pgmap v914: 158 pgs: 158 active+clean; 1.9 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T22:58:30.185 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:30.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:30.311 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:30.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:30.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:30 smithi178 bash[29889]: cluster 2024-09-13T22:58:28.919139+0000 mgr.y (mgr.14613) 774 : cluster [DBG] pgmap v914: 158 pgs: 158 active+clean; 1.9 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T22:58:30.425 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:30 smithi178 bash[22840]: cluster 2024-09-13T22:58:28.919139+0000 mgr.y (mgr.14613) 774 : cluster [DBG] pgmap v914: 158 pgs: 158 active+clean; 1.9 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T22:58:30.449 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:30.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:30.578 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:30.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.713 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:30.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:30.715 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 24 16 36333 36317 94.55 85.8125 0.010335 0.0105705 2024-09-13T22:58:30.739 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:30.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:30.848 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:30.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:30.976 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:30.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:31.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:31.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:31.243 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:31.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:31.370 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:31.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:31.498 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:31.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:31.621 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:31.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 25 16 37605 37589 93.9478 79.5 0.019215 0.010638 2024-09-13T22:58:31.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:31.860 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:31.890 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:31.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:31.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:32.032 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:32.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:32.174 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:32.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:32.307 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:32.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:32 smithi192 bash[24608]: cluster 2024-09-13T22:58:30.919857+0000 mgr.y (mgr.14613) 775 : cluster [DBG] pgmap v915: 158 pgs: 158 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T22:58:32.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:32.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:32 smithi178 bash[29889]: cluster 2024-09-13T22:58:30.919857+0000 mgr.y (mgr.14613) 775 : cluster [DBG] pgmap v915: 158 pgs: 158 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T22:58:32.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:32 smithi178 bash[22840]: cluster 2024-09-13T22:58:30.919857+0000 mgr.y (mgr.14613) 775 : cluster [DBG] pgmap v915: 158 pgs: 158 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T22:58:32.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:32.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:32.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:32.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:32.694 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:32.715 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 26 16 38927 38911 93.5121 82.625 0.00940358 0.0106877 2024-09-13T22:58:32.779 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:32.780 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:32.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:32.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:32.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:32.951 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:32.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:33.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:33.085 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:33.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:33.224 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:33.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:33.352 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[29889]: cluster 2024-09-13T22:58:31.888403+0000 osd.3 (osd.3) 385 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[29889]: cluster 2024-09-13T22:58:31.888471+0000 osd.3 (osd.3) 386 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:33.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[29889]: cluster 2024-09-13T22:58:32.165804+0000 mon.a (mon.0) 1672 : cluster [DBG] osdmap e409: 8 total, 8 up, 6 in 2024-09-13T22:58:33.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[29889]: cluster 2024-09-13T22:58:33.135426+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e410: 8 total, 8 up, 6 in 2024-09-13T22:58:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[22840]: cluster 2024-09-13T22:58:31.888403+0000 osd.3 (osd.3) 385 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[22840]: cluster 2024-09-13T22:58:31.888471+0000 osd.3 (osd.3) 386 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[22840]: cluster 2024-09-13T22:58:32.165804+0000 mon.a (mon.0) 1672 : cluster [DBG] osdmap e409: 8 total, 8 up, 6 in 2024-09-13T22:58:33.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:33 smithi178 bash[22840]: cluster 2024-09-13T22:58:33.135426+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e410: 8 total, 8 up, 6 in 2024-09-13T22:58:33.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:33.485 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:33.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:33.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:33 smithi192 bash[24608]: cluster 2024-09-13T22:58:31.888403+0000 osd.3 (osd.3) 385 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:33.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:33 smithi192 bash[24608]: cluster 2024-09-13T22:58:31.888471+0000 osd.3 (osd.3) 386 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:33.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:33 smithi192 bash[24608]: cluster 2024-09-13T22:58:32.165804+0000 mon.a (mon.0) 1672 : cluster [DBG] osdmap e409: 8 total, 8 up, 6 in 2024-09-13T22:58:33.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:33 smithi192 bash[24608]: cluster 2024-09-13T22:58:33.135426+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e410: 8 total, 8 up, 6 in 2024-09-13T22:58:33.615 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:33.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 27 16 39960 39944 92.4398 64.5625 0.002292 0.0107731 2024-09-13T22:58:33.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:33.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:33.872 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:33.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:33.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:34.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:34.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:34.138 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:34.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:34.267 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:34.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:34.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:34.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:34 smithi178 bash[22840]: cluster 2024-09-13T22:58:32.920292+0000 mgr.y (mgr.14613) 776 : cluster [DBG] pgmap v917: 158 pgs: 158 active+clean; 2.1 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T22:58:34.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:34 smithi178 bash[29889]: cluster 2024-09-13T22:58:32.920292+0000 mgr.y (mgr.14613) 776 : cluster [DBG] pgmap v917: 158 pgs: 158 active+clean; 2.1 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T22:58:34.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:34.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:34 smithi192 bash[24608]: cluster 2024-09-13T22:58:32.920292+0000 mgr.y (mgr.14613) 776 : cluster [DBG] pgmap v917: 158 pgs: 158 active+clean; 2.1 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T22:58:34.518 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:34.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:34.647 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:34.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 28 16 40593 40577 90.5515 39.5625 0.0154045 0.0110366 2024-09-13T22:58:34.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:34.787 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:34.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:34.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:34.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:35.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:35.046 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:35.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:35.174 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:35.277 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:35.287 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: cluster 2024-09-13T22:58:33.985119+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: cluster 2024-09-13T22:58:33.985255+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: cluster 2024-09-13T22:58:34.217906+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e411: 8 total, 8 up, 6 in 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: audit 2024-09-13T22:58:34.404719+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: audit 2024-09-13T22:58:34.408147+0000 mon.b (mon.1) 268 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: cluster 2024-09-13T22:58:34.679561+0000 osd.2 (osd.2) 295 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[22840]: cluster 2024-09-13T22:58:34.679700+0000 osd.2 (osd.2) 296 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: cluster 2024-09-13T22:58:33.985119+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: cluster 2024-09-13T22:58:33.985255+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: cluster 2024-09-13T22:58:34.217906+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e411: 8 total, 8 up, 6 in 2024-09-13T22:58:35.288 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: audit 2024-09-13T22:58:34.404719+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:35.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: audit 2024-09-13T22:58:34.408147+0000 mon.b (mon.1) 268 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:35.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: cluster 2024-09-13T22:58:34.679561+0000 osd.2 (osd.2) 295 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:35.289 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:35 smithi178 bash[29889]: cluster 2024-09-13T22:58:34.679700+0000 osd.2 (osd.2) 296 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:35.302 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:35.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: cluster 2024-09-13T22:58:33.985119+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: cluster 2024-09-13T22:58:33.985255+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: cluster 2024-09-13T22:58:34.217906+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e411: 8 total, 8 up, 6 in 2024-09-13T22:58:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: audit 2024-09-13T22:58:34.404719+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: audit 2024-09-13T22:58:34.408147+0000 mon.b (mon.1) 268 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: cluster 2024-09-13T22:58:34.679561+0000 osd.2 (osd.2) 295 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[24608]: cluster 2024-09-13T22:58:34.679700+0000 osd.2 (osd.2) 296 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:35.431 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:35.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:35.567 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:35.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:35.697 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:35.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 29 16 41709 41693 89.8339 69.75 0.00263583 0.0110721 2024-09-13T22:58:35.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:35.824 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:35.927 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:35.959 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:35.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:36.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:36.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:58:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:58:36.085 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:36.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:36.212 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:36.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:36.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:36.473 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:36.576 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:36.585 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:36 smithi178 bash[29889]: cluster 2024-09-13T22:58:34.922132+0000 mgr.y (mgr.14613) 777 : cluster [DBG] pgmap v920: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T22:58:36.585 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:36 smithi178 bash[29889]: cluster 2024-09-13T22:58:35.412499+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e412: 8 total, 8 up, 6 in 2024-09-13T22:58:36.585 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:36 smithi178 bash[22840]: cluster 2024-09-13T22:58:34.922132+0000 mgr.y (mgr.14613) 777 : cluster [DBG] pgmap v920: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T22:58:36.585 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:36 smithi178 bash[22840]: cluster 2024-09-13T22:58:35.412499+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e412: 8 total, 8 up, 6 in 2024-09-13T22:58:36.601 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:36.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:36.714 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:36 smithi192 bash[24608]: cluster 2024-09-13T22:58:34.922132+0000 mgr.y (mgr.14613) 777 : cluster [DBG] pgmap v920: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T22:58:36.714 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:36 smithi192 bash[24608]: cluster 2024-09-13T22:58:35.412499+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e412: 8 total, 8 up, 6 in 2024-09-13T22:58:36.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 30 16 42589 42573 88.6727 55 0.0117607 0.0112708 2024-09-13T22:58:36.730 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:36.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:36.857 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:36.959 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:36.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:36.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:37.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:37.115 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:37.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:37.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:37.243 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:37.346 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:37.384 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:37.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:37.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:37 smithi178 bash[29889]: audit 2024-09-13T22:58:35.891830+0000 mgr.y (mgr.14613) 778 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:37.497 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:37 smithi178 bash[22840]: audit 2024-09-13T22:58:35.891830+0000 mgr.y (mgr.14613) 778 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:37.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:37.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:37 smithi192 bash[24608]: audit 2024-09-13T22:58:35.891830+0000 mgr.y (mgr.14613) 778 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:37.641 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:37.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 31 16 43722 43706 88.0963 70.8125 0.00290809 0.0113079 2024-09-13T22:58:37.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:37.775 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:37.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:37.906 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:37.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:38.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:38.036 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:38.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:38.190 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:38.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:38.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:38.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:38.453 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:38.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:38.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:38 smithi192 bash[24608]: cluster 2024-09-13T22:58:36.923287+0000 mgr.y (mgr.14613) 779 : cluster [DBG] pgmap v922: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T22:58:38.578 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:38.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:38 smithi178 bash[29889]: cluster 2024-09-13T22:58:36.923287+0000 mgr.y (mgr.14613) 779 : cluster [DBG] pgmap v922: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T22:58:38.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:38 smithi178 bash[22840]: cluster 2024-09-13T22:58:36.923287+0000 mgr.y (mgr.14613) 779 : cluster [DBG] pgmap v922: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T22:58:38.707 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:38.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 32 15 44755 44740 87.3627 64.625 0.00594337 0.0114412 2024-09-13T22:58:38.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:38.833 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:38.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:38.958 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:38.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:39.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:39.083 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:39.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:39.209 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:39.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:39.341 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:39.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:39.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:39.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:39.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:39.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:39.716 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:39.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 33 16 45384 45368 85.9048 39.25 0.0199977 0.0115845 2024-09-13T22:58:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:39.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:39.842 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:39.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:39.967 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:39.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:40.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:40.082 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:58:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:58:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:58:40.106 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:40.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:40.233 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:40.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:40.359 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:40.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:40.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:40.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:40.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:40 smithi192 bash[24608]: cluster 2024-09-13T22:58:38.924047+0000 mgr.y (mgr.14613) 780 : cluster [DBG] pgmap v923: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:58:40.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:40 smithi192 bash[24608]: audit 2024-09-13T22:58:39.652926+0000 mon.b (mon.1) 269 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:58:40.625 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:40.716 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 34 16 46261 46245 84.9903 54.8125 0.0131386 0.0117594 2024-09-13T22:58:40.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:40.747 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:40.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:40.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:40.912 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:40 smithi178 bash[22840]: cluster 2024-09-13T22:58:38.924047+0000 mgr.y (mgr.14613) 780 : cluster [DBG] pgmap v923: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:58:40.913 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:40 smithi178 bash[22840]: audit 2024-09-13T22:58:39.652926+0000 mon.b (mon.1) 269 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:58:40.913 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:40 smithi178 bash[29889]: cluster 2024-09-13T22:58:38.924047+0000 mgr.y (mgr.14613) 780 : cluster [DBG] pgmap v923: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T22:58:40.913 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:40 smithi178 bash[29889]: audit 2024-09-13T22:58:39.652926+0000 mon.b (mon.1) 269 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:58:40.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:40.994 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:40.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:41.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:41.117 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:41.120 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:58:41.120 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:58:41.158 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:58:41.159 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:58:41.171 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: debug 2024-09-13T22:58:41.057+0000 7f8b8e05d640 -1 mon.a@0(leader).osd e412 definitely_dead 0 2024-09-13T22:58:41.201 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:58:41.201 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:58:41.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:41.232 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:58:41.232 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:58:41.241 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:41.262 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:58:41.262 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:58:41.299 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:58:41.300 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:58:41.341 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:58:41.341 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:58:41.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:41.365 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:58:41.365 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T22:58:41.365 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T22:58:41.365 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T22:58:41.365 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:58:41.365 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:41.371 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:58:41.371 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:58:41.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:41.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:41 smithi192 bash[24608]: audit 2024-09-13T22:58:40.669093+0000 mon.b (mon.1) 270 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:58:41.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:41 smithi192 bash[24608]: audit 2024-09-13T22:58:40.670142+0000 mon.b (mon.1) 271 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:58:41.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:41 smithi192 bash[24608]: audit 2024-09-13T22:58:40.683368+0000 mon.a (mon.0) 1677 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:41.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:41 smithi192 bash[24608]: audit 2024-09-13T22:58:41.060063+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? v1:172.21.15.178:0/952875783' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-13T22:58:41.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:41 smithi192 bash[24608]: audit 2024-09-13T22:58:41.060975+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.25705 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-13T22:58:41.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:41 smithi192 bash[24608]: audit 2024-09-13T22:58:41.088892+0000 mon.a (mon.0) 1679 : audit [DBG] from='client.? v1:172.21.15.178:0/375760224' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:41.493 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:41.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:41.616 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:41.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[29889]: audit 2024-09-13T22:58:40.669093+0000 mon.b (mon.1) 270 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:58:41.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[29889]: audit 2024-09-13T22:58:40.670142+0000 mon.b (mon.1) 271 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:58:41.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[29889]: audit 2024-09-13T22:58:40.683368+0000 mon.a (mon.0) 1677 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:41.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[29889]: audit 2024-09-13T22:58:41.060063+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? v1:172.21.15.178:0/952875783' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-13T22:58:41.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[29889]: audit 2024-09-13T22:58:41.060975+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.25705 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-13T22:58:41.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[29889]: audit 2024-09-13T22:58:41.088892+0000 mon.a (mon.0) 1679 : audit [DBG] from='client.? v1:172.21.15.178:0/375760224' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:41.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: audit 2024-09-13T22:58:40.669093+0000 mon.b (mon.1) 270 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:58:41.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: audit 2024-09-13T22:58:40.670142+0000 mon.b (mon.1) 271 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:58:41.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: audit 2024-09-13T22:58:40.683368+0000 mon.a (mon.0) 1677 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:41.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: audit 2024-09-13T22:58:41.060063+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? v1:172.21.15.178:0/952875783' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-13T22:58:41.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: audit 2024-09-13T22:58:41.060975+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.25705 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-13T22:58:41.635 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:41 smithi178 bash[22840]: audit 2024-09-13T22:58:41.088892+0000 mon.a (mon.0) 1679 : audit [DBG] from='client.? v1:172.21.15.178:0/375760224' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:58:41.697 INFO:teuthology.orchestra.run.smithi178.stderr:marked in osd.6. 2024-09-13T22:58:41.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 35 16 47195 47179 84.2298 58.375 0.0273065 0.0118611 2024-09-13T22:58:41.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:41.740 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:41.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:41.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:41.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:41.993 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:41.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:42.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:42.124 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:42.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:42.253 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:42.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:42.382 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:42.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:42.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:42.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:42.701 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T22:58:42.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 36 16 47203 47187 81.9042 0.5 0.0481178 0.0118656 2024-09-13T22:58:42.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:42.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:42.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: cluster 2024-09-13T22:58:40.924659+0000 mgr.y (mgr.14613) 781 : cluster [DBG] pgmap v924: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-13T22:58:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: cluster 2024-09-13T22:58:41.362899+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: cluster 2024-09-13T22:58:41.362964+0000 osd.1 (osd.1) 399 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: cluster 2024-09-13T22:58:41.624143+0000 osd.5 (osd.5) 447 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:42.886 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: cluster 2024-09-13T22:58:41.624193+0000 osd.5 (osd.5) 448 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:42.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: audit 2024-09-13T22:58:41.692367+0000 mon.a (mon.0) 1680 : audit [INF] from='client.25705 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T22:58:42.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: audit 2024-09-13T22:58:41.706190+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.178:0/1874299349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:42.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: cluster 2024-09-13T22:58:41.721500+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-09-13T22:58:42.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:42 smithi192 bash[24608]: audit 2024-09-13T22:58:41.725935+0000 mon.a (mon.0) 1682 : audit [INF] from='client.25838 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:42.899 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:42.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: cluster 2024-09-13T22:58:40.924659+0000 mgr.y (mgr.14613) 781 : cluster [DBG] pgmap v924: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: cluster 2024-09-13T22:58:41.362899+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: cluster 2024-09-13T22:58:41.362964+0000 osd.1 (osd.1) 399 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: cluster 2024-09-13T22:58:41.624143+0000 osd.5 (osd.5) 447 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: cluster 2024-09-13T22:58:41.624193+0000 osd.5 (osd.5) 448 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: audit 2024-09-13T22:58:41.692367+0000 mon.a (mon.0) 1680 : audit [INF] from='client.25705 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: audit 2024-09-13T22:58:41.706190+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.178:0/1874299349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: cluster 2024-09-13T22:58:41.721500+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[29889]: audit 2024-09-13T22:58:41.725935+0000 mon.a (mon.0) 1682 : audit [INF] from='client.25838 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: cluster 2024-09-13T22:58:40.924659+0000 mgr.y (mgr.14613) 781 : cluster [DBG] pgmap v924: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-13T22:58:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: cluster 2024-09-13T22:58:41.362899+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: cluster 2024-09-13T22:58:41.362964+0000 osd.1 (osd.1) 399 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: cluster 2024-09-13T22:58:41.624143+0000 osd.5 (osd.5) 447 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: cluster 2024-09-13T22:58:41.624193+0000 osd.5 (osd.5) 448 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: audit 2024-09-13T22:58:41.692367+0000 mon.a (mon.0) 1680 : audit [INF] from='client.25705 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: audit 2024-09-13T22:58:41.706190+0000 mon.c (mon.2) 82 : audit [INF] from='client.? v1:172.21.15.178:0/1874299349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: cluster 2024-09-13T22:58:41.721500+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-09-13T22:58:42.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:42 smithi178 bash[22840]: audit 2024-09-13T22:58:41.725935+0000 mon.a (mon.0) 1682 : audit [INF] from='client.25838 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:58:43.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:43.029 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:43.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:43.160 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:43.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:43.289 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:43.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:43.420 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:43.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:43.548 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:43.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:43.678 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:43.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 37 16 47203 47187 79.6907 0 - 0.0118656 2024-09-13T22:58:43.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:43.808 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:43.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:43.930 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:43.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:44.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:44.048 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:43 smithi192 bash[24608]: cluster 2024-09-13T22:58:42.692499+0000 mon.a (mon.0) 1683 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:58:44.048 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:43 smithi192 bash[24608]: audit 2024-09-13T22:58:42.694189+0000 mon.a (mon.0) 1684 : audit [INF] from='client.25838 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:58:44.048 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:43 smithi192 bash[24608]: cluster 2024-09-13T22:58:42.698241+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-09-13T22:58:44.048 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:43 smithi192 bash[24608]: cluster 2024-09-13T22:58:43.703216+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-09-13T22:58:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:44.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:44.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[29889]: cluster 2024-09-13T22:58:42.692499+0000 mon.a (mon.0) 1683 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:58:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[29889]: audit 2024-09-13T22:58:42.694189+0000 mon.a (mon.0) 1684 : audit [INF] from='client.25838 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:58:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[29889]: cluster 2024-09-13T22:58:42.698241+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-09-13T22:58:44.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[29889]: cluster 2024-09-13T22:58:43.703216+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-09-13T22:58:44.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[22840]: cluster 2024-09-13T22:58:42.692499+0000 mon.a (mon.0) 1683 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:58:44.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[22840]: audit 2024-09-13T22:58:42.694189+0000 mon.a (mon.0) 1684 : audit [INF] from='client.25838 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T22:58:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[22840]: cluster 2024-09-13T22:58:42.698241+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-09-13T22:58:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:43 smithi178 bash[22840]: cluster 2024-09-13T22:58:43.703216+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-09-13T22:58:44.213 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:44.319 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:44.338 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:44.440 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:44.468 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:44.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:44.617 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:44.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 38 16 47655 47639 78.3371 14.125 0.0205408 0.0127579 2024-09-13T22:58:44.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:44.748 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:44.853 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:44.882 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:44.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:44.987 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:45.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:45.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:45.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:44 smithi192 bash[24608]: cluster 2024-09-13T22:58:42.925288+0000 mgr.y (mgr.14613) 782 : cluster [DBG] pgmap v927: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 681 B/s rd, 43 MiB/s wr, 688 op/s 2024-09-13T22:58:45.152 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:45.185 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:44 smithi178 bash[29889]: cluster 2024-09-13T22:58:42.925288+0000 mgr.y (mgr.14613) 782 : cluster [DBG] pgmap v927: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 681 B/s rd, 43 MiB/s wr, 688 op/s 2024-09-13T22:58:45.186 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:44 smithi178 bash[22840]: cluster 2024-09-13T22:58:42.925288+0000 mgr.y (mgr.14613) 782 : cluster [DBG] pgmap v927: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 681 B/s rd, 43 MiB/s wr, 688 op/s 2024-09-13T22:58:45.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:45.287 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:45.391 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:45.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:45.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:45.556 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:45.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:45.695 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:45.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 39 16 48549 48533 77.7611 55.875 0.00592555 0.0128529 2024-09-13T22:58:45.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:45.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:45 smithi178 bash[29889]: cluster 2024-09-13T22:58:44.813900+0000 mon.a (mon.0) 1687 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-09-13T22:58:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:45 smithi178 bash[29889]: cluster 2024-09-13T22:58:44.925746+0000 mgr.y (mgr.14613) 783 : cluster [DBG] pgmap v930: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 remapped+peering, 1 activating+undersized+remapped, 3 active+undersized+remapped, 146 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 588 op/s; 17726/141846 objects degraded (12.497%); 2927/141846 objects misplaced (2.064%); 48 KiB/s, 22 objects/s recovering 2024-09-13T22:58:45.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:45 smithi178 bash[22840]: cluster 2024-09-13T22:58:44.813900+0000 mon.a (mon.0) 1687 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-09-13T22:58:45.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:45 smithi178 bash[22840]: cluster 2024-09-13T22:58:44.925746+0000 mgr.y (mgr.14613) 783 : cluster [DBG] pgmap v930: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 remapped+peering, 1 activating+undersized+remapped, 3 active+undersized+remapped, 146 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 588 op/s; 17726/141846 objects degraded (12.497%); 2927/141846 objects misplaced (2.064%); 48 KiB/s, 22 objects/s recovering 2024-09-13T22:58:45.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:45.960 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:45.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:46.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:46.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:58:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:58:46.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:45 smithi192 bash[24608]: cluster 2024-09-13T22:58:44.813900+0000 mon.a (mon.0) 1687 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-09-13T22:58:46.073 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:45 smithi192 bash[24608]: cluster 2024-09-13T22:58:44.925746+0000 mgr.y (mgr.14613) 783 : cluster [DBG] pgmap v930: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 remapped+peering, 1 activating+undersized+remapped, 3 active+undersized+remapped, 146 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 588 op/s; 17726/141846 objects degraded (12.497%); 2927/141846 objects misplaced (2.064%); 48 KiB/s, 22 objects/s recovering 2024-09-13T22:58:46.091 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:46.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:46.229 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:46.271 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T22:58:46.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:46.356 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:46.412 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-13T22:58:46.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:46.533 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:46.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:46.672 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:46.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:58:46.719162+0000 min lat: 0.00117945 max lat: 2.36455 avg lat: 0.0129477 2024-09-13T22:58:46.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:58:46.718 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 40 16 49431 49415 77.1951 55.125 0.00630212 0.0129477 2024-09-13T22:58:46.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:46.803 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:46.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:46 smithi192 bash[24608]: cluster 2024-09-13T22:58:45.805672+0000 mon.a (mon.0) 1688 : cluster [WRN] Health check failed: Degraded data redundancy: 17726/141846 objects degraded (12.497%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:46 smithi192 bash[24608]: cluster 2024-09-13T22:58:45.818280+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-09-13T22:58:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:46 smithi192 bash[24608]: audit 2024-09-13T22:58:45.901905+0000 mgr.y (mgr.14613) 784 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:46.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:46.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:46.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:47.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:47.074 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:47.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:46 smithi178 bash[22840]: cluster 2024-09-13T22:58:45.805672+0000 mon.a (mon.0) 1688 : cluster [WRN] Health check failed: Degraded data redundancy: 17726/141846 objects degraded (12.497%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:46 smithi178 bash[22840]: cluster 2024-09-13T22:58:45.818280+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-09-13T22:58:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:46 smithi178 bash[22840]: audit 2024-09-13T22:58:45.901905+0000 mgr.y (mgr.14613) 784 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:46 smithi178 bash[29889]: cluster 2024-09-13T22:58:45.805672+0000 mon.a (mon.0) 1688 : cluster [WRN] Health check failed: Degraded data redundancy: 17726/141846 objects degraded (12.497%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:46 smithi178 bash[29889]: cluster 2024-09-13T22:58:45.818280+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-09-13T22:58:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:46 smithi178 bash[29889]: audit 2024-09-13T22:58:45.901905+0000 mgr.y (mgr.14613) 784 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:47.208 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:47.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:47.336 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:47.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:47.467 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:47.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:47.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:47.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:47.717 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 41 16 50363 50347 76.7329 58.25 0.0138246 0.0130267 2024-09-13T22:58:47.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:47.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:47.872 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:47.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:47.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:48.001 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:48.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:48.114 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:47 smithi192 bash[24608]: cluster 2024-09-13T22:58:46.926243+0000 mgr.y (mgr.14613) 785 : cluster [DBG] pgmap v932: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 remapped+peering, 1 activating+undersized+remapped, 3 active+undersized+remapped, 145 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 581 op/s; 17726/142992 objects degraded (12.396%); 5737/142992 objects misplaced (4.012%); 1.4 MiB/s, 43 objects/s recovering 2024-09-13T22:58:48.127 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:48.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:47 smithi178 bash[22840]: cluster 2024-09-13T22:58:46.926243+0000 mgr.y (mgr.14613) 785 : cluster [DBG] pgmap v932: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 remapped+peering, 1 activating+undersized+remapped, 3 active+undersized+remapped, 145 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 581 op/s; 17726/142992 objects degraded (12.396%); 5737/142992 objects misplaced (4.012%); 1.4 MiB/s, 43 objects/s recovering 2024-09-13T22:58:48.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:47 smithi178 bash[29889]: cluster 2024-09-13T22:58:46.926243+0000 mgr.y (mgr.14613) 785 : cluster [DBG] pgmap v932: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 remapped+peering, 1 activating+undersized+remapped, 3 active+undersized+remapped, 145 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 581 op/s; 17726/142992 objects degraded (12.396%); 5737/142992 objects misplaced (4.012%); 1.4 MiB/s, 43 objects/s recovering 2024-09-13T22:58:48.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:48.257 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:48.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:58:48.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.395 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:48.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:48.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T22:58:48.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:48.564 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:48.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:48.711 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:48.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 42 15 51231 51216 76.1963 54.3125 0.0178079 0.0131171 2024-09-13T22:58:48.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:48.846 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:48.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:48.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:48.997 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:49.103 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:49.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:49.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:49.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:49.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:49.391 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:49 smithi178 bash[22840]: cluster 2024-09-13T22:58:48.138111+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-09-13T22:58:49.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:49 smithi178 bash[29889]: cluster 2024-09-13T22:58:48.138111+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-09-13T22:58:49.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:49 smithi192 bash[24608]: cluster 2024-09-13T22:58:48.138111+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-09-13T22:58:49.412 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:49.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:49.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:49.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:49.674 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:49.719 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 43 16 52167 52151 75.7832 58.4375 0.0069357 0.0131891 2024-09-13T22:58:49.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:49.817 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:49.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:49.968 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:49.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:50.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:50.082 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:58:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:58:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:58:50.100 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:50.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:50.237 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:50.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:50.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:50.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:50.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:50.644 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:50 smithi178 bash[29889]: cluster 2024-09-13T22:58:48.927061+0000 mgr.y (mgr.14613) 786 : cluster [DBG] pgmap v934: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+undersized+remapped, 147 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.9 KiB/s rd, 61 MiB/s wr, 970 op/s; 14757/150819 objects degraded (9.785%); 8471/150819 objects misplaced (5.617%); 3.9 MiB/s, 79 objects/s recovering 2024-09-13T22:58:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:50 smithi178 bash[29889]: audit 2024-09-13T22:58:49.430674+0000 mon.a (mon.0) 1691 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:50 smithi178 bash[29889]: audit 2024-09-13T22:58:49.432115+0000 mon.b (mon.1) 273 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:50.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:50 smithi178 bash[22840]: cluster 2024-09-13T22:58:48.927061+0000 mgr.y (mgr.14613) 786 : cluster [DBG] pgmap v934: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+undersized+remapped, 147 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.9 KiB/s rd, 61 MiB/s wr, 970 op/s; 14757/150819 objects degraded (9.785%); 8471/150819 objects misplaced (5.617%); 3.9 MiB/s, 79 objects/s recovering 2024-09-13T22:58:50.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:50 smithi178 bash[22840]: audit 2024-09-13T22:58:49.430674+0000 mon.a (mon.0) 1691 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:50.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:50 smithi178 bash[22840]: audit 2024-09-13T22:58:49.432115+0000 mon.b (mon.1) 273 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:50.719 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 44 16 53084 53068 75.3634 57.3125 0.0139451 0.0132602 2024-09-13T22:58:50.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:50.777 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:50.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:50.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:50 smithi192 bash[24608]: cluster 2024-09-13T22:58:48.927061+0000 mgr.y (mgr.14613) 786 : cluster [DBG] pgmap v934: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+undersized+remapped, 147 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.9 KiB/s rd, 61 MiB/s wr, 970 op/s; 14757/150819 objects degraded (9.785%); 8471/150819 objects misplaced (5.617%); 3.9 MiB/s, 79 objects/s recovering 2024-09-13T22:58:50.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:50 smithi192 bash[24608]: audit 2024-09-13T22:58:49.430674+0000 mon.a (mon.0) 1691 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:58:50.891 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:50 smithi192 bash[24608]: audit 2024-09-13T22:58:49.432115+0000 mon.b (mon.1) 273 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:58:50.908 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:50.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:51.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:51.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:51.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.182 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:51.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:51.195 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:51.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:51.309 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:51.412 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:58:51.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T22:58:51.413 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_4 2024-09-13T22:58:51.413 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_4 by 10 min 8 2024-09-13T22:58:51.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T22:58:51.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:51.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:51.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:51.553 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:51 smithi192 bash[24608]: cluster 2024-09-13T22:58:51.430190+0000 mon.a (mon.0) 1692 : cluster [WRN] Health check update: Degraded data redundancy: 14757/154059 objects degraded (9.579%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:51.568 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:51.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:51 smithi178 bash[29889]: cluster 2024-09-13T22:58:51.430190+0000 mon.a (mon.0) 1692 : cluster [WRN] Health check update: Degraded data redundancy: 14757/154059 objects degraded (9.579%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:51 smithi178 bash[22840]: cluster 2024-09-13T22:58:51.430190+0000 mon.a (mon.0) 1692 : cluster [WRN] Health check update: Degraded data redundancy: 14757/154059 objects degraded (9.579%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:51.693 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:51.719 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 45 16 53904 53888 74.8274 51.25 0.0141244 0.0133546 2024-09-13T22:58:51.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:51.842 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:51.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:51.976 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:51.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:52.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:52.106 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:52.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:52.236 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:52.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:52.372 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:52.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:52.509 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:52.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.644 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:52.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:52 smithi192 bash[24608]: cluster 2024-09-13T22:58:50.927794+0000 mgr.y (mgr.14613) 787 : cluster [DBG] pgmap v935: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 836 B/s rd, 42 MiB/s wr, 665 op/s; 14757/154059 objects degraded (9.579%); 8114/154059 objects misplaced (5.267%); 6.9 MiB/s, 111 objects/s recovering 2024-09-13T22:58:52.719 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 46 16 54814 54798 74.437 56.875 0.0130682 0.0134267 2024-09-13T22:58:52.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:52.756 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:52 smithi178 bash[29889]: cluster 2024-09-13T22:58:50.927794+0000 mgr.y (mgr.14613) 787 : cluster [DBG] pgmap v935: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 836 B/s rd, 42 MiB/s wr, 665 op/s; 14757/154059 objects degraded (9.579%); 8114/154059 objects misplaced (5.267%); 6.9 MiB/s, 111 objects/s recovering 2024-09-13T22:58:52.758 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:52 smithi178 bash[22840]: cluster 2024-09-13T22:58:50.927794+0000 mgr.y (mgr.14613) 787 : cluster [DBG] pgmap v935: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 836 B/s rd, 42 MiB/s wr, 665 op/s; 14757/154059 objects degraded (9.579%); 8114/154059 objects misplaced (5.267%); 6.9 MiB/s, 111 objects/s recovering 2024-09-13T22:58:52.774 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:52.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:52.900 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:52.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:53.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:53.028 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:53.137 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:53.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:53.231 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:53.232 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:53.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:53.293 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:53.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:53.417 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:53.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:53.553 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:53.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:53.680 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:53.719 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 47 16 55626 55610 73.933 50.75 0.011547 0.0135188 2024-09-13T22:58:53.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.823 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:53.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:53.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:53.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:53.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:53.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:54.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:54.072 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:54.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:54.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:54.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:54.337 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:54.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:54.473 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:54.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:54.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:54 smithi192 bash[24608]: cluster 2024-09-13T22:58:52.928411+0000 mgr.y (mgr.14613) 788 : cluster [DBG] pgmap v936: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 36 MiB/s wr, 581 op/s; 14757/155784 objects degraded (9.473%); 7901/155784 objects misplaced (5.072%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T22:58:54.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:54.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:54.720 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 48 16 56420 56404 73.4266 49.625 0.020444 0.0136135 2024-09-13T22:58:54.728 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:54 smithi178 bash[29889]: cluster 2024-09-13T22:58:52.928411+0000 mgr.y (mgr.14613) 788 : cluster [DBG] pgmap v936: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 36 MiB/s wr, 581 op/s; 14757/155784 objects degraded (9.473%); 7901/155784 objects misplaced (5.072%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T22:58:54.729 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:54 smithi178 bash[22840]: cluster 2024-09-13T22:58:52.928411+0000 mgr.y (mgr.14613) 788 : cluster [DBG] pgmap v936: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 36 MiB/s wr, 581 op/s; 14757/155784 objects degraded (9.473%); 7901/155784 objects misplaced (5.072%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T22:58:54.752 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:54.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:54.879 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:54.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:54.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:55.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:55.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:55.149 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:55.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:55.265 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:58:55.278 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:55.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:55.407 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:55.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:55.528 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:55.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:55.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:55.720 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 49 15 57110 57095 72.8095 43.1875 0.268863 0.0137128 2024-09-13T22:58:55.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:55.782 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:55.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:55.922 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:55.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:56.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:56.069 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:56.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:56.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:58:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:58:56.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:58:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:58:56.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:56.327 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:56.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:56.463 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:56.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:56.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:56.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:56 smithi178 bash[29889]: cluster 2024-09-13T22:58:54.931170+0000 mgr.y (mgr.14613) 789 : cluster [DBG] pgmap v937: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 933 op/s; 14757/167355 objects degraded (8.818%); 7284/167355 objects misplaced (4.352%); 10 MiB/s, 165 objects/s recovering 2024-09-13T22:58:56.603 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:56 smithi178 bash[22840]: cluster 2024-09-13T22:58:54.931170+0000 mgr.y (mgr.14613) 789 : cluster [DBG] pgmap v937: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 933 op/s; 14757/167355 objects degraded (8.818%); 7284/167355 objects misplaced (4.352%); 10 MiB/s, 165 objects/s recovering 2024-09-13T22:58:56.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:56.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 50 16 57851 57835 72.2771 46.25 0.0299213 0.0138284 2024-09-13T22:58:56.722 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:56.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:56.861 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:56.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:56 smithi192 bash[24608]: cluster 2024-09-13T22:58:54.931170+0000 mgr.y (mgr.14613) 789 : cluster [DBG] pgmap v937: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 933 op/s; 14757/167355 objects degraded (8.818%); 7284/167355 objects misplaced (4.352%); 10 MiB/s, 165 objects/s recovering 2024-09-13T22:58:56.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:56.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:56.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:57.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:57.120 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:57.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:57.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:57.243 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:57.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:57.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:57.480 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:57.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:57.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:57.601 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:57.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:57 smithi178 bash[29889]: audit 2024-09-13T22:58:55.910209+0000 mgr.y (mgr.14613) 790 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:57 smithi178 bash[29889]: cluster 2024-09-13T22:58:56.870535+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:57 smithi178 bash[29889]: cluster 2024-09-13T22:58:56.870586+0000 osd.7 (osd.7) 417 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:57 smithi178 bash[22840]: audit 2024-09-13T22:58:55.910209+0000 mgr.y (mgr.14613) 790 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:57 smithi178 bash[22840]: cluster 2024-09-13T22:58:56.870535+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:57 smithi178 bash[22840]: cluster 2024-09-13T22:58:56.870586+0000 osd.7 (osd.7) 417 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:57.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:57.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 51 16 58677 58661 71.872 51.625 0.0133481 0.0139063 2024-09-13T22:58:57.733 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:57.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:57.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:57.865 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:58:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:57.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:58:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:57 smithi192 bash[24608]: audit 2024-09-13T22:58:55.910209+0000 mgr.y (mgr.14613) 790 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:58:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:57 smithi192 bash[24608]: cluster 2024-09-13T22:58:56.870535+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:58:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:57 smithi192 bash[24608]: cluster 2024-09-13T22:58:56.870586+0000 osd.7 (osd.7) 417 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:58:57.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:58.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:58.153 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:58:58.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:58.282 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:58.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:58.411 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:58.514 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:58.555 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:58:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:58:58.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:58.692 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:58:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:58:58.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 52 16 59564 59548 71.5559 55.4375 0.0159268 0.0139688 2024-09-13T22:58:58.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:58.806 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:58 smithi192 bash[24608]: cluster 2024-09-13T22:58:56.931784+0000 mgr.y (mgr.14613) 791 : cluster [DBG] pgmap v938: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 860 op/s; 14757/168801 objects degraded (8.742%); 7063/168801 objects misplaced (4.184%); 10 MiB/s, 164 objects/s recovering 2024-09-13T22:58:58.806 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:58 smithi192 bash[24608]: cluster 2024-09-13T22:58:57.498265+0000 mon.a (mon.0) 1693 : cluster [WRN] Health check update: Degraded data redundancy: 14757/168801 objects degraded (8.742%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:58.806 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:58 smithi192 bash[24608]: cluster 2024-09-13T22:58:57.517250+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-09-13T22:58:58.806 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:58:58 smithi192 bash[24608]: cluster 2024-09-13T22:58:58.142852+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-09-13T22:58:58.829 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:58:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:58.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:58:58.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[22840]: cluster 2024-09-13T22:58:56.931784+0000 mgr.y (mgr.14613) 791 : cluster [DBG] pgmap v938: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 860 op/s; 14757/168801 objects degraded (8.742%); 7063/168801 objects misplaced (4.184%); 10 MiB/s, 164 objects/s recovering 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[22840]: cluster 2024-09-13T22:58:57.498265+0000 mon.a (mon.0) 1693 : cluster [WRN] Health check update: Degraded data redundancy: 14757/168801 objects degraded (8.742%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[22840]: cluster 2024-09-13T22:58:57.517250+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[22840]: cluster 2024-09-13T22:58:58.142852+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[29889]: cluster 2024-09-13T22:58:56.931784+0000 mgr.y (mgr.14613) 791 : cluster [DBG] pgmap v938: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 860 op/s; 14757/168801 objects degraded (8.742%); 7063/168801 objects misplaced (4.184%); 10 MiB/s, 164 objects/s recovering 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[29889]: cluster 2024-09-13T22:58:57.498265+0000 mon.a (mon.0) 1693 : cluster [WRN] Health check update: Degraded data redundancy: 14757/168801 objects degraded (8.742%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[29889]: cluster 2024-09-13T22:58:57.517250+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-09-13T22:58:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:58:58 smithi178 bash[29889]: cluster 2024-09-13T22:58:58.142852+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-09-13T22:58:58.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:59.063 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:58:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:59.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:58:59.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:59.189 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:58:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:59.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:58:59.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:59.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 53 16 60447 60431 71.247 55.1875 0.0200572 0.0140294 2024-09-13T22:58:59.969 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:58:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:58:59.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:00.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:00.084 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:58:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:58:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:59:00.120 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:00.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:00.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:00.249 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:00.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:00.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:00.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:00.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:00.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:00.660 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:00.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:00.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 54 16 61286 61270 70.8987 52.4375 0.0692844 0.0140938 2024-09-13T22:59:00.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:00.776 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:00 smithi178 bash[22840]: cluster 2024-09-13T22:58:58.932469+0000 mgr.y (mgr.14613) 792 : cluster [DBG] pgmap v941: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s; 14757/172437 objects degraded (8.558%); 6869/172437 objects misplaced (3.983%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T22:59:00.776 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:00 smithi178 bash[29889]: cluster 2024-09-13T22:58:58.932469+0000 mgr.y (mgr.14613) 792 : cluster [DBG] pgmap v941: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s; 14757/172437 objects degraded (8.558%); 6869/172437 objects misplaced (3.983%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T22:59:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:00 smithi192 bash[24608]: cluster 2024-09-13T22:58:58.932469+0000 mgr.y (mgr.14613) 792 : cluster [DBG] pgmap v941: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s; 14757/172437 objects degraded (8.558%); 6869/172437 objects misplaced (3.983%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T22:59:01.148 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:01.251 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:01.272 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:01.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:01.400 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:01.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:01.534 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:01.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:01.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:01.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 55 16 62004 61988 70.4255 44.875 0.00574744 0.0141849 2024-09-13T22:59:01.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:01.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:01.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:01.919 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:01.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:02.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:02.057 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:02.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:02.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:02.189 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:02.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:02.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:02.315 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:02.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:02.418 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:02.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:02 smithi178 bash[22840]: cluster 2024-09-13T22:59:00.933174+0000 mgr.y (mgr.14613) 793 : cluster [DBG] pgmap v942: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s; 14757/179211 objects degraded (8.234%); 6391/179211 objects misplaced (3.566%); 12 MiB/s, 188 objects/s recovering 2024-09-13T22:59:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:02 smithi178 bash[29889]: cluster 2024-09-13T22:59:00.933174+0000 mgr.y (mgr.14613) 793 : cluster [DBG] pgmap v942: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s; 14757/179211 objects degraded (8.234%); 6391/179211 objects misplaced (3.566%); 12 MiB/s, 188 objects/s recovering 2024-09-13T22:59:02.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 56 16 62715 62699 69.9614 44.4375 0.0302962 0.0142876 2024-09-13T22:59:02.877 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:02.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:02.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:02 smithi192 bash[24608]: cluster 2024-09-13T22:59:00.933174+0000 mgr.y (mgr.14613) 793 : cluster [DBG] pgmap v942: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s; 14757/179211 objects degraded (8.234%); 6391/179211 objects misplaced (3.566%); 12 MiB/s, 188 objects/s recovering 2024-09-13T22:59:02.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:03.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:03.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:03.104 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:03.367 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:03.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:03.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:03.504 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:03.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:03.624 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:03.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:03.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:03 smithi178 bash[29889]: cluster 2024-09-13T22:59:03.137427+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Degraded data redundancy: 14757/179211 objects degraded (8.234%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:03.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:03 smithi178 bash[22840]: cluster 2024-09-13T22:59:03.137427+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Degraded data redundancy: 14757/179211 objects degraded (8.234%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:03.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 57 16 63479 63463 69.5717 47.75 0.00783558 0.0143666 2024-09-13T22:59:03.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:03 smithi192 bash[24608]: cluster 2024-09-13T22:59:03.137427+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Degraded data redundancy: 14757/179211 objects degraded (8.234%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:03.753 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:03.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:03.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:03.877 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:03.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:03.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:04.005 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:04.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:04.147 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:04.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:04.294 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:04.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:04.425 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:04.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:04.556 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:04.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:04.678 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:04.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 58 16 64169 64153 69.1157 43.125 0.0066001 0.0144311 2024-09-13T22:59:04.736 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:04 smithi178 bash[22840]: cluster 2024-09-13T22:59:02.933826+0000 mgr.y (mgr.14613) 794 : cluster [DBG] pgmap v943: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 35 MiB/s wr, 557 op/s; 14757/180732 objects degraded (8.165%); 6147/180732 objects misplaced (3.401%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T22:59:04.736 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:04 smithi178 bash[22840]: audit 2024-09-13T22:59:04.449553+0000 mon.b (mon.1) 274 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:04.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:04.806 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:04.811 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:04 smithi192 bash[24608]: cluster 2024-09-13T22:59:02.933826+0000 mgr.y (mgr.14613) 794 : cluster [DBG] pgmap v943: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 35 MiB/s wr, 557 op/s; 14757/180732 objects degraded (8.165%); 6147/180732 objects misplaced (3.401%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T22:59:04.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:04 smithi192 bash[24608]: audit 2024-09-13T22:59:04.449553+0000 mon.b (mon.1) 274 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:04.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:04.923 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:04 smithi178 bash[29889]: cluster 2024-09-13T22:59:02.933826+0000 mgr.y (mgr.14613) 794 : cluster [DBG] pgmap v943: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 35 MiB/s wr, 557 op/s; 14757/180732 objects degraded (8.165%); 6147/180732 objects misplaced (3.401%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T22:59:04.923 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:04 smithi178 bash[29889]: audit 2024-09-13T22:59:04.449553+0000 mon.b (mon.1) 274 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:04.945 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:04.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:05.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:05.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:05.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:05.211 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:05.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:05.338 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:05.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:05.462 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:05.565 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:05.586 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:05.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:05.710 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:05.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 59 16 64788 64772 68.5999 38.6875 0.00332226 0.0145323 2024-09-13T22:59:05.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.849 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:05.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:05.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:05.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:59:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:59:05.976 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:05.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:06.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:06.120 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:06.225 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:06.253 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:06.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:06.387 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:06.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:06.513 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:06.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:06.645 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:06 smithi178 bash[29889]: cluster 2024-09-13T22:59:04.934852+0000 mgr.y (mgr.14613) 795 : cluster [DBG] pgmap v944: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 917 op/s; 14757/190797 objects degraded (7.734%); 5324/190797 objects misplaced (2.790%); 14 MiB/s, 217 objects/s recovering 2024-09-13T22:59:06.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:06 smithi178 bash[22840]: cluster 2024-09-13T22:59:04.934852+0000 mgr.y (mgr.14613) 795 : cluster [DBG] pgmap v944: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 917 op/s; 14757/190797 objects degraded (7.734%); 5324/190797 objects misplaced (2.790%); 14 MiB/s, 217 objects/s recovering 2024-09-13T22:59:06.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:59:06.723499+0000 min lat: 0.00117945 max lat: 2.36455 avg lat: 0.0146419 2024-09-13T22:59:06.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:59:06.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 60 16 65553 65537 68.2534 47.8125 0.0112706 0.0146419 2024-09-13T22:59:06.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:06.757 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:06 smithi192 bash[24608]: cluster 2024-09-13T22:59:04.934852+0000 mgr.y (mgr.14613) 795 : cluster [DBG] pgmap v944: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 917 op/s; 14757/190797 objects degraded (7.734%); 5324/190797 objects misplaced (2.790%); 14 MiB/s, 217 objects/s recovering 2024-09-13T22:59:06.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:06.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:06.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:06.905 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:06.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:07.009 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:07.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:07.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:07.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:07.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:07.303 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:07.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:07.432 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:07.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:07.565 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:07.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:07.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:07.721 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 61 16 66232 66216 67.8302 42.4375 0.00551645 0.0147262 2024-09-13T22:59:07.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:07.820 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:07.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:07 smithi192 bash[24608]: audit 2024-09-13T22:59:05.918025+0000 mgr.y (mgr.14613) 796 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:07.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:07.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:07 smithi178 bash[29889]: audit 2024-09-13T22:59:05.918025+0000 mgr.y (mgr.14613) 796 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:07.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:07 smithi178 bash[22840]: audit 2024-09-13T22:59:05.918025+0000 mgr.y (mgr.14613) 796 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:07.959 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:07.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:08.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:08.083 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:08.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:08.215 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:08.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:08.345 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:08.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:08.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:08.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:08.618 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:08.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:08 smithi192 bash[24608]: cluster 2024-09-13T22:59:06.935753+0000 mgr.y (mgr.14613) 797 : cluster [DBG] pgmap v945: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 825 op/s; 14757/192102 objects degraded (7.682%); 5057/192102 objects misplaced (2.632%); 13 MiB/s, 212 objects/s recovering 2024-09-13T22:59:08.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:08 smithi192 bash[24608]: cluster 2024-09-13T22:59:08.140113+0000 mon.a (mon.0) 1697 : cluster [WRN] Health check update: Degraded data redundancy: 14757/192102 objects degraded (7.682%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:08.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 62 16 66768 66752 67.2765 33.5 0.00803739 0.0148565 2024-09-13T22:59:08.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:08.746 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:08.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:08.867 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:08 smithi178 bash[22840]: cluster 2024-09-13T22:59:06.935753+0000 mgr.y (mgr.14613) 797 : cluster [DBG] pgmap v945: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 825 op/s; 14757/192102 objects degraded (7.682%); 5057/192102 objects misplaced (2.632%); 13 MiB/s, 212 objects/s recovering 2024-09-13T22:59:08.867 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:08 smithi178 bash[22840]: cluster 2024-09-13T22:59:08.140113+0000 mon.a (mon.0) 1697 : cluster [WRN] Health check update: Degraded data redundancy: 14757/192102 objects degraded (7.682%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:08.867 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:08 smithi178 bash[29889]: cluster 2024-09-13T22:59:06.935753+0000 mgr.y (mgr.14613) 797 : cluster [DBG] pgmap v945: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 825 op/s; 14757/192102 objects degraded (7.682%); 5057/192102 objects misplaced (2.632%); 13 MiB/s, 212 objects/s recovering 2024-09-13T22:59:08.867 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:08 smithi178 bash[29889]: cluster 2024-09-13T22:59:08.140113+0000 mon.a (mon.0) 1697 : cluster [WRN] Health check update: Degraded data redundancy: 14757/192102 objects degraded (7.682%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:08.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:09.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:09.030 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:09.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.161 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:09.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:09.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:09.288 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:09.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:09.416 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T22:59:09.416 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 20, 2024-09-13T22:59:09.416 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 600, 2024-09-13T22:59:09.416 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:45708 10.e e332/322 10:7b58b151:::benchmark_data_smithi178_111521_object45707:head v 332'2786, mlcod=331'2784)", 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:53:57.364657+0000", 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 312.04728154999998, 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.62913611300000005, 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.417 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364657+0000", 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364657+0000", 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364657+0000", 2024-09-13T22:59:09.418 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2399999999999999e-07 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364690+0000", 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.3556999999999997e-05 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364700+0000", 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6989999999999998e-06 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.419 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364707+0000", 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6220000000000003e-06 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364757+0000", 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.9854000000000001e-05 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.420 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.364770+0000", 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2942000000000001e-05 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.993678+0000", 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.62890795399999999 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.421 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.993793+0000", 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011526100000000001 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:45714 10.e e332/322 10:7280052f:::benchmark_data_smithi178_111521_object45713:head v 332'2787, mlcod=331'2784)", 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:53:57.369190+0000", 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 312.04274835299998, 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.63241816200000001, 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.422 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369190+0000", 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369190+0000", 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.423 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369192+0000", 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6920000000000001e-06 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369233+0000", 2024-09-13T22:59:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.1182e-05 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.991051+0000", 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.621817752 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.991062+0000", 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1395e-05 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.425 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.994002+0000", 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002940561 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.994054+0000", 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1946e-05 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:58.001516+0000", 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0074615929999999999 2024-09-13T22:59:09.426 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:58.001608+0000", 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.2040999999999995e-05 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:45713 10.8 e332/322 10:116a53b7:::benchmark_data_smithi178_111521_object45712:head v 332'2825, mlcod=331'2823)", 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:53:57.369768+0000", 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 312.04217073799998, 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.63461843500000004, 2024-09-13T22:59:09.427 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369768+0000", 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.428 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369768+0000", 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369768+0000", 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.44e-07 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.369795+0000", 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6424999999999999e-05 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.429 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.989374+0000", 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.61957933200000004 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.989393+0000", 2024-09-13T22:59:09.430 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8617000000000001e-05 2024-09-13T22:59:09.431 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.431 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.989537+0000", 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014482000000000001 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:57.989601+0000", 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3379000000000004e-05 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:58.004305+0000", 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014704637 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.433 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:53:58.004386+0000", 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0581000000000007e-05 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.434 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:102354 10.e e344/322 10:7d098ade:::benchmark_data_smithi178_111521_object102353:head v 344'6350, mlcod=344'6348)", 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:54:53.873982+0000", 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 255.53795638700001, 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.320772112, 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.873982+0000", 2024-09-13T22:59:09.435 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.873982+0000", 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.873983+0000", 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8899999999999998e-07 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.436 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.874025+0000", 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2354000000000001e-05 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.874031+0000", 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.648e-06 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.874036+0000", 2024-09-13T22:59:09.437 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3569999999999997e-06 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.874097+0000", 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1468000000000002e-05 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.874118+0000", 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0931999999999999e-05 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.438 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.878980+0000", 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0048611690000000003 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194754+0000", 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.31577439499999999 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.439 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:102360 10.9 e344/322 10:99ec171f:::benchmark_data_smithi178_111521_object102359:head v 344'6433, mlcod=344'6431)", 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:54:53.875012+0000", 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 255.53692597700001, 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.32034292199999997, 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875012+0000", 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.440 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875012+0000", 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875014+0000", 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5149999999999999e-06 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875090+0000", 2024-09-13T22:59:09.441 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6438000000000002e-05 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875100+0000", 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6749999999999997e-06 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875106+0000", 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4400000000000002e-06 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.442 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875162+0000", 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.5201000000000002e-05 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875179+0000", 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7774e-05 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.195242+0000", 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.32006277399999999 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.443 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.195355+0000", 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000113105 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:102369 10.8 e344/322 10:132a24f4:::benchmark_data_smithi178_111521_object102368:head v 344'6344, mlcod=344'6342)", 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:54:53.875051+0000", 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 255.536887713, 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.31976212300000001, 2024-09-13T22:59:09.444 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875051+0000", 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875051+0000", 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.445 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875052+0000", 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.486e-06 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875074+0000", 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2254000000000001e-05 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875083+0000", 2024-09-13T22:59:09.446 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3089999999999996e-06 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875088+0000", 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.339e-06 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875141+0000", 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.2757999999999999e-05 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.875154+0000", 2024-09-13T22:59:09.447 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3474e-05 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194719+0000", 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.31956499199999999 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194813+0000", 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3511000000000006e-05 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.448 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:102361 10.3 e344/322 10:c3588c03:::benchmark_data_smithi178_111521_object102360:head v 344'6418, mlcod=344'6416)", 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:54:53.876294+0000", 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 255.53564392499999, 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.32553449099999998, 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876294+0000", 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.449 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876294+0000", 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876296+0000", 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3990000000000001e-06 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876425+0000", 2024-09-13T22:59:09.450 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012920999999999999 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876435+0000", 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7880000000000001e-06 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876441+0000", 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.55e-06 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.451 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876498+0000", 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6895999999999999e-05 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.876515+0000", 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6850999999999998e-05 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.201761+0000", 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.32524642999999998 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.452 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.201829+0000", 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7366999999999994e-05 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:102371 10.e e344/322 10:78abb7b0:::benchmark_data_smithi178_111521_object102370:head v 344'6351, mlcod=344'6348)", 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:54:53.879425+0000", 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 255.532513497, 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.32485833800000002, 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.453 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.879425+0000", 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.879425+0000", 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.454 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.879426+0000", 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.372e-06 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.879489+0000", 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2848000000000005e-05 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194669+0000", 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.315179653 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.455 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194696+0000", 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.7056e-05 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194824+0000", 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000128248 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194880+0000", 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6563e-05 2024-09-13T22:59:09.456 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.204116+0000", 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0092356279999999992 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.204283+0000", 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016697000000000001 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.457 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:102372 10.8 e344/322 10:1e292453:::benchmark_data_smithi178_111521_object102371:head v 344'6345, mlcod=344'6342)", 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:54:53.880591+0000", 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 255.53134689000001, 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.32033763700000001, 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.458 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.880591+0000", 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.880591+0000", 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.880593+0000", 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6810000000000001e-06 2024-09-13T22:59:09.459 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:53.880617+0000", 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4372e-05 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194683+0000", 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.31406519900000002 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194709+0000", 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6270000000000001e-05 2024-09-13T22:59:09.460 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.194914+0000", 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00020526999999999999 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.195184+0000", 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00026957199999999999 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.200862+0000", 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0056783939999999998 2024-09-13T22:59:09.461 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:54:54.200929+0000", 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6878999999999994e-05 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "MOSDPGPush(10.a 365/358 [PushOp(10:5ece7ea1:::benchmark_data_smithi178_111521_object96748:head, version: 342'6213, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:5ece7ea1:::benchmark_data_smithi178_111521_object96748:head@342'6213, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:22.607032+0000", 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 166.80490662700001, 2024-09-13T22:59:09.462 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.57274647999999995, 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "started", 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.607032+0000", 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.607032+0000", 2024-09-13T22:59:09.463 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.607032+0000", 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.7400000000000002e-07 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.607147+0000", 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011432799999999999 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.464 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.607160+0000", 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3057000000000001e-05 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.607170+0000", 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7399999999999999e-06 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179558+0000", 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.57238812400000005 2024-09-13T22:59:09.465 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179573+0000", 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5404e-05 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179778+0000", 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00020495299999999999 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.466 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:177550 10.3 e365/322 10:c7459afa:::benchmark_data_smithi178_111521_object66814:head v 365'11170, mlcod=365'11168)", 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:22.608298+0000", 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 166.80364002100001, 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.57543654499999997, 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.467 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.608298+0000", 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.608298+0000", 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.608300+0000", 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.331e-06 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.468 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.608307+0000", 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5209999999999997e-06 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179807+0000", 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.571500173 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179817+0000", 2024-09-13T22:59:09.469 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.8300000000000008e-06 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179879+0000", 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1529e-05 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179901+0000", 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2597999999999999e-05 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.183695+0000", 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0037940869999999998 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.183735+0000", 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9475999999999999e-05 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:177552 10.5 e365/322 10:a8b60773:::benchmark_data_smithi178_111521_object66816:head v 365'11000, mlcod=365'10998)", 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:56:22.611294+0000", 2024-09-13T22:59:09.471 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 166.80064379300001, 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.57035836900000003, 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.611294+0000", 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.611294+0000", 2024-09-13T22:59:09.472 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.611296+0000", 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5179999999999999e-06 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:22.611302+0000", 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.387e-06 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.473 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179787+0000", 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.56848499299999999 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179794+0000", 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1709999999999999e-06 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179859+0000", 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.5130999999999997e-05 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.474 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.179882+0000", 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3034e-05 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.181610+0000", 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00172879 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:23.181653+0000", 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2345000000000002e-05 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.475 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:209554 10.f e382/358 10:f6a63969:::benchmark_data_smithi178_111521_object98818:head v 382'12848, mlcod=382'12845)", 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:57:04.235390+0000", 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 125.17654784200001, 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.46438689399999999, 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235390+0000", 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235390+0000", 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.477 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235391+0000", 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.3600000000000003e-07 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235397+0000", 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6450000000000002e-06 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.478 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235403+0000", 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.46e-06 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235408+0000", 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0000000000000004e-06 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.238292+0000", 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002883287 2024-09-13T22:59:09.479 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.238315+0000", 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3416000000000001e-05 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699737+0000", 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.461422049 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699777+0000", 2024-09-13T22:59:09.480 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0300999999999999e-05 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:209557 10.7 e382/376 10:e7cab47a:::benchmark_data_smithi178_111521_object98821:head v 382'13361, mlcod=382'13358)", 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:57:04.235422+0000", 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 125.176516353, 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.46438148600000001, 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.481 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235422+0000", 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235422+0000", 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235422+0000", 2024-09-13T22:59:09.482 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.1300000000000001e-07 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235426+0000", 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9140000000000001e-06 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235431+0000", 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6909999999999996e-06 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.483 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.235435+0000", 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0980000000000004e-06 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.238951+0000", 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0035159409999999999 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.238970+0000", 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.9021e-05 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.484 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699781+0000", 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.460810791 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699803+0000", 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2617e-05 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.485 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:209560 10.f e382/358 10:f68fe75b:::benchmark_data_smithi178_111521_object98824:head v 382'12849, mlcod=382'12845)", 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:57:04.237347+0000", 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 125.17459111399999, 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.46261695600000002, 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.237347+0000", 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.486 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.237347+0000", 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.237352+0000", 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.3930000000000001e-06 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.237358+0000", 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9410000000000001e-06 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.487 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.237368+0000", 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.9399999999999997e-06 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.237376+0000", 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.4590000000000001e-06 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.240296+0000", 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002920565 2024-09-13T22:59:09.488 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.240317+0000", 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.0698000000000001e-05 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699807+0000", 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.45948972700000001 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.489 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699964+0000", 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000157233 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_repop(client.25538.0:209565 10.a e382/367 10:55d1fa1c:::benchmark_data_smithi178_111521_object98829:head v 382'13394, mlcod=382'13392)", 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:57:04.242682+0000", 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 125.169256663, 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.457275759, 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.490 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.242682+0000", 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.242682+0000", 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.491 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.242682+0000", 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.2300000000000001e-07 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.242688+0000", 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3589999999999996e-06 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.242695+0000", 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6590000000000001e-06 2024-09-13T22:59:09.492 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.242702+0000", 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.0140000000000001e-06 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.698026+0000", 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.45532372700000001 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.698045+0000", 2024-09-13T22:59:09.493 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8811e-05 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699928+0000", 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0018834189999999999 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:57:04.699957+0000", 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.9247e-05 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.494 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47191 11.d 11:bafe891b:::benchmark_data_smithi178_128208_object47190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.700085+0000", 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 27.711853291000001, 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2997564449999999, 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47191 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.495 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700085+0000", 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700085+0000", 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700086+0000", 2024-09-13T22:59:09.496 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.203e-06 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700250+0000", 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016370999999999999 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700261+0000", 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0934000000000001e-05 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.497 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700268+0000", 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1720000000000001e-06 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700435+0000", 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016700800000000001 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700641+0000", 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00020648899999999999 2024-09-13T22:59:09.498 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,5", 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700771+0000", 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012961400000000001 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.703154+0000", 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0023831899999999999 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.706217+0000", 2024-09-13T22:59:09.499 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0030629899999999998 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.724880+0000", 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.018662959 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.724893+0000", 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2578000000000001e-05 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.500 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.727348+0000", 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0024551779999999 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.727370+0000", 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1855e-05 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:43.999744+0000", 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2723744340000001 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.501 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:43.999841+0000", 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.7131000000000002e-05 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47201 11.d 11:b8e528a3:::benchmark_data_smithi178_128208_object47200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.743434+0000", 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 27.668503912999999, 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2739111090000002, 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.502 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47201 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743434+0000", 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.503 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743434+0000", 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743437+0000", 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.5500000000000001e-06 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743462+0000", 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.4941999999999999e-05 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.504 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743479+0000", 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7448999999999998e-05 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743487+0000", 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0129999999999993e-06 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743568+0000", 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0869000000000001e-05 2024-09-13T22:59:09.505 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743576+0000", 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0450000000000001e-06 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.727641+0000", 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.98406438299999999 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:59:09.506 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.727656+0000", 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5347999999999999e-05 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.003736+0000", 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.276079846 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.003989+0000", 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00025359500000000001 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.507 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,6", 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.004272+0000", 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00028225800000000002 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.009597+0000", 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0053255560000000004 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.009642+0000", 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.5138e-05 2024-09-13T22:59:09.508 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.014540+0000", 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0048975659999999999 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.014637+0000", 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6699000000000004e-05 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017345+0000", 2024-09-13T22:59:09.509 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0027088519999999999 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47203 11.d 11:be04a449:::benchmark_data_smithi178_128208_object47202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.743503+0000", 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 27.668435553999998, 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2883100700000001, 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:59:09.510 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47203 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743503+0000", 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743503+0000", 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.511 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743503+0000", 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3e-07 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743521+0000", 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8074e-05 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743526+0000", 2024-09-13T22:59:09.512 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8250000000000004e-06 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743530+0000", 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.5499999999999999e-06 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743591+0000", 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1253999999999993e-05 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:59:09.513 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743595+0000", 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.4009999999999999e-06 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.727745+0000", 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.98415002500000004 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.727758+0000", 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2466000000000001e-05 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.514 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.006658+0000", 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2788997069999999 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.006818+0000", 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016008600000000001 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,6", 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.007002+0000", 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00018433699999999999 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.515 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017444+0000", 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010442237 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017502+0000", 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.7905999999999998e-05 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.516 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.029245+0000", 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.011743188999999999 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.029282+0000", 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6958000000000001e-05 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.031813+0000", 2024-09-13T22:59:09.517 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0025304250000000002 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:66197 11.d 11:b318e491:::benchmark_data_smithi178_128208_object66196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:59:07.644390+0000", 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 1.7675481669999999, 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.27142877100000001, 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T22:59:09.518 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 66197 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.644390+0000", 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.644390+0000", 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.519 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.644393+0000", 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.033e-06 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.644422+0000", 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.8819999999999999e-05 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.644438+0000", 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5854000000000001e-05 2024-09-13T22:59:09.520 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.644447+0000", 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.0410000000000003e-06 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.645148+0000", 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00070104599999999998 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.645313+0000", 2024-09-13T22:59:09.521 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000165413 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,6", 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.645477+0000", 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016407699999999999 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.648232+0000", 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002754187 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.522 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.652588+0000", 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0043567459999999999 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.653666+0000", 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001077227 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.653723+0000", 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.7917000000000001e-05 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T22:59:09.523 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:59:07.915819+0000", 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.26209540999999997 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T22:59:09.524 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T22:59:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:09.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:09.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:09.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:09.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 63 16 67539 67523 66.9734 48.1875 0.0268038 0.0149245 2024-09-13T22:59:09.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:09.780 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:09.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:09.913 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:09.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:10.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:10.042 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:10.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:10.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:10.214 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:59:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:59:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:59:10.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:10.301 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:10.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:10.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:10.496 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T22:59:10.499 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T22:59:10.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:10.553 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:10.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:10.678 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T22:59:10.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:10.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 64 16 68126 68110 66.5002 36.6875 0.0496282 0.0150281 2024-09-13T22:59:10.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:10.826 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:10.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:10 smithi192 bash[24608]: cluster 2024-09-13T22:59:08.936440+0000 mgr.y (mgr.14613) 798 : cluster [DBG] pgmap v946: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 948 B/s rd, 46 MiB/s wr, 739 op/s; 14757/192729 objects degraded (7.657%); 5057/192729 objects misplaced (2.624%); 12 MiB/s, 185 objects/s recovering 2024-09-13T22:59:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:10 smithi192 bash[24608]: audit 2024-09-13T22:59:09.761281+0000 mon.b (mon.1) 275 : audit [INF] from='client.? v1:172.21.15.178:0/1165388010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:59:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:10 smithi192 bash[24608]: audit 2024-09-13T22:59:09.761540+0000 mon.a (mon.0) 1698 : audit [INF] from='client.25720 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:59:10.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:10 smithi178 bash[29889]: cluster 2024-09-13T22:59:08.936440+0000 mgr.y (mgr.14613) 798 : cluster [DBG] pgmap v946: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 948 B/s rd, 46 MiB/s wr, 739 op/s; 14757/192729 objects degraded (7.657%); 5057/192729 objects misplaced (2.624%); 12 MiB/s, 185 objects/s recovering 2024-09-13T22:59:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:10 smithi178 bash[29889]: audit 2024-09-13T22:59:09.761281+0000 mon.b (mon.1) 275 : audit [INF] from='client.? v1:172.21.15.178:0/1165388010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:59:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:10 smithi178 bash[29889]: audit 2024-09-13T22:59:09.761540+0000 mon.a (mon.0) 1698 : audit [INF] from='client.25720 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:59:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:10 smithi178 bash[22840]: cluster 2024-09-13T22:59:08.936440+0000 mgr.y (mgr.14613) 798 : cluster [DBG] pgmap v946: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 948 B/s rd, 46 MiB/s wr, 739 op/s; 14757/192729 objects degraded (7.657%); 5057/192729 objects misplaced (2.624%); 12 MiB/s, 185 objects/s recovering 2024-09-13T22:59:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:10 smithi178 bash[22840]: audit 2024-09-13T22:59:09.761281+0000 mon.b (mon.1) 275 : audit [INF] from='client.? v1:172.21.15.178:0/1165388010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:59:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:10 smithi178 bash[22840]: audit 2024-09-13T22:59:09.761540+0000 mon.a (mon.0) 1698 : audit [INF] from='client.25720 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T22:59:10.957 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:10.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:11.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:11.077 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:11.169 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:59:11.170 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:59:11.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:11.195 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:59:11.195 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:59:11.203 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:11.215 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:59:11.215 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:59:11.237 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:59:11.237 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:59:11.259 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:59:11.260 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:59:11.282 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:59:11.282 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:59:11.306 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:59:11.306 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:59:11.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:11.327 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:59:11.328 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:59:11.328 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:11.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:11.451 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:11.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:11.582 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[29889]: audit 2024-09-13T22:59:10.482189+0000 mgr.y (mgr.14613) 799 : audit [DBG] from='client.25726 v1:172.21.15.178:0/3551930116' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:59:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[29889]: cluster 2024-09-13T22:59:10.558666+0000 mon.a (mon.0) 1699 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:59:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[29889]: audit 2024-09-13T22:59:10.672328+0000 mon.a (mon.0) 1700 : audit [INF] from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:59:11.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[29889]: cluster 2024-09-13T22:59:10.683634+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-09-13T22:59:11.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[29889]: audit 2024-09-13T22:59:11.054313+0000 mon.a (mon.0) 1702 : audit [DBG] from='client.? v1:172.21.15.178:0/1248742998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:59:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[22840]: audit 2024-09-13T22:59:10.482189+0000 mgr.y (mgr.14613) 799 : audit [DBG] from='client.25726 v1:172.21.15.178:0/3551930116' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:59:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[22840]: cluster 2024-09-13T22:59:10.558666+0000 mon.a (mon.0) 1699 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:59:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[22840]: audit 2024-09-13T22:59:10.672328+0000 mon.a (mon.0) 1700 : audit [INF] from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:59:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[22840]: cluster 2024-09-13T22:59:10.683634+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-09-13T22:59:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:11 smithi178 bash[22840]: audit 2024-09-13T22:59:11.054313+0000 mon.a (mon.0) 1702 : audit [DBG] from='client.? v1:172.21.15.178:0/1248742998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:59:11.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:11.705 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:11.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 65 16 68293 68277 65.6378 10.4375 0.00788638 0.015026 2024-09-13T22:59:11.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:11.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:11 smithi192 bash[24608]: audit 2024-09-13T22:59:10.482189+0000 mgr.y (mgr.14613) 799 : audit [DBG] from='client.25726 v1:172.21.15.178:0/3551930116' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T22:59:11.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:11 smithi192 bash[24608]: cluster 2024-09-13T22:59:10.558666+0000 mon.a (mon.0) 1699 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T22:59:11.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:11 smithi192 bash[24608]: audit 2024-09-13T22:59:10.672328+0000 mon.a (mon.0) 1700 : audit [INF] from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T22:59:11.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:11 smithi192 bash[24608]: cluster 2024-09-13T22:59:10.683634+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-09-13T22:59:11.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:11 smithi192 bash[24608]: audit 2024-09-13T22:59:11.054313+0000 mon.a (mon.0) 1702 : audit [DBG] from='client.? v1:172.21.15.178:0/1248742998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:59:11.840 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:11.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:11.972 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:11.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:12.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:12.101 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:12.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:12.237 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:12.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:12.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:12.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:12.500 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:12.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:12.637 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:12.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 66 16 69028 69012 65.3393 45.9375 0.0234533 0.0152983 2024-09-13T22:59:12.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:12.764 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:12.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:12.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:12 smithi178 bash[22840]: cluster 2024-09-13T22:59:10.936959+0000 mgr.y (mgr.14613) 800 : cluster [DBG] pgmap v948: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 726 op/s; 14757/200985 objects degraded (7.342%); 3899/200985 objects misplaced (1.940%); 16 MiB/s, 249 objects/s recovering 2024-09-13T22:59:12.881 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:12 smithi178 bash[22840]: cluster 2024-09-13T22:59:11.682138+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-09-13T22:59:12.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:12 smithi178 bash[29889]: cluster 2024-09-13T22:59:10.936959+0000 mgr.y (mgr.14613) 800 : cluster [DBG] pgmap v948: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 726 op/s; 14757/200985 objects degraded (7.342%); 3899/200985 objects misplaced (1.940%); 16 MiB/s, 249 objects/s recovering 2024-09-13T22:59:12.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:12 smithi178 bash[29889]: cluster 2024-09-13T22:59:11.682138+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-09-13T22:59:12.895 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:12.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:13.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:13.010 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:12 smithi192 bash[24608]: cluster 2024-09-13T22:59:10.936959+0000 mgr.y (mgr.14613) 800 : cluster [DBG] pgmap v948: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 726 op/s; 14757/200985 objects degraded (7.342%); 3899/200985 objects misplaced (1.940%); 16 MiB/s, 249 objects/s recovering 2024-09-13T22:59:13.010 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:12 smithi192 bash[24608]: cluster 2024-09-13T22:59:11.682138+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-09-13T22:59:13.023 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:13.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:13.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:13.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:13.293 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:13.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:13.415 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:13.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:13.541 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:13.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:13.668 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:13.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 67 16 69713 69697 65.003 42.8125 0.0266475 0.0153771 2024-09-13T22:59:13.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:13.789 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:13.893 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:13.913 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:13.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:14.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:14.040 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:14.045 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:13 smithi178 bash[22840]: cluster 2024-09-13T22:59:13.141189+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: Degraded data redundancy: 14757/200985 objects degraded (7.342%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:14.045 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:13 smithi178 bash[29889]: cluster 2024-09-13T22:59:13.141189+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: Degraded data redundancy: 14757/200985 objects degraded (7.342%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:14.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:14.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:13 smithi192 bash[24608]: cluster 2024-09-13T22:59:13.141189+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: Degraded data redundancy: 14757/200985 objects degraded (7.342%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:14.171 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:14.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.310 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:14.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.432 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:14.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.574 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:14.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:14.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:14.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 68 16 70211 70195 64.5048 31.125 0.0306077 0.0154954 2024-09-13T22:59:14.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.825 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:14 smithi178 bash[22840]: cluster 2024-09-13T22:59:12.937542+0000 mgr.y (mgr.14613) 801 : cluster [DBG] pgmap v950: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 29 MiB/s wr, 472 op/s; 14757/202116 objects degraded (7.301%); 3596/202116 objects misplaced (1.779%); 13 MiB/s, 215 objects/s recovering 2024-09-13T22:59:14.825 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:14 smithi178 bash[29889]: cluster 2024-09-13T22:59:12.937542+0000 mgr.y (mgr.14613) 801 : cluster [DBG] pgmap v950: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 29 MiB/s wr, 472 op/s; 14757/202116 objects degraded (7.301%); 3596/202116 objects misplaced (1.779%); 13 MiB/s, 215 objects/s recovering 2024-09-13T22:59:14.842 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:14.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:14.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:15.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:15.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:15.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:14 smithi192 bash[24608]: cluster 2024-09-13T22:59:12.937542+0000 mgr.y (mgr.14613) 801 : cluster [DBG] pgmap v950: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 29 MiB/s wr, 472 op/s; 14757/202116 objects degraded (7.301%); 3596/202116 objects misplaced (1.779%); 13 MiB/s, 215 objects/s recovering 2024-09-13T22:59:15.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:15.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:15.261 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:15.364 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:15.387 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:15.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:15.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:15.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:15.649 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:15.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 69 16 70777 70761 64.0826 35.375 0.0113242 0.0155976 2024-09-13T22:59:15.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:15.781 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:15.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T22:59:15.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:15.918 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:15.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:16.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:16.072 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:16.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:16.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:59:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:59:16.201 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:16.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:16.340 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:16.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:16.466 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:16.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:16.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:16.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:16.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 70 16 71552 71536 63.859 48.4375 0.0168445 0.0156532 2024-09-13T22:59:16.741 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:16.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[22840]: cluster 2024-09-13T22:59:14.938850+0000 mgr.y (mgr.14613) 802 : cluster [DBG] pgmap v951: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 726 op/s; 11849/209505 objects degraded (5.656%); 5850/209505 objects misplaced (2.792%); 17 MiB/s, 264 objects/s recovering 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[22840]: cluster 2024-09-13T22:59:15.356650+0000 osd.4 (osd.4) 276 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[22840]: cluster 2024-09-13T22:59:15.356741+0000 osd.4 (osd.4) 277 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[22840]: cluster 2024-09-13T22:59:15.711304+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[29889]: cluster 2024-09-13T22:59:14.938850+0000 mgr.y (mgr.14613) 802 : cluster [DBG] pgmap v951: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 726 op/s; 11849/209505 objects degraded (5.656%); 5850/209505 objects misplaced (2.792%); 17 MiB/s, 264 objects/s recovering 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[29889]: cluster 2024-09-13T22:59:15.356650+0000 osd.4 (osd.4) 276 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[29889]: cluster 2024-09-13T22:59:15.356741+0000 osd.4 (osd.4) 277 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:16.857 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:16 smithi178 bash[29889]: cluster 2024-09-13T22:59:15.711304+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-09-13T22:59:16.878 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:16.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:16.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:17.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:17.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:17.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:16 smithi192 bash[24608]: cluster 2024-09-13T22:59:14.938850+0000 mgr.y (mgr.14613) 802 : cluster [DBG] pgmap v951: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 726 op/s; 11849/209505 objects degraded (5.656%); 5850/209505 objects misplaced (2.792%); 17 MiB/s, 264 objects/s recovering 2024-09-13T22:59:17.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:16 smithi192 bash[24608]: cluster 2024-09-13T22:59:15.356650+0000 osd.4 (osd.4) 276 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:17.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:16 smithi192 bash[24608]: cluster 2024-09-13T22:59:15.356741+0000 osd.4 (osd.4) 277 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:17.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:16 smithi192 bash[24608]: cluster 2024-09-13T22:59:15.711304+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-09-13T22:59:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.174 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:17.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:17.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:17.299 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:17.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.437 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:17.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:17.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:17.570 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:17.673 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.705 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:17.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:17.722 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 71 16 72365 72349 63.6752 50.8125 0.0157613 0.0156977 2024-09-13T22:59:17.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:17.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:17 smithi192 bash[24608]: audit 2024-09-13T22:59:15.928206+0000 mgr.y (mgr.14613) 803 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:17.830 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:17.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:17.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:17 smithi178 bash[22840]: audit 2024-09-13T22:59:15.928206+0000 mgr.y (mgr.14613) 803 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:17.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:17 smithi178 bash[29889]: audit 2024-09-13T22:59:15.928206+0000 mgr.y (mgr.14613) 803 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:17.969 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:17.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:18.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:18.097 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:18.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:18.232 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:18.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:18.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:18.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:18.494 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:18.599 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:18.627 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:18.723 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 72 16 72812 72796 63.1789 27.9375 0.0140292 0.0158223 2024-09-13T22:59:18.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:18.756 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:18.862 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:18.911 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:18.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:19.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:19.061 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:19.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:18 smithi192 bash[24608]: cluster 2024-09-13T22:59:16.940026+0000 mgr.y (mgr.14613) 804 : cluster [DBG] pgmap v953: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 817 B/s rd, 32 MiB/s wr, 514 op/s; 11849/210648 objects degraded (5.625%); 5517/210648 objects misplaced (2.619%); 13 MiB/s, 206 objects/s recovering 2024-09-13T22:59:19.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:18 smithi192 bash[24608]: cluster 2024-09-13T22:59:18.142301+0000 mon.a (mon.0) 1706 : cluster [WRN] Health check update: Degraded data redundancy: 11849/210648 objects degraded (5.625%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:19.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:18 smithi192 bash[24608]: cluster 2024-09-13T22:59:18.151722+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-09-13T22:59:19.169 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:19.212 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:19.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[22840]: cluster 2024-09-13T22:59:16.940026+0000 mgr.y (mgr.14613) 804 : cluster [DBG] pgmap v953: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 817 B/s rd, 32 MiB/s wr, 514 op/s; 11849/210648 objects degraded (5.625%); 5517/210648 objects misplaced (2.619%); 13 MiB/s, 206 objects/s recovering 2024-09-13T22:59:19.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[22840]: cluster 2024-09-13T22:59:18.142301+0000 mon.a (mon.0) 1706 : cluster [WRN] Health check update: Degraded data redundancy: 11849/210648 objects degraded (5.625%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:19.313 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[22840]: cluster 2024-09-13T22:59:18.151722+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-09-13T22:59:19.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[29889]: cluster 2024-09-13T22:59:16.940026+0000 mgr.y (mgr.14613) 804 : cluster [DBG] pgmap v953: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 817 B/s rd, 32 MiB/s wr, 514 op/s; 11849/210648 objects degraded (5.625%); 5517/210648 objects misplaced (2.619%); 13 MiB/s, 206 objects/s recovering 2024-09-13T22:59:19.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[29889]: cluster 2024-09-13T22:59:18.142301+0000 mon.a (mon.0) 1706 : cluster [WRN] Health check update: Degraded data redundancy: 11849/210648 objects degraded (5.625%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:19.314 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[29889]: cluster 2024-09-13T22:59:18.151722+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-09-13T22:59:19.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:19.345 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:19.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:19.473 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:19.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:19.602 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:19.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:19.723 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 73 16 73521 73505 62.92 44.3125 0.0168967 0.0158868 2024-09-13T22:59:19.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:19.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:19.864 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:59:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:59:19.865 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[29889]: audit 2024-09-13T22:59:19.467731+0000 mon.b (mon.1) 276 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:19.865 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:19 smithi178 bash[22840]: audit 2024-09-13T22:59:19.467731+0000 mon.b (mon.1) 276 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:19.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:19.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:19.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:19 smithi192 bash[24608]: audit 2024-09-13T22:59:19.467731+0000 mon.b (mon.1) 276 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.016 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:20.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:20.118 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:20.152 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:20.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.298 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:20.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:20.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:20.423 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:20.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:20.546 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:20.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:20.679 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:20.723 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 74 16 74350 74334 62.7699 51.8125 0.0325868 0.0159246 2024-09-13T22:59:20.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:20.789 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:20 smithi192 bash[24608]: cluster 2024-09-13T22:59:18.940675+0000 mgr.y (mgr.14613) 805 : cluster [DBG] pgmap v955: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 705 B/s rd, 26 MiB/s wr, 420 op/s; 11849/211251 objects degraded (5.609%); 5517/211251 objects misplaced (2.612%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T22:59:20.805 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:20.832 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":946,"stamp":"2024-09-13T22:59:08.935936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4197784091,"num_objects":64243,"num_object_clones":0,"num_object_copies":192729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14757,"num_objects_misplaced":5057,"num_objects_unfound":0,"num_objects_dirty":64243,"num_whiteouts":0,"num_read":2848,"num_read_kb":2600,"num_write":65052,"num_write_kb":4098861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4205,"num_bytes_recovered":245113699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66664,"ondisk_log_size":66664,"up":474,"acting":466,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":482,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17190892,"kb_used_data":12516156,"kb_used_omap":10,"kb_used_meta":4674613,"kb_avail":639025172,"statfs":{"total":671965249536,"available":654361776128,"internally_reserved":0,"allocated":12816543744,"data_stored":12812729730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4786803819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,7,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":522715136,"num_objects":7976,"num_object_clones":0,"num_object_copies":23928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2006,"num_objects_unfound":0,"num_objects_dirty":7976,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7976,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2006,"num_bytes_recovered":131465216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.797006"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":763,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157975+0000","last_change":"2024-09-13T22:58:42.763562+0000","last_active":"2024-09-13T22:58:58.157975+0000","last_peered":"2024-09-13T22:58:58.157975+0000","last_clean":"2024-09-13T22:58:58.157975+0000","last_became_active":"2024-09-13T22:58:42.763279+0000","last_became_peered":"2024-09-13T22:58:42.763279+0000","last_unstale":"2024-09-13T22:58:58.157975+0000","last_undegraded":"2024-09-13T22:58:58.157975+0000","last_fullsized":"2024-09-13T22:58:58.157975+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.827330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.827330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:38.447663+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":818,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161561+0000","last_change":"2024-09-13T22:58:42.740974+0000","last_active":"2024-09-13T22:58:58.161561+0000","last_peered":"2024-09-13T22:58:58.161561+0000","last_clean":"2024-09-13T22:58:58.161561+0000","last_became_active":"2024-09-13T22:58:42.740638+0000","last_became_peered":"2024-09-13T22:58:42.740638+0000","last_unstale":"2024-09-13T22:58:58.161561+0000","last_undegraded":"2024-09-13T22:58:58.161561+0000","last_fullsized":"2024-09-13T22:58:58.161561+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.259089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:22.232806+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.259089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.229452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":855,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158967+0000","last_change":"2024-09-13T22:58:42.783796+0000","last_active":"2024-09-13T22:58:58.158967+0000","last_peered":"2024-09-13T22:58:58.158967+0000","last_clean":"2024-09-13T22:58:58.158967+0000","last_became_active":"2024-09-13T22:58:42.747843+0000","last_became_peered":"2024-09-13T22:58:42.747843+0000","last_unstale":"2024-09-13T22:58:58.158967+0000","last_undegraded":"2024-09-13T22:58:58.158967+0000","last_fullsized":"2024-09-13T22:58:58.158967+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'25","last_scrub_stamp":"2024-09-13T22:57:46.573001+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.573001+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.241918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.166995+0000","last_change":"2024-09-13T22:58:42.747300+0000","last_active":"2024-09-13T22:58:58.166995+0000","last_peered":"2024-09-13T22:58:58.166995+0000","last_clean":"2024-09-13T22:58:58.166995+0000","last_became_active":"2024-09-13T22:58:42.745341+0000","last_became_peered":"2024-09-13T22:58:42.745341+0000","last_unstale":"2024-09-13T22:58:58.166995+0000","last_undegraded":"2024-09-13T22:58:58.166995+0000","last_fullsized":"2024-09-13T22:58:58.166995+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:00.773837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T22:57:00.773837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":765,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159056+0000","last_change":"2024-09-13T22:56:28.054932+0000","last_active":"2024-09-13T22:58:58.159056+0000","last_peered":"2024-09-13T22:58:58.159056+0000","last_clean":"2024-09-13T22:58:58.159056+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T22:58:58.159056+0000","last_undegraded":"2024-09-13T22:58:58.159056+0000","last_fullsized":"2024-09-13T22:58:58.159056+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:28.051743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.051743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":826,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161317+0000","last_change":"2024-09-13T22:57:06.415408+0000","last_active":"2024-09-13T22:58:58.161317+0000","last_peered":"2024-09-13T22:58:58.161317+0000","last_clean":"2024-09-13T22:58:58.161317+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T22:58:58.161317+0000","last_undegraded":"2024-09-13T22:58:58.161317+0000","last_fullsized":"2024-09-13T22:58:58.161317+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:57:06.415278+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.415278+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":763,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.149279+0000","last_change":"2024-09-13T22:57:49.171117+0000","last_active":"2024-09-13T22:58:58.149279+0000","last_peered":"2024-09-13T22:58:58.149279+0000","last_clean":"2024-09-13T22:58:58.149279+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T22:58:58.149279+0000","last_undegraded":"2024-09-13T22:58:58.149279+0000","last_fullsized":"2024-09-13T22:58:58.149279+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.170845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.170845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.075808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":281,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160365+0000","last_change":"2024-09-13T22:57:09.115091+0000","last_active":"2024-09-13T22:58:58.160365+0000","last_peered":"2024-09-13T22:58:58.160365+0000","last_clean":"2024-09-13T22:58:58.160365+0000","last_became_active":"2024-09-13T22:55:38.143783+0000","last_became_peered":"2024-09-13T22:55:38.143783+0000","last_unstale":"2024-09-13T22:58:58.160365+0000","last_undegraded":"2024-09-13T22:58:58.160365+0000","last_fullsized":"2024-09-13T22:58:58.160365+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.114979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.114979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":775,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164820+0000","last_change":"2024-09-13T22:58:42.771418+0000","last_active":"2024-09-13T22:58:58.164820+0000","last_peered":"2024-09-13T22:58:58.164820+0000","last_clean":"2024-09-13T22:58:58.164820+0000","last_became_active":"2024-09-13T22:58:42.770668+0000","last_became_peered":"2024-09-13T22:58:42.770668+0000","last_unstale":"2024-09-13T22:58:58.164820+0000","last_undegraded":"2024-09-13T22:58:58.164820+0000","last_fullsized":"2024-09-13T22:58:58.164820+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:20.805535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:20.805535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":953,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161796+0000","last_change":"2024-09-13T22:58:42.775997+0000","last_active":"2024-09-13T22:58:58.161796+0000","last_peered":"2024-09-13T22:58:58.161796+0000","last_clean":"2024-09-13T22:58:58.161796+0000","last_became_active":"2024-09-13T22:58:42.728658+0000","last_became_peered":"2024-09-13T22:58:42.728658+0000","last_unstale":"2024-09-13T22:58:58.161796+0000","last_undegraded":"2024-09-13T22:58:58.161796+0000","last_fullsized":"2024-09-13T22:58:58.161796+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:56:41.965747+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T22:56:41.965747+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":765,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157821+0000","last_change":"2024-09-13T22:55:38.153026+0000","last_active":"2024-09-13T22:58:58.157821+0000","last_peered":"2024-09-13T22:58:58.157821+0000","last_clean":"2024-09-13T22:58:58.157821+0000","last_became_active":"2024-09-13T22:55:38.151599+0000","last_became_peered":"2024-09-13T22:55:38.151599+0000","last_unstale":"2024-09-13T22:58:58.157821+0000","last_undegraded":"2024-09-13T22:58:58.157821+0000","last_fullsized":"2024-09-13T22:58:58.157821+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:21.721055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T22:55:21.721055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":767,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160273+0000","last_change":"2024-09-13T22:58:14.162918+0000","last_active":"2024-09-13T22:58:58.160273+0000","last_peered":"2024-09-13T22:58:58.160273+0000","last_clean":"2024-09-13T22:58:58.160273+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T22:58:58.160273+0000","last_undegraded":"2024-09-13T22:58:58.160273+0000","last_fullsized":"2024-09-13T22:58:58.160273+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:14.162730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.162730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:17.036097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":763,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.169201+0000","last_change":"2024-09-13T22:54:29.857171+0000","last_active":"2024-09-13T22:58:58.169201+0000","last_peered":"2024-09-13T22:58:58.169201+0000","last_clean":"2024-09-13T22:58:58.169201+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T22:58:58.169201+0000","last_undegraded":"2024-09-13T22:58:58.169201+0000","last_fullsized":"2024-09-13T22:58:58.169201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:29.857063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:29.857063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"397'24","reported_seq":343,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161248+0000","last_change":"2024-09-13T22:58:42.802871+0000","last_active":"2024-09-13T22:58:58.161248+0000","last_peered":"2024-09-13T22:58:58.161248+0000","last_clean":"2024-09-13T22:58:58.161248+0000","last_became_active":"2024-09-13T22:58:42.768934+0000","last_became_peered":"2024-09-13T22:58:42.768934+0000","last_unstale":"2024-09-13T22:58:58.161248+0000","last_undegraded":"2024-09-13T22:58:58.161248+0000","last_fullsized":"2024-09-13T22:58:58.161248+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T22:58:19.363461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:19.363461+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:43.594831+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":771,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160557+0000","last_change":"2024-09-13T22:58:09.016326+0000","last_active":"2024-09-13T22:58:58.160557+0000","last_peered":"2024-09-13T22:58:58.160557+0000","last_clean":"2024-09-13T22:58:58.160557+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T22:58:58.160557+0000","last_undegraded":"2024-09-13T22:58:58.160557+0000","last_fullsized":"2024-09-13T22:58:58.160557+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:09.016219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.016219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.276915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":768,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159769+0000","last_change":"2024-09-13T22:57:51.203622+0000","last_active":"2024-09-13T22:58:58.159769+0000","last_peered":"2024-09-13T22:58:58.159769+0000","last_clean":"2024-09-13T22:58:58.159769+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T22:58:58.159769+0000","last_undegraded":"2024-09-13T22:58:58.159769+0000","last_fullsized":"2024-09-13T22:58:58.159769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.203544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.203544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"420'4028","reported_seq":4084,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:03.769710+0000","last_change":"2024-09-13T22:57:49.649946+0000","last_active":"2024-09-13T22:59:03.769710+0000","last_peered":"2024-09-13T22:59:03.769710+0000","last_clean":"2024-09-13T22:59:03.769710+0000","last_became_active":"2024-09-13T22:57:49.649466+0000","last_became_peered":"2024-09-13T22:57:49.649466+0000","last_unstale":"2024-09-13T22:59:03.769710+0000","last_undegraded":"2024-09-13T22:59:03.769710+0000","last_fullsized":"2024-09-13T22:59:03.769710+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.651547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263979008,"num_objects":4028,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4028,"num_write_kb":257792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":826,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160119+0000","last_change":"2024-09-13T22:56:37.051234+0000","last_active":"2024-09-13T22:58:58.160119+0000","last_peered":"2024-09-13T22:58:58.160119+0000","last_clean":"2024-09-13T22:58:58.160119+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T22:58:58.160119+0000","last_undegraded":"2024-09-13T22:58:58.160119+0000","last_fullsized":"2024-09-13T22:58:58.160119+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:37.051143+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:37.051143+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":43,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162498+0000","last_change":"2024-09-13T22:58:42.778291+0000","last_active":"2024-09-13T22:58:58.162498+0000","last_peered":"2024-09-13T22:58:58.162498+0000","last_clean":"2024-09-13T22:58:58.162498+0000","last_became_active":"2024-09-13T22:58:42.749141+0000","last_became_peered":"2024-09-13T22:58:42.749141+0000","last_unstale":"2024-09-13T22:58:58.162498+0000","last_undegraded":"2024-09-13T22:58:58.162498+0000","last_fullsized":"2024-09-13T22:58:58.162498+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:57:48.143715+0000","last_deep_scrub":"228'44","last_deep_scrub_stamp":"2024-09-13T22:54:34.302937+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.143715+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.589944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":782,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158589+0000","last_change":"2024-09-13T22:58:16.344650+0000","last_active":"2024-09-13T22:58:58.158589+0000","last_peered":"2024-09-13T22:58:58.158589+0000","last_clean":"2024-09-13T22:58:58.158589+0000","last_became_active":"2024-09-13T22:55:38.140796+0000","last_became_peered":"2024-09-13T22:55:38.140796+0000","last_unstale":"2024-09-13T22:58:58.158589+0000","last_undegraded":"2024-09-13T22:58:58.158589+0000","last_fullsized":"2024-09-13T22:58:58.158589+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:16.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:58:16.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.752947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":802,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150100+0000","last_change":"2024-09-13T22:58:42.743933+0000","last_active":"2024-09-13T22:58:58.150100+0000","last_peered":"2024-09-13T22:58:58.150100+0000","last_clean":"2024-09-13T22:58:58.150100+0000","last_became_active":"2024-09-13T22:58:42.743564+0000","last_became_peered":"2024-09-13T22:58:42.743564+0000","last_unstale":"2024-09-13T22:58:58.150100+0000","last_undegraded":"2024-09-13T22:58:58.150100+0000","last_fullsized":"2024-09-13T22:58:58.150100+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.123464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.707036+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":698,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162051+0000","last_change":"2024-09-13T22:58:42.748052+0000","last_active":"2024-09-13T22:58:58.162051+0000","last_peered":"2024-09-13T22:58:58.162051+0000","last_clean":"2024-09-13T22:58:58.162051+0000","last_became_active":"2024-09-13T22:58:42.747455+0000","last_became_peered":"2024-09-13T22:58:42.747455+0000","last_unstale":"2024-09-13T22:58:58.162051+0000","last_undegraded":"2024-09-13T22:58:58.162051+0000","last_fullsized":"2024-09-13T22:58:58.162051+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:58.481749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:58.481749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"420'3934","reported_seq":5708,"reported_epoch":420,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:59:03.797477+0000","last_change":"2024-09-13T22:58:43.993989+0000","last_active":"2024-09-13T22:59:03.797477+0000","last_peered":"2024-09-13T22:59:03.797477+0000","last_clean":"2024-09-13T22:58:41.700563+0000","last_became_active":"2024-09-13T22:58:43.975938+0000","last_became_peered":"2024-09-13T22:58:43.975938+0000","last_unstale":"2024-09-13T22:59:03.797477+0000","last_undegraded":"2024-09-13T22:59:03.797477+0000","last_fullsized":"2024-09-13T22:58:43.706289+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257818624,"num_objects":3934,"num_object_clones":0,"num_object_copies":11802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2109,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":251776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":56229888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":2109},{"shards":"1,7","objects":1825}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":1043,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157695+0000","last_change":"2024-09-13T22:58:02.221879+0000","last_active":"2024-09-13T22:58:58.157695+0000","last_peered":"2024-09-13T22:58:58.157695+0000","last_clean":"2024-09-13T22:58:58.157695+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T22:58:58.157695+0000","last_undegraded":"2024-09-13T22:58:58.157695+0000","last_fullsized":"2024-09-13T22:58:58.157695+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:02.221654+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.221654+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:20.958768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":846,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158253+0000","last_change":"2024-09-13T22:55:38.114946+0000","last_active":"2024-09-13T22:58:58.158253+0000","last_peered":"2024-09-13T22:58:58.158253+0000","last_clean":"2024-09-13T22:58:58.158253+0000","last_became_active":"2024-09-13T22:55:38.095029+0000","last_became_peered":"2024-09-13T22:55:38.095029+0000","last_unstale":"2024-09-13T22:58:58.158253+0000","last_undegraded":"2024-09-13T22:58:58.158253+0000","last_fullsized":"2024-09-13T22:58:58.158253+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:55:13.915384+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T22:55:13.915384+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":772,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150271+0000","last_change":"2024-09-13T22:55:35.535365+0000","last_active":"2024-09-13T22:58:58.150271+0000","last_peered":"2024-09-13T22:58:58.150271+0000","last_clean":"2024-09-13T22:58:58.150271+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T22:58:58.150271+0000","last_undegraded":"2024-09-13T22:58:58.150271+0000","last_fullsized":"2024-09-13T22:58:58.150271+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:35.535254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:35.535254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":495,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161554+0000","last_change":"2024-09-13T22:57:06.962170+0000","last_active":"2024-09-13T22:58:58.161554+0000","last_peered":"2024-09-13T22:58:58.161554+0000","last_clean":"2024-09-13T22:58:58.161554+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T22:58:58.161554+0000","last_undegraded":"2024-09-13T22:58:58.161554+0000","last_fullsized":"2024-09-13T22:58:58.161554+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'12","last_scrub_stamp":"2024-09-13T22:57:06.962075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.962075+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":281,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.153321+0000","last_change":"2024-09-13T22:58:04.163362+0000","last_active":"2024-09-13T22:58:58.153321+0000","last_peered":"2024-09-13T22:58:58.153321+0000","last_clean":"2024-09-13T22:58:58.153321+0000","last_became_active":"2024-09-13T22:55:38.103505+0000","last_became_peered":"2024-09-13T22:55:38.103505+0000","last_unstale":"2024-09-13T22:58:58.153321+0000","last_undegraded":"2024-09-13T22:58:58.153321+0000","last_fullsized":"2024-09-13T22:58:58.153321+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.163173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.163173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.621623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"420'3992","reported_seq":4048,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:03.723672+0000","last_change":"2024-09-13T22:57:49.644230+0000","last_active":"2024-09-13T22:59:03.723672+0000","last_peered":"2024-09-13T22:59:03.723672+0000","last_clean":"2024-09-13T22:59:03.723672+0000","last_became_active":"2024-09-13T22:57:49.643953+0000","last_became_peered":"2024-09-13T22:57:49.643953+0000","last_unstale":"2024-09-13T22:59:03.723672+0000","last_undegraded":"2024-09-13T22:59:03.723672+0000","last_fullsized":"2024-09-13T22:59:03.723672+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:17.126537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261619712,"num_objects":3992,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":255488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":852,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160520+0000","last_change":"2024-09-13T22:58:45.828688+0000","last_active":"2024-09-13T22:58:58.160520+0000","last_peered":"2024-09-13T22:58:58.160520+0000","last_clean":"2024-09-13T22:58:58.160520+0000","last_became_active":"2024-09-13T22:58:45.828166+0000","last_became_peered":"2024-09-13T22:58:45.828166+0000","last_unstale":"2024-09-13T22:58:58.160520+0000","last_undegraded":"2024-09-13T22:58:58.160520+0000","last_fullsized":"2024-09-13T22:58:58.160520+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:12.165912+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.165912+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.127090+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"397'39","reported_seq":821,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.163325+0000","last_change":"2024-09-13T22:55:29.816311+0000","last_active":"2024-09-13T22:58:58.163325+0000","last_peered":"2024-09-13T22:58:58.163325+0000","last_clean":"2024-09-13T22:58:58.163325+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T22:58:58.163325+0000","last_undegraded":"2024-09-13T22:58:58.163325+0000","last_fullsized":"2024-09-13T22:58:58.163325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:55:29.816212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.816212+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.149425+0000","last_change":"2024-09-13T22:57:50.137862+0000","last_active":"2024-09-13T22:58:58.149425+0000","last_peered":"2024-09-13T22:58:58.149425+0000","last_clean":"2024-09-13T22:58:58.149425+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T22:58:58.149425+0000","last_undegraded":"2024-09-13T22:58:58.149425+0000","last_fullsized":"2024-09-13T22:58:58.149425+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.137581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.137581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:02.936942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"351'15","reported_seq":457,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:01.803727+0000","last_change":"2024-09-13T22:57:02.049588+0000","last_active":"2024-09-13T22:59:01.803727+0000","last_peered":"2024-09-13T22:59:01.803727+0000","last_clean":"2024-09-13T22:59:01.803727+0000","last_became_active":"2024-09-13T22:55:38.106061+0000","last_became_peered":"2024-09-13T22:55:38.106061+0000","last_unstale":"2024-09-13T22:59:01.803727+0000","last_undegraded":"2024-09-13T22:59:01.803727+0000","last_fullsized":"2024-09-13T22:59:01.803727+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'15","last_scrub_stamp":"2024-09-13T22:57:02.049279+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T22:57:02.049279+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":760,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161461+0000","last_change":"2024-09-13T22:57:11.104485+0000","last_active":"2024-09-13T22:58:58.161461+0000","last_peered":"2024-09-13T22:58:58.161461+0000","last_clean":"2024-09-13T22:58:58.161461+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T22:58:58.161461+0000","last_undegraded":"2024-09-13T22:58:58.161461+0000","last_fullsized":"2024-09-13T22:58:58.161461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:11.104365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T22:57:11.104365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"420'3986","reported_seq":6113,"reported_epoch":420,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:59:06.429174+0000","last_change":"2024-09-13T22:58:44.036587+0000","last_active":"2024-09-13T22:59:06.429174+0000","last_peered":"2024-09-13T22:59:06.429174+0000","last_clean":"2024-09-13T22:58:41.669774+0000","last_became_active":"2024-09-13T22:58:43.968409+0000","last_became_peered":"2024-09-13T22:58:43.968409+0000","last_unstale":"2024-09-13T22:59:06.429174+0000","last_undegraded":"2024-09-13T22:59:06.429174+0000","last_fullsized":"2024-09-13T22:58:43.700865+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226496,"num_objects":3986,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1865,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":255104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1034,"num_bytes_recovered":67764224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[0,3],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":1865},{"shards":"0,3","objects":2121}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":373,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164386+0000","last_change":"2024-09-13T22:57:50.496802+0000","last_active":"2024-09-13T22:58:58.164386+0000","last_peered":"2024-09-13T22:58:58.164386+0000","last_clean":"2024-09-13T22:58:58.164386+0000","last_became_active":"2024-09-13T22:55:38.124814+0000","last_became_peered":"2024-09-13T22:55:38.124814+0000","last_unstale":"2024-09-13T22:58:58.164386+0000","last_undegraded":"2024-09-13T22:58:58.164386+0000","last_fullsized":"2024-09-13T22:58:58.164386+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T22:57:50.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.496702+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.914857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159040+0000","last_change":"2024-09-13T22:57:51.208483+0000","last_active":"2024-09-13T22:58:58.159040+0000","last_peered":"2024-09-13T22:58:58.159040+0000","last_clean":"2024-09-13T22:58:58.159040+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T22:58:58.159040+0000","last_undegraded":"2024-09-13T22:58:58.159040+0000","last_fullsized":"2024-09-13T22:58:58.159040+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.208413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.208413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:07.011784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":591,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159029+0000","last_change":"2024-09-13T22:58:42.753044+0000","last_active":"2024-09-13T22:58:58.159029+0000","last_peered":"2024-09-13T22:58:58.159029+0000","last_clean":"2024-09-13T22:58:58.159029+0000","last_became_active":"2024-09-13T22:58:42.746002+0000","last_became_peered":"2024-09-13T22:58:42.746002+0000","last_unstale":"2024-09-13T22:58:58.159029+0000","last_undegraded":"2024-09-13T22:58:58.159029+0000","last_fullsized":"2024-09-13T22:58:58.159029+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.848113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.848113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159232+0000","last_change":"2024-09-13T22:58:42.746884+0000","last_active":"2024-09-13T22:58:58.159232+0000","last_peered":"2024-09-13T22:58:58.159232+0000","last_clean":"2024-09-13T22:58:58.159232+0000","last_became_active":"2024-09-13T22:58:42.746385+0000","last_became_peered":"2024-09-13T22:58:42.746385+0000","last_unstale":"2024-09-13T22:58:58.159232+0000","last_undegraded":"2024-09-13T22:58:58.159232+0000","last_fullsized":"2024-09-13T22:58:58.159232+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.718269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:33.793814+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"420'3937","reported_seq":3992,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:03.788196+0000","last_change":"2024-09-13T22:57:49.642973+0000","last_active":"2024-09-13T22:59:03.788196+0000","last_peered":"2024-09-13T22:59:03.788196+0000","last_clean":"2024-09-13T22:59:03.788196+0000","last_became_active":"2024-09-13T22:57:49.642612+0000","last_became_peered":"2024-09-13T22:57:49.642612+0000","last_unstale":"2024-09-13T22:59:03.788196+0000","last_undegraded":"2024-09-13T22:59:03.788196+0000","last_fullsized":"2024-09-13T22:59:03.788196+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258015232,"num_objects":3937,"num_object_clones":0,"num_object_copies":11811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3937,"num_write_kb":251968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":817,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158400+0000","last_change":"2024-09-13T22:58:18.833271+0000","last_active":"2024-09-13T22:58:58.158400+0000","last_peered":"2024-09-13T22:58:58.158400+0000","last_clean":"2024-09-13T22:58:58.158400+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T22:58:58.158400+0000","last_undegraded":"2024-09-13T22:58:58.158400+0000","last_fullsized":"2024-09-13T22:58:58.158400+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:18.833192+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T22:58:18.833192+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:28.251911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":891,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.171433+0000","last_change":"2024-09-13T22:57:58.252181+0000","last_active":"2024-09-13T22:58:58.171433+0000","last_peered":"2024-09-13T22:58:58.171433+0000","last_clean":"2024-09-13T22:58:58.171433+0000","last_became_active":"2024-09-13T22:55:38.135796+0000","last_became_peered":"2024-09-13T22:55:38.135796+0000","last_unstale":"2024-09-13T22:58:58.171433+0000","last_undegraded":"2024-09-13T22:58:58.171433+0000","last_fullsized":"2024-09-13T22:58:58.171433+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'36","last_scrub_stamp":"2024-09-13T22:57:58.251941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:58.251941+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:05.244239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161212+0000","last_change":"2024-09-13T22:58:42.748161+0000","last_active":"2024-09-13T22:58:58.161212+0000","last_peered":"2024-09-13T22:58:58.161212+0000","last_clean":"2024-09-13T22:58:58.161212+0000","last_became_active":"2024-09-13T22:58:42.747048+0000","last_became_peered":"2024-09-13T22:58:42.747048+0000","last_unstale":"2024-09-13T22:58:58.161212+0000","last_undegraded":"2024-09-13T22:58:58.161212+0000","last_fullsized":"2024-09-13T22:58:58.161212+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.243067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.243067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:10.576315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":600,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160311+0000","last_change":"2024-09-13T22:58:42.755687+0000","last_active":"2024-09-13T22:58:58.160311+0000","last_peered":"2024-09-13T22:58:58.160311+0000","last_clean":"2024-09-13T22:58:58.160311+0000","last_became_active":"2024-09-13T22:58:42.721344+0000","last_became_peered":"2024-09-13T22:58:42.721344+0000","last_unstale":"2024-09-13T22:58:58.160311+0000","last_undegraded":"2024-09-13T22:58:58.160311+0000","last_fullsized":"2024-09-13T22:58:58.160311+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:58:11.043740+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:58:11.043740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:25.875060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"351'14","reported_seq":1056,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.163002+0000","last_change":"2024-09-13T22:58:02.750352+0000","last_active":"2024-09-13T22:58:58.163002+0000","last_peered":"2024-09-13T22:58:58.163002+0000","last_clean":"2024-09-13T22:58:58.163002+0000","last_became_active":"2024-09-13T22:55:38.087590+0000","last_became_peered":"2024-09-13T22:55:38.087590+0000","last_unstale":"2024-09-13T22:58:58.163002+0000","last_undegraded":"2024-09-13T22:58:58.163002+0000","last_fullsized":"2024-09-13T22:58:58.163002+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'14","last_scrub_stamp":"2024-09-13T22:58:02.750208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.750208+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:22.642062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":765,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158197+0000","last_change":"2024-09-13T22:58:42.771114+0000","last_active":"2024-09-13T22:58:58.158197+0000","last_peered":"2024-09-13T22:58:58.158197+0000","last_clean":"2024-09-13T22:58:58.158197+0000","last_became_active":"2024-09-13T22:58:42.770701+0000","last_became_peered":"2024-09-13T22:58:42.770701+0000","last_unstale":"2024-09-13T22:58:58.158197+0000","last_undegraded":"2024-09-13T22:58:58.158197+0000","last_fullsized":"2024-09-13T22:58:58.158197+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.267276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.267276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:30.718218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"420'4019","reported_seq":4075,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:04.072148+0000","last_change":"2024-09-13T22:57:49.650986+0000","last_active":"2024-09-13T22:59:04.072148+0000","last_peered":"2024-09-13T22:59:04.072148+0000","last_clean":"2024-09-13T22:59:04.072148+0000","last_became_active":"2024-09-13T22:57:49.650245+0000","last_became_peered":"2024-09-13T22:57:49.650245+0000","last_unstale":"2024-09-13T22:59:04.072148+0000","last_undegraded":"2024-09-13T22:59:04.072148+0000","last_fullsized":"2024-09-13T22:59:04.072148+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.292803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":12057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4019,"num_write_kb":257216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":642,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164253+0000","last_change":"2024-09-13T22:58:45.844049+0000","last_active":"2024-09-13T22:58:58.164253+0000","last_peered":"2024-09-13T22:58:58.164253+0000","last_clean":"2024-09-13T22:58:58.164253+0000","last_became_active":"2024-09-13T22:58:45.843410+0000","last_became_peered":"2024-09-13T22:58:45.843410+0000","last_unstale":"2024-09-13T22:58:58.164253+0000","last_undegraded":"2024-09-13T22:58:58.164253+0000","last_fullsized":"2024-09-13T22:58:58.164253+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:14.492071+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.492071+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:38.284448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":804,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162423+0000","last_change":"2024-09-13T22:56:59.524045+0000","last_active":"2024-09-13T22:58:58.162423+0000","last_peered":"2024-09-13T22:58:58.162423+0000","last_clean":"2024-09-13T22:58:58.162423+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T22:58:58.162423+0000","last_undegraded":"2024-09-13T22:58:58.162423+0000","last_fullsized":"2024-09-13T22:58:58.162423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:56:59.523938+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:56:59.523938+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":832,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.153612+0000","last_change":"2024-09-13T22:58:42.781273+0000","last_active":"2024-09-13T22:58:58.153612+0000","last_peered":"2024-09-13T22:58:58.153612+0000","last_clean":"2024-09-13T22:58:58.153612+0000","last_became_active":"2024-09-13T22:58:42.742912+0000","last_became_peered":"2024-09-13T22:58:42.742912+0000","last_unstale":"2024-09-13T22:58:58.153612+0000","last_undegraded":"2024-09-13T22:58:58.153612+0000","last_fullsized":"2024-09-13T22:58:58.153612+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T22:58:15.074737+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.074737+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.949551+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":2028,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:02.083730+0000","last_change":"2024-09-13T22:57:55.678130+0000","last_active":"2024-09-13T22:59:02.083730+0000","last_peered":"2024-09-13T22:59:02.083730+0000","last_clean":"2024-09-13T22:59:02.083730+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T22:59:02.083730+0000","last_undegraded":"2024-09-13T22:59:02.083730+0000","last_fullsized":"2024-09-13T22:59:02.083730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T22:57:55.678034+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.678034+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:03.773483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1256,"num_read_kb":1251,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":280,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161017+0000","last_change":"2024-09-13T22:57:49.593322+0000","last_active":"2024-09-13T22:58:58.161017+0000","last_peered":"2024-09-13T22:58:58.161017+0000","last_clean":"2024-09-13T22:58:58.161017+0000","last_became_active":"2024-09-13T22:55:38.124054+0000","last_became_peered":"2024-09-13T22:55:38.124054+0000","last_unstale":"2024-09-13T22:58:58.161017+0000","last_undegraded":"2024-09-13T22:58:58.161017+0000","last_fullsized":"2024-09-13T22:58:58.161017+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.593224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.593224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.823607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":277,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.167261+0000","last_change":"2024-09-13T22:58:42.748742+0000","last_active":"2024-09-13T22:58:58.167261+0000","last_peered":"2024-09-13T22:58:58.167261+0000","last_clean":"2024-09-13T22:58:58.167261+0000","last_became_active":"2024-09-13T22:58:42.747444+0000","last_became_peered":"2024-09-13T22:58:42.747444+0000","last_unstale":"2024-09-13T22:58:58.167261+0000","last_undegraded":"2024-09-13T22:58:58.167261+0000","last_fullsized":"2024-09-13T22:58:58.167261+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:31.153155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T22:55:31.153155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"420'4021","reported_seq":4077,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:03.761669+0000","last_change":"2024-09-13T22:57:49.647079+0000","last_active":"2024-09-13T22:59:03.761669+0000","last_peered":"2024-09-13T22:59:03.761669+0000","last_clean":"2024-09-13T22:59:03.761669+0000","last_became_active":"2024-09-13T22:57:49.646766+0000","last_became_peered":"2024-09-13T22:57:49.646766+0000","last_unstale":"2024-09-13T22:59:03.761669+0000","last_undegraded":"2024-09-13T22:59:03.761669+0000","last_fullsized":"2024-09-13T22:59:03.761669+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.932769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":257344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":463,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160743+0000","last_change":"2024-09-13T22:58:45.831947+0000","last_active":"2024-09-13T22:58:58.160743+0000","last_peered":"2024-09-13T22:58:58.160743+0000","last_clean":"2024-09-13T22:58:58.160743+0000","last_became_active":"2024-09-13T22:58:45.829511+0000","last_became_peered":"2024-09-13T22:58:45.829511+0000","last_unstale":"2024-09-13T22:58:58.160743+0000","last_undegraded":"2024-09-13T22:58:58.160743+0000","last_fullsized":"2024-09-13T22:58:58.160743+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:28.769564+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.769564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1130,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150672+0000","last_change":"2024-09-13T22:58:09.054534+0000","last_active":"2024-09-13T22:58:58.150672+0000","last_peered":"2024-09-13T22:58:58.150672+0000","last_clean":"2024-09-13T22:58:58.150672+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T22:58:58.150672+0000","last_undegraded":"2024-09-13T22:58:58.150672+0000","last_fullsized":"2024-09-13T22:58:58.150672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'64","last_scrub_stamp":"2024-09-13T22:58:09.054428+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:55:36.587996+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.054428+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.603201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":783,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157082+0000","last_change":"2024-09-13T22:58:07.245450+0000","last_active":"2024-09-13T22:58:58.157082+0000","last_peered":"2024-09-13T22:58:58.157082+0000","last_clean":"2024-09-13T22:58:58.157082+0000","last_became_active":"2024-09-13T22:55:38.145203+0000","last_became_peered":"2024-09-13T22:55:38.145203+0000","last_unstale":"2024-09-13T22:58:58.157082+0000","last_undegraded":"2024-09-13T22:58:58.157082+0000","last_fullsized":"2024-09-13T22:58:58.157082+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.245370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.245370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:11.129975+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162427+0000","last_change":"2024-09-13T22:58:42.741243+0000","last_active":"2024-09-13T22:58:58.162427+0000","last_peered":"2024-09-13T22:58:58.162427+0000","last_clean":"2024-09-13T22:58:58.162427+0000","last_became_active":"2024-09-13T22:58:42.740862+0000","last_became_peered":"2024-09-13T22:58:42.740862+0000","last_unstale":"2024-09-13T22:58:58.162427+0000","last_undegraded":"2024-09-13T22:58:58.162427+0000","last_fullsized":"2024-09-13T22:58:58.162427+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.310671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.310671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:32.722159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":773,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.149107+0000","last_change":"2024-09-13T22:58:42.742935+0000","last_active":"2024-09-13T22:58:58.149107+0000","last_peered":"2024-09-13T22:58:58.149107+0000","last_clean":"2024-09-13T22:58:58.149107+0000","last_became_active":"2024-09-13T22:58:42.742606+0000","last_became_peered":"2024-09-13T22:58:42.742606+0000","last_unstale":"2024-09-13T22:58:58.149107+0000","last_undegraded":"2024-09-13T22:58:58.149107+0000","last_fullsized":"2024-09-13T22:58:58.149107+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:55.737268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:55.737268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"420'4083","reported_seq":4139,"reported_epoch":420,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T22:59:04.783132+0000","last_change":"2024-09-13T22:58:44.000052+0000","last_active":"2024-09-13T22:59:04.783132+0000","last_peered":"2024-09-13T22:59:04.783132+0000","last_clean":"2024-09-13T22:58:41.699643+0000","last_became_active":"2024-09-13T22:58:43.992207+0000","last_became_peered":"2024-09-13T22:58:43.992207+0000","last_unstale":"2024-09-13T22:59:04.783132+0000","last_undegraded":"2024-09-13T22:58:43.828083+0000","last_fullsized":"2024-09-13T22:58:43.704318+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267583488,"num_objects":4083,"num_object_clones":0,"num_object_copies":12249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":261312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,1],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2","objects":1057},{"shards":"1,2,5","objects":3025}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":466,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161395+0000","last_change":"2024-09-13T22:58:45.833296+0000","last_active":"2024-09-13T22:58:58.161395+0000","last_peered":"2024-09-13T22:58:58.161395+0000","last_clean":"2024-09-13T22:58:58.161395+0000","last_became_active":"2024-09-13T22:58:45.832840+0000","last_became_peered":"2024-09-13T22:58:45.832840+0000","last_unstale":"2024-09-13T22:58:58.161395+0000","last_undegraded":"2024-09-13T22:58:58.161395+0000","last_fullsized":"2024-09-13T22:58:58.161395+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:30.160971+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:21.295886+0000","last_clean_scrub_stamp":"2024-09-13T22:56:30.160971+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1101,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:01.803814+0000","last_change":"2024-09-13T22:58:09.995562+0000","last_active":"2024-09-13T22:59:01.803814+0000","last_peered":"2024-09-13T22:59:01.803814+0000","last_clean":"2024-09-13T22:59:01.803814+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T22:59:01.803814+0000","last_undegraded":"2024-09-13T22:59:01.803814+0000","last_fullsized":"2024-09-13T22:59:01.803814+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'48","last_scrub_stamp":"2024-09-13T22:58:09.995445+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.995445+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:28.734156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":790,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150747+0000","last_change":"2024-09-13T22:57:06.876493+0000","last_active":"2024-09-13T22:58:58.150747+0000","last_peered":"2024-09-13T22:58:58.150747+0000","last_clean":"2024-09-13T22:58:58.150747+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T22:58:58.150747+0000","last_undegraded":"2024-09-13T22:58:58.150747+0000","last_fullsized":"2024-09-13T22:58:58.150747+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T22:57:06.876301+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T22:54:26.175576+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.876301+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":817,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150217+0000","last_change":"2024-09-13T22:58:06.134146+0000","last_active":"2024-09-13T22:58:58.150217+0000","last_peered":"2024-09-13T22:58:58.150217+0000","last_clean":"2024-09-13T22:58:58.150217+0000","last_became_active":"2024-09-13T22:55:38.114687+0000","last_became_peered":"2024-09-13T22:55:38.114687+0000","last_unstale":"2024-09-13T22:58:58.150217+0000","last_undegraded":"2024-09-13T22:58:58.150217+0000","last_fullsized":"2024-09-13T22:58:58.150217+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T22:58:06.133998+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T22:55:21.433994+0000","last_clean_scrub_stamp":"2024-09-13T22:58:06.133998+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:31.928770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"414'14","reported_seq":1071,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160004+0000","last_change":"2024-09-13T22:58:42.782423+0000","last_active":"2024-09-13T22:58:58.160004+0000","last_peered":"2024-09-13T22:58:58.160004+0000","last_clean":"2024-09-13T22:58:58.160004+0000","last_became_active":"2024-09-13T22:58:42.740603+0000","last_became_peered":"2024-09-13T22:58:42.740603+0000","last_unstale":"2024-09-13T22:58:58.160004+0000","last_undegraded":"2024-09-13T22:58:58.160004+0000","last_fullsized":"2024-09-13T22:58:58.160004+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'13","last_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_deep_scrub":"358'13","last_deep_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_clean_scrub_stamp":"2024-09-13T22:58:05.208554+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.060623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159567+0000","last_change":"2024-09-13T22:57:56.259013+0000","last_active":"2024-09-13T22:58:58.159567+0000","last_peered":"2024-09-13T22:58:58.159567+0000","last_clean":"2024-09-13T22:58:58.159567+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T22:58:58.159567+0000","last_undegraded":"2024-09-13T22:58:58.159567+0000","last_fullsized":"2024-09-13T22:58:58.159567+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:56.258783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:56.258783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.527327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"420'4017","reported_seq":4073,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:03.935617+0000","last_change":"2024-09-13T22:57:49.647535+0000","last_active":"2024-09-13T22:59:03.935617+0000","last_peered":"2024-09-13T22:59:03.935617+0000","last_clean":"2024-09-13T22:59:03.935617+0000","last_became_active":"2024-09-13T22:57:49.646701+0000","last_became_peered":"2024-09-13T22:57:49.646701+0000","last_unstale":"2024-09-13T22:59:03.935617+0000","last_undegraded":"2024-09-13T22:59:03.935617+0000","last_fullsized":"2024-09-13T22:59:03.935617+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:08.172073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263258112,"num_objects":4017,"num_object_clones":0,"num_object_copies":12051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4017,"num_write_kb":257088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":826,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160064+0000","last_change":"2024-09-13T22:57:57.050275+0000","last_active":"2024-09-13T22:58:58.160064+0000","last_peered":"2024-09-13T22:58:58.160064+0000","last_clean":"2024-09-13T22:58:58.160064+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T22:58:58.160064+0000","last_undegraded":"2024-09-13T22:58:58.160064+0000","last_fullsized":"2024-09-13T22:58:58.160064+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:57.050042+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.050042+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.549861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"397'39","reported_seq":132,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164038+0000","last_change":"2024-09-13T22:58:42.777359+0000","last_active":"2024-09-13T22:58:58.164038+0000","last_peered":"2024-09-13T22:58:58.164038+0000","last_clean":"2024-09-13T22:58:58.164038+0000","last_became_active":"2024-09-13T22:58:42.733200+0000","last_became_peered":"2024-09-13T22:58:42.733200+0000","last_unstale":"2024-09-13T22:58:58.164038+0000","last_undegraded":"2024-09-13T22:58:58.164038+0000","last_fullsized":"2024-09-13T22:58:58.164038+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'29","last_scrub_stamp":"2024-09-13T22:57:45.567559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:45.567559+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:02.697633+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":801,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161006+0000","last_change":"2024-09-13T22:58:42.742254+0000","last_active":"2024-09-13T22:58:58.161006+0000","last_peered":"2024-09-13T22:58:58.161006+0000","last_clean":"2024-09-13T22:58:58.161006+0000","last_became_active":"2024-09-13T22:58:42.741601+0000","last_became_peered":"2024-09-13T22:58:42.741601+0000","last_unstale":"2024-09-13T22:58:58.161006+0000","last_undegraded":"2024-09-13T22:58:58.161006+0000","last_fullsized":"2024-09-13T22:58:58.161006+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:17.058474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:58:17.058474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:24.020245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":804,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160816+0000","last_change":"2024-09-13T22:58:42.740841+0000","last_active":"2024-09-13T22:58:58.160816+0000","last_peered":"2024-09-13T22:58:58.160816+0000","last_clean":"2024-09-13T22:58:58.160816+0000","last_became_active":"2024-09-13T22:58:42.740401+0000","last_became_peered":"2024-09-13T22:58:42.740401+0000","last_unstale":"2024-09-13T22:58:58.160816+0000","last_undegraded":"2024-09-13T22:58:58.160816+0000","last_fullsized":"2024-09-13T22:58:58.160816+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.483697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.483697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":782,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158681+0000","last_change":"2024-09-13T22:58:42.746942+0000","last_active":"2024-09-13T22:58:58.158681+0000","last_peered":"2024-09-13T22:58:58.158681+0000","last_clean":"2024-09-13T22:58:58.158681+0000","last_became_active":"2024-09-13T22:58:42.746572+0000","last_became_peered":"2024-09-13T22:58:42.746572+0000","last_unstale":"2024-09-13T22:58:58.158681+0000","last_undegraded":"2024-09-13T22:58:58.158681+0000","last_fullsized":"2024-09-13T22:58:58.158681+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:56.539208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.539208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"420'3968","reported_seq":4030,"reported_epoch":420,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T22:59:03.772125+0000","last_change":"2024-09-13T22:58:43.999316+0000","last_active":"2024-09-13T22:59:03.772125+0000","last_peered":"2024-09-13T22:59:03.772125+0000","last_clean":"2024-09-13T22:58:41.690313+0000","last_became_active":"2024-09-13T22:58:43.991578+0000","last_became_peered":"2024-09-13T22:58:43.991578+0000","last_unstale":"2024-09-13T22:59:03.772125+0000","last_undegraded":"2024-09-13T22:58:43.904905+0000","last_fullsized":"2024-09-13T22:58:43.702881+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":3968,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3,5","objects":2908},{"shards":"3,5","objects":1060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1070,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158183+0000","last_change":"2024-09-13T22:56:54.138074+0000","last_active":"2024-09-13T22:58:58.158183+0000","last_peered":"2024-09-13T22:58:58.158183+0000","last_clean":"2024-09-13T22:58:58.158183+0000","last_became_active":"2024-09-13T22:55:41.103393+0000","last_became_peered":"2024-09-13T22:55:41.103393+0000","last_unstale":"2024-09-13T22:58:58.158183+0000","last_undegraded":"2024-09-13T22:58:58.158183+0000","last_fullsized":"2024-09-13T22:58:58.158183+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:56:54.137952+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:56:54.137952+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":127,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160033+0000","last_change":"2024-09-13T22:58:42.777890+0000","last_active":"2024-09-13T22:58:58.160033+0000","last_peered":"2024-09-13T22:58:58.160033+0000","last_clean":"2024-09-13T22:58:58.160033+0000","last_became_active":"2024-09-13T22:58:42.727647+0000","last_became_peered":"2024-09-13T22:58:42.727647+0000","last_unstale":"2024-09-13T22:58:58.160033+0000","last_undegraded":"2024-09-13T22:58:58.160033+0000","last_fullsized":"2024-09-13T22:58:58.160033+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'38","last_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.760160+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"75'1","reported_seq":822,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164474+0000","last_change":"2024-09-13T22:57:48.508060+0000","last_active":"2024-09-13T22:58:58.164474+0000","last_peered":"2024-09-13T22:58:58.164474+0000","last_clean":"2024-09-13T22:58:58.164474+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T22:58:58.164474+0000","last_undegraded":"2024-09-13T22:58:58.164474+0000","last_fullsized":"2024-09-13T22:58:58.164474+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:57:48.507763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.507763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:16.560730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":805,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164484+0000","last_change":"2024-09-13T22:58:15.506098+0000","last_active":"2024-09-13T22:58:58.164484+0000","last_peered":"2024-09-13T22:58:58.164484+0000","last_clean":"2024-09-13T22:58:58.164484+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T22:58:58.164484+0000","last_undegraded":"2024-09-13T22:58:58.164484+0000","last_fullsized":"2024-09-13T22:58:58.164484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.505982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.505982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:40.780684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":765,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161431+0000","last_change":"2024-09-13T22:58:42.766673+0000","last_active":"2024-09-13T22:58:58.161431+0000","last_peered":"2024-09-13T22:58:58.161431+0000","last_clean":"2024-09-13T22:58:58.161431+0000","last_became_active":"2024-09-13T22:58:42.764039+0000","last_became_peered":"2024-09-13T22:58:42.764039+0000","last_unstale":"2024-09-13T22:58:58.161431+0000","last_undegraded":"2024-09-13T22:58:58.161431+0000","last_fullsized":"2024-09-13T22:58:58.161431+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:18.683688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:18.683688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"420'3958","reported_seq":4015,"reported_epoch":420,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T22:59:03.926066+0000","last_change":"2024-09-13T22:58:44.054081+0000","last_active":"2024-09-13T22:59:03.926066+0000","last_peered":"2024-09-13T22:59:03.926066+0000","last_clean":"2024-09-13T22:58:41.669910+0000","last_became_active":"2024-09-13T22:58:44.045997+0000","last_became_peered":"2024-09-13T22:58:44.045997+0000","last_unstale":"2024-09-13T22:59:03.926066+0000","last_undegraded":"2024-09-13T22:58:43.833353+0000","last_fullsized":"2024-09-13T22:58:43.782513+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3958,"num_write_kb":253312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,0],"avail_no_missing":["3","0","1"],"object_location_counts":[{"shards":"0,1,3","objects":2979},{"shards":"0,3","objects":979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":870,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.152752+0000","last_change":"2024-09-13T22:58:45.844296+0000","last_active":"2024-09-13T22:58:58.152752+0000","last_peered":"2024-09-13T22:58:58.152752+0000","last_clean":"2024-09-13T22:58:58.152752+0000","last_became_active":"2024-09-13T22:58:45.843305+0000","last_became_peered":"2024-09-13T22:58:45.843305+0000","last_unstale":"2024-09-13T22:58:58.152752+0000","last_undegraded":"2024-09-13T22:58:58.152752+0000","last_fullsized":"2024-09-13T22:58:58.152752+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:52.050564+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T22:57:52.050564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"397'42","reported_seq":915,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.175735+0000","last_change":"2024-09-13T22:58:15.305788+0000","last_active":"2024-09-13T22:58:58.175735+0000","last_peered":"2024-09-13T22:58:58.175735+0000","last_clean":"2024-09-13T22:58:58.175735+0000","last_became_active":"2024-09-13T22:55:38.098695+0000","last_became_peered":"2024-09-13T22:55:38.098695+0000","last_unstale":"2024-09-13T22:58:58.175735+0000","last_undegraded":"2024-09-13T22:58:58.175735+0000","last_fullsized":"2024-09-13T22:58:58.175735+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-09-13T22:58:15.305297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.305297+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:37.735805+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":540,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158200+0000","last_change":"2024-09-13T22:55:11.632270+0000","last_active":"2024-09-13T22:58:58.158200+0000","last_peered":"2024-09-13T22:58:58.158200+0000","last_clean":"2024-09-13T22:58:58.158200+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T22:58:58.158200+0000","last_undegraded":"2024-09-13T22:58:58.158200+0000","last_fullsized":"2024-09-13T22:58:58.158200+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:11.632162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T22:55:11.632162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":772,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161468+0000","last_change":"2024-09-13T22:57:03.031136+0000","last_active":"2024-09-13T22:58:58.161468+0000","last_peered":"2024-09-13T22:58:58.161468+0000","last_clean":"2024-09-13T22:58:58.161468+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T22:58:58.161468+0000","last_undegraded":"2024-09-13T22:58:58.161468+0000","last_fullsized":"2024-09-13T22:58:58.161468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:03.031031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T22:57:03.031031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":769,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150004+0000","last_change":"2024-09-13T22:58:07.109173+0000","last_active":"2024-09-13T22:58:58.150004+0000","last_peered":"2024-09-13T22:58:58.150004+0000","last_clean":"2024-09-13T22:58:58.150004+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T22:58:58.150004+0000","last_undegraded":"2024-09-13T22:58:58.150004+0000","last_fullsized":"2024-09-13T22:58:58.150004+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.109070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:18.659096+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.109070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.174273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"420'3929","reported_seq":3988,"reported_epoch":420,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T22:59:03.774025+0000","last_change":"2024-09-13T22:58:44.008850+0000","last_active":"2024-09-13T22:59:03.774025+0000","last_peered":"2024-09-13T22:59:03.774025+0000","last_clean":"2024-09-13T22:58:41.694610+0000","last_became_active":"2024-09-13T22:58:43.997767+0000","last_became_peered":"2024-09-13T22:58:43.997767+0000","last_unstale":"2024-09-13T22:59:03.774025+0000","last_undegraded":"2024-09-13T22:58:43.851555+0000","last_fullsized":"2024-09-13T22:58:43.702978+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3929,"log_dups_size":0,"ondisk_log_size":3929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257490944,"num_objects":3929,"num_object_clones":0,"num_object_copies":11787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3929,"num_write_kb":251456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":2919},{"shards":"3,5","objects":1010}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":842,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.168642+0000","last_change":"2024-09-13T22:58:42.787694+0000","last_active":"2024-09-13T22:58:58.168642+0000","last_peered":"2024-09-13T22:58:58.168642+0000","last_clean":"2024-09-13T22:58:58.168642+0000","last_became_active":"2024-09-13T22:58:42.746819+0000","last_became_peered":"2024-09-13T22:58:42.746819+0000","last_unstale":"2024-09-13T22:58:58.168642+0000","last_undegraded":"2024-09-13T22:58:58.168642+0000","last_fullsized":"2024-09-13T22:58:58.168642+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:58:09.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.777653+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.088363+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":781,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160724+0000","last_change":"2024-09-13T22:56:44.954891+0000","last_active":"2024-09-13T22:58:58.160724+0000","last_peered":"2024-09-13T22:58:58.160724+0000","last_clean":"2024-09-13T22:58:58.160724+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T22:58:58.160724+0000","last_undegraded":"2024-09-13T22:58:58.160724+0000","last_fullsized":"2024-09-13T22:58:58.160724+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:44.954773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:44.954773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1053,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159321+0000","last_change":"2024-09-13T22:58:45.830925+0000","last_active":"2024-09-13T22:58:58.159321+0000","last_peered":"2024-09-13T22:58:58.159321+0000","last_clean":"2024-09-13T22:58:58.159321+0000","last_became_active":"2024-09-13T22:58:45.829938+0000","last_became_peered":"2024-09-13T22:58:45.829938+0000","last_unstale":"2024-09-13T22:58:58.159321+0000","last_undegraded":"2024-09-13T22:58:58.159321+0000","last_fullsized":"2024-09-13T22:58:58.159321+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:55:26.889228+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:55:26.889228+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":757,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.169258+0000","last_change":"2024-09-13T22:55:32.846798+0000","last_active":"2024-09-13T22:58:58.169258+0000","last_peered":"2024-09-13T22:58:58.169258+0000","last_clean":"2024-09-13T22:58:58.169258+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T22:58:58.169258+0000","last_undegraded":"2024-09-13T22:58:58.169258+0000","last_fullsized":"2024-09-13T22:58:58.169258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:32.846649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T22:55:32.846649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":786,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161388+0000","last_change":"2024-09-13T22:56:33.129911+0000","last_active":"2024-09-13T22:58:58.161388+0000","last_peered":"2024-09-13T22:58:58.161388+0000","last_clean":"2024-09-13T22:58:58.161388+0000","last_became_active":"2024-09-13T22:55:38.101749+0000","last_became_peered":"2024-09-13T22:55:38.101749+0000","last_unstale":"2024-09-13T22:58:58.161388+0000","last_undegraded":"2024-09-13T22:58:58.161388+0000","last_fullsized":"2024-09-13T22:58:58.161388+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:33.129825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:33.129825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"420'3969","reported_seq":4027,"reported_epoch":420,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T22:59:04.046088+0000","last_change":"2024-09-13T22:58:44.007213+0000","last_active":"2024-09-13T22:59:04.046088+0000","last_peered":"2024-09-13T22:59:04.046088+0000","last_clean":"2024-09-13T22:58:41.695079+0000","last_became_active":"2024-09-13T22:58:44.001130+0000","last_became_peered":"2024-09-13T22:58:44.001130+0000","last_unstale":"2024-09-13T22:59:04.046088+0000","last_undegraded":"2024-09-13T22:58:43.721528+0000","last_fullsized":"2024-09-13T22:58:43.702774+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":11907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":254016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":2925},{"shards":"5,7","objects":1044}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":878,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164709+0000","last_change":"2024-09-13T22:58:42.803636+0000","last_active":"2024-09-13T22:58:58.164709+0000","last_peered":"2024-09-13T22:58:58.164709+0000","last_clean":"2024-09-13T22:58:58.164709+0000","last_became_active":"2024-09-13T22:58:42.764099+0000","last_became_peered":"2024-09-13T22:58:42.764099+0000","last_unstale":"2024-09-13T22:58:58.164709+0000","last_undegraded":"2024-09-13T22:58:58.164709+0000","last_fullsized":"2024-09-13T22:58:58.164709+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-09-13T22:58:14.318902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.318902+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:26.072332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":813,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.165988+0000","last_change":"2024-09-13T22:58:12.453817+0000","last_active":"2024-09-13T22:58:58.165988+0000","last_peered":"2024-09-13T22:58:58.165988+0000","last_clean":"2024-09-13T22:58:58.165988+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T22:58:58.165988+0000","last_undegraded":"2024-09-13T22:58:58.165988+0000","last_fullsized":"2024-09-13T22:58:58.165988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:12.453747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.453747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.100029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":797,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164253+0000","last_change":"2024-09-13T22:58:07.404829+0000","last_active":"2024-09-13T22:58:58.164253+0000","last_peered":"2024-09-13T22:58:58.164253+0000","last_clean":"2024-09-13T22:58:58.164253+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T22:58:58.164253+0000","last_undegraded":"2024-09-13T22:58:58.164253+0000","last_fullsized":"2024-09-13T22:58:58.164253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.404723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.404723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:35.558468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":744,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159229+0000","last_change":"2024-09-13T22:54:25.819955+0000","last_active":"2024-09-13T22:58:58.159229+0000","last_peered":"2024-09-13T22:58:58.159229+0000","last_clean":"2024-09-13T22:58:58.159229+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T22:58:58.159229+0000","last_undegraded":"2024-09-13T22:58:58.159229+0000","last_fullsized":"2024-09-13T22:58:58.159229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:54:25.819823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T22:54:25.819823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"420'3966","reported_seq":7718,"reported_epoch":420,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T22:59:04.108254+0000","last_change":"2024-09-13T22:58:44.021198+0000","last_active":"2024-09-13T22:59:04.108254+0000","last_peered":"2024-09-13T22:59:04.108254+0000","last_clean":"2024-09-13T22:58:41.696868+0000","last_became_active":"2024-09-13T22:58:43.992072+0000","last_became_peered":"2024-09-13T22:58:43.992072+0000","last_unstale":"2024-09-13T22:59:04.108254+0000","last_undegraded":"2024-09-13T22:59:04.108254+0000","last_fullsized":"2024-09-13T22:58:43.704046+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259915776,"num_objects":3966,"num_object_clones":0,"num_object_copies":11898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1083,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3966,"num_write_kb":253824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1847,"num_bytes_recovered":121044992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,2],"avail_no_missing":["7","1","2"],"object_location_counts":[{"shards":"1,2,7","objects":1083},{"shards":"2,7","objects":2883}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"397'30","reported_seq":857,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161962+0000","last_change":"2024-09-13T22:58:42.795628+0000","last_active":"2024-09-13T22:58:58.161962+0000","last_peered":"2024-09-13T22:58:58.161962+0000","last_clean":"2024-09-13T22:58:58.161962+0000","last_became_active":"2024-09-13T22:58:42.731142+0000","last_became_peered":"2024-09-13T22:58:42.731142+0000","last_unstale":"2024-09-13T22:58:58.161962+0000","last_undegraded":"2024-09-13T22:58:58.161962+0000","last_fullsized":"2024-09-13T22:58:58.161962+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:57:53.059034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.059034+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":798,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.151093+0000","last_change":"2024-09-13T22:57:07.882465+0000","last_active":"2024-09-13T22:58:58.151093+0000","last_peered":"2024-09-13T22:58:58.151093+0000","last_clean":"2024-09-13T22:58:58.151093+0000","last_became_active":"2024-09-13T22:55:38.093381+0000","last_became_peered":"2024-09-13T22:55:38.093381+0000","last_unstale":"2024-09-13T22:58:58.151093+0000","last_undegraded":"2024-09-13T22:58:58.151093+0000","last_fullsized":"2024-09-13T22:58:58.151093+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.882167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.882167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":493,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.167213+0000","last_change":"2024-09-13T22:55:38.098852+0000","last_active":"2024-09-13T22:58:58.167213+0000","last_peered":"2024-09-13T22:58:58.167213+0000","last_clean":"2024-09-13T22:58:58.167213+0000","last_became_active":"2024-09-13T22:55:38.098515+0000","last_became_peered":"2024-09-13T22:55:38.098515+0000","last_unstale":"2024-09-13T22:58:58.167213+0000","last_undegraded":"2024-09-13T22:58:58.167213+0000","last_fullsized":"2024-09-13T22:58:58.167213+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:15.398485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:15.398485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":777,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.174220+0000","last_change":"2024-09-13T22:58:16.384185+0000","last_active":"2024-09-13T22:58:58.174220+0000","last_peered":"2024-09-13T22:58:58.174220+0000","last_clean":"2024-09-13T22:58:58.174220+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T22:58:58.174220+0000","last_undegraded":"2024-09-13T22:58:58.174220+0000","last_fullsized":"2024-09-13T22:58:58.174220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:16.383923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:58:16.383923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:33.067080+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"420'4195","reported_seq":4251,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:08.674372+0000","last_change":"2024-09-13T22:57:49.646353+0000","last_active":"2024-09-13T22:59:08.674372+0000","last_peered":"2024-09-13T22:59:08.674372+0000","last_clean":"2024-09-13T22:59:08.674372+0000","last_became_active":"2024-09-13T22:57:49.645497+0000","last_became_peered":"2024-09-13T22:57:49.645497+0000","last_unstale":"2024-09-13T22:59:08.674372+0000","last_undegraded":"2024-09-13T22:59:08.674372+0000","last_fullsized":"2024-09-13T22:59:08.674372+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.493617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274923520,"num_objects":4195,"num_object_clones":0,"num_object_copies":12585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":268480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":894,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157810+0000","last_change":"2024-09-13T22:56:57.107636+0000","last_active":"2024-09-13T22:58:58.157810+0000","last_peered":"2024-09-13T22:58:58.157810+0000","last_clean":"2024-09-13T22:58:58.157810+0000","last_became_active":"2024-09-13T22:55:38.151658+0000","last_became_peered":"2024-09-13T22:55:38.151658+0000","last_unstale":"2024-09-13T22:58:58.157810+0000","last_undegraded":"2024-09-13T22:58:58.157810+0000","last_fullsized":"2024-09-13T22:58:58.157810+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:56:57.107495+0000","last_deep_scrub":"228'34","last_deep_scrub_stamp":"2024-09-13T22:53:27.660042+0000","last_clean_scrub_stamp":"2024-09-13T22:56:57.107495+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":355,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159137+0000","last_change":"2024-09-13T22:55:19.900066+0000","last_active":"2024-09-13T22:58:58.159137+0000","last_peered":"2024-09-13T22:58:58.159137+0000","last_clean":"2024-09-13T22:58:58.159137+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T22:58:58.159137+0000","last_undegraded":"2024-09-13T22:58:58.159137+0000","last_fullsized":"2024-09-13T22:58:58.159137+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:19.899953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:32.054441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:19.899953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"414'16","reported_seq":1076,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157477+0000","last_change":"2024-09-13T22:58:42.788693+0000","last_active":"2024-09-13T22:58:58.157477+0000","last_peered":"2024-09-13T22:58:58.157477+0000","last_clean":"2024-09-13T22:58:58.157477+0000","last_became_active":"2024-09-13T22:58:42.758407+0000","last_became_peered":"2024-09-13T22:58:42.758407+0000","last_unstale":"2024-09-13T22:58:58.157477+0000","last_undegraded":"2024-09-13T22:58:58.157477+0000","last_fullsized":"2024-09-13T22:58:58.157477+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'13","last_scrub_stamp":"2024-09-13T22:54:27.850029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:54:27.850029+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160577+0000","last_change":"2024-09-13T22:58:42.734657+0000","last_active":"2024-09-13T22:58:58.160577+0000","last_peered":"2024-09-13T22:58:58.160577+0000","last_clean":"2024-09-13T22:58:58.160577+0000","last_became_active":"2024-09-13T22:58:42.732057+0000","last_became_peered":"2024-09-13T22:58:42.732057+0000","last_unstale":"2024-09-13T22:58:58.160577+0000","last_undegraded":"2024-09-13T22:58:58.160577+0000","last_fullsized":"2024-09-13T22:58:58.160577+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:03.020207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T22:58:03.020207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.175169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"420'4042","reported_seq":4098,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:06.408878+0000","last_change":"2024-09-13T22:57:49.644003+0000","last_active":"2024-09-13T22:59:06.408878+0000","last_peered":"2024-09-13T22:59:06.408878+0000","last_clean":"2024-09-13T22:59:06.408878+0000","last_became_active":"2024-09-13T22:57:49.643715+0000","last_became_peered":"2024-09-13T22:57:49.643715+0000","last_unstale":"2024-09-13T22:59:06.408878+0000","last_undegraded":"2024-09-13T22:59:06.408878+0000","last_fullsized":"2024-09-13T22:59:06.408878+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":397,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:57:48.602963+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.602963+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.750653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896512,"num_objects":4042,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4042,"num_write_kb":258688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"397'27","reported_seq":865,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158615+0000","last_change":"2024-09-13T22:57:53.254550+0000","last_active":"2024-09-13T22:58:58.158615+0000","last_peered":"2024-09-13T22:58:58.158615+0000","last_clean":"2024-09-13T22:58:58.158615+0000","last_became_active":"2024-09-13T22:55:38.151005+0000","last_became_peered":"2024-09-13T22:55:38.151005+0000","last_unstale":"2024-09-13T22:58:58.158615+0000","last_undegraded":"2024-09-13T22:58:58.158615+0000","last_fullsized":"2024-09-13T22:58:58.158615+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T22:57:53.254254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.254254+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":405,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161483+0000","last_change":"2024-09-13T22:58:01.252621+0000","last_active":"2024-09-13T22:58:58.161483+0000","last_peered":"2024-09-13T22:58:58.161483+0000","last_clean":"2024-09-13T22:58:58.161483+0000","last_became_active":"2024-09-13T22:55:38.136788+0000","last_became_peered":"2024-09-13T22:55:38.136788+0000","last_unstale":"2024-09-13T22:58:58.161483+0000","last_undegraded":"2024-09-13T22:58:58.161483+0000","last_fullsized":"2024-09-13T22:58:58.161483+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:58:01.252317+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:58:01.252317+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:12.966114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":364,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159849+0000","last_change":"2024-09-13T22:57:46.251728+0000","last_active":"2024-09-13T22:58:58.159849+0000","last_peered":"2024-09-13T22:58:58.159849+0000","last_clean":"2024-09-13T22:58:58.159849+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T22:58:58.159849+0000","last_undegraded":"2024-09-13T22:58:58.159849+0000","last_fullsized":"2024-09-13T22:58:58.159849+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:46.251423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.719665+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.251423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:08.732783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":760,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158101+0000","last_change":"2024-09-13T22:57:50.594198+0000","last_active":"2024-09-13T22:58:58.158101+0000","last_peered":"2024-09-13T22:58:58.158101+0000","last_clean":"2024-09-13T22:58:58.158101+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T22:58:58.158101+0000","last_undegraded":"2024-09-13T22:58:58.158101+0000","last_fullsized":"2024-09-13T22:58:58.158101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.594120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.594120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":898,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157287+0000","last_change":"2024-09-13T22:58:42.790531+0000","last_active":"2024-09-13T22:58:58.157287+0000","last_peered":"2024-09-13T22:58:58.157287+0000","last_clean":"2024-09-13T22:58:58.157287+0000","last_became_active":"2024-09-13T22:58:42.758655+0000","last_became_peered":"2024-09-13T22:58:42.758655+0000","last_unstale":"2024-09-13T22:58:58.157287+0000","last_undegraded":"2024-09-13T22:58:58.157287+0000","last_fullsized":"2024-09-13T22:58:58.157287+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'24","last_scrub_stamp":"2024-09-13T22:56:52.280906+0000","last_deep_scrub":"228'24","last_deep_scrub_stamp":"2024-09-13T22:53:21.566725+0000","last_clean_scrub_stamp":"2024-09-13T22:56:52.280906+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":805,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164763+0000","last_change":"2024-09-13T22:56:57.677342+0000","last_active":"2024-09-13T22:58:58.164763+0000","last_peered":"2024-09-13T22:58:58.164763+0000","last_clean":"2024-09-13T22:58:58.164763+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T22:58:58.164763+0000","last_undegraded":"2024-09-13T22:58:58.164763+0000","last_fullsized":"2024-09-13T22:58:58.164763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:57.675268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T22:56:57.675268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":755,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162287+0000","last_change":"2024-09-13T22:55:34.729429+0000","last_active":"2024-09-13T22:58:58.162287+0000","last_peered":"2024-09-13T22:58:58.162287+0000","last_clean":"2024-09-13T22:58:58.162287+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T22:58:58.162287+0000","last_undegraded":"2024-09-13T22:58:58.162287+0000","last_fullsized":"2024-09-13T22:58:58.162287+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:34.729322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:55:34.729322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":819,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.165597+0000","last_change":"2024-09-13T22:57:01.727570+0000","last_active":"2024-09-13T22:58:58.165597+0000","last_peered":"2024-09-13T22:58:58.165597+0000","last_clean":"2024-09-13T22:58:58.165597+0000","last_became_active":"2024-09-13T22:55:38.126360+0000","last_became_peered":"2024-09-13T22:55:38.126360+0000","last_unstale":"2024-09-13T22:58:58.165597+0000","last_undegraded":"2024-09-13T22:58:58.165597+0000","last_fullsized":"2024-09-13T22:58:58.165597+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:01.727464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T22:57:01.727464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":29,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161673+0000","last_change":"2024-09-13T22:58:42.758306+0000","last_active":"2024-09-13T22:58:58.161673+0000","last_peered":"2024-09-13T22:58:58.161673+0000","last_clean":"2024-09-13T22:58:58.161673+0000","last_became_active":"2024-09-13T22:58:42.736178+0000","last_became_peered":"2024-09-13T22:58:42.736178+0000","last_unstale":"2024-09-13T22:58:58.161673+0000","last_undegraded":"2024-09-13T22:58:58.161673+0000","last_fullsized":"2024-09-13T22:58:58.161673+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T22:57:54.173839+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.173839+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:58.741530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162275+0000","last_change":"2024-09-13T22:58:42.745550+0000","last_active":"2024-09-13T22:58:58.162275+0000","last_peered":"2024-09-13T22:58:58.162275+0000","last_clean":"2024-09-13T22:58:58.162275+0000","last_became_active":"2024-09-13T22:58:42.744948+0000","last_became_peered":"2024-09-13T22:58:42.744948+0000","last_unstale":"2024-09-13T22:58:58.162275+0000","last_undegraded":"2024-09-13T22:58:58.162275+0000","last_fullsized":"2024-09-13T22:58:58.162275+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:51.610149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T22:56:51.610149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":771,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161331+0000","last_change":"2024-09-13T22:58:04.051815+0000","last_active":"2024-09-13T22:58:58.161331+0000","last_peered":"2024-09-13T22:58:58.161331+0000","last_clean":"2024-09-13T22:58:58.161331+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T22:58:58.161331+0000","last_undegraded":"2024-09-13T22:58:58.161331+0000","last_fullsized":"2024-09-13T22:58:58.161331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.051696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.051696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.541486+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":781,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161347+0000","last_change":"2024-09-13T22:58:42.769155+0000","last_active":"2024-09-13T22:58:58.161347+0000","last_peered":"2024-09-13T22:58:58.161347+0000","last_clean":"2024-09-13T22:58:58.161347+0000","last_became_active":"2024-09-13T22:58:42.768571+0000","last_became_peered":"2024-09-13T22:58:42.768571+0000","last_unstale":"2024-09-13T22:58:58.161347+0000","last_undegraded":"2024-09-13T22:58:58.161347+0000","last_fullsized":"2024-09-13T22:58:58.161347+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.349704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.349704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:37.274401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":893,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158462+0000","last_change":"2024-09-13T22:58:42.792666+0000","last_active":"2024-09-13T22:58:58.158462+0000","last_peered":"2024-09-13T22:58:58.158462+0000","last_clean":"2024-09-13T22:58:58.158462+0000","last_became_active":"2024-09-13T22:58:42.759372+0000","last_became_peered":"2024-09-13T22:58:42.759372+0000","last_unstale":"2024-09-13T22:58:58.158462+0000","last_undegraded":"2024-09-13T22:58:58.158462+0000","last_fullsized":"2024-09-13T22:58:58.158462+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'19","last_scrub_stamp":"2024-09-13T22:55:30.790120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:30.790120+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":783,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160904+0000","last_change":"2024-09-13T22:56:35.136404+0000","last_active":"2024-09-13T22:58:58.160904+0000","last_peered":"2024-09-13T22:58:58.160904+0000","last_clean":"2024-09-13T22:58:58.160904+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T22:58:58.160904+0000","last_undegraded":"2024-09-13T22:58:58.160904+0000","last_fullsized":"2024-09-13T22:58:58.160904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:35.136329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T22:56:35.136329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":1043,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:59:01.803645+0000","last_change":"2024-09-13T22:56:36.091025+0000","last_active":"2024-09-13T22:59:01.803645+0000","last_peered":"2024-09-13T22:59:01.803645+0000","last_clean":"2024-09-13T22:59:01.803645+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T22:59:01.803645+0000","last_undegraded":"2024-09-13T22:59:01.803645+0000","last_fullsized":"2024-09-13T22:59:01.803645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T22:56:36.090908+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T22:53:18.805172+0000","last_clean_scrub_stamp":"2024-09-13T22:56:36.090908+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":768,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161164+0000","last_change":"2024-09-13T22:57:55.222402+0000","last_active":"2024-09-13T22:58:58.161164+0000","last_peered":"2024-09-13T22:58:58.161164+0000","last_clean":"2024-09-13T22:58:58.161164+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T22:58:58.161164+0000","last_undegraded":"2024-09-13T22:58:58.161164+0000","last_fullsized":"2024-09-13T22:58:58.161164+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:55.222315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.222315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.342997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":811,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161879+0000","last_change":"2024-09-13T22:57:53.159155+0000","last_active":"2024-09-13T22:58:58.161879+0000","last_peered":"2024-09-13T22:58:58.161879+0000","last_clean":"2024-09-13T22:58:58.161879+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T22:58:58.161879+0000","last_undegraded":"2024-09-13T22:58:58.161879+0000","last_fullsized":"2024-09-13T22:58:58.161879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T22:57:53.159058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.159058+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:58:59.582286+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161323+0000","last_change":"2024-09-13T22:55:38.138681+0000","last_active":"2024-09-13T22:58:58.161323+0000","last_peered":"2024-09-13T22:58:58.161323+0000","last_clean":"2024-09-13T22:58:58.161323+0000","last_became_active":"2024-09-13T22:55:38.138035+0000","last_became_peered":"2024-09-13T22:55:38.138035+0000","last_unstale":"2024-09-13T22:58:58.161323+0000","last_undegraded":"2024-09-13T22:58:58.161323+0000","last_fullsized":"2024-09-13T22:58:58.161323+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:36.129393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:36.129393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":1031,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162737+0000","last_change":"2024-09-13T22:58:14.754552+0000","last_active":"2024-09-13T22:58:58.162737+0000","last_peered":"2024-09-13T22:58:58.162737+0000","last_clean":"2024-09-13T22:58:58.162737+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T22:58:58.162737+0000","last_undegraded":"2024-09-13T22:58:58.162737+0000","last_fullsized":"2024-09-13T22:58:58.162737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:58:14.754430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.754430+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:23.336001+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":289,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.164956+0000","last_change":"2024-09-13T22:57:47.490067+0000","last_active":"2024-09-13T22:58:58.164956+0000","last_peered":"2024-09-13T22:58:58.164956+0000","last_clean":"2024-09-13T22:58:58.164956+0000","last_became_active":"2024-09-13T22:55:38.098486+0000","last_became_peered":"2024-09-13T22:55:38.098486+0000","last_unstale":"2024-09-13T22:58:58.164956+0000","last_undegraded":"2024-09-13T22:58:58.164956+0000","last_fullsized":"2024-09-13T22:58:58.164956+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.489929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.489929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":841,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161605+0000","last_change":"2024-09-13T22:55:38.169420+0000","last_active":"2024-09-13T22:58:58.161605+0000","last_peered":"2024-09-13T22:58:58.161605+0000","last_clean":"2024-09-13T22:58:58.161605+0000","last_became_active":"2024-09-13T22:55:38.144744+0000","last_became_peered":"2024-09-13T22:55:38.144744+0000","last_unstale":"2024-09-13T22:58:58.161605+0000","last_undegraded":"2024-09-13T22:58:58.161605+0000","last_fullsized":"2024-09-13T22:58:58.161605+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:55:28.849442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:28.849442+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.163031+0000","last_change":"2024-09-13T22:58:42.749921+0000","last_active":"2024-09-13T22:58:58.163031+0000","last_peered":"2024-09-13T22:58:58.163031+0000","last_clean":"2024-09-13T22:58:58.163031+0000","last_became_active":"2024-09-13T22:58:42.749062+0000","last_became_peered":"2024-09-13T22:58:42.749062+0000","last_unstale":"2024-09-13T22:58:58.163031+0000","last_undegraded":"2024-09-13T22:58:58.163031+0000","last_fullsized":"2024-09-13T22:58:58.163031+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:54.689197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.689197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.021866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":756,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158158+0000","last_change":"2024-09-13T22:55:27.845977+0000","last_active":"2024-09-13T22:58:58.158158+0000","last_peered":"2024-09-13T22:58:58.158158+0000","last_clean":"2024-09-13T22:58:58.158158+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T22:58:58.158158+0000","last_undegraded":"2024-09-13T22:58:58.158158+0000","last_fullsized":"2024-09-13T22:58:58.158158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:27.845519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T22:55:27.845519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":769,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159921+0000","last_change":"2024-09-13T22:58:13.326329+0000","last_active":"2024-09-13T22:58:58.159921+0000","last_peered":"2024-09-13T22:58:58.159921+0000","last_clean":"2024-09-13T22:58:58.159921+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T22:58:58.159921+0000","last_undegraded":"2024-09-13T22:58:58.159921+0000","last_fullsized":"2024-09-13T22:58:58.159921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.326193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.326193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:43.161929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":397,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.169194+0000","last_change":"2024-09-13T22:58:42.799745+0000","last_active":"2024-09-13T22:58:58.169194+0000","last_peered":"2024-09-13T22:58:58.169194+0000","last_clean":"2024-09-13T22:58:58.169194+0000","last_became_active":"2024-09-13T22:58:42.767781+0000","last_became_peered":"2024-09-13T22:58:42.767781+0000","last_unstale":"2024-09-13T22:58:58.169194+0000","last_undegraded":"2024-09-13T22:58:58.169194+0000","last_fullsized":"2024-09-13T22:58:58.169194+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'20","last_scrub_stamp":"2024-09-13T22:54:30.095316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:54:30.095316+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.151019+0000","last_change":"2024-09-13T22:58:00.147479+0000","last_active":"2024-09-13T22:58:58.151019+0000","last_peered":"2024-09-13T22:58:58.151019+0000","last_clean":"2024-09-13T22:58:58.151019+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T22:58:58.151019+0000","last_undegraded":"2024-09-13T22:58:58.151019+0000","last_fullsized":"2024-09-13T22:58:58.151019+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:00.147366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T22:58:00.147366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.200027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":772,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.149969+0000","last_change":"2024-09-13T22:58:03.162993+0000","last_active":"2024-09-13T22:58:58.149969+0000","last_peered":"2024-09-13T22:58:58.149969+0000","last_clean":"2024-09-13T22:58:58.149969+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T22:58:58.149969+0000","last_undegraded":"2024-09-13T22:58:58.149969+0000","last_fullsized":"2024-09-13T22:58:58.149969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:03.162920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T22:58:03.162920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:22.631813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":752,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160919+0000","last_change":"2024-09-13T22:57:07.530820+0000","last_active":"2024-09-13T22:58:58.160919+0000","last_peered":"2024-09-13T22:58:58.160919+0000","last_clean":"2024-09-13T22:58:58.160919+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T22:58:58.160919+0000","last_undegraded":"2024-09-13T22:58:58.160919+0000","last_fullsized":"2024-09-13T22:58:58.160919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.530747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.530747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"397'18","reported_seq":811,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161401+0000","last_change":"2024-09-13T22:58:14.333753+0000","last_active":"2024-09-13T22:58:58.161401+0000","last_peered":"2024-09-13T22:58:58.161401+0000","last_clean":"2024-09-13T22:58:58.161401+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T22:58:58.161401+0000","last_undegraded":"2024-09-13T22:58:58.161401+0000","last_fullsized":"2024-09-13T22:58:58.161401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'18","last_scrub_stamp":"2024-09-13T22:58:14.333490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.333490+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:39.855178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161613+0000","last_change":"2024-09-13T22:58:42.727064+0000","last_active":"2024-09-13T22:58:58.161613+0000","last_peered":"2024-09-13T22:58:58.161613+0000","last_clean":"2024-09-13T22:58:58.161613+0000","last_became_active":"2024-09-13T22:58:42.726569+0000","last_became_peered":"2024-09-13T22:58:42.726569+0000","last_unstale":"2024-09-13T22:58:58.161613+0000","last_undegraded":"2024-09-13T22:58:58.161613+0000","last_fullsized":"2024-09-13T22:58:58.161613+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:00.019958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:00.019958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:06.684228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161557+0000","last_change":"2024-09-13T22:58:42.752271+0000","last_active":"2024-09-13T22:58:58.161557+0000","last_peered":"2024-09-13T22:58:58.161557+0000","last_clean":"2024-09-13T22:58:58.161557+0000","last_became_active":"2024-09-13T22:58:42.748403+0000","last_became_peered":"2024-09-13T22:58:42.748403+0000","last_unstale":"2024-09-13T22:58:58.161557+0000","last_undegraded":"2024-09-13T22:58:58.161557+0000","last_fullsized":"2024-09-13T22:58:58.161557+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.200989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.200989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:26.086950+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.157297+0000","last_change":"2024-09-13T22:57:10.040687+0000","last_active":"2024-09-13T22:58:58.157297+0000","last_peered":"2024-09-13T22:58:58.157297+0000","last_clean":"2024-09-13T22:58:58.157297+0000","last_became_active":"2024-09-13T22:55:38.135406+0000","last_became_peered":"2024-09-13T22:55:38.135406+0000","last_unstale":"2024-09-13T22:58:58.157297+0000","last_undegraded":"2024-09-13T22:58:58.157297+0000","last_fullsized":"2024-09-13T22:58:58.157297+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:10.040498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:10.040498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":809,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161469+0000","last_change":"2024-09-13T22:57:46.014205+0000","last_active":"2024-09-13T22:58:58.161469+0000","last_peered":"2024-09-13T22:58:58.161469+0000","last_clean":"2024-09-13T22:58:58.161469+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T22:58:58.161469+0000","last_undegraded":"2024-09-13T22:58:58.161469+0000","last_fullsized":"2024-09-13T22:58:58.161469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:57:46.013979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.013979+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161124+0000","last_change":"2024-09-13T22:58:42.742636+0000","last_active":"2024-09-13T22:58:58.161124+0000","last_peered":"2024-09-13T22:58:58.161124+0000","last_clean":"2024-09-13T22:58:58.161124+0000","last_became_active":"2024-09-13T22:58:42.742229+0000","last_became_peered":"2024-09-13T22:58:42.742229+0000","last_unstale":"2024-09-13T22:58:58.161124+0000","last_undegraded":"2024-09-13T22:58:58.161124+0000","last_fullsized":"2024-09-13T22:58:58.161124+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.134261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.134261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:33.799429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":786,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158223+0000","last_change":"2024-09-13T22:58:42.764230+0000","last_active":"2024-09-13T22:58:58.158223+0000","last_peered":"2024-09-13T22:58:58.158223+0000","last_clean":"2024-09-13T22:58:58.158223+0000","last_became_active":"2024-09-13T22:58:42.763883+0000","last_became_peered":"2024-09-13T22:58:42.763883+0000","last_unstale":"2024-09-13T22:58:58.158223+0000","last_undegraded":"2024-09-13T22:58:58.158223+0000","last_fullsized":"2024-09-13T22:58:58.158223+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:55.145267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:56:55.145267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":772,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158196+0000","last_change":"2024-09-13T22:58:42.749489+0000","last_active":"2024-09-13T22:58:58.158196+0000","last_peered":"2024-09-13T22:58:58.158196+0000","last_clean":"2024-09-13T22:58:58.158196+0000","last_became_active":"2024-09-13T22:58:42.749118+0000","last_became_peered":"2024-09-13T22:58:42.749118+0000","last_unstale":"2024-09-13T22:58:58.158196+0000","last_undegraded":"2024-09-13T22:58:58.158196+0000","last_fullsized":"2024-09-13T22:58:58.158196+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:59.732688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:57:59.732688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:21.260894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.169301+0000","last_change":"2024-09-13T22:56:53.116102+0000","last_active":"2024-09-13T22:58:58.169301+0000","last_peered":"2024-09-13T22:58:58.169301+0000","last_clean":"2024-09-13T22:58:58.169301+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T22:58:58.169301+0000","last_undegraded":"2024-09-13T22:58:58.169301+0000","last_fullsized":"2024-09-13T22:58:58.169301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_clean_scrub_stamp":"2024-09-13T22:56:53.115999+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":123,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.163977+0000","last_change":"2024-09-13T22:58:42.785378+0000","last_active":"2024-09-13T22:58:58.163977+0000","last_peered":"2024-09-13T22:58:58.163977+0000","last_clean":"2024-09-13T22:58:58.163977+0000","last_became_active":"2024-09-13T22:58:42.740500+0000","last_became_peered":"2024-09-13T22:58:42.740500+0000","last_unstale":"2024-09-13T22:58:58.163977+0000","last_undegraded":"2024-09-13T22:58:58.163977+0000","last_fullsized":"2024-09-13T22:58:58.163977+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-13T22:57:10.942679+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T22:57:10.942679+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":809,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160187+0000","last_change":"2024-09-13T22:58:42.747672+0000","last_active":"2024-09-13T22:58:58.160187+0000","last_peered":"2024-09-13T22:58:58.160187+0000","last_clean":"2024-09-13T22:58:58.160187+0000","last_became_active":"2024-09-13T22:58:42.746984+0000","last_became_peered":"2024-09-13T22:58:42.746984+0000","last_unstale":"2024-09-13T22:58:58.160187+0000","last_undegraded":"2024-09-13T22:58:58.160187+0000","last_fullsized":"2024-09-13T22:58:58.160187+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:34.086426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T22:56:34.086426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"414'17","reported_seq":32,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161342+0000","last_change":"2024-09-13T22:58:42.768259+0000","last_active":"2024-09-13T22:58:58.161342+0000","last_peered":"2024-09-13T22:58:58.161342+0000","last_clean":"2024-09-13T22:58:58.161342+0000","last_became_active":"2024-09-13T22:58:42.743192+0000","last_became_peered":"2024-09-13T22:58:42.743192+0000","last_unstale":"2024-09-13T22:58:58.161342+0000","last_undegraded":"2024-09-13T22:58:58.161342+0000","last_fullsized":"2024-09-13T22:58:58.161342+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'14","last_scrub_stamp":"2024-09-13T22:55:25.529898+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T22:55:25.529898+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150738+0000","last_change":"2024-09-13T22:57:09.960429+0000","last_active":"2024-09-13T22:58:58.150738+0000","last_peered":"2024-09-13T22:58:58.150738+0000","last_clean":"2024-09-13T22:58:58.150738+0000","last_became_active":"2024-09-13T22:55:38.096392+0000","last_became_peered":"2024-09-13T22:55:38.096392+0000","last_unstale":"2024-09-13T22:58:58.150738+0000","last_undegraded":"2024-09-13T22:58:58.150738+0000","last_fullsized":"2024-09-13T22:58:58.150738+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:09.960363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:09.960363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":809,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.163178+0000","last_change":"2024-09-13T22:57:37.364148+0000","last_active":"2024-09-13T22:58:58.163178+0000","last_peered":"2024-09-13T22:58:58.163178+0000","last_clean":"2024-09-13T22:58:58.163178+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T22:58:58.163178+0000","last_undegraded":"2024-09-13T22:58:58.163178+0000","last_fullsized":"2024-09-13T22:58:58.163178+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'15","last_scrub_stamp":"2024-09-13T22:55:35.746045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:35.746045+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":776,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162916+0000","last_change":"2024-09-13T22:58:42.792046+0000","last_active":"2024-09-13T22:58:58.162916+0000","last_peered":"2024-09-13T22:58:58.162916+0000","last_clean":"2024-09-13T22:58:58.162916+0000","last_became_active":"2024-09-13T22:58:42.748914+0000","last_became_peered":"2024-09-13T22:58:42.748914+0000","last_unstale":"2024-09-13T22:58:58.162916+0000","last_undegraded":"2024-09-13T22:58:58.162916+0000","last_fullsized":"2024-09-13T22:58:58.162916+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T22:54:35.340609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:54:35.340609+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":774,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.161299+0000","last_change":"2024-09-13T22:55:32.033215+0000","last_active":"2024-09-13T22:58:58.161299+0000","last_peered":"2024-09-13T22:58:58.161299+0000","last_clean":"2024-09-13T22:58:58.161299+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T22:58:58.161299+0000","last_undegraded":"2024-09-13T22:58:58.161299+0000","last_fullsized":"2024-09-13T22:58:58.161299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:32.033129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T22:55:32.033129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160834+0000","last_change":"2024-09-13T22:58:42.750451+0000","last_active":"2024-09-13T22:58:58.160834+0000","last_peered":"2024-09-13T22:58:58.160834+0000","last_clean":"2024-09-13T22:58:58.160834+0000","last_became_active":"2024-09-13T22:58:42.748790+0000","last_became_peered":"2024-09-13T22:58:42.748790+0000","last_unstale":"2024-09-13T22:58:58.160834+0000","last_undegraded":"2024-09-13T22:58:58.160834+0000","last_fullsized":"2024-09-13T22:58:58.160834+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.025258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.025258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:25.719759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":23,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162613+0000","last_change":"2024-09-13T22:58:42.746378+0000","last_active":"2024-09-13T22:58:58.162613+0000","last_peered":"2024-09-13T22:58:58.162613+0000","last_clean":"2024-09-13T22:58:58.162613+0000","last_became_active":"2024-09-13T22:58:42.746020+0000","last_became_peered":"2024-09-13T22:58:42.746020+0000","last_unstale":"2024-09-13T22:58:58.162613+0000","last_undegraded":"2024-09-13T22:58:58.162613+0000","last_fullsized":"2024-09-13T22:58:58.162613+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:01.014315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:01.014315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":838,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.160640+0000","last_change":"2024-09-13T22:58:12.352800+0000","last_active":"2024-09-13T22:58:58.160640+0000","last_peered":"2024-09-13T22:58:58.160640+0000","last_clean":"2024-09-13T22:58:58.160640+0000","last_became_active":"2024-09-13T22:55:38.138369+0000","last_became_peered":"2024-09-13T22:55:38.138369+0000","last_unstale":"2024-09-13T22:58:58.160640+0000","last_undegraded":"2024-09-13T22:58:58.160640+0000","last_fullsized":"2024-09-13T22:58:58.160640+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'11","last_scrub_stamp":"2024-09-13T22:58:12.352675+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.352675+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:37.199068+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":789,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.151002+0000","last_change":"2024-09-13T22:57:57.115157+0000","last_active":"2024-09-13T22:58:58.151002+0000","last_peered":"2024-09-13T22:58:58.151002+0000","last_clean":"2024-09-13T22:58:58.151002+0000","last_became_active":"2024-09-13T22:55:38.094235+0000","last_became_peered":"2024-09-13T22:55:38.094235+0000","last_unstale":"2024-09-13T22:58:58.151002+0000","last_undegraded":"2024-09-13T22:58:58.151002+0000","last_fullsized":"2024-09-13T22:58:58.151002+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:57.114953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:17.313617+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.114953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:19.976193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":774,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.162196+0000","last_change":"2024-09-13T22:58:42.779680+0000","last_active":"2024-09-13T22:58:58.162196+0000","last_peered":"2024-09-13T22:58:58.162196+0000","last_clean":"2024-09-13T22:58:58.162196+0000","last_became_active":"2024-09-13T22:58:42.748249+0000","last_became_peered":"2024-09-13T22:58:42.748249+0000","last_unstale":"2024-09-13T22:58:58.162196+0000","last_undegraded":"2024-09-13T22:58:58.162196+0000","last_fullsized":"2024-09-13T22:58:58.162196+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T22:55:10.933585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:55:10.933585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":775,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.158410+0000","last_change":"2024-09-13T22:58:42.769670+0000","last_active":"2024-09-13T22:58:58.158410+0000","last_peered":"2024-09-13T22:58:58.158410+0000","last_clean":"2024-09-13T22:58:58.158410+0000","last_became_active":"2024-09-13T22:58:42.769318+0000","last_became_peered":"2024-09-13T22:58:42.769318+0000","last_unstale":"2024-09-13T22:58:58.158410+0000","last_undegraded":"2024-09-13T22:58:58.158410+0000","last_fullsized":"2024-09-13T22:58:58.158410+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:24.878070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:55:24.878070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":617,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.150062+0000","last_change":"2024-09-13T22:57:54.101195+0000","last_active":"2024-09-13T22:58:58.150062+0000","last_peered":"2024-09-13T22:58:58.150062+0000","last_clean":"2024-09-13T22:58:58.150062+0000","last_became_active":"2024-09-13T22:55:38.099550+0000","last_became_peered":"2024-09-13T22:55:38.099550+0000","last_unstale":"2024-09-13T22:58:58.150062+0000","last_undegraded":"2024-09-13T22:58:58.150062+0000","last_fullsized":"2024-09-13T22:58:58.150062+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T22:57:54.100973+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.100973+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:59:15.423659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":266,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-13T22:58:58.159320+0000","last_change":"2024-09-13T22:55:38.133820+0000","last_active":"2024-09-13T22:58:58.159320+0000","last_peered":"2024-09-13T22:58:58.159320+0000","last_clean":"2024-09-13T22:58:58.159320+0000","last_became_active":"2024-09-13T22:55:38.133425+0000","last_became_peered":"2024-09-13T22:55:38.133425+0000","last_unstale":"2024-09-13T22:58:58.159320+0000","last_undegraded":"2024-09-13T22:58:58.159320+0000","last_fullsized":"2024-09-13T22:58:58.159320+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:55:28.200985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T22:55:28.200985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4197187584,"num_objects":64044,"num_object_clones":0,"num_object_copies":192132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14757,"num_objects_misplaced":5057,"num_objects_unfound":0,"num_objects_dirty":64044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64044,"num_write_kb":4098816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3740,"num_bytes_recovered":245104640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12859342848,"data_stored":12859342848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64044,"ondisk_log_size":64044,"up":48,"acting":40,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1256,"num_read_kb":1251,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1555,"num_read_kb":1312,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":432,"num_bytes_recovered":8470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":12866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1055,"ondisk_log_size":1055,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809097,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2444936,"kb_used_data":1773752,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91300216,"statfs":{"total":95995035648,"available":93491421184,"internally_reserved":0,"allocated":1816322048,"data_stored":1815766954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":687274442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136148,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335856,"kb_used_data":745184,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92409296,"statfs":{"total":95995035648,"available":94627119104,"internally_reserved":0,"allocated":763068416,"data_stored":762535686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604830150},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528608,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2068792,"kb_used_data":1446184,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91676360,"statfs":{"total":95995035648,"available":93876592640,"internally_reserved":0,"allocated":1480892416,"data_stored":1480352312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888362,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084595,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2786356,"kb_used_data":1966628,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":90958796,"statfs":{"total":95995035648,"available":93141807104,"internally_reserved":0,"allocated":2013827072,"data_stored":2013291439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":839383495},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378943,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2300860,"kb_used_data":1737964,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":91444292,"statfs":{"total":95995035648,"available":93638955008,"internally_reserved":0,"allocated":1779675136,"data_stored":1779131688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575177,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3545936,"kb_used_data":2851136,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":90199216,"statfs":{"total":95995035648,"available":92363997184,"internally_reserved":0,"allocated":2919563264,"data_stored":2918999517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738708,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2708156,"kb_used_data":1995308,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91036996,"statfs":{"total":95995035648,"available":93221883904,"internally_reserved":0,"allocated":2043195392,"data_stored":2042652134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2057240576,"data_stored":2057240576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2929917952,"data_stored":2929917952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1778647040,"data_stored":1778647040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2012610560,"data_stored":2012610560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1481834496,"data_stored":1481834496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":771227648,"data_stored":771227648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1827864576,"data_stored":1827864576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T22:59:20.845 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-13T22:59:20.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:20.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:20 smithi178 bash[29889]: cluster 2024-09-13T22:59:18.940675+0000 mgr.y (mgr.14613) 805 : cluster [DBG] pgmap v955: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 705 B/s rd, 26 MiB/s wr, 420 op/s; 11849/211251 objects degraded (5.609%); 5517/211251 objects misplaced (2.612%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T22:59:20.920 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:20 smithi178 bash[22840]: cluster 2024-09-13T22:59:18.940675+0000 mgr.y (mgr.14613) 805 : cluster [DBG] pgmap v955: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 705 B/s rd, 26 MiB/s wr, 420 op/s; 11849/211251 objects degraded (5.609%); 5517/211251 objects misplaced (2.612%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T22:59:20.938 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:20.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:21.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:21.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:21.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:21.215 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:21.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T22:59:21.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:21.346 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:21.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:21.505 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:21.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:21.664 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:21.723 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 75 16 75129 75113 62.5821 48.6875 0.0297119 0.0159732 2024-09-13T22:59:21.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.802 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:21.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:21.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:21.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:21.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:22.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:22.059 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:22.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:22.191 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:22.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:22.327 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:22.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:22.458 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:22.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:22.590 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:22.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:22.719 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:22.723 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 76 15 75942 75927 62.428 50.875 0.0217416 0.0160108 2024-09-13T22:59:22.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T22:59:22.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:22.915 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:22.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:23.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:23.032 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:22 smithi178 bash[22840]: cluster 2024-09-13T22:59:20.941415+0000 mgr.y (mgr.14613) 806 : cluster [DBG] pgmap v956: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 709 op/s; 11849/219129 objects degraded (5.407%); 5106/219129 objects misplaced (2.330%); 11 MiB/s, 174 objects/s recovering 2024-09-13T22:59:23.032 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:22 smithi178 bash[29889]: cluster 2024-09-13T22:59:20.941415+0000 mgr.y (mgr.14613) 806 : cluster [DBG] pgmap v956: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 709 op/s; 11849/219129 objects degraded (5.407%); 5106/219129 objects misplaced (2.330%); 11 MiB/s, 174 objects/s recovering 2024-09-13T22:59:23.069 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:22 smithi192 bash[24608]: cluster 2024-09-13T22:59:20.941415+0000 mgr.y (mgr.14613) 806 : cluster [DBG] pgmap v956: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 709 op/s; 11849/219129 objects degraded (5.407%); 5106/219129 objects misplaced (2.330%); 11 MiB/s, 174 objects/s recovering 2024-09-13T22:59:23.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:23.209 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:23.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:23.357 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:23.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:23.484 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:23.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:23.627 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:23.723 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 77 16 76724 76708 62.2511 48.8125 0.061386 0.0160552 2024-09-13T22:59:23.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:23.754 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:23.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:23.860 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:23.917 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:23.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:24.023 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:24.044 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:24.063 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:23 smithi178 bash[29889]: cluster 2024-09-13T22:59:23.144724+0000 mon.a (mon.0) 1708 : cluster [WRN] Health check update: Degraded data redundancy: 11849/219129 objects degraded (5.407%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:24.064 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:23 smithi178 bash[22840]: cluster 2024-09-13T22:59:23.144724+0000 mon.a (mon.0) 1708 : cluster [WRN] Health check update: Degraded data redundancy: 11849/219129 objects degraded (5.407%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:24.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:24.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:23 smithi192 bash[24608]: cluster 2024-09-13T22:59:23.144724+0000 mon.a (mon.0) 1708 : cluster [WRN] Health check update: Degraded data redundancy: 11849/219129 objects degraded (5.407%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:24.174 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:24.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:24.307 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:24.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.450 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:24.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:24.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:24.569 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:24.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:24.703 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:24.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 78 16 77427 77411 62.0163 43.9375 0.0232907 0.016118 2024-09-13T22:59:24.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:24 smithi192 bash[24608]: cluster 2024-09-13T22:59:22.941898+0000 mgr.y (mgr.14613) 807 : cluster [DBG] pgmap v957: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 28 MiB/s wr, 455 op/s; 11849/220428 objects degraded (5.375%); 4731/220428 objects misplaced (2.146%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T22:59:24.826 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:24.928 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:24.941 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:24 smithi178 bash[29889]: cluster 2024-09-13T22:59:22.941898+0000 mgr.y (mgr.14613) 807 : cluster [DBG] pgmap v957: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 28 MiB/s wr, 455 op/s; 11849/220428 objects degraded (5.375%); 4731/220428 objects misplaced (2.146%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T22:59:24.942 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:24 smithi178 bash[22840]: cluster 2024-09-13T22:59:22.941898+0000 mgr.y (mgr.14613) 807 : cluster [DBG] pgmap v957: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 28 MiB/s wr, 455 op/s; 11849/220428 objects degraded (5.375%); 4731/220428 objects misplaced (2.146%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T22:59:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:24.975 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:24.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:25.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:25.105 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:25.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:25.234 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:25.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:25.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:25.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:25.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:25.600 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:25.623 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:25.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 79 16 78237 78221 61.8721 50.625 0.0243277 0.0161553 2024-09-13T22:59:25.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:25.764 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:25.846 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T22:59:25.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T22:59:25.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:25.891 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:25.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:25.898 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:25.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.015 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:26.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:26.126 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:59:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:59:26.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:26.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:26.281 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:26.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:26.412 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:26.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.547 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:26.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:26.677 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:26.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:2024-09-13T22:59:26.725892+0000 min lat: 0.00117945 max lat: 2.36455 avg lat: 0.0161964 2024-09-13T22:59:26.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T22:59:26.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 80 16 79029 79013 61.7174 49.5 0.0199199 0.0161964 2024-09-13T22:59:26.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.793 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:26 smithi178 bash[29889]: cluster 2024-09-13T22:59:24.942712+0000 mgr.y (mgr.14613) 808 : cluster [DBG] pgmap v958: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 48 MiB/s wr, 766 op/s; 11849/230712 objects degraded (5.136%); 4259/230712 objects misplaced (1.846%); 11 MiB/s, 172 objects/s recovering 2024-09-13T22:59:26.793 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:26 smithi178 bash[22840]: cluster 2024-09-13T22:59:24.942712+0000 mgr.y (mgr.14613) 808 : cluster [DBG] pgmap v958: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 48 MiB/s wr, 766 op/s; 11849/230712 objects degraded (5.136%); 4259/230712 objects misplaced (1.846%); 11 MiB/s, 172 objects/s recovering 2024-09-13T22:59:26.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:26.918 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:26.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:26.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:27.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:27.060 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:26 smithi192 bash[24608]: cluster 2024-09-13T22:59:24.942712+0000 mgr.y (mgr.14613) 808 : cluster [DBG] pgmap v958: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 48 MiB/s wr, 766 op/s; 11849/230712 objects degraded (5.136%); 4259/230712 objects misplaced (1.846%); 11 MiB/s, 172 objects/s recovering 2024-09-13T22:59:27.074 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:27.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:27.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:27.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:27.322 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:27.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:27.444 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:27.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:27.570 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:27.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:27.697 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:27.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 81 16 79654 79638 61.4376 39.0625 0.00983906 0.0162583 2024-09-13T22:59:27.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:27.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:27.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:27 smithi192 bash[24608]: audit 2024-09-13T22:59:25.938071+0000 mgr.y (mgr.14613) 809 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:27.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:27.944 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:27.945 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:27 smithi178 bash[29889]: audit 2024-09-13T22:59:25.938071+0000 mgr.y (mgr.14613) 809 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:27.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:27 smithi178 bash[22840]: audit 2024-09-13T22:59:25.938071+0000 mgr.y (mgr.14613) 809 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:27.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:28.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:28.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:28.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:28.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:28.197 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:28.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:28.331 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:28.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:28.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:28.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:28.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:28.605 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:28.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:28.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 82 16 80076 80060 61.0101 26.375 0.00724793 0.0163854 2024-09-13T22:59:28.735 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:28.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:28.867 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:28.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:28.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:29.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:29.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:29.132 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:28 smithi192 bash[24608]: cluster 2024-09-13T22:59:26.943212+0000 mgr.y (mgr.14613) 810 : cluster [DBG] pgmap v959: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 718 op/s; 11849/232167 objects degraded (5.104%); 3875/232167 objects misplaced (1.669%); 10 MiB/s, 164 objects/s recovering 2024-09-13T22:59:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:28 smithi192 bash[24608]: cluster 2024-09-13T22:59:28.145643+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check update: Degraded data redundancy: 11849/232167 objects degraded (5.104%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:29.159 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:28 smithi178 bash[22840]: cluster 2024-09-13T22:59:26.943212+0000 mgr.y (mgr.14613) 810 : cluster [DBG] pgmap v959: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 718 op/s; 11849/232167 objects degraded (5.104%); 3875/232167 objects misplaced (1.669%); 10 MiB/s, 164 objects/s recovering 2024-09-13T22:59:29.159 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:28 smithi178 bash[22840]: cluster 2024-09-13T22:59:28.145643+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check update: Degraded data redundancy: 11849/232167 objects degraded (5.104%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:29.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:28 smithi178 bash[29889]: cluster 2024-09-13T22:59:26.943212+0000 mgr.y (mgr.14613) 810 : cluster [DBG] pgmap v959: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 718 op/s; 11849/232167 objects degraded (5.104%); 3875/232167 objects misplaced (1.669%); 10 MiB/s, 164 objects/s recovering 2024-09-13T22:59:29.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:28 smithi178 bash[29889]: cluster 2024-09-13T22:59:28.145643+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check update: Degraded data redundancy: 11849/232167 objects degraded (5.104%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:29.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:29.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:29.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:29.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:29.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:29.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:29.658 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:29.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 83 16 80906 80890 60.9 51.875 0.00605398 0.0164142 2024-09-13T22:59:29.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:29.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:29.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:29.918 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:29.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:29.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:59:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:59:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:59:29.971 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:30.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:30.033 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 22:59:29 smithi192 bash[50858]: logger=cleanup t=2024-09-13T22:59:29.983083173Z level=info msg="Completed cleanup jobs" duration=888.259µs 2024-09-13T22:59:30.047 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:30.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:30.183 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:30.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:30.310 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:30.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:30.440 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:30.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:30.582 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:30.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:30.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 84 16 81716 81700 60.7776 50.625 0.00619456 0.0164455 2024-09-13T22:59:30.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:30.842 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:30.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:30.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:30.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:31.036 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:30 smithi178 bash[29889]: cluster 2024-09-13T22:59:28.944259+0000 mgr.y (mgr.14613) 811 : cluster [DBG] pgmap v960: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 948 B/s rd, 43 MiB/s wr, 686 op/s; 11849/232848 objects degraded (5.089%); 3875/232848 objects misplaced (1.664%); 9.5 MiB/s, 152 objects/s recovering 2024-09-13T22:59:31.036 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:30 smithi178 bash[22840]: cluster 2024-09-13T22:59:28.944259+0000 mgr.y (mgr.14613) 811 : cluster [DBG] pgmap v960: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 948 B/s rd, 43 MiB/s wr, 686 op/s; 11849/232848 objects degraded (5.089%); 3875/232848 objects misplaced (1.664%); 9.5 MiB/s, 152 objects/s recovering 2024-09-13T22:59:31.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:31.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:31.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:30 smithi192 bash[24608]: cluster 2024-09-13T22:59:28.944259+0000 mgr.y (mgr.14613) 811 : cluster [DBG] pgmap v960: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 948 B/s rd, 43 MiB/s wr, 686 op/s; 11849/232848 objects degraded (5.089%); 3875/232848 objects misplaced (1.664%); 9.5 MiB/s, 152 objects/s recovering 2024-09-13T22:59:31.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:31.235 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:31.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:31.365 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:31.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:31.493 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:31.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:31.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:31.724 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 85 16 82390 82374 60.5581 42.125 0.0142603 0.0164925 2024-09-13T22:59:31.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:31.764 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:31.867 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:31.898 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:31.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:31.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:32.021 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:32.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:32.155 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:32.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:32.284 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:32.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:32.412 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:32.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:32.540 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:32.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:32.666 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:32.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 86 16 82551 82535 59.971 10.0625 0.00362086 0.0165045 2024-09-13T22:59:32.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.801 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:32.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:32.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:32.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:32 smithi192 bash[24608]: cluster 2024-09-13T22:59:30.945079+0000 mgr.y (mgr.14613) 812 : cluster [DBG] pgmap v961: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s; 11849/241212 objects degraded (4.912%); 3466/241212 objects misplaced (1.437%); 11 MiB/s, 170 objects/s recovering 2024-09-13T22:59:32.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:32 smithi192 bash[24608]: cluster 2024-09-13T22:59:31.789914+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-09-13T22:59:32.931 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:32.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:33.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:33.056 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:33.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:33.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:32 smithi178 bash[29889]: cluster 2024-09-13T22:59:30.945079+0000 mgr.y (mgr.14613) 812 : cluster [DBG] pgmap v961: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s; 11849/241212 objects degraded (4.912%); 3466/241212 objects misplaced (1.437%); 11 MiB/s, 170 objects/s recovering 2024-09-13T22:59:33.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:32 smithi178 bash[29889]: cluster 2024-09-13T22:59:31.789914+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-09-13T22:59:33.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:32 smithi178 bash[22840]: cluster 2024-09-13T22:59:30.945079+0000 mgr.y (mgr.14613) 812 : cluster [DBG] pgmap v961: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s; 11849/241212 objects degraded (4.912%); 3466/241212 objects misplaced (1.437%); 11 MiB/s, 170 objects/s recovering 2024-09-13T22:59:33.170 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:32 smithi178 bash[22840]: cluster 2024-09-13T22:59:31.789914+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-09-13T22:59:33.187 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:33.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:33.317 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:33.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:33.460 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:33.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:33.592 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:33.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:33.722 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:33.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 87 16 83065 83049 59.6508 32.125 0.0133267 0.0167571 2024-09-13T22:59:33.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:33.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:33 smithi192 bash[24608]: cluster 2024-09-13T22:59:32.801725+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-09-13T22:59:33.835 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:33 smithi192 bash[24608]: cluster 2024-09-13T22:59:33.146960+0000 mon.a (mon.0) 1712 : cluster [WRN] Health check update: Degraded data redundancy: 11849/241212 objects degraded (4.912%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:33.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:33.980 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:33.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:34.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:34.108 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:34.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:33 smithi178 bash[22840]: cluster 2024-09-13T22:59:32.801725+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-09-13T22:59:34.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:33 smithi178 bash[22840]: cluster 2024-09-13T22:59:33.146960+0000 mon.a (mon.0) 1712 : cluster [WRN] Health check update: Degraded data redundancy: 11849/241212 objects degraded (4.912%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:34.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:33 smithi178 bash[29889]: cluster 2024-09-13T22:59:32.801725+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-09-13T22:59:34.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:33 smithi178 bash[29889]: cluster 2024-09-13T22:59:33.146960+0000 mon.a (mon.0) 1712 : cluster [WRN] Health check update: Degraded data redundancy: 11849/241212 objects degraded (4.912%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:34.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:34.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:34.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:34.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:34.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:34.489 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:34.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:34.616 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:34.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:34.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 88 16 83891 83875 59.5595 51.625 0.0225492 0.0167839 2024-09-13T22:59:34.751 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:34.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:34.877 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:34.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:34.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:34.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:34 smithi192 bash[24608]: cluster 2024-09-13T22:59:32.945585+0000 mgr.y (mgr.14613) 813 : cluster [DBG] pgmap v964: 158 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 490 op/s; 11849/242469 objects degraded (4.887%); 2996/242469 objects misplaced (1.236%); 9.9 MiB/s, 157 objects/s recovering 2024-09-13T22:59:34.992 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:34 smithi192 bash[24608]: audit 2024-09-13T22:59:34.486041+0000 mon.b (mon.1) 277 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:35.002 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:35.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:35.115 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:34 smithi178 bash[29889]: cluster 2024-09-13T22:59:32.945585+0000 mgr.y (mgr.14613) 813 : cluster [DBG] pgmap v964: 158 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 490 op/s; 11849/242469 objects degraded (4.887%); 2996/242469 objects misplaced (1.236%); 9.9 MiB/s, 157 objects/s recovering 2024-09-13T22:59:35.115 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:34 smithi178 bash[29889]: audit 2024-09-13T22:59:34.486041+0000 mon.b (mon.1) 277 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:35.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:34 smithi178 bash[22840]: cluster 2024-09-13T22:59:32.945585+0000 mgr.y (mgr.14613) 813 : cluster [DBG] pgmap v964: 158 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 490 op/s; 11849/242469 objects degraded (4.887%); 2996/242469 objects misplaced (1.236%); 9.9 MiB/s, 157 objects/s recovering 2024-09-13T22:59:35.115 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:34 smithi178 bash[22840]: audit 2024-09-13T22:59:34.486041+0000 mon.b (mon.1) 277 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:35.134 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:35.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.259 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:35.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:35.402 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:35.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:35.538 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:35.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:35.673 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:35.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 89 16 84686 84670 59.4486 49.6875 0.00935536 0.0168152 2024-09-13T22:59:35.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.813 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:35.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.944 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:35.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:36.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:36.070 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:36.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:59:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:59:36.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:36.193 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:36.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:36.317 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:36.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:36.444 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:36.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:36.572 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:36.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:36.698 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:36.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout: 90 16 85027 85011 59.0248 21.3125 0.00328382 0.0168031 2024-09-13T22:59:36.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:36.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:36.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:36.951 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:36.952 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:36 smithi192 bash[24608]: cluster 2024-09-13T22:59:34.946621+0000 mgr.y (mgr.14613) 814 : cluster [DBG] pgmap v965: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s; 8930/250620 objects degraded (3.563%); 4983/250620 objects misplaced (1.988%); 14 MiB/s, 226 objects/s recovering 2024-09-13T22:59:36.952 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:36 smithi192 bash[24608]: cluster 2024-09-13T22:59:35.738242+0000 osd.6 (osd.6) 310 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:36.952 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:36 smithi192 bash[24608]: cluster 2024-09-13T22:59:35.738308+0000 osd.6 (osd.6) 311 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:36.952 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:36 smithi192 bash[24608]: cluster 2024-09-13T22:59:35.814802+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-09-13T22:59:36.953 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:36 smithi192 bash[24608]: audit 2024-09-13T22:59:35.946087+0000 mgr.y (mgr.14613) 815 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:36.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:37.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:37.076 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:37.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:37.199 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:37.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: cluster 2024-09-13T22:59:34.946621+0000 mgr.y (mgr.14613) 814 : cluster [DBG] pgmap v965: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s; 8930/250620 objects degraded (3.563%); 4983/250620 objects misplaced (1.988%); 14 MiB/s, 226 objects/s recovering 2024-09-13T22:59:37.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: cluster 2024-09-13T22:59:35.738242+0000 osd.6 (osd.6) 310 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:37.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: cluster 2024-09-13T22:59:35.738308+0000 osd.6 (osd.6) 311 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:37.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: cluster 2024-09-13T22:59:35.814802+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-09-13T22:59:37.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: audit 2024-09-13T22:59:35.946087+0000 mgr.y (mgr.14613) 815 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:37.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:37.329 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:37.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:37.457 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:37.459 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: cluster 2024-09-13T22:59:34.946621+0000 mgr.y (mgr.14613) 814 : cluster [DBG] pgmap v965: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s; 8930/250620 objects degraded (3.563%); 4983/250620 objects misplaced (1.988%); 14 MiB/s, 226 objects/s recovering 2024-09-13T22:59:37.459 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: cluster 2024-09-13T22:59:35.738242+0000 osd.6 (osd.6) 310 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:37.459 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: cluster 2024-09-13T22:59:35.738308+0000 osd.6 (osd.6) 311 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:37.459 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: cluster 2024-09-13T22:59:35.814802+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-09-13T22:59:37.459 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: audit 2024-09-13T22:59:35.946087+0000 mgr.y (mgr.14613) 815 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:37.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:37.582 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:37.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:37.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total time run: 90.1893 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Total writes made: 85027 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Write size: 65536 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Object size: 65536 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Bandwidth (MB/sec): 58.9226 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Bandwidth: 29.1037 2024-09-13T22:59:37.725 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max bandwidth (MB/sec): 157.875 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min bandwidth (MB/sec): 0 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average IOPS: 942 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev IOPS: 465.659 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max IOPS: 2526 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min IOPS: 0 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Average Latency(s): 0.016964 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Stddev Latency(s): 0.0445864 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Max latency(s): 2.36455 2024-09-13T22:59:37.726 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Min latency(s): 0.00117945 2024-09-13T22:59:37.747 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T22:59:37.809 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:37.820 INFO:teuthology.orchestra.run.smithi178.stdout:ERROR: (22) Invalid argument 2024-09-13T22:59:37.820 INFO:teuthology.orchestra.run.smithi178.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T22:59:37.830 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:37.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:37.972 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:37.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:38.010 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T22:59:38.011 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T22:59:38.045 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T22:59:38.045 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T22:59:38.068 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T22:59:38.069 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T22:59:38.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:38.111 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T22:59:38.111 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T22:59:38.112 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:38.142 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T22:59:38.143 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T22:59:38.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:37 smithi192 bash[24608]: cluster 2024-09-13T22:59:36.822562+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-09-13T22:59:38.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:37 smithi192 bash[24608]: cluster 2024-09-13T22:59:36.947200+0000 mgr.y (mgr.14613) 816 : cluster [DBG] pgmap v968: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 992 B/s rd, 35 MiB/s wr, 562 op/s; 8930/251169 objects degraded (3.555%); 4983/251169 objects misplaced (1.984%); 11 MiB/s, 180 objects/s recovering 2024-09-13T22:59:38.175 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T22:59:38.176 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T22:59:38.211 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T22:59:38.211 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T22:59:38.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:38.240 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:38.261 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T22:59:38.261 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T22:59:38.276 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: cluster 2024-09-13T22:59:36.822562+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-09-13T22:59:38.276 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[29889]: cluster 2024-09-13T22:59:36.947200+0000 mgr.y (mgr.14613) 816 : cluster [DBG] pgmap v968: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 992 B/s rd, 35 MiB/s wr, 562 op/s; 8930/251169 objects degraded (3.555%); 4983/251169 objects misplaced (1.984%); 11 MiB/s, 180 objects/s recovering 2024-09-13T22:59:38.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: cluster 2024-09-13T22:59:36.822562+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-09-13T22:59:38.276 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:37 smithi178 bash[22840]: cluster 2024-09-13T22:59:36.947200+0000 mgr.y (mgr.14613) 816 : cluster [DBG] pgmap v968: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 992 B/s rd, 35 MiB/s wr, 562 op/s; 8930/251169 objects degraded (3.555%); 4983/251169 objects misplaced (1.984%); 11 MiB/s, 180 objects/s recovering 2024-09-13T22:59:38.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:38.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:38.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:38.512 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:38.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:38.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:38.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:38.780 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:38.832 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T22:59:38.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:38.907 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:38.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:39.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:39.018 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[29889]: audit 2024-09-13T22:59:37.968000+0000 mon.a (mon.0) 1715 : audit [DBG] from='client.? v1:172.21.15.178:0/3302657931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:59:39.018 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[29889]: cluster 2024-09-13T22:59:38.149767+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check update: Degraded data redundancy: 8930/251169 objects degraded (3.555%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:39.018 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[29889]: audit 2024-09-13T22:59:38.334690+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.178:0/141260715' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:59:39.018 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[29889]: audit 2024-09-13T22:59:38.335522+0000 mon.a (mon.0) 1717 : audit [INF] from='client.25862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:59:39.019 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[22840]: audit 2024-09-13T22:59:37.968000+0000 mon.a (mon.0) 1715 : audit [DBG] from='client.? v1:172.21.15.178:0/3302657931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:59:39.019 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[22840]: cluster 2024-09-13T22:59:38.149767+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check update: Degraded data redundancy: 8930/251169 objects degraded (3.555%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:39.019 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[22840]: audit 2024-09-13T22:59:38.334690+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.178:0/141260715' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:59:39.019 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:38 smithi178 bash[22840]: audit 2024-09-13T22:59:38.335522+0000 mon.a (mon.0) 1717 : audit [INF] from='client.25862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:59:39.037 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:39.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:39.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:38 smithi192 bash[24608]: audit 2024-09-13T22:59:37.968000+0000 mon.a (mon.0) 1715 : audit [DBG] from='client.? v1:172.21.15.178:0/3302657931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T22:59:39.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:38 smithi192 bash[24608]: cluster 2024-09-13T22:59:38.149767+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check update: Degraded data redundancy: 8930/251169 objects degraded (3.555%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:39.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:38 smithi192 bash[24608]: audit 2024-09-13T22:59:38.334690+0000 mon.c (mon.2) 83 : audit [INF] from='client.? v1:172.21.15.178:0/141260715' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:59:39.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:38 smithi192 bash[24608]: audit 2024-09-13T22:59:38.335522+0000 mon.a (mon.0) 1717 : audit [INF] from='client.25862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T22:59:39.164 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:39.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:39.299 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:39.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:39.439 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:39.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:39.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:39.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:39.693 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:39.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:39.839 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:39.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:39.983 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:39.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:40.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:40.100 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:39 smithi192 bash[24608]: cluster 2024-09-13T22:59:38.825897+0000 mon.a (mon.0) 1718 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:59:40.100 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:39 smithi192 bash[24608]: audit 2024-09-13T22:59:38.830175+0000 mon.a (mon.0) 1719 : audit [INF] from='client.25862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:59:40.100 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:39 smithi192 bash[24608]: cluster 2024-09-13T22:59:38.833311+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-09-13T22:59:40.100 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:39 smithi192 bash[24608]: cluster 2024-09-13T22:59:38.947614+0000 mgr.y (mgr.14613) 817 : cluster [DBG] pgmap v970: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 510 op/s; 8930/251367 objects degraded (3.553%); 4983/251367 objects misplaced (1.982%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T22:59:40.121 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:40.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[22840]: cluster 2024-09-13T22:59:38.825897+0000 mon.a (mon.0) 1718 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:59:40.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[22840]: audit 2024-09-13T22:59:38.830175+0000 mon.a (mon.0) 1719 : audit [INF] from='client.25862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:59:40.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[22840]: cluster 2024-09-13T22:59:38.833311+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-09-13T22:59:40.161 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[22840]: cluster 2024-09-13T22:59:38.947614+0000 mgr.y (mgr.14613) 817 : cluster [DBG] pgmap v970: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 510 op/s; 8930/251367 objects degraded (3.553%); 4983/251367 objects misplaced (1.982%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T22:59:40.161 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:59:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:59:40.161 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[29889]: cluster 2024-09-13T22:59:38.825897+0000 mon.a (mon.0) 1718 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T22:59:40.161 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[29889]: audit 2024-09-13T22:59:38.830175+0000 mon.a (mon.0) 1719 : audit [INF] from='client.25862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T22:59:40.161 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[29889]: cluster 2024-09-13T22:59:38.833311+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-09-13T22:59:40.161 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:39 smithi178 bash[29889]: cluster 2024-09-13T22:59:38.947614+0000 mgr.y (mgr.14613) 817 : cluster [DBG] pgmap v970: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 510 op/s; 8930/251367 objects degraded (3.553%); 4983/251367 objects misplaced (1.982%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T22:59:40.226 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:40.256 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:40.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:40.424 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:40.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:40.557 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:40.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:40.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:40.691 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:40.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:40.821 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:40.849 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T22:59:40.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:40.955 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:40 smithi178 bash[22840]: audit 2024-09-13T22:59:39.859160+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? v1:172.21.15.178:0/3179276095' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:59:40.955 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:40 smithi178 bash[22840]: audit 2024-09-13T22:59:40.793221+0000 mon.b (mon.1) 278 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:59:40.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:40 smithi178 bash[29889]: audit 2024-09-13T22:59:39.859160+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? v1:172.21.15.178:0/3179276095' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:59:40.956 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:40 smithi178 bash[29889]: audit 2024-09-13T22:59:40.793221+0000 mon.b (mon.1) 278 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:59:40.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:40.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:41.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:40 smithi192 bash[24608]: audit 2024-09-13T22:59:39.859160+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? v1:172.21.15.178:0/3179276095' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T22:59:41.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:40 smithi192 bash[24608]: audit 2024-09-13T22:59:40.793221+0000 mon.b (mon.1) 278 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T22:59:41.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:41.246 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:41.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:41.494 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.552 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:41.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:41.702 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:41.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:41.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:41.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:41.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:42.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:42.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:42.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:41 smithi192 bash[24608]: audit 2024-09-13T22:59:40.850733+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.178:0/3179276095' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:59:42.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:41 smithi192 bash[24608]: cluster 2024-09-13T22:59:40.855090+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-09-13T22:59:42.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:41 smithi192 bash[24608]: cluster 2024-09-13T22:59:40.948102+0000 mgr.y (mgr.14613) 818 : cluster [DBG] pgmap v972: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 21 MiB/s wr, 491 op/s; 5905/253194 objects degraded (2.332%); 7052/253194 objects misplaced (2.785%); 12 MiB/s, 186 objects/s recovering 2024-09-13T22:59:42.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:41 smithi192 bash[24608]: audit 2024-09-13T22:59:41.523081+0000 mon.a (mon.0) 1724 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:42.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:41 smithi192 bash[24608]: audit 2024-09-13T22:59:41.538930+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:42.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:42.250 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:42.265 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[29889]: audit 2024-09-13T22:59:40.850733+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.178:0/3179276095' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:59:42.265 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[29889]: cluster 2024-09-13T22:59:40.855090+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-09-13T22:59:42.265 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[29889]: cluster 2024-09-13T22:59:40.948102+0000 mgr.y (mgr.14613) 818 : cluster [DBG] pgmap v972: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 21 MiB/s wr, 491 op/s; 5905/253194 objects degraded (2.332%); 7052/253194 objects misplaced (2.785%); 12 MiB/s, 186 objects/s recovering 2024-09-13T22:59:42.265 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[29889]: audit 2024-09-13T22:59:41.523081+0000 mon.a (mon.0) 1724 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:42.265 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[29889]: audit 2024-09-13T22:59:41.538930+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:42.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[22840]: audit 2024-09-13T22:59:40.850733+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.178:0/3179276095' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T22:59:42.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[22840]: cluster 2024-09-13T22:59:40.855090+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-09-13T22:59:42.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[22840]: cluster 2024-09-13T22:59:40.948102+0000 mgr.y (mgr.14613) 818 : cluster [DBG] pgmap v972: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 21 MiB/s wr, 491 op/s; 5905/253194 objects degraded (2.332%); 7052/253194 objects misplaced (2.785%); 12 MiB/s, 186 objects/s recovering 2024-09-13T22:59:42.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[22840]: audit 2024-09-13T22:59:41.523081+0000 mon.a (mon.0) 1724 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:42.266 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:41 smithi178 bash[22840]: audit 2024-09-13T22:59:41.538930+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:42.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:42.383 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:42.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:42.510 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:42.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:42.635 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:42.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:42.785 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:42.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:42.919 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:42.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:43.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:43.045 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:43.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:43.173 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:43.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:43.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:43.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: audit 2024-09-13T22:59:42.054359+0000 mon.a (mon.0) 1726 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: audit 2024-09-13T22:59:42.072838+0000 mon.a (mon.0) 1727 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: cluster 2024-09-13T22:59:42.434946+0000 osd.0 (osd.0) 407 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:43.318 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: cluster 2024-09-13T22:59:42.435018+0000 osd.0 (osd.0) 408 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: audit 2024-09-13T22:59:43.009692+0000 mon.b (mon.1) 279 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: audit 2024-09-13T22:59:43.014137+0000 mon.b (mon.1) 280 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[29889]: audit 2024-09-13T22:59:43.021512+0000 mon.a (mon.0) 1728 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: audit 2024-09-13T22:59:42.054359+0000 mon.a (mon.0) 1726 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: audit 2024-09-13T22:59:42.072838+0000 mon.a (mon.0) 1727 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: cluster 2024-09-13T22:59:42.434946+0000 osd.0 (osd.0) 407 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: cluster 2024-09-13T22:59:42.435018+0000 osd.0 (osd.0) 408 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: audit 2024-09-13T22:59:43.009692+0000 mon.b (mon.1) 279 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:59:43.319 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: audit 2024-09-13T22:59:43.014137+0000 mon.b (mon.1) 280 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:59:43.320 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:43 smithi178 bash[22840]: audit 2024-09-13T22:59:43.021512+0000 mon.a (mon.0) 1728 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: audit 2024-09-13T22:59:42.054359+0000 mon.a (mon.0) 1726 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: audit 2024-09-13T22:59:42.072838+0000 mon.a (mon.0) 1727 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: cluster 2024-09-13T22:59:42.434946+0000 osd.0 (osd.0) 407 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: cluster 2024-09-13T22:59:42.435018+0000 osd.0 (osd.0) 408 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: audit 2024-09-13T22:59:43.009692+0000 mon.b (mon.1) 279 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: audit 2024-09-13T22:59:43.014137+0000 mon.b (mon.1) 280 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T22:59:43.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:43 smithi192 bash[24608]: audit 2024-09-13T22:59:43.021512+0000 mon.a (mon.0) 1728 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:43.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:43.439 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:43.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:43.562 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:43.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:43.682 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:43.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:43.808 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:43.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:43.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:43.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:44.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:44.081 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:44.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:44.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:44.221 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[29889]: cluster 2024-09-13T22:59:42.948820+0000 mgr.y (mgr.14613) 819 : cluster [DBG] pgmap v973: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1002 B/s rd, 15 MiB/s wr, 408 op/s; 5905/252708 objects degraded (2.337%); 7052/252708 objects misplaced (2.791%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[29889]: cluster 2024-09-13T22:59:43.080421+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[29889]: cluster 2024-09-13T22:59:43.150680+0000 mon.a (mon.0) 1730 : cluster [WRN] Health check update: Degraded data redundancy: 5905/252708 objects degraded (2.337%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[29889]: cluster 2024-09-13T22:59:43.155131+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[22840]: cluster 2024-09-13T22:59:42.948820+0000 mgr.y (mgr.14613) 819 : cluster [DBG] pgmap v973: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1002 B/s rd, 15 MiB/s wr, 408 op/s; 5905/252708 objects degraded (2.337%); 7052/252708 objects misplaced (2.791%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[22840]: cluster 2024-09-13T22:59:43.080421+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[22840]: cluster 2024-09-13T22:59:43.150680+0000 mon.a (mon.0) 1730 : cluster [WRN] Health check update: Degraded data redundancy: 5905/252708 objects degraded (2.337%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:44.222 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:44 smithi178 bash[22840]: cluster 2024-09-13T22:59:43.155131+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-09-13T22:59:44.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:44.348 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:44.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:44 smithi192 bash[24608]: cluster 2024-09-13T22:59:42.948820+0000 mgr.y (mgr.14613) 819 : cluster [DBG] pgmap v973: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1002 B/s rd, 15 MiB/s wr, 408 op/s; 5905/252708 objects degraded (2.337%); 7052/252708 objects misplaced (2.791%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T22:59:44.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:44 smithi192 bash[24608]: cluster 2024-09-13T22:59:43.080421+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-09-13T22:59:44.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:44 smithi192 bash[24608]: cluster 2024-09-13T22:59:43.150680+0000 mon.a (mon.0) 1730 : cluster [WRN] Health check update: Degraded data redundancy: 5905/252708 objects degraded (2.337%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T22:59:44.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:44 smithi192 bash[24608]: cluster 2024-09-13T22:59:43.155131+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-09-13T22:59:44.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:44.479 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:44.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:44.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:44.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:44.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:44.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:44.876 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:44.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:44.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:45.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:45.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:45.143 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:45.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:45.269 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:45.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:45.397 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:45.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:45.524 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:45.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:45.661 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:45.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:45.798 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:45.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:45.924 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:45.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:46.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:46.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:46.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:59:45 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:59:46.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:46.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:46.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:46.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:46 smithi192 bash[24608]: cluster 2024-09-13T22:59:44.949818+0000 mgr.y (mgr.14613) 820 : cluster [DBG] pgmap v976: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.8 KiB/s rd, 14 MiB/s wr, 970 op/s; 5905/242274 objects degraded (2.437%); 6441/242274 objects misplaced (2.659%); 16 MiB/s, 261 objects/s recovering 2024-09-13T22:59:46.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:46.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:46.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:46 smithi178 bash[29889]: cluster 2024-09-13T22:59:44.949818+0000 mgr.y (mgr.14613) 820 : cluster [DBG] pgmap v976: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.8 KiB/s rd, 14 MiB/s wr, 970 op/s; 5905/242274 objects degraded (2.437%); 6441/242274 objects misplaced (2.659%); 16 MiB/s, 261 objects/s recovering 2024-09-13T22:59:46.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:46 smithi178 bash[22840]: cluster 2024-09-13T22:59:44.949818+0000 mgr.y (mgr.14613) 820 : cluster [DBG] pgmap v976: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.8 KiB/s rd, 14 MiB/s wr, 970 op/s; 5905/242274 objects degraded (2.437%); 6441/242274 objects misplaced (2.659%); 16 MiB/s, 261 objects/s recovering 2024-09-13T22:59:46.467 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:46.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:46.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:46.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:46.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:46.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:46.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:46.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:46.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:46.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:47.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:47.111 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[29889]: cluster 2024-09-13T22:59:45.449468+0000 osd.3 (osd.3) 387 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[29889]: cluster 2024-09-13T22:59:45.449557+0000 osd.3 (osd.3) 388 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[29889]: audit 2024-09-13T22:59:45.954110+0000 mgr.y (mgr.14613) 821 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[29889]: cluster 2024-09-13T22:59:46.095236+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[22840]: cluster 2024-09-13T22:59:45.449468+0000 osd.3 (osd.3) 387 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[22840]: cluster 2024-09-13T22:59:45.449557+0000 osd.3 (osd.3) 388 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[22840]: audit 2024-09-13T22:59:45.954110+0000 mgr.y (mgr.14613) 821 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:47.112 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:47 smithi178 bash[22840]: cluster 2024-09-13T22:59:46.095236+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-09-13T22:59:47.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:47.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:47.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:47.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:47.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:47 smithi192 bash[24608]: cluster 2024-09-13T22:59:45.449468+0000 osd.3 (osd.3) 387 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:47.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:47 smithi192 bash[24608]: cluster 2024-09-13T22:59:45.449557+0000 osd.3 (osd.3) 388 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:47.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:47 smithi192 bash[24608]: audit 2024-09-13T22:59:45.954110+0000 mgr.y (mgr.14613) 821 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:47.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:47 smithi192 bash[24608]: cluster 2024-09-13T22:59:46.095236+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-09-13T22:59:47.390 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:47.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:47.523 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:47.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:47.664 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:47.768 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:47.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:47.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:47.930 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:47.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:48.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:48.061 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:48.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:48.177 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:48 smithi192 bash[24608]: cluster 2024-09-13T22:59:46.950450+0000 mgr.y (mgr.14613) 822 : cluster [DBG] pgmap v978: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 650 op/s; 5905/241503 objects degraded (2.445%); 6441/241503 objects misplaced (2.667%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T22:59:48.187 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:48.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:48.314 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:48.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:48.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:48 smithi178 bash[29889]: cluster 2024-09-13T22:59:46.950450+0000 mgr.y (mgr.14613) 822 : cluster [DBG] pgmap v978: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 650 op/s; 5905/241503 objects degraded (2.445%); 6441/241503 objects misplaced (2.667%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T22:59:48.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:48 smithi178 bash[22840]: cluster 2024-09-13T22:59:46.950450+0000 mgr.y (mgr.14613) 822 : cluster [DBG] pgmap v978: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 650 op/s; 5905/241503 objects degraded (2.445%); 6441/241503 objects misplaced (2.667%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T22:59:48.455 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:48.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:48.597 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:48.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:48.724 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:48.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:48.854 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:48.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:48.979 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:48.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:49.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:49.107 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:49.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:49.235 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:49.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:49.359 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:49.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:49 smithi192 bash[24608]: cluster 2024-09-13T22:59:48.152224+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check update: Degraded data redundancy: 5905/241503 objects degraded (2.445%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:49.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:49 smithi192 bash[24608]: cluster 2024-09-13T22:59:48.158324+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-09-13T22:59:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:49.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:49 smithi178 bash[29889]: cluster 2024-09-13T22:59:48.152224+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check update: Degraded data redundancy: 5905/241503 objects degraded (2.445%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:49.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:49 smithi178 bash[29889]: cluster 2024-09-13T22:59:48.158324+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-09-13T22:59:49.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:49 smithi178 bash[22840]: cluster 2024-09-13T22:59:48.152224+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check update: Degraded data redundancy: 5905/241503 objects degraded (2.445%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:49.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:49 smithi178 bash[22840]: cluster 2024-09-13T22:59:48.158324+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-09-13T22:59:49.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:49.491 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:49.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:49.619 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:49.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:49.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:49.747 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:49.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:49.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:49.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:49.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:50.020 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:50.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:50.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:59:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:59:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T22:59:50.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.278 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:50.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:50.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:50.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T22:59:50.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.544 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:50.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T22:59:50.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:50.660 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:50 smithi178 bash[29889]: cluster 2024-09-13T22:59:48.951013+0000 mgr.y (mgr.14613) 823 : cluster [DBG] pgmap v980: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 684 op/s; 5905/240663 objects degraded (2.454%); 6441/240663 objects misplaced (2.676%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T22:59:50.660 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:50 smithi178 bash[29889]: audit 2024-09-13T22:59:49.505906+0000 mon.a (mon.0) 1735 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:50.660 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:50 smithi178 bash[29889]: audit 2024-09-13T22:59:49.508822+0000 mon.b (mon.1) 281 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:50.661 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:50 smithi178 bash[22840]: cluster 2024-09-13T22:59:48.951013+0000 mgr.y (mgr.14613) 823 : cluster [DBG] pgmap v980: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 684 op/s; 5905/240663 objects degraded (2.454%); 6441/240663 objects misplaced (2.676%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T22:59:50.661 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:50 smithi178 bash[22840]: audit 2024-09-13T22:59:49.505906+0000 mon.a (mon.0) 1735 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:50.661 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:50 smithi178 bash[22840]: audit 2024-09-13T22:59:49.508822+0000 mon.b (mon.1) 281 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:50.704 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:50.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:50 smithi192 bash[24608]: cluster 2024-09-13T22:59:48.951013+0000 mgr.y (mgr.14613) 823 : cluster [DBG] pgmap v980: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 684 op/s; 5905/240663 objects degraded (2.454%); 6441/240663 objects misplaced (2.676%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T22:59:50.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:50 smithi192 bash[24608]: audit 2024-09-13T22:59:49.505906+0000 mon.a (mon.0) 1735 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T22:59:50.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:50 smithi192 bash[24608]: audit 2024-09-13T22:59:49.508822+0000 mon.b (mon.1) 281 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T22:59:50.849 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:50.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.974 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:50.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:51.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:51.103 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:51.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:51.237 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:51.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:51.363 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:51.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:51.498 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:51.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:51.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:51.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:51.759 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:51.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:51.895 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:51.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:52.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.042 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:52.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:52.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:52.178 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:52.283 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:52.383 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:52.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:52.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:52.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T22:59:52.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T22:59:52.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:52.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:52.642 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:52 smithi178 bash[29889]: cluster 2024-09-13T22:59:50.951872+0000 mgr.y (mgr.14613) 824 : cluster [DBG] pgmap v981: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 988 op/s; 5925/229611 objects degraded (2.580%); 5776/229611 objects misplaced (2.516%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:59:52.643 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:52 smithi178 bash[22840]: cluster 2024-09-13T22:59:50.951872+0000 mgr.y (mgr.14613) 824 : cluster [DBG] pgmap v981: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 988 op/s; 5925/229611 objects degraded (2.580%); 5776/229611 objects misplaced (2.516%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:59:52.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:52.794 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:52.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:52.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:52 smithi192 bash[24608]: cluster 2024-09-13T22:59:50.951872+0000 mgr.y (mgr.14613) 824 : cluster [DBG] pgmap v981: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 988 op/s; 5925/229611 objects degraded (2.580%); 5776/229611 objects misplaced (2.516%); 10 MiB/s, 161 objects/s recovering 2024-09-13T22:59:52.953 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:52.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:53.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:53.086 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:53.190 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:53.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:53.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:53.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:53.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:53.495 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:53.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:53.626 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:53.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:53.740 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:53 smithi178 bash[22840]: cluster 2024-09-13T22:59:52.805157+0000 osd.2 (osd.2) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:53.740 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:53 smithi178 bash[22840]: cluster 2024-09-13T22:59:52.805227+0000 osd.2 (osd.2) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:53.740 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:53 smithi178 bash[22840]: cluster 2024-09-13T22:59:53.153555+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: Degraded data redundancy: 5925/229611 objects degraded (2.580%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:53.740 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:53 smithi178 bash[29889]: cluster 2024-09-13T22:59:52.805157+0000 osd.2 (osd.2) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:53.741 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:53 smithi178 bash[29889]: cluster 2024-09-13T22:59:52.805227+0000 osd.2 (osd.2) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:53.741 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:53 smithi178 bash[29889]: cluster 2024-09-13T22:59:53.153555+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: Degraded data redundancy: 5925/229611 objects degraded (2.580%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:53.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:53.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:53.903 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:53.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:53 smithi192 bash[24608]: cluster 2024-09-13T22:59:52.805157+0000 osd.2 (osd.2) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:53 smithi192 bash[24608]: cluster 2024-09-13T22:59:52.805227+0000 osd.2 (osd.2) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:53.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:53 smithi192 bash[24608]: cluster 2024-09-13T22:59:53.153555+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: Degraded data redundancy: 5925/229611 objects degraded (2.580%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:54.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:54.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:54.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:54.193 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:54.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:54.325 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:54.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:54.457 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:54.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:54.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:54 smithi192 bash[24608]: cluster 2024-09-13T22:59:52.952491+0000 mgr.y (mgr.14613) 825 : cluster [DBG] pgmap v982: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 556 op/s; 5925/228918 objects degraded (2.588%); 5776/228918 objects misplaced (2.523%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T22:59:54.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:54 smithi192 bash[24608]: cluster 2024-09-13T22:59:53.534104+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-09-13T22:59:54.580 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:54.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:54.708 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:54.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:54.832 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:54.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:54 smithi178 bash[22840]: cluster 2024-09-13T22:59:52.952491+0000 mgr.y (mgr.14613) 825 : cluster [DBG] pgmap v982: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 556 op/s; 5925/228918 objects degraded (2.588%); 5776/228918 objects misplaced (2.523%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T22:59:54.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:54 smithi178 bash[22840]: cluster 2024-09-13T22:59:53.534104+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-09-13T22:59:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:54 smithi178 bash[29889]: cluster 2024-09-13T22:59:52.952491+0000 mgr.y (mgr.14613) 825 : cluster [DBG] pgmap v982: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 556 op/s; 5925/228918 objects degraded (2.588%); 5776/228918 objects misplaced (2.523%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T22:59:54.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:54 smithi178 bash[29889]: cluster 2024-09-13T22:59:53.534104+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-09-13T22:59:54.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:54.972 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:54.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:55.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:55.102 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:55.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:55.240 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:55.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:55.363 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:55.364 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:55.370 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:55.474 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:55.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:55.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:55.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:55 smithi192 bash[24608]: cluster 2024-09-13T22:59:53.849869+0000 osd.5 (osd.5) 449 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:55.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:55 smithi192 bash[24608]: cluster 2024-09-13T22:59:53.849937+0000 osd.5 (osd.5) 450 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:55.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:55 smithi192 bash[24608]: cluster 2024-09-13T22:59:54.541665+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-09-13T22:59:55.630 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:55.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:55.756 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:55.782 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:55 smithi178 bash[29889]: cluster 2024-09-13T22:59:53.849869+0000 osd.5 (osd.5) 449 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:55.783 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:55 smithi178 bash[29889]: cluster 2024-09-13T22:59:53.849937+0000 osd.5 (osd.5) 450 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:55.783 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:55 smithi178 bash[29889]: cluster 2024-09-13T22:59:54.541665+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-09-13T22:59:55.784 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:55 smithi178 bash[22840]: cluster 2024-09-13T22:59:53.849869+0000 osd.5 (osd.5) 449 : cluster [DBG] purged_snaps scrub starts 2024-09-13T22:59:55.784 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:55 smithi178 bash[22840]: cluster 2024-09-13T22:59:53.849937+0000 osd.5 (osd.5) 450 : cluster [DBG] purged_snaps scrub ok 2024-09-13T22:59:55.784 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:55 smithi178 bash[22840]: cluster 2024-09-13T22:59:54.541665+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-09-13T22:59:55.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:55.885 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:55.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:55.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:56.019 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:56.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:56.148 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:56.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:56.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:56.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:56.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 22:59:55 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T22:59:56.424 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:56.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:56.549 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:56.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:56 smithi192 bash[24608]: cluster 2024-09-13T22:59:54.953188+0000 mgr.y (mgr.14613) 826 : cluster [DBG] pgmap v985: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.22k op/s; 6022/216726 objects degraded (2.779%); 5026/216726 objects misplaced (2.319%); 12 MiB/s, 190 objects/s recovering 2024-09-13T22:59:56.680 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:56.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:56.811 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:56.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:56.923 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:56 smithi178 bash[29889]: cluster 2024-09-13T22:59:54.953188+0000 mgr.y (mgr.14613) 826 : cluster [DBG] pgmap v985: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.22k op/s; 6022/216726 objects degraded (2.779%); 5026/216726 objects misplaced (2.319%); 12 MiB/s, 190 objects/s recovering 2024-09-13T22:59:56.923 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:56 smithi178 bash[22840]: cluster 2024-09-13T22:59:54.953188+0000 mgr.y (mgr.14613) 826 : cluster [DBG] pgmap v985: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.22k op/s; 6022/216726 objects degraded (2.779%); 5026/216726 objects misplaced (2.319%); 12 MiB/s, 190 objects/s recovering 2024-09-13T22:59:56.939 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:56.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:57.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:57.064 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:57.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:57.188 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:57.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:57.321 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T22:59:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T22:59:57.399 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:57.399 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T22:59:57.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:57.465 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:57.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:57.593 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:57.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:57 smithi192 bash[24608]: audit 2024-09-13T22:59:55.962180+0000 mgr.y (mgr.14613) 827 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:57.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:57.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:57 smithi178 bash[29889]: audit 2024-09-13T22:59:55.962180+0000 mgr.y (mgr.14613) 827 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:57.706 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:57 smithi178 bash[22840]: audit 2024-09-13T22:59:55.962180+0000 mgr.y (mgr.14613) 827 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T22:59:57.726 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:57.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:57.851 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:57.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:57.972 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:57.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:58.075 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:58.111 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T22:59:58.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:58.235 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:58.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:58.359 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:58.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:58.479 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:58.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:58.610 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T22:59:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T22:59:58.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:58.736 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T22:59:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:58 smithi192 bash[24608]: cluster 2024-09-13T22:59:56.955928+0000 mgr.y (mgr.14613) 828 : cluster [DBG] pgmap v986: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6022/215973 objects degraded (2.788%); 5026/215973 objects misplaced (2.327%); 10 MiB/s, 162 objects/s recovering 2024-09-13T22:59:58.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 22:59:58 smithi192 bash[24608]: cluster 2024-09-13T22:59:58.154417+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 6022/215973 objects degraded (2.788%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T22:59:58.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:58.851 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:58 smithi178 bash[22840]: cluster 2024-09-13T22:59:56.955928+0000 mgr.y (mgr.14613) 828 : cluster [DBG] pgmap v986: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6022/215973 objects degraded (2.788%); 5026/215973 objects misplaced (2.327%); 10 MiB/s, 162 objects/s recovering 2024-09-13T22:59:58.851 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 22:59:58 smithi178 bash[22840]: cluster 2024-09-13T22:59:58.154417+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 6022/215973 objects degraded (2.788%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:58.852 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:58 smithi178 bash[29889]: cluster 2024-09-13T22:59:56.955928+0000 mgr.y (mgr.14613) 828 : cluster [DBG] pgmap v986: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6022/215973 objects degraded (2.788%); 5026/215973 objects misplaced (2.327%); 10 MiB/s, 162 objects/s recovering 2024-09-13T22:59:58.852 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 22:59:58 smithi178 bash[29889]: cluster 2024-09-13T22:59:58.154417+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 6022/215973 objects degraded (2.788%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T22:59:58.880 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:58.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:58.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:59.015 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:59.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:59.142 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T22:59:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T22:59:59.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:59.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:59.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:59.437 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T22:59:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T22:59:59.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:59.578 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T22:59:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T22:59:59.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:59.707 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:59.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:59.834 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T22:59:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T22:59:59.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:59.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T22:59:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T22:59:59.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:00.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:00.079 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 22:59:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:22:59:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:00.099 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:00.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:00.243 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:00.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:00.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:00.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:00.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:00.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:00.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:00.652 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:00.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:00.790 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:00.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:00.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T22:59:58.956525+0000 mgr.y (mgr.14613) 829 : cluster [DBG] pgmap v987: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 601 op/s; 6022/215190 objects degraded (2.798%); 5026/215190 objects misplaced (2.336%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000118+0000 mon.a (mon.0) 1740 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 6022/215190 objects degraded (2.798%), 3 pgs degraded, 5 pgs undersized 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000144+0000 mon.a (mon.0) 1741 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 6022/215190 objects degraded (2.798%), 3 pgs degraded, 5 pgs undersized 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000152+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 11.5 is stuck undersized for 75s, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,5] 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000160+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 11.8 is stuck undersized for 75s, current state active+recovering+undersized+remapped, last acting [3,5] 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000168+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 11.9 is stuck undersized for 75s, current state active+recovery_wait+undersized+degraded+remapped, last acting [3,0] 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000176+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 11.a is stuck undersized for 75s, current state active+recovering+undersized+remapped, last acting [3,5] 2024-09-13T23:00:00.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.000185+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 11.d is stuck undersized for 75s, current state active+recovering+undersized+degraded+remapped, last acting [2,1] 2024-09-13T23:00:00.917 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:00.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:01.014 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T22:59:58.956525+0000 mgr.y (mgr.14613) 829 : cluster [DBG] pgmap v987: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 601 op/s; 6022/215190 objects degraded (2.798%); 5026/215190 objects misplaced (2.336%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000118+0000 mon.a (mon.0) 1740 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 6022/215190 objects degraded (2.798%), 3 pgs degraded, 5 pgs undersized 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000144+0000 mon.a (mon.0) 1741 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 6022/215190 objects degraded (2.798%), 3 pgs degraded, 5 pgs undersized 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000152+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 11.5 is stuck undersized for 75s, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,5] 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000160+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 11.8 is stuck undersized for 75s, current state active+recovering+undersized+remapped, last acting [3,5] 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000168+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 11.9 is stuck undersized for 75s, current state active+recovery_wait+undersized+degraded+remapped, last acting [3,0] 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000176+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 11.a is stuck undersized for 75s, current state active+recovering+undersized+remapped, last acting [3,5] 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.000185+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 11.d is stuck undersized for 75s, current state active+recovering+undersized+degraded+remapped, last acting [2,1] 2024-09-13T23:00:01.015 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T22:59:58.956525+0000 mgr.y (mgr.14613) 829 : cluster [DBG] pgmap v987: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 601 op/s; 6022/215190 objects degraded (2.798%); 5026/215190 objects misplaced (2.336%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000118+0000 mon.a (mon.0) 1740 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 6022/215190 objects degraded (2.798%), 3 pgs degraded, 5 pgs undersized 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000144+0000 mon.a (mon.0) 1741 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 6022/215190 objects degraded (2.798%), 3 pgs degraded, 5 pgs undersized 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000152+0000 mon.a (mon.0) 1742 : cluster [WRN] pg 11.5 is stuck undersized for 75s, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,5] 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000160+0000 mon.a (mon.0) 1743 : cluster [WRN] pg 11.8 is stuck undersized for 75s, current state active+recovering+undersized+remapped, last acting [3,5] 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000168+0000 mon.a (mon.0) 1744 : cluster [WRN] pg 11.9 is stuck undersized for 75s, current state active+recovery_wait+undersized+degraded+remapped, last acting [3,0] 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000176+0000 mon.a (mon.0) 1745 : cluster [WRN] pg 11.a is stuck undersized for 75s, current state active+recovering+undersized+remapped, last acting [3,5] 2024-09-13T23:00:01.016 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.000185+0000 mon.a (mon.0) 1746 : cluster [WRN] pg 11.d is stuck undersized for 75s, current state active+recovering+undersized+degraded+remapped, last acting [2,1] 2024-09-13T23:00:01.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:01.050 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:01.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:01.175 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:01.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:01.297 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:01.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:01.421 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:01.524 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:01.553 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:01.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:01.687 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:01.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:01.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:01.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:01.968 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:01.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:02.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:02.093 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:02.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:02.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:02.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:02.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:02.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:02.499 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:02.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:02.637 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:02.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:02 smithi192 bash[24608]: cluster 2024-09-13T23:00:00.958112+0000 mgr.y (mgr.14613) 830 : cluster [DBG] pgmap v988: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 6047/204549 objects degraded (2.956%); 4323/204549 objects misplaced (2.113%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:02.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:02.776 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:02.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:02.903 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:02.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:02.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:02 smithi178 bash[29889]: cluster 2024-09-13T23:00:00.958112+0000 mgr.y (mgr.14613) 830 : cluster [DBG] pgmap v988: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 6047/204549 objects degraded (2.956%); 4323/204549 objects misplaced (2.113%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:02.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:02 smithi178 bash[22840]: cluster 2024-09-13T23:00:00.958112+0000 mgr.y (mgr.14613) 830 : cluster [DBG] pgmap v988: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 6047/204549 objects degraded (2.956%); 4323/204549 objects misplaced (2.113%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:03.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:03.046 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:03.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:03.176 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:03.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:03.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:03.415 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:03.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:03.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:03.561 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:03.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:03.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:03.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:03.808 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:03 smithi192 bash[24608]: cluster 2024-09-13T23:00:02.450619+0000 osd.1 (osd.1) 400 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:03.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:03 smithi192 bash[24608]: cluster 2024-09-13T23:00:02.450670+0000 osd.1 (osd.1) 401 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:03.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:03 smithi192 bash[24608]: cluster 2024-09-13T23:00:02.573652+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-09-13T23:00:03.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:03 smithi192 bash[24608]: cluster 2024-09-13T23:00:03.155104+0000 mon.a (mon.0) 1748 : cluster [WRN] Health check update: Degraded data redundancy: 6047/204549 objects degraded (2.956%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:03.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:03 smithi192 bash[24608]: cluster 2024-09-13T23:00:03.162119+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-09-13T23:00:03.819 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:03.867 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[29889]: cluster 2024-09-13T23:00:02.450619+0000 osd.1 (osd.1) 400 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:03.867 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[29889]: cluster 2024-09-13T23:00:02.450670+0000 osd.1 (osd.1) 401 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:03.868 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[29889]: cluster 2024-09-13T23:00:02.573652+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-09-13T23:00:03.868 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[29889]: cluster 2024-09-13T23:00:03.155104+0000 mon.a (mon.0) 1748 : cluster [WRN] Health check update: Degraded data redundancy: 6047/204549 objects degraded (2.956%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:03.868 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[29889]: cluster 2024-09-13T23:00:03.162119+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-09-13T23:00:03.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:03.948 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:03.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:04.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:04.082 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[22840]: cluster 2024-09-13T23:00:02.450619+0000 osd.1 (osd.1) 400 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[22840]: cluster 2024-09-13T23:00:02.450670+0000 osd.1 (osd.1) 401 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[22840]: cluster 2024-09-13T23:00:02.573652+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-09-13T23:00:04.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[22840]: cluster 2024-09-13T23:00:03.155104+0000 mon.a (mon.0) 1748 : cluster [WRN] Health check update: Degraded data redundancy: 6047/204549 objects degraded (2.956%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:04.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:03 smithi178 bash[22840]: cluster 2024-09-13T23:00:03.162119+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-09-13T23:00:04.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:04.211 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:04.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:04.341 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:04.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:04.473 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:04.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:04.606 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:04.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:04.710 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:04.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:04.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:04.856 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:04 smithi192 bash[24608]: cluster 2024-09-13T23:00:02.958669+0000 mgr.y (mgr.14613) 831 : cluster [DBG] pgmap v990: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 607 B/s rd, 0 B/s wr, 546 op/s; 6047/203811 objects degraded (2.967%); 4323/203811 objects misplaced (2.121%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T23:00:04.857 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:04 smithi192 bash[24608]: audit 2024-09-13T23:00:04.527934+0000 mon.b (mon.1) 282 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:04.871 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:04.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:04 smithi178 bash[29889]: cluster 2024-09-13T23:00:02.958669+0000 mgr.y (mgr.14613) 831 : cluster [DBG] pgmap v990: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 607 B/s rd, 0 B/s wr, 546 op/s; 6047/203811 objects degraded (2.967%); 4323/203811 objects misplaced (2.121%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T23:00:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:04 smithi178 bash[29889]: audit 2024-09-13T23:00:04.527934+0000 mon.b (mon.1) 282 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:04.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:04 smithi178 bash[22840]: cluster 2024-09-13T23:00:02.958669+0000 mgr.y (mgr.14613) 831 : cluster [DBG] pgmap v990: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 607 B/s rd, 0 B/s wr, 546 op/s; 6047/203811 objects degraded (2.967%); 4323/203811 objects misplaced (2.121%); 6.4 MiB/s, 103 objects/s recovering 2024-09-13T23:00:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:04 smithi178 bash[22840]: audit 2024-09-13T23:00:04.527934+0000 mon.b (mon.1) 282 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:04.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:05.012 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:05.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.143 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:05.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:05.267 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:05.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:05.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.529 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:05.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:05.663 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:05.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.793 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:05.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:05.922 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:05.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:06.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:06.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:06.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:00:05 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:00:06.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:06.177 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:06.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:06.316 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:06.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:06.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:06.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:06.577 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:06.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:06.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:06 smithi192 bash[24608]: cluster 2024-09-13T23:00:04.959738+0000 mgr.y (mgr.14613) 832 : cluster [DBG] pgmap v992: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 952 op/s; 6075/193137 objects degraded (3.145%); 3404/193137 objects misplaced (1.762%); 12 MiB/s, 196 objects/s recovering 2024-09-13T23:00:06.704 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:06.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:06.834 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:06.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:06 smithi178 bash[22840]: cluster 2024-09-13T23:00:04.959738+0000 mgr.y (mgr.14613) 832 : cluster [DBG] pgmap v992: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 952 op/s; 6075/193137 objects degraded (3.145%); 3404/193137 objects misplaced (1.762%); 12 MiB/s, 196 objects/s recovering 2024-09-13T23:00:06.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:06 smithi178 bash[29889]: cluster 2024-09-13T23:00:04.959738+0000 mgr.y (mgr.14613) 832 : cluster [DBG] pgmap v992: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 952 op/s; 6075/193137 objects degraded (3.145%); 3404/193137 objects misplaced (1.762%); 12 MiB/s, 196 objects/s recovering 2024-09-13T23:00:06.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:06.974 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:06.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:07.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:07.105 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:07.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:07.233 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:07.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:07.356 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:07.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:07.486 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:07.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:07.634 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:07.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:07.772 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:07.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:07.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:07 smithi192 bash[24608]: audit 2024-09-13T23:00:05.970949+0000 mgr.y (mgr.14613) 833 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:07.885 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:07 smithi192 bash[24608]: cluster 2024-09-13T23:00:06.592517+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-09-13T23:00:07.900 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:07.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:07.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:07 smithi178 bash[29889]: audit 2024-09-13T23:00:05.970949+0000 mgr.y (mgr.14613) 833 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:07.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:07 smithi178 bash[29889]: cluster 2024-09-13T23:00:06.592517+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-09-13T23:00:07.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:07 smithi178 bash[22840]: audit 2024-09-13T23:00:05.970949+0000 mgr.y (mgr.14613) 833 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:07.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:07 smithi178 bash[22840]: cluster 2024-09-13T23:00:06.592517+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-09-13T23:00:08.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:08.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:08.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:08.160 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:08.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:08.286 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:08.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:08.417 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:08.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:08.546 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:08.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:08.674 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:08 smithi178 bash[29889]: cluster 2024-09-13T23:00:06.960312+0000 mgr.y (mgr.14613) 834 : cluster [DBG] pgmap v994: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 675 op/s; 6075/192399 objects degraded (3.158%); 3404/192399 objects misplaced (1.769%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T23:00:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:08 smithi178 bash[29889]: cluster 2024-09-13T23:00:07.603298+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-09-13T23:00:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:08 smithi178 bash[29889]: cluster 2024-09-13T23:00:08.156493+0000 mon.a (mon.0) 1752 : cluster [WRN] Health check update: Degraded data redundancy: 6075/192399 objects degraded (3.158%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:08 smithi178 bash[22840]: cluster 2024-09-13T23:00:06.960312+0000 mgr.y (mgr.14613) 834 : cluster [DBG] pgmap v994: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 675 op/s; 6075/192399 objects degraded (3.158%); 3404/192399 objects misplaced (1.769%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T23:00:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:08 smithi178 bash[22840]: cluster 2024-09-13T23:00:07.603298+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-09-13T23:00:08.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:08 smithi178 bash[22840]: cluster 2024-09-13T23:00:08.156493+0000 mon.a (mon.0) 1752 : cluster [WRN] Health check update: Degraded data redundancy: 6075/192399 objects degraded (3.158%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:08.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:08.803 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:08.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:08 smithi192 bash[24608]: cluster 2024-09-13T23:00:06.960312+0000 mgr.y (mgr.14613) 834 : cluster [DBG] pgmap v994: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 675 op/s; 6075/192399 objects degraded (3.158%); 3404/192399 objects misplaced (1.769%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T23:00:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:08 smithi192 bash[24608]: cluster 2024-09-13T23:00:07.603298+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-09-13T23:00:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:08 smithi192 bash[24608]: cluster 2024-09-13T23:00:08.156493+0000 mon.a (mon.0) 1752 : cluster [WRN] Health check update: Degraded data redundancy: 6075/192399 objects degraded (3.158%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:08.925 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:08.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:09.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:09.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:09.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:09.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:09.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:09.332 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:09.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:09.455 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:09.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:09.591 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:09.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:09.717 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:09.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:09.847 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:09.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:00:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:00:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:09.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:09.979 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:09.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:10.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:10.108 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:10.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:10.239 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:10.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:10.376 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:10.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:10.503 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:10.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:10.635 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:10.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:10.767 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:10.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:10.879 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:10 smithi192 bash[24608]: cluster 2024-09-13T23:00:08.961253+0000 mgr.y (mgr.14613) 835 : cluster [DBG] pgmap v996: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 666 op/s; 6075/191820 objects degraded (3.167%); 3404/191820 objects misplaced (1.775%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T23:00:10.894 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:10.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:10 smithi178 bash[29889]: cluster 2024-09-13T23:00:08.961253+0000 mgr.y (mgr.14613) 835 : cluster [DBG] pgmap v996: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 666 op/s; 6075/191820 objects degraded (3.167%); 3404/191820 objects misplaced (1.775%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T23:00:10.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:10 smithi178 bash[22840]: cluster 2024-09-13T23:00:08.961253+0000 mgr.y (mgr.14613) 835 : cluster [DBG] pgmap v996: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 666 op/s; 6075/191820 objects degraded (3.167%); 3404/191820 objects misplaced (1.775%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T23:00:11.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:11.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:11.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:11.157 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:11.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:11.282 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:11.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:11.407 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:11.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:11.538 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:11.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:11.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:11.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:11.774 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:11 smithi192 bash[24608]: cluster 2024-09-13T23:00:11.092161+0000 osd.7 (osd.7) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:11.774 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:11 smithi192 bash[24608]: cluster 2024-09-13T23:00:11.092223+0000 osd.7 (osd.7) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:11.787 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:11.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:11.929 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:11.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:12.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:12.058 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:12.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:12.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:11 smithi178 bash[29889]: cluster 2024-09-13T23:00:11.092161+0000 osd.7 (osd.7) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:12.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:11 smithi178 bash[29889]: cluster 2024-09-13T23:00:11.092223+0000 osd.7 (osd.7) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:12.172 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:11 smithi178 bash[22840]: cluster 2024-09-13T23:00:11.092161+0000 osd.7 (osd.7) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:12.172 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:11 smithi178 bash[22840]: cluster 2024-09-13T23:00:11.092223+0000 osd.7 (osd.7) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:12.192 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:12.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:12.344 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:12.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:12.470 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:12.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:12.621 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:12.715 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:12 smithi178 bash[29889]: cluster 2024-09-13T23:00:10.962222+0000 mgr.y (mgr.14613) 836 : cluster [DBG] pgmap v997: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 153 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 876 op/s; 4511/183315 objects degraded (2.461%); 3817/183315 objects misplaced (2.082%); 16 MiB/s, 261 objects/s recovering 2024-09-13T23:00:12.715 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:12 smithi178 bash[29889]: cluster 2024-09-13T23:00:11.823532+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-09-13T23:00:12.716 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:12 smithi178 bash[22840]: cluster 2024-09-13T23:00:10.962222+0000 mgr.y (mgr.14613) 836 : cluster [DBG] pgmap v997: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 153 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 876 op/s; 4511/183315 objects degraded (2.461%); 3817/183315 objects misplaced (2.082%); 16 MiB/s, 261 objects/s recovering 2024-09-13T23:00:12.716 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:12 smithi178 bash[22840]: cluster 2024-09-13T23:00:11.823532+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-09-13T23:00:12.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:12.755 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:12.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:12.868 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:12 smithi192 bash[24608]: cluster 2024-09-13T23:00:10.962222+0000 mgr.y (mgr.14613) 836 : cluster [DBG] pgmap v997: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 153 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 876 op/s; 4511/183315 objects degraded (2.461%); 3817/183315 objects misplaced (2.082%); 16 MiB/s, 261 objects/s recovering 2024-09-13T23:00:12.869 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:12 smithi192 bash[24608]: cluster 2024-09-13T23:00:11.823532+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-09-13T23:00:12.885 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:12.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:12.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:13.015 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:13.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:13.145 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:13.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:13.281 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:13.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:13.431 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:13.494 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T23:00:13.495 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [], 2024-09-13T23:00:13.495 INFO:teuthology.orchestra.run.smithi178.stdout: "complaint_time": 30, 2024-09-13T23:00:13.495 INFO:teuthology.orchestra.run.smithi178.stdout: "num_blocked_ops": 0 2024-09-13T23:00:13.495 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T23:00:13.531 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T23:00:13.531 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T23:00:13.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:13.557 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:13.565 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T23:00:13.565 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T23:00:13.601 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T23:00:13.601 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T23:00:13.651 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T23:00:13.652 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T23:00:13.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:13.682 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:13.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:13 smithi178 bash[29889]: cluster 2024-09-13T23:00:13.157722+0000 mon.a (mon.0) 1754 : cluster [WRN] Health check update: Degraded data redundancy: 4511/183315 objects degraded (2.461%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:13.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:13 smithi178 bash[29889]: cluster 2024-09-13T23:00:13.168084+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-09-13T23:00:13.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:13 smithi178 bash[29889]: audit 2024-09-13T23:00:13.489653+0000 mon.a (mon.0) 1756 : audit [DBG] from='client.? v1:172.21.15.178:0/1218697791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:00:13.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:13 smithi178 bash[22840]: cluster 2024-09-13T23:00:13.157722+0000 mon.a (mon.0) 1754 : cluster [WRN] Health check update: Degraded data redundancy: 4511/183315 objects degraded (2.461%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:13.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:13 smithi178 bash[22840]: cluster 2024-09-13T23:00:13.168084+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-09-13T23:00:13.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:13 smithi178 bash[22840]: audit 2024-09-13T23:00:13.489653+0000 mon.a (mon.0) 1756 : audit [DBG] from='client.? v1:172.21.15.178:0/1218697791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:00:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:13.697 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T23:00:13.697 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T23:00:13.732 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T23:00:13.732 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T23:00:13.764 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T23:00:13.764 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T23:00:13.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:13.806 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T23:00:13.806 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T23:00:13.825 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:13.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:13 smithi192 bash[24608]: cluster 2024-09-13T23:00:13.157722+0000 mon.a (mon.0) 1754 : cluster [WRN] Health check update: Degraded data redundancy: 4511/183315 objects degraded (2.461%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:13.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:13 smithi192 bash[24608]: cluster 2024-09-13T23:00:13.168084+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-09-13T23:00:13.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:13 smithi192 bash[24608]: audit 2024-09-13T23:00:13.489653+0000 mon.a (mon.0) 1756 : audit [DBG] from='client.? v1:172.21.15.178:0/1218697791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:00:13.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:13.961 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:13.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:14.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.100 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:14.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:14.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:14.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:14.341 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:14.373 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:14.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:14.514 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:14.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:14.628 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T23:00:14.629 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T23:00:14.640 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:14.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:14.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:14.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:14.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[22840]: cluster 2024-09-13T23:00:12.962745+0000 mgr.y (mgr.14613) 837 : cluster [DBG] pgmap v999: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 153 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 503 op/s; 4511/182796 objects degraded (2.468%); 3817/182796 objects misplaced (2.088%); 11 MiB/s, 180 objects/s recovering 2024-09-13T23:00:14.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[22840]: audit 2024-09-13T23:00:13.890618+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.178:0/689478817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:00:14.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[22840]: audit 2024-09-13T23:00:13.891084+0000 mon.a (mon.0) 1757 : audit [INF] from='client.25747 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[22840]: audit 2024-09-13T23:00:13.991317+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.178:0/1459583203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[22840]: audit 2024-09-13T23:00:13.991930+0000 mon.a (mon.0) 1758 : audit [INF] from='client.25883 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[29889]: cluster 2024-09-13T23:00:12.962745+0000 mgr.y (mgr.14613) 837 : cluster [DBG] pgmap v999: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 153 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 503 op/s; 4511/182796 objects degraded (2.468%); 3817/182796 objects misplaced (2.088%); 11 MiB/s, 180 objects/s recovering 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[29889]: audit 2024-09-13T23:00:13.890618+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.178:0/689478817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[29889]: audit 2024-09-13T23:00:13.891084+0000 mon.a (mon.0) 1757 : audit [INF] from='client.25747 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[29889]: audit 2024-09-13T23:00:13.991317+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.178:0/1459583203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:00:14.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:14 smithi178 bash[29889]: audit 2024-09-13T23:00:13.991930+0000 mon.a (mon.0) 1758 : audit [INF] from='client.25883 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:00:14.902 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:14 smithi192 bash[24608]: cluster 2024-09-13T23:00:12.962745+0000 mgr.y (mgr.14613) 837 : cluster [DBG] pgmap v999: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 153 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 803 B/s rd, 0 B/s wr, 503 op/s; 4511/182796 objects degraded (2.468%); 3817/182796 objects misplaced (2.088%); 11 MiB/s, 180 objects/s recovering 2024-09-13T23:00:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:14 smithi192 bash[24608]: audit 2024-09-13T23:00:13.890618+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.178:0/689478817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:00:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:14 smithi192 bash[24608]: audit 2024-09-13T23:00:13.891084+0000 mon.a (mon.0) 1757 : audit [INF] from='client.25747 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:00:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:14 smithi192 bash[24608]: audit 2024-09-13T23:00:13.991317+0000 mon.c (mon.2) 84 : audit [INF] from='client.? v1:172.21.15.178:0/1459583203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:00:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:14 smithi192 bash[24608]: audit 2024-09-13T23:00:13.991930+0000 mon.a (mon.0) 1758 : audit [INF] from='client.25883 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:00:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:14.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:15.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:15.037 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:15.140 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:15.164 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:15.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:15.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:15.435 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:15.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:15.572 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:15.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[29889]: cluster 2024-09-13T23:00:14.621245+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[29889]: audit 2024-09-13T23:00:14.623860+0000 mon.a (mon.0) 1760 : audit [INF] from='client.25747 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[29889]: audit 2024-09-13T23:00:14.624072+0000 mon.a (mon.0) 1761 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[29889]: cluster 2024-09-13T23:00:14.633908+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[22840]: cluster 2024-09-13T23:00:14.621245+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[22840]: audit 2024-09-13T23:00:14.623860+0000 mon.a (mon.0) 1760 : audit [INF] from='client.25747 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:00:15.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[22840]: audit 2024-09-13T23:00:14.624072+0000 mon.a (mon.0) 1761 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T23:00:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:15 smithi178 bash[22840]: cluster 2024-09-13T23:00:14.633908+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-09-13T23:00:15.700 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:15.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:15.831 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:15 smithi192 bash[24608]: cluster 2024-09-13T23:00:14.621245+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:00:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:15 smithi192 bash[24608]: audit 2024-09-13T23:00:14.623860+0000 mon.a (mon.0) 1760 : audit [INF] from='client.25747 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:00:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:15 smithi192 bash[24608]: audit 2024-09-13T23:00:14.624072+0000 mon.a (mon.0) 1761 : audit [INF] from='client.25883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T23:00:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:15 smithi192 bash[24608]: cluster 2024-09-13T23:00:14.633908+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-09-13T23:00:15.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:15.960 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:15.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:16.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:16.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:16.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:16.230 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:16.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:00:15 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:00:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:16.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:16.373 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:16.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:16.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:16.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:16.644 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:16.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:16.772 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:16.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:16 smithi192 bash[24608]: cluster 2024-09-13T23:00:14.963331+0000 mgr.y (mgr.14613) 838 : cluster [DBG] pgmap v1002: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.10k op/s; 4511/172059 objects degraded (2.622%); 2816/172059 objects misplaced (1.637%); 22 MiB/s, 358 objects/s recovering 2024-09-13T23:00:16.911 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:16.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:16.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:16 smithi178 bash[29889]: cluster 2024-09-13T23:00:14.963331+0000 mgr.y (mgr.14613) 838 : cluster [DBG] pgmap v1002: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.10k op/s; 4511/172059 objects degraded (2.622%); 2816/172059 objects misplaced (1.637%); 22 MiB/s, 358 objects/s recovering 2024-09-13T23:00:16.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:16 smithi178 bash[22840]: cluster 2024-09-13T23:00:14.963331+0000 mgr.y (mgr.14613) 838 : cluster [DBG] pgmap v1002: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.10k op/s; 4511/172059 objects degraded (2.622%); 2816/172059 objects misplaced (1.637%); 22 MiB/s, 358 objects/s recovering 2024-09-13T23:00:17.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:17.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:17.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:17.169 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:17.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:17.305 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:17.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:17.434 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:17.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:17.573 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:17.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:17.698 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:17.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:17.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:17.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:17 smithi192 bash[24608]: audit 2024-09-13T23:00:15.978217+0000 mgr.y (mgr.14613) 839 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:17.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:17 smithi178 bash[29889]: audit 2024-09-13T23:00:15.978217+0000 mgr.y (mgr.14613) 839 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:17.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:17 smithi178 bash[22840]: audit 2024-09-13T23:00:15.978217+0000 mgr.y (mgr.14613) 839 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:17.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:17.963 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:17.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:18.065 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:18.096 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:18.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:18.236 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:18.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:18.369 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:18.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:18.493 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:18.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:18.632 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:18.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:18.765 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:18.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:18.879 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:18 smithi192 bash[24608]: cluster 2024-09-13T23:00:16.963818+0000 mgr.y (mgr.14613) 840 : cluster [DBG] pgmap v1003: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 668 op/s; 4511/171300 objects degraded (2.633%); 2816/171300 objects misplaced (1.644%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:18.879 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:18 smithi192 bash[24608]: cluster 2024-09-13T23:00:18.159365+0000 mon.a (mon.0) 1763 : cluster [WRN] Health check update: Degraded data redundancy: 4511/171300 objects degraded (2.633%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:18.879 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:18 smithi192 bash[24608]: cluster 2024-09-13T23:00:18.165011+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-09-13T23:00:18.895 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:18.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:18.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:18.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:18 smithi178 bash[29889]: cluster 2024-09-13T23:00:16.963818+0000 mgr.y (mgr.14613) 840 : cluster [DBG] pgmap v1003: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 668 op/s; 4511/171300 objects degraded (2.633%); 2816/171300 objects misplaced (1.644%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:18.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:18 smithi178 bash[29889]: cluster 2024-09-13T23:00:18.159365+0000 mon.a (mon.0) 1763 : cluster [WRN] Health check update: Degraded data redundancy: 4511/171300 objects degraded (2.633%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:18.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:18 smithi178 bash[29889]: cluster 2024-09-13T23:00:18.165011+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-09-13T23:00:18.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:18 smithi178 bash[22840]: cluster 2024-09-13T23:00:16.963818+0000 mgr.y (mgr.14613) 840 : cluster [DBG] pgmap v1003: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 668 op/s; 4511/171300 objects degraded (2.633%); 2816/171300 objects misplaced (1.644%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:18.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:18 smithi178 bash[22840]: cluster 2024-09-13T23:00:18.159365+0000 mon.a (mon.0) 1763 : cluster [WRN] Health check update: Degraded data redundancy: 4511/171300 objects degraded (2.633%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:18.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:18 smithi178 bash[22840]: cluster 2024-09-13T23:00:18.165011+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-09-13T23:00:19.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:19.025 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:19.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:19.153 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:19.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:19.279 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:19.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:19.404 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:19.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:19.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:19.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:19.648 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:19.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:19.774 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:19.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:19.899 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:19.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:20.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:20.031 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:20.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:20.163 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:20.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:00:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:00:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:20.271 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:20.296 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:20.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:20.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:20.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:20.558 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:20.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:20.682 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:20.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T23:00:20.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:20.810 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:20.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:20 smithi192 bash[24608]: cluster 2024-09-13T23:00:18.964434+0000 mgr.y (mgr.14613) 841 : cluster [DBG] pgmap v1005: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 4511/170448 objects degraded (2.647%); 2816/170448 objects misplaced (1.652%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:20.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:20 smithi192 bash[24608]: cluster 2024-09-13T23:00:19.535135+0000 osd.4 (osd.4) 278 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:20.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:20 smithi192 bash[24608]: cluster 2024-09-13T23:00:19.535199+0000 osd.4 (osd.4) 279 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:20.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:20 smithi192 bash[24608]: audit 2024-09-13T23:00:19.547593+0000 mon.a (mon.0) 1765 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:20.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:20 smithi192 bash[24608]: audit 2024-09-13T23:00:19.549111+0000 mon.b (mon.1) 284 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:20.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:20 smithi192 bash[24608]: cluster 2024-09-13T23:00:19.669143+0000 mon.a (mon.0) 1766 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-09-13T23:00:20.912 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:20.943 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:20.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[22840]: cluster 2024-09-13T23:00:18.964434+0000 mgr.y (mgr.14613) 841 : cluster [DBG] pgmap v1005: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 4511/170448 objects degraded (2.647%); 2816/170448 objects misplaced (1.652%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:20.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[22840]: cluster 2024-09-13T23:00:19.535135+0000 osd.4 (osd.4) 278 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:20.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[22840]: cluster 2024-09-13T23:00:19.535199+0000 osd.4 (osd.4) 279 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:20.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[22840]: audit 2024-09-13T23:00:19.547593+0000 mon.a (mon.0) 1765 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:20.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[22840]: audit 2024-09-13T23:00:19.549111+0000 mon.b (mon.1) 284 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:20.945 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[22840]: cluster 2024-09-13T23:00:19.669143+0000 mon.a (mon.0) 1766 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-09-13T23:00:20.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[29889]: cluster 2024-09-13T23:00:18.964434+0000 mgr.y (mgr.14613) 841 : cluster [DBG] pgmap v1005: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 4511/170448 objects degraded (2.647%); 2816/170448 objects misplaced (1.652%); 10 MiB/s, 166 objects/s recovering 2024-09-13T23:00:20.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[29889]: cluster 2024-09-13T23:00:19.535135+0000 osd.4 (osd.4) 278 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:20.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[29889]: cluster 2024-09-13T23:00:19.535199+0000 osd.4 (osd.4) 279 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:20.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[29889]: audit 2024-09-13T23:00:19.547593+0000 mon.a (mon.0) 1765 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:20.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[29889]: audit 2024-09-13T23:00:19.549111+0000 mon.b (mon.1) 284 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:20.946 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:20 smithi178 bash[29889]: cluster 2024-09-13T23:00:19.669143+0000 mon.a (mon.0) 1766 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-09-13T23:00:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:20.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:21.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:21.069 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:21.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:21.199 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:21.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:21.325 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:21.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:21.449 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:21.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:21.575 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:21.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:21.707 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:21.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.848 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:21.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:21.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:21.974 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:21.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:22.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:22.103 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:22.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:22.244 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:22.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:22.374 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:22.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T23:00:22.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:22.508 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:22.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:22.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:22.763 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:22 smithi178 bash[22840]: cluster 2024-09-13T23:00:20.965157+0000 mgr.y (mgr.14613) 842 : cluster [DBG] pgmap v1007: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 807 B/s rd, 0 B/s wr, 768 op/s; 4511/158226 objects degraded (2.851%); 1688/158226 objects misplaced (1.067%); 14 MiB/s, 221 objects/s recovering 2024-09-13T23:00:22.763 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:22 smithi178 bash[29889]: cluster 2024-09-13T23:00:20.965157+0000 mgr.y (mgr.14613) 842 : cluster [DBG] pgmap v1007: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 807 B/s rd, 0 B/s wr, 768 op/s; 4511/158226 objects degraded (2.851%); 1688/158226 objects misplaced (1.067%); 14 MiB/s, 221 objects/s recovering 2024-09-13T23:00:22.785 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:22.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:22.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:22 smithi192 bash[24608]: cluster 2024-09-13T23:00:20.965157+0000 mgr.y (mgr.14613) 842 : cluster [DBG] pgmap v1007: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 807 B/s rd, 0 B/s wr, 768 op/s; 4511/158226 objects degraded (2.851%); 1688/158226 objects misplaced (1.067%); 14 MiB/s, 221 objects/s recovering 2024-09-13T23:00:22.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:22.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:23.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:23.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:23.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T23:00:23.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:23.193 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:23.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:23.327 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:23.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:23.453 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:23.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:23.619 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:23.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:23.748 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:23.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:23.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:23 smithi192 bash[24608]: cluster 2024-09-13T23:00:23.161010+0000 mon.a (mon.0) 1767 : cluster [WRN] Health check update: Degraded data redundancy: 4511/158226 objects degraded (2.851%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:23.862 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:23 smithi192 bash[24608]: cluster 2024-09-13T23:00:23.173441+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-09-13T23:00:23.875 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:23.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:23.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:23 smithi178 bash[22840]: cluster 2024-09-13T23:00:23.161010+0000 mon.a (mon.0) 1767 : cluster [WRN] Health check update: Degraded data redundancy: 4511/158226 objects degraded (2.851%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:23.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:23 smithi178 bash[22840]: cluster 2024-09-13T23:00:23.173441+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-09-13T23:00:23.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:23 smithi178 bash[29889]: cluster 2024-09-13T23:00:23.161010+0000 mon.a (mon.0) 1767 : cluster [WRN] Health check update: Degraded data redundancy: 4511/158226 objects degraded (2.851%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:23.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:23 smithi178 bash[29889]: cluster 2024-09-13T23:00:23.173441+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-09-13T23:00:23.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:24.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:24.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:24.146 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:24.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:24.280 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:24.382 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:24.418 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:24.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T23:00:24.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:24.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:24.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:24.714 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:24.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:24.827 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:24 smithi178 bash[29889]: cluster 2024-09-13T23:00:22.965983+0000 mgr.y (mgr.14613) 843 : cluster [DBG] pgmap v1008: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s; 4511/157401 objects degraded (2.866%); 1688/157401 objects misplaced (1.072%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T23:00:24.827 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:24 smithi178 bash[22840]: cluster 2024-09-13T23:00:22.965983+0000 mgr.y (mgr.14613) 843 : cluster [DBG] pgmap v1008: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s; 4511/157401 objects degraded (2.866%); 1688/157401 objects misplaced (1.072%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T23:00:24.847 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:24.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:24 smithi192 bash[24608]: cluster 2024-09-13T23:00:22.965983+0000 mgr.y (mgr.14613) 843 : cluster [DBG] pgmap v1008: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s; 4511/157401 objects degraded (2.866%); 1688/157401 objects misplaced (1.072%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T23:00:24.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:24.981 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:24.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:25.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:25.112 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:25.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:25.242 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:25.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:25.373 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:25.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:25.508 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:25.597 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:25.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:25.639 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:25.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:25.766 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:25.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:25.897 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:25.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:26.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:26.039 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:26.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:26.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:00:25 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:00:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.172 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:26.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:26.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:26.304 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:26.408 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:26.436 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:26.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:26.566 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:26.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:26.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:26.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:26.828 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:26.829 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:26 smithi178 bash[29889]: cluster 2024-09-13T23:00:24.967041+0000 mgr.y (mgr.14613) 844 : cluster [DBG] pgmap v1010: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.25k op/s; 4331/145725 objects degraded (2.972%); 420/145725 objects misplaced (0.288%); 22 MiB/s, 378 objects/s recovering 2024-09-13T23:00:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:26.837 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:26 smithi178 bash[22840]: cluster 2024-09-13T23:00:24.967041+0000 mgr.y (mgr.14613) 844 : cluster [DBG] pgmap v1010: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.25k op/s; 4331/145725 objects degraded (2.972%); 420/145725 objects misplaced (0.288%); 22 MiB/s, 378 objects/s recovering 2024-09-13T23:00:26.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:26 smithi192 bash[24608]: cluster 2024-09-13T23:00:24.967041+0000 mgr.y (mgr.14613) 844 : cluster [DBG] pgmap v1010: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.25k op/s; 4331/145725 objects degraded (2.972%); 420/145725 objects misplaced (0.288%); 22 MiB/s, 378 objects/s recovering 2024-09-13T23:00:26.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:26.958 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:26.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:27.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:27.094 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:27.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:27.225 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:27.329 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:27.362 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:27.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:27.507 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:27.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:27.636 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:27.636 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:27 smithi178 bash[29889]: audit 2024-09-13T23:00:25.988610+0000 mgr.y (mgr.14613) 845 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:27.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:27 smithi178 bash[22840]: audit 2024-09-13T23:00:25.988610+0000 mgr.y (mgr.14613) 845 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:27.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:27.759 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:27.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:27.870 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:27 smithi192 bash[24608]: audit 2024-09-13T23:00:25.988610+0000 mgr.y (mgr.14613) 845 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:27.881 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:27.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:27.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:28.004 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:28.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:28.132 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:28.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:28.265 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:28.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:28.393 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:28.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:28.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:28.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:28.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:28.646 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:28.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:28.768 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:28.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:28.882 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:28 smithi192 bash[24608]: cluster 2024-09-13T23:00:26.967710+0000 mgr.y (mgr.14613) 846 : cluster [DBG] pgmap v1011: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 4331/144843 objects degraded (2.990%); 420/144843 objects misplaced (0.290%); 19 MiB/s, 321 objects/s recovering 2024-09-13T23:00:28.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:28 smithi192 bash[24608]: cluster 2024-09-13T23:00:27.576800+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-09-13T23:00:28.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:28 smithi192 bash[24608]: cluster 2024-09-13T23:00:28.162478+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check update: Degraded data redundancy: 4331/144843 objects degraded (2.990%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:28.883 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:28 smithi192 bash[24608]: cluster 2024-09-13T23:00:28.168028+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-09-13T23:00:28.898 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:28.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:29.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:29.026 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:29.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[29889]: cluster 2024-09-13T23:00:26.967710+0000 mgr.y (mgr.14613) 846 : cluster [DBG] pgmap v1011: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 4331/144843 objects degraded (2.990%); 420/144843 objects misplaced (0.290%); 19 MiB/s, 321 objects/s recovering 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[29889]: cluster 2024-09-13T23:00:27.576800+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[29889]: cluster 2024-09-13T23:00:28.162478+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check update: Degraded data redundancy: 4331/144843 objects degraded (2.990%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[29889]: cluster 2024-09-13T23:00:28.168028+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[22840]: cluster 2024-09-13T23:00:26.967710+0000 mgr.y (mgr.14613) 846 : cluster [DBG] pgmap v1011: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 4331/144843 objects degraded (2.990%); 420/144843 objects misplaced (0.290%); 19 MiB/s, 321 objects/s recovering 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[22840]: cluster 2024-09-13T23:00:27.576800+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[22840]: cluster 2024-09-13T23:00:28.162478+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check update: Degraded data redundancy: 4331/144843 objects degraded (2.990%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:29.139 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:28 smithi178 bash[22840]: cluster 2024-09-13T23:00:28.168028+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-09-13T23:00:29.151 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:29.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:29.281 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:29.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:29.411 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:29.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:29.567 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:29.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:29.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:29.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:29.832 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:29.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:29.946 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:00:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:00:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:29.965 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:29.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:30.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:30.089 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:30.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:30.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:30.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:30 smithi178 bash[29889]: cluster 2024-09-13T23:00:28.968768+0000 mgr.y (mgr.14613) 847 : cluster [DBG] pgmap v1014: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 745 op/s; 4331/144000 objects degraded (3.008%); 420/144000 objects misplaced (0.292%); 13 MiB/s, 241 objects/s recovering 2024-09-13T23:00:30.331 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:30 smithi178 bash[29889]: cluster 2024-09-13T23:00:29.176018+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-09-13T23:00:30.331 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:30 smithi178 bash[22840]: cluster 2024-09-13T23:00:28.968768+0000 mgr.y (mgr.14613) 847 : cluster [DBG] pgmap v1014: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 745 op/s; 4331/144000 objects degraded (3.008%); 420/144000 objects misplaced (0.292%); 13 MiB/s, 241 objects/s recovering 2024-09-13T23:00:30.332 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:30 smithi178 bash[22840]: cluster 2024-09-13T23:00:29.176018+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-09-13T23:00:30.346 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:30.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:30 smithi192 bash[24608]: cluster 2024-09-13T23:00:28.968768+0000 mgr.y (mgr.14613) 847 : cluster [DBG] pgmap v1014: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 745 op/s; 4331/144000 objects degraded (3.008%); 420/144000 objects misplaced (0.292%); 13 MiB/s, 241 objects/s recovering 2024-09-13T23:00:30.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:30 smithi192 bash[24608]: cluster 2024-09-13T23:00:29.176018+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-09-13T23:00:30.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:30.475 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:30.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:30.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:30.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:30.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:30.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:30.860 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:30.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:30.992 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:30.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:31.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:31.122 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:31.225 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:31.254 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:31.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:31.389 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:31.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:31.510 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:31.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:31.635 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:31.738 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:31.767 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:31.770 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:31 smithi178 bash[29889]: cluster 2024-09-13T23:00:30.582181+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-09-13T23:00:31.771 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:31 smithi178 bash[22840]: cluster 2024-09-13T23:00:30.582181+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-09-13T23:00:31.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:31.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:31.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:31.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:31 smithi192 bash[24608]: cluster 2024-09-13T23:00:30.582181+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-09-13T23:00:32.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:32.039 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:32.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:32.165 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:32.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:32.294 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:32.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:32.425 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:32.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:32.552 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:32.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:32.680 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:32.764 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:32 smithi178 bash[22840]: cluster 2024-09-13T23:00:30.969574+0000 mgr.y (mgr.14613) 848 : cluster [DBG] pgmap v1017: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 2731/134241 objects degraded (2.034%); 307/134241 objects misplaced (0.229%); 8.9 MiB/s, 428 objects/s recovering 2024-09-13T23:00:32.765 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:32 smithi178 bash[22840]: cluster 2024-09-13T23:00:31.588800+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-09-13T23:00:32.765 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:32 smithi178 bash[29889]: cluster 2024-09-13T23:00:30.969574+0000 mgr.y (mgr.14613) 848 : cluster [DBG] pgmap v1017: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 2731/134241 objects degraded (2.034%); 307/134241 objects misplaced (0.229%); 8.9 MiB/s, 428 objects/s recovering 2024-09-13T23:00:32.765 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:32 smithi178 bash[29889]: cluster 2024-09-13T23:00:31.588800+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-09-13T23:00:32.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:32.816 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:32 smithi192 bash[24608]: cluster 2024-09-13T23:00:30.969574+0000 mgr.y (mgr.14613) 848 : cluster [DBG] pgmap v1017: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 2731/134241 objects degraded (2.034%); 307/134241 objects misplaced (0.229%); 8.9 MiB/s, 428 objects/s recovering 2024-09-13T23:00:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:32 smithi192 bash[24608]: cluster 2024-09-13T23:00:31.588800+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-09-13T23:00:32.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:32.939 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:32.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:33.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.064 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:33.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:33.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.344 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:33.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:33.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:33.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:33.610 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:33.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:33.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:33.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.857 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:33 smithi178 bash[29889]: cluster 2024-09-13T23:00:33.163798+0000 mon.a (mon.0) 1775 : cluster [WRN] Health check update: Degraded data redundancy: 2731/134241 objects degraded (2.034%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:33.857 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:33 smithi178 bash[29889]: cluster 2024-09-13T23:00:33.171616+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-09-13T23:00:33.861 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:33 smithi178 bash[22840]: cluster 2024-09-13T23:00:33.163798+0000 mon.a (mon.0) 1775 : cluster [WRN] Health check update: Degraded data redundancy: 2731/134241 objects degraded (2.034%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:33.861 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:33 smithi178 bash[22840]: cluster 2024-09-13T23:00:33.171616+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-09-13T23:00:33.880 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:33.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:33.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:33 smithi192 bash[24608]: cluster 2024-09-13T23:00:33.163798+0000 mon.a (mon.0) 1775 : cluster [WRN] Health check update: Degraded data redundancy: 2731/134241 objects degraded (2.034%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T23:00:33.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:33 smithi192 bash[24608]: cluster 2024-09-13T23:00:33.171616+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-09-13T23:00:33.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:34.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:34.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:34.136 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:34.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:34.267 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:34.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:34.398 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:34.501 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:34.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:34.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:34.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:34 smithi192 bash[24608]: cluster 2024-09-13T23:00:32.970099+0000 mgr.y (mgr.14613) 849 : cluster [DBG] pgmap v1019: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 780 op/s; 2731/133608 objects degraded (2.044%); 307/133608 objects misplaced (0.230%); 7.4 MiB/s, 356 objects/s recovering 2024-09-13T23:00:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:34 smithi192 bash[24608]: audit 2024-09-13T23:00:34.561929+0000 mon.b (mon.1) 285 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:34.666 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:34.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:34.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:34.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:34.923 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:34.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:34.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:34 smithi178 bash[29889]: cluster 2024-09-13T23:00:32.970099+0000 mgr.y (mgr.14613) 849 : cluster [DBG] pgmap v1019: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 780 op/s; 2731/133608 objects degraded (2.044%); 307/133608 objects misplaced (0.230%); 7.4 MiB/s, 356 objects/s recovering 2024-09-13T23:00:34.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:34 smithi178 bash[29889]: audit 2024-09-13T23:00:34.561929+0000 mon.b (mon.1) 285 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:34.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:34 smithi178 bash[22840]: cluster 2024-09-13T23:00:32.970099+0000 mgr.y (mgr.14613) 849 : cluster [DBG] pgmap v1019: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 780 op/s; 2731/133608 objects degraded (2.044%); 307/133608 objects misplaced (0.230%); 7.4 MiB/s, 356 objects/s recovering 2024-09-13T23:00:34.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:34 smithi178 bash[22840]: audit 2024-09-13T23:00:34.561929+0000 mon.b (mon.1) 285 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:35.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:35.051 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:35.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:35.180 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:35.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:35.313 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:35.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:35.451 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:35.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:35.580 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:35.585 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T23:00:35.585 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T23:00:35.620 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T23:00:35.620 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T23:00:35.665 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T23:00:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T23:00:35.685 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:35.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:35.721 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T23:00:35.722 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T23:00:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: "size": 20, 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 600, 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: "ops": [ 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160548 10.f 10:f451908b:::benchmark_data_smithi178_111521_object49812:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.219271+0000", 2024-09-13T23:00:35.760 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.531799057, 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3127255920000001, 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160548 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.761 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219271+0000", 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219271+0000", 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.762 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219272+0000", 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.77e-06 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219285+0000", 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3072e-05 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219296+0000", 2024-09-13T23:00:35.763 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0868e-05 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219303+0000", 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6340000000000003e-06 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219354+0000", 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0804999999999999e-05 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.764 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.219369+0000", 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5006e-05 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218147+0000", 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99877783799999997 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218161+0000", 2024-09-13T23:00:35.765 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4151000000000001e-05 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.530243+0000", 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3120817810000001 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.531996+0000", 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001753667 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.766 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160568 10.7 10:e5973984:::benchmark_data_smithi178_111521_object49832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.244871+0000", 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.50619887300002, 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2871166450000002, 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.767 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160568 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.244871+0000", 2024-09-13T23:00:35.768 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.244871+0000", 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.244873+0000", 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.542e-06 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.769 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.244880+0000", 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.759e-06 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.244896+0000", 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6328999999999999e-05 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.244904+0000", 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.6290000000000001e-06 2024-09-13T23:00:35.770 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.245153+0000", 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00024883700000000002 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.245161+0000", 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9060000000000002e-06 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218269+0000", 2024-09-13T23:00:35.771 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.97310817599999999 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218273+0000", 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2930000000000002e-06 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.485459+0000", 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2671861719999999 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.772 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.487678+0000", 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0022183340000000002 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 2,5,6", 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.488006+0000", 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00032816699999999998 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.500741+0000", 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.012734618 2024-09-13T23:00:35.773 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.500904+0000", 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016301999999999999 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.500971+0000", 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7113999999999995e-05 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.774 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.531856+0000", 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.030884852000000001 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.531905+0000", 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.8980999999999997e-05 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.531987+0000", 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.2916000000000004e-05 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.775 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160580 10.f 10:f4ce83f6:::benchmark_data_smithi178_111521_object49844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.257995+0000", 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.49307498100001, 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3098952220000002, 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.776 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160580 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.257995+0000", 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.257995+0000", 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.777 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.257997+0000", 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.967e-06 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258003+0000", 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.331e-06 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258014+0000", 2024-09-13T23:00:35.778 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0828e-05 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258022+0000", 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8169999999999999e-06 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258114+0000", 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.2616999999999998e-05 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.779 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258125+0000", 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0798999999999999e-05 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218199+0000", 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.96007383700000004 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218207+0000", 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7100000000000007e-06 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.780 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.532725+0000", 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3145187810000001 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.533249+0000", 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00052338199999999997 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.781 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.533552+0000", 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00030363500000000002 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.556316+0000", 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.022763597 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.556358+0000", 2024-09-13T23:00:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.1647999999999998e-05 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.563586+0000", 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0072288300000000003 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.567711+0000", 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0041242049999999997 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.783 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.567769+0000", 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8489999999999997e-05 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.567890+0000", 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000120748 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.784 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160581 10.7 10:e4a3cd48:::benchmark_data_smithi178_111521_object49845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.258802+0000", 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.49226817300001, 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2737789940000002, 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160581 2024-09-13T23:00:35.785 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258802+0000", 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258802+0000", 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.786 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258803+0000", 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6199999999999999e-06 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258818+0000", 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4983e-05 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258832+0000", 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3596e-05 2024-09-13T23:00:35.787 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258843+0000", 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0908e-05 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258950+0000", 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010739 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.788 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.258962+0000", 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2082999999999999e-05 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218285+0000", 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.95932324599999996 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218289+0000", 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.8399999999999997e-06 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.501154+0000", 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2828650939999999 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.501278+0000", 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000123984 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 2,5,6", 2024-09-13T23:00:35.790 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.501387+0000", 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000109094 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.531995+0000", 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.030607269999999999 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.532332+0000", 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00033734699999999999 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.791 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.532378+0000", 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.5796e-05 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.532419+0000", 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0815000000000001e-05 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.532477+0000", 2024-09-13T23:00:35.792 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8576999999999998e-05 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.532581+0000", 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010335099999999999 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160584 10.f 10:f8ebef4d:::benchmark_data_smithi178_111521_object49848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.259134+0000", 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.49193558500002, 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3090685820000001, 2024-09-13T23:00:35.793 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160584 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259134+0000", 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.794 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259134+0000", 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259136+0000", 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4300000000000001e-06 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259140+0000", 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.5589999999999999e-06 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.795 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259152+0000", 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1946999999999999e-05 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259162+0000", 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6569999999999992e-06 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259208+0000", 2024-09-13T23:00:35.796 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.6309e-05 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.259217+0000", 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3559999999999999e-06 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218220+0000", 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.95900295599999996 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.797 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218224+0000", 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9609999999999999e-06 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.541837+0000", 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3236126969999999 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.541976+0000", 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013875000000000001 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.798 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.542125+0000", 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014953300000000001 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.556559+0000", 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.014434037 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.563628+0000", 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.007068375 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.799 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.563642+0000", 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3978999999999999e-05 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.567936+0000", 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0042940749999999996 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568093+0000", 2024-09-13T23:00:35.800 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000157563 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568203+0000", 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000109398 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160590 10.7 10:eaf17787:::benchmark_data_smithi178_111521_object49854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.265011+0000", 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.486059195, 2024-09-13T23:00:35.801 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.313623867, 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160590 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265011+0000", 2024-09-13T23:00:35.802 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265011+0000", 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265013+0000", 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3549999999999999e-06 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.803 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265019+0000", 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.5400000000000001e-06 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265033+0000", 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3526999999999999e-05 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265041+0000", 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.3420000000000008e-06 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.804 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265095+0000", 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.3446000000000003e-05 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265102+0000", 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.2289999999999997e-06 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218324+0000", 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.95322209599999996 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.805 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218332+0000", 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.089e-06 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.558920+0000", 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3405880589999999 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.559031+0000", 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011069300000000001 2024-09-13T23:00:35.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 2,5,6", 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.559152+0000", 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000121178 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568079+0000", 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0089269569999999993 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.807 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568586+0000", 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00050717399999999999 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.577645+0000", 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0090584610000000003 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.577950+0000", 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00030549400000000001 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.808 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.578495+0000", 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00054507299999999998 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.578634+0000", 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000139154 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.809 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.25538.0:160591 10.f 10:fb1f7aa7:::benchmark_data_smithi178_111521_object49855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:55:59.265074+0000", 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 276.48599593500001, 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3033812259999999, 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.25538", 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 160591 2024-09-13T23:00:35.810 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265074+0000", 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265074+0000", 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.811 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265075+0000", 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4099999999999997e-07 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265078+0000", 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.382e-06 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265085+0000", 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1609999999999997e-06 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.812 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265091+0000", 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.6400000000000002e-06 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265120+0000", 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.8937000000000001e-05 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:55:59.265124+0000", 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2309999999999998e-06 2024-09-13T23:00:35.813 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218247+0000", 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.95312266700000003 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:00.218255+0000", 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.7180000000000002e-06 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.814 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.553433+0000", 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3351773140000001 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.553560+0000", 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012678799999999999 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 1,2,6", 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.553695+0000", 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000135852 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.815 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.563648+0000", 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0099522459999999997 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.563691+0000", 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.3059000000000002e-05 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.816 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.563732+0000", 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.1227999999999999e-05 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568250+0000", 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0045181509999999998 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568344+0000", 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3796999999999997e-05 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.817 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:56:01.568455+0000", 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00011111399999999999 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47181 11.9 11:9e1aeba3:::benchmark_data_smithi178_128208_object47180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.690301+0000", 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 114.06076876500001, 2024-09-13T23:00:35.818 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3636829499999998, 2024-09-13T23:00:35.819 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.819 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.819 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47181 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690301+0000", 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.820 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690301+0000", 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690303+0000", 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.6589999999999999e-06 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690328+0000", 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.5870000000000001e-05 2024-09-13T23:00:35.821 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690343+0000", 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4195999999999999e-05 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690351+0000", 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0909999999999993e-06 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.822 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690514+0000", 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016296 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.690908+0000", 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00039478400000000002 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.691055+0000", 2024-09-13T23:00:35.823 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000146895 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.694355+0000", 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0032993749999999998 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695973+0000", 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.001618469 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.824 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.730609+0000", 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.034636062000000002 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.730617+0000", 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9359999999999999e-06 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.722702+0000", 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99208436 2024-09-13T23:00:35.825 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.722716+0000", 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4504e-05 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.053913+0000", 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.331197221 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.053984+0000", 2024-09-13T23:00:35.826 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.0568000000000006e-05 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47185 11.9 11:91c6c8a3:::benchmark_data_smithi178_128208_object47184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.695824+0000", 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 114.055245547, 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3602069530000001, 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.827 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47185 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695824+0000", 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695824+0000", 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.828 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695826+0000", 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.542e-06 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695849+0000", 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3448999999999999e-05 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.829 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695858+0000", 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1339999999999993e-06 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.695864+0000", 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.0449999999999997e-06 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.696005+0000", 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014095899999999999 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.830 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.696110+0000", 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000104521 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.696196+0000", 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.6489999999999994e-05 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.699917+0000", 2024-09-13T23:00:35.831 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.003720723 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.700730+0000", 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00081345699999999996 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.730853+0000", 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.030122804 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.832 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.730864+0000", 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1018e-05 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.722785+0000", 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.99192075999999996 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.722795+0000", 2024-09-13T23:00:35.833 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.8339999999999994e-06 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.055946+0000", 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.333151161 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.056031+0000", 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.5055999999999993e-05 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.834 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47189 11.8 11:1d806ad1:::benchmark_data_smithi178_128208_object47188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.701239+0000", 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 114.049830852, 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3186114299999998, 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.835 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47189 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.701239+0000", 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.701239+0000", 2024-09-13T23:00:35.836 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.701241+0000", 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.182e-06 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.701272+0000", 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.0818999999999997e-05 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.837 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.701288+0000", 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5943e-05 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.701296+0000", 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.1079999999999993e-06 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.703372+0000", 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0020762200000000001 2024-09-13T23:00:35.838 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.703519+0000", 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00014653600000000001 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,5", 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.703658+0000", 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013909299999999999 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.839 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.722154+0000", 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.018496077 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.722168+0000", 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4636e-05 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.706506+0000", 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.98433775999999995 2024-09-13T23:00:35.840 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.706520+0000", 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4092e-05 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:43.998595+0000", 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.292074266 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.841 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for degraded object", 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:43.999176+0000", 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00058104200000000004 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.019721+0000", 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.020545188999999998 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.019850+0000", 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000129467 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.842 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47196 11.9 11:928cb5b0:::benchmark_data_smithi178_128208_object47195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.742511+0000", 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 114.008559181, 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3270441370000001, 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47196 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.843 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742511+0000", 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742511+0000", 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742512+0000", 2024-09-13T23:00:35.844 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3349999999999999e-06 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742539+0000", 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.6741e-05 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742548+0000", 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.9400000000000008e-06 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.845 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742556+0000", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0320000000000003e-06 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743391+0000", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00083558799999999996 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.743399+0000", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8869999999999992e-06 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.722822+0000", 2024-09-13T23:00:35.846 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.97942268300000002 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.722830+0000", 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0730000000000005e-06 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.056089+0000", 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3332594499999999 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.847 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.056200+0000", 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000110414 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 0,6", 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.056303+0000", 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00010317100000000001 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.063006+0000", 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0067030229999999998 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.848 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.064738+0000", 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0017321890000000001 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.066480+0000", 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0017420560000000001 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.066531+0000", 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.0852e-05 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.849 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.069555+0000", 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.003023703 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47198 11.8 11:170b5e93:::benchmark_data_smithi178_128208_object47197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.742568+0000", 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 114.008502014, 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2821253320000001, 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.850 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47198 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742568+0000", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742568+0000", 2024-09-13T23:00:35.851 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742568+0000", 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.1200000000000002e-07 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742590+0000", 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.1650000000000001e-05 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742598+0000", 2024-09-13T23:00:35.852 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8679999999999999e-06 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742605+0000", 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.289e-06 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742956+0000", 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00035109799999999999 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.853 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742964+0000", 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5170000000000002e-06 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.716847+0000", 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.97388356799999998 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.716860+0000", 2024-09-13T23:00:35.854 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2554999999999999e-05 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.001807+0000", 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.284947166 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.001984+0000", 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00017677599999999999 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.002162+0000", 2024-09-13T23:00:35.855 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000177737 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.004601+0000", 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0024391339999999999 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.012915+0000", 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0083145289999999993 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.019922+0000", 2024-09-13T23:00:35.856 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0070066540000000002 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.019986+0000", 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3947000000000003e-05 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.024693+0000", 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0047071320000000002 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.857 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:47199 11.a 11:5f594222:::benchmark_data_smithi178_128208_object47198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T22:58:41.742614+0000", 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 114.00845534600001, 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2771027730000002, 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 47199 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.858 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742614+0000", 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742614+0000", 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742615+0000", 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4499999999999997e-07 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.859 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742638+0000", 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.3017999999999999e-05 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742838+0000", 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000200003 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742846+0000", 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.7849999999999992e-06 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.860 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742901+0000", 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.4727e-05 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:41.742912+0000", 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1678999999999999e-05 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.712529+0000", 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.96961642199999998 2024-09-13T23:00:35.861 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:42.712544+0000", 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.5084e-05 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.006570+0000", 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2940261319999999 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.006709+0000", 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013928000000000001 2024-09-13T23:00:35.862 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.006842+0000", 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000132898 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017046+0000", 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010204181 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017114+0000", 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7967999999999997e-05 2024-09-13T23:00:35.863 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017131+0000", 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.7147e-05 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.017342+0000", 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00021043400000000001 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.864 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T22:58:44.019717+0000", 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0023753699999999999 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107710 11.8 11:1ab65455:::benchmark_data_smithi178_128208_object22680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.584128+0000", 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 29.166941411, 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.037444601, 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.865 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "reached pg", 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107710 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.584128+0000", 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.866 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.584128+0000", 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.584129+0000", 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4200000000000004e-07 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.584133+0000", 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.7100000000000001e-06 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.867 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.584141+0000", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.605e-06 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.584146+0000", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8690000000000001e-06 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.587953+0000", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0038064040000000002 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.868 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.588090+0000", 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013720099999999999 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.588221+0000", 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00013132700000000001 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.592424+0000", 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0042025279999999996 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.869 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.594493+0000", 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.002069581 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.740919+0000", 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.14642560199999999 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.740933+0000", 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.3491e-05 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.870 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.621457+0000", 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.880524325 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.621573+0000", 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000116016 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.871 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107727 11.8 11:13c0f8bb:::benchmark_data_smithi178_128208_object22697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.716890+0000", 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 29.034179875, 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.91360812800000002, 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107727 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.872 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.716890+0000", 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.716890+0000", 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.716892+0000", 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 2.2680000000000001e-06 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.873 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.716898+0000", 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8409999999999996e-06 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.716910+0000", 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.2536e-05 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.716920+0000", 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.1090000000000004e-06 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.874 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.741043+0000", 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.024123051 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.741053+0000", 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.9639999999999998e-06 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.621958+0000", 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.88090589500000005 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.875 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.622034+0000", 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.5978999999999999e-05 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.622103+0000", 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.8448000000000006e-05 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630282+0000", 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0081793439999999999 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.876 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630324+0000", 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2098000000000002e-05 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630404+0000", 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9775000000000001e-05 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.877 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630455+0000", 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1088999999999998e-05 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630498+0000", 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.2731000000000001e-05 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107763 11.8 11:1cac8ce1:::benchmark_data_smithi178_128208_object22733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-13T23:00:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.764034+0000", 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 28.987035425999998, 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.87005613000000004, 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107763 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.879 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764034+0000", 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764034+0000", 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764035+0000", 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.004e-06 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.880 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764039+0000", 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.7510000000000002e-06 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764047+0000", 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.9500000000000001e-06 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764053+0000", 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8329999999999999e-06 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.881 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764178+0000", 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012481499999999999 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.764186+0000", 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.8720000000000008e-06 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.623903+0000", 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.85971682400000005 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.882 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.623962+0000", 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.9042999999999997e-05 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.624027+0000", 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.4635999999999996e-05 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630356+0000", 2024-09-13T23:00:35.883 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0063288449999999996 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630555+0000", 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00019930600000000001 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.633812+0000", 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0032568079999999999 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.884 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.634046+0000", 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00023424700000000001 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.634090+0000", 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.4196000000000002e-05 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.885 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107770 11.8 11:14969aa4:::benchmark_data_smithi178_128208_object22740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.768045+0000", 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 28.983024533999998, 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.86622980900000002, 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107770 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.886 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768045+0000", 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768045+0000", 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768046+0000", 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6000000000000003e-07 2024-09-13T23:00:35.887 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768049+0000", 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.0599999999999999e-06 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768056+0000", 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7970000000000003e-06 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768062+0000", 2024-09-13T23:00:35.888 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.3450000000000002e-06 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768330+0000", 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00026763500000000001 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.768339+0000", 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3819999999999993e-06 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.889 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.625141+0000", 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.85680213199999999 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.625199+0000", 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.8189000000000001e-05 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.625261+0000", 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1954999999999999e-05 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.890 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630572+0000", 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0053104930000000003 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630677+0000", 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000105609 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.634146+0000", 2024-09-13T23:00:35.891 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0034682020000000001 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.634234+0000", 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.8738000000000003e-05 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.634275+0000", 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.0611999999999997e-05 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.892 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107780 11.8 11:19c3ce14:::benchmark_data_smithi178_128208_object22750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.779146+0000", 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 28.971924147999999, 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.85885118100000002, 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.893 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107780 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.779146+0000", 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.779146+0000", 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.894 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.779147+0000", 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.8610000000000001e-06 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.779153+0000", 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 5.1460000000000004e-06 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.779162+0000", 2024-09-13T23:00:35.895 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.6409999999999997e-06 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.779169+0000", 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.7429999999999996e-06 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.780688+0000", 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0015188039999999999 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.896 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.780697+0000", 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.4839999999999993e-06 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.626225+0000", 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.84552775800000002 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.626314+0000", 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.9109000000000004e-05 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.897 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.626393+0000", 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 7.8646999999999994e-05 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630584+0000", 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0041908429999999997 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.637808+0000", 2024-09-13T23:00:35.898 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0072246990000000002 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.637929+0000", 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012084899999999999 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.637965+0000", 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6167000000000001e-05 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.899 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.637997+0000", 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.1430000000000002e-05 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107841 11.8 11:1564e088:::benchmark_data_smithi178_128208_object22811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.846104+0000", 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 28.904965820000001, 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.79202662400000001, 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.900 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107841 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.846104+0000", 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.901 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.846104+0000", 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.846105+0000", 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.4799999999999997e-07 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.846108+0000", 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.6710000000000001e-06 2024-09-13T23:00:35.902 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.846115+0000", 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.9419999999999998e-06 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.846122+0000", 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.1739999999999997e-06 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.903 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.848834+0000", 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0027125859999999999 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.848843+0000", 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 9.3409999999999992e-06 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.628484+0000", 2024-09-13T23:00:35.904 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.77964021900000002 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.628610+0000", 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012635200000000001 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.628738+0000", 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00012835 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.905 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.633772+0000", 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0050335969999999999 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.637880+0000", 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0041078190000000004 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.638046+0000", 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00016633300000000001 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.906 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.638091+0000", 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 4.4475999999999999e-05 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.638130+0000", 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9916000000000002e-05 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "description": "osd_op(client.15909.0:107858 11.8 11:11c8cee1:::benchmark_data_smithi178_128208_object22828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "initiated_at": "2024-09-13T23:00:06.864810+0000", 2024-09-13T23:00:35.907 INFO:teuthology.orchestra.run.smithi178.stdout: "age": 28.886259937999998, 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.78599870800000005, 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "continuous": false, 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "type_data": { 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "client_info": { 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "client": "client.15909", 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "tid": 107858 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "events": [ 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "initiated", 2024-09-13T23:00:35.908 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.864810+0000", 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "header_read", 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.864810+0000", 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "throttled", 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.864811+0000", 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.4610000000000001e-06 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.909 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "all_read", 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.864815+0000", 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 3.9280000000000003e-06 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "dispatched", 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.864927+0000", 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.000111895 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "queued_for_pg", 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.864937+0000", 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.0220999999999999e-05 2024-09-13T23:00:35.910 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.865124+0000", 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.00018674699999999999 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for peered", 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:06.865135+0000", 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 1.1031e-05 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "reached_pg", 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630746+0000", 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.76561133999999997 2024-09-13T23:00:35.911 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "started", 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630813+0000", 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.6489000000000007e-05 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "waiting for subops from 5,6", 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.630875+0000", 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.2592e-05 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "op_commit", 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.634105+0000", 2024-09-13T23:00:35.912 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0032297269999999999 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.644673+0000", 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.010567338000000001 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.650659+0000", 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 0.0059865939999999996 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "commit_sent", 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.650740+0000", 2024-09-13T23:00:35.913 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 8.0982000000000005e-05 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: "event": "done", 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: "time": "2024-09-13T23:00:07.650808+0000", 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: "duration": 6.8362999999999996e-05 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-09-13T23:00:35.914 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-09-13T23:00:35.915 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T23:00:35.915 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T23:00:35.915 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T23:00:35.916 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T23:00:35.916 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T23:00:35.916 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T23:00:35.916 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T23:00:35.917 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T23:00:35.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:35 smithi178 bash[29889]: audit 2024-09-13T23:00:35.542477+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.178:0/673706653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:00:35.917 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:35 smithi178 bash[22840]: audit 2024-09-13T23:00:35.542477+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.178:0/673706653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:00:35.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:35.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:35 smithi192 bash[24608]: audit 2024-09-13T23:00:35.542477+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.178:0/673706653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:00:35.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:35.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:36.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:36.085 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:36.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:36.210 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:36.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:36.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:00:35 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:00:36.341 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:36.444 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:36.477 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:36.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:36.606 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:36.617 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T23:00:36.622 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is set 2024-09-13T23:00:36.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:36.736 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:36.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:36.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[22840]: cluster 2024-09-13T23:00:34.971076+0000 mgr.y (mgr.14613) 850 : cluster [DBG] pgmap v1021: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.14k op/s; 2523/122046 objects degraded (2.067%); 96/122046 objects misplaced (0.079%); 8.4 MiB/s, 200 objects/s recovering 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[22840]: audit 2024-09-13T23:00:35.970194+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.178:0/949785636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[22840]: audit 2024-09-13T23:00:35.971217+0000 mon.a (mon.0) 1777 : audit [INF] from='client.25768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[22840]: audit 2024-09-13T23:00:36.095373+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.178:0/3418795641' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[29889]: cluster 2024-09-13T23:00:34.971076+0000 mgr.y (mgr.14613) 850 : cluster [DBG] pgmap v1021: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.14k op/s; 2523/122046 objects degraded (2.067%); 96/122046 objects misplaced (0.079%); 8.4 MiB/s, 200 objects/s recovering 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[29889]: audit 2024-09-13T23:00:35.970194+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.178:0/949785636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[29889]: audit 2024-09-13T23:00:35.971217+0000 mon.a (mon.0) 1777 : audit [INF] from='client.25768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:00:36.853 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:36 smithi178 bash[29889]: audit 2024-09-13T23:00:36.095373+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.178:0/3418795641' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:00:36.875 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:36.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:36.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:36 smithi192 bash[24608]: cluster 2024-09-13T23:00:34.971076+0000 mgr.y (mgr.14613) 850 : cluster [DBG] pgmap v1021: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.14k op/s; 2523/122046 objects degraded (2.067%); 96/122046 objects misplaced (0.079%); 8.4 MiB/s, 200 objects/s recovering 2024-09-13T23:00:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:36 smithi192 bash[24608]: audit 2024-09-13T23:00:35.970194+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.178:0/949785636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:00:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:36 smithi192 bash[24608]: audit 2024-09-13T23:00:35.971217+0000 mon.a (mon.0) 1777 : audit [INF] from='client.25768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:00:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:36 smithi192 bash[24608]: audit 2024-09-13T23:00:36.095373+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.178:0/3418795641' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:00:36.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:37.006 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:37.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:37.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:37.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:37.264 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:37.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:37.395 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:37.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:37.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:37.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:37.648 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:37.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:37.773 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:37.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:37.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[29889]: audit 2024-09-13T23:00:35.998125+0000 mgr.y (mgr.14613) 851 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:37.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[29889]: cluster 2024-09-13T23:00:36.613589+0000 mon.a (mon.0) 1779 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:00:37.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[29889]: audit 2024-09-13T23:00:36.615193+0000 mon.a (mon.0) 1780 : audit [INF] from='client.25768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T23:00:37.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[29889]: audit 2024-09-13T23:00:36.615433+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.178:0/3418795641' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:00:37.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[29889]: cluster 2024-09-13T23:00:36.620487+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-09-13T23:00:37.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[22840]: audit 2024-09-13T23:00:35.998125+0000 mgr.y (mgr.14613) 851 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:37.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[22840]: cluster 2024-09-13T23:00:36.613589+0000 mon.a (mon.0) 1779 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:00:37.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[22840]: audit 2024-09-13T23:00:36.615193+0000 mon.a (mon.0) 1780 : audit [INF] from='client.25768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T23:00:37.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[22840]: audit 2024-09-13T23:00:36.615433+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.178:0/3418795641' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:00:37.888 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:37 smithi178 bash[22840]: cluster 2024-09-13T23:00:36.620487+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-09-13T23:00:37.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:37.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:37 smithi192 bash[24608]: audit 2024-09-13T23:00:35.998125+0000 mgr.y (mgr.14613) 851 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:37 smithi192 bash[24608]: cluster 2024-09-13T23:00:36.613589+0000 mon.a (mon.0) 1779 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:00:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:37 smithi192 bash[24608]: audit 2024-09-13T23:00:36.615193+0000 mon.a (mon.0) 1780 : audit [INF] from='client.25768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T23:00:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:37 smithi192 bash[24608]: audit 2024-09-13T23:00:36.615433+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.178:0/3418795641' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:00:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:37 smithi192 bash[24608]: cluster 2024-09-13T23:00:36.620487+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-09-13T23:00:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:37.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:38.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:38.035 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:38.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:38.167 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:38.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:38.303 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:38.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:38.428 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:38.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:38.555 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:38.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:38.686 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:38.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:38.819 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:38.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:38 smithi192 bash[24608]: cluster 2024-09-13T23:00:36.977867+0000 mgr.y (mgr.14613) 852 : cluster [DBG] pgmap v1023: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 722 op/s; 2523/121224 objects degraded (2.081%); 96/121224 objects misplaced (0.079%); 2.2 MiB/s, 69 objects/s recovering 2024-09-13T23:00:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:38 smithi192 bash[24608]: cluster 2024-09-13T23:00:38.166360+0000 mon.a (mon.0) 1783 : cluster [WRN] Health check update: Degraded data redundancy: 2523/121224 objects degraded (2.081%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:38 smithi192 bash[24608]: cluster 2024-09-13T23:00:38.174962+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-09-13T23:00:38.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:38.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:38 smithi178 bash[29889]: cluster 2024-09-13T23:00:36.977867+0000 mgr.y (mgr.14613) 852 : cluster [DBG] pgmap v1023: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 722 op/s; 2523/121224 objects degraded (2.081%); 96/121224 objects misplaced (0.079%); 2.2 MiB/s, 69 objects/s recovering 2024-09-13T23:00:38.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:38 smithi178 bash[29889]: cluster 2024-09-13T23:00:38.166360+0000 mon.a (mon.0) 1783 : cluster [WRN] Health check update: Degraded data redundancy: 2523/121224 objects degraded (2.081%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:38.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:38 smithi178 bash[29889]: cluster 2024-09-13T23:00:38.174962+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-09-13T23:00:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:38 smithi178 bash[22840]: cluster 2024-09-13T23:00:36.977867+0000 mgr.y (mgr.14613) 852 : cluster [DBG] pgmap v1023: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 722 op/s; 2523/121224 objects degraded (2.081%); 96/121224 objects misplaced (0.079%); 2.2 MiB/s, 69 objects/s recovering 2024-09-13T23:00:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:38 smithi178 bash[22840]: cluster 2024-09-13T23:00:38.166360+0000 mon.a (mon.0) 1783 : cluster [WRN] Health check update: Degraded data redundancy: 2523/121224 objects degraded (2.081%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:38 smithi178 bash[22840]: cluster 2024-09-13T23:00:38.174962+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-09-13T23:00:38.945 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:38.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:39.049 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:39.075 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:39.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.207 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:39.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:39.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.472 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:39.576 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:39.600 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:39.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.737 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:39.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:39.853 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:00:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:00:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:39.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:39.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:40.011 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:40.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:40.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:40.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:40.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:40.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:40.404 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:40.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:40.526 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:40.629 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:40.659 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:40.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:40.773 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:40 smithi192 bash[24608]: cluster 2024-09-13T23:00:38.978481+0000 mgr.y (mgr.14613) 853 : cluster [DBG] pgmap v1025: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 736 op/s; 2523/120354 objects degraded (2.096%); 96/120354 objects misplaced (0.080%); 2.2 MiB/s, 69 objects/s recovering 2024-09-13T23:00:40.785 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:40.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:40.910 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:40.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:40.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:40 smithi178 bash[29889]: cluster 2024-09-13T23:00:38.978481+0000 mgr.y (mgr.14613) 853 : cluster [DBG] pgmap v1025: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 736 op/s; 2523/120354 objects degraded (2.096%); 96/120354 objects misplaced (0.080%); 2.2 MiB/s, 69 objects/s recovering 2024-09-13T23:00:40.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:40 smithi178 bash[22840]: cluster 2024-09-13T23:00:38.978481+0000 mgr.y (mgr.14613) 853 : cluster [DBG] pgmap v1025: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 736 op/s; 2523/120354 objects degraded (2.096%); 96/120354 objects misplaced (0.080%); 2.2 MiB/s, 69 objects/s recovering 2024-09-13T23:00:41.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:41.034 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:41.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:41.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:41.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:41.290 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:41.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:41.429 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:41.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.568 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:41.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:41.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:41.687 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:41.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:41.821 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:41.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:41.947 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:41.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:42.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:42.073 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:42.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:42.199 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:42.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:42.325 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:42.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:42.458 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:42.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:42.599 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:42.704 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:42.731 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:42.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:42.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:42.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:42.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:42 smithi192 bash[24608]: cluster 2024-09-13T23:00:40.980077+0000 mgr.y (mgr.14613) 854 : cluster [DBG] pgmap v1026: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 2401/108054 objects degraded (2.222%); 2.4 MiB/s, 81 objects/s recovering 2024-09-13T23:00:42.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:42 smithi178 bash[22840]: cluster 2024-09-13T23:00:40.980077+0000 mgr.y (mgr.14613) 854 : cluster [DBG] pgmap v1026: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 2401/108054 objects degraded (2.222%); 2.4 MiB/s, 81 objects/s recovering 2024-09-13T23:00:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:42 smithi178 bash[29889]: cluster 2024-09-13T23:00:40.980077+0000 mgr.y (mgr.14613) 854 : cluster [DBG] pgmap v1026: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 2401/108054 objects degraded (2.222%); 2.4 MiB/s, 81 objects/s recovering 2024-09-13T23:00:42.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T23:00:42.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:42.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T23:00:43.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:43.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:43.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:43.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:43.270 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:43.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:43.420 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:43.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:43.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:43.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:43 smithi192 bash[24608]: audit 2024-09-13T23:00:43.144940+0000 mon.b (mon.1) 287 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:00:43.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:43 smithi192 bash[24608]: cluster 2024-09-13T23:00:43.167889+0000 mon.a (mon.0) 1785 : cluster [WRN] Health check update: Degraded data redundancy: 2401/108054 objects degraded (2.222%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:43.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:43.706 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:43.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:43.846 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:43 smithi178 bash[29889]: audit 2024-09-13T23:00:43.144940+0000 mon.b (mon.1) 287 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:00:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:43 smithi178 bash[29889]: cluster 2024-09-13T23:00:43.167889+0000 mon.a (mon.0) 1785 : cluster [WRN] Health check update: Degraded data redundancy: 2401/108054 objects degraded (2.222%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:43.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:43 smithi178 bash[22840]: audit 2024-09-13T23:00:43.144940+0000 mon.b (mon.1) 287 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:00:43.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:43 smithi178 bash[22840]: cluster 2024-09-13T23:00:43.167889+0000 mon.a (mon.0) 1785 : cluster [WRN] Health check update: Degraded data redundancy: 2401/108054 objects degraded (2.222%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:43.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:43.985 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:43.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:44.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.115 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:44.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:44.250 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:44.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.380 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:44.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.511 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:44.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:44.638 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:44.746 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:44.756 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:44 smithi178 bash[29889]: cluster 2024-09-13T23:00:42.981051+0000 mgr.y (mgr.14613) 855 : cluster [DBG] pgmap v1027: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 618 op/s; 2401/107190 objects degraded (2.240%); 687 KiB/s, 27 objects/s recovering 2024-09-13T23:00:44.757 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:44 smithi178 bash[22840]: cluster 2024-09-13T23:00:42.981051+0000 mgr.y (mgr.14613) 855 : cluster [DBG] pgmap v1027: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 618 op/s; 2401/107190 objects degraded (2.240%); 687 KiB/s, 27 objects/s recovering 2024-09-13T23:00:44.776 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:44.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:44 smithi192 bash[24608]: cluster 2024-09-13T23:00:42.981051+0000 mgr.y (mgr.14613) 855 : cluster [DBG] pgmap v1027: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 618 op/s; 2401/107190 objects degraded (2.240%); 687 KiB/s, 27 objects/s recovering 2024-09-13T23:00:44.908 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:44.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:44.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T23:00:44.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T23:00:45.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:45.065 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:45.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:45.229 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:45.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:45.434 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:45.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:45.561 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:45.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:45.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:45 smithi192 bash[24608]: cluster 2024-09-13T23:00:45.005601+0000 osd.0 (osd.0) 409 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:45.671 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:45 smithi192 bash[24608]: cluster 2024-09-13T23:00:45.005668+0000 osd.0 (osd.0) 410 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:45.688 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:45.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:45.826 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:45.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:45.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:45 smithi178 bash[22840]: cluster 2024-09-13T23:00:45.005601+0000 osd.0 (osd.0) 409 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:45.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:45 smithi178 bash[22840]: cluster 2024-09-13T23:00:45.005668+0000 osd.0 (osd.0) 410 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:45 smithi178 bash[29889]: cluster 2024-09-13T23:00:45.005601+0000 osd.0 (osd.0) 409 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:45 smithi178 bash[29889]: cluster 2024-09-13T23:00:45.005668+0000 osd.0 (osd.0) 410 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:45.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:45.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:46.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:46.095 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:46.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:46.223 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:46.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:46.352 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:46.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:00:46 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:00:46.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:46.481 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:46.586 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:46.611 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:46.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:46.747 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:46.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:46.861 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:46 smithi178 bash[29889]: cluster 2024-09-13T23:00:44.983746+0000 mgr.y (mgr.14613) 856 : cluster [DBG] pgmap v1028: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 2127/93849 objects degraded (2.266%); 658 KiB/s, 58 objects/s recovering 2024-09-13T23:00:46.861 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:46 smithi178 bash[29889]: cluster 2024-09-13T23:00:45.674800+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-09-13T23:00:46.864 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:46 smithi178 bash[22840]: cluster 2024-09-13T23:00:44.983746+0000 mgr.y (mgr.14613) 856 : cluster [DBG] pgmap v1028: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 2127/93849 objects degraded (2.266%); 658 KiB/s, 58 objects/s recovering 2024-09-13T23:00:46.864 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:46 smithi178 bash[22840]: cluster 2024-09-13T23:00:45.674800+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-09-13T23:00:46.896 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:46.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:46 smithi192 bash[24608]: cluster 2024-09-13T23:00:44.983746+0000 mgr.y (mgr.14613) 856 : cluster [DBG] pgmap v1028: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 2127/93849 objects degraded (2.266%); 658 KiB/s, 58 objects/s recovering 2024-09-13T23:00:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:46 smithi192 bash[24608]: cluster 2024-09-13T23:00:45.674800+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-09-13T23:00:47.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:47.022 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:47.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:47.148 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:47.251 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:47.287 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:47.391 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:47.419 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:47.524 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:47.547 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:47.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:47.680 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:47.784 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:47.795 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:47 smithi178 bash[29889]: audit 2024-09-13T23:00:46.007018+0000 mgr.y (mgr.14613) 857 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:47.795 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:47 smithi178 bash[22840]: audit 2024-09-13T23:00:46.007018+0000 mgr.y (mgr.14613) 857 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:47.813 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:47.875 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:47.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:47.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:47 smithi192 bash[24608]: audit 2024-09-13T23:00:46.007018+0000 mgr.y (mgr.14613) 857 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:47.940 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:47.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:48.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:48.070 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:48.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:48.196 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:48.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:48.363 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:00:48.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:48.468 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:48.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:48.598 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:48.704 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:48.734 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:48.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:48.865 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:48.900 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:48.900 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:00:48.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:48 smithi192 bash[24608]: cluster 2024-09-13T23:00:46.984323+0000 mgr.y (mgr.14613) 858 : cluster [DBG] pgmap v1030: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 2127/92928 objects degraded (2.289%); 625 KiB/s, 55 objects/s recovering 2024-09-13T23:00:48.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:48 smithi192 bash[24608]: cluster 2024-09-13T23:00:48.168677+0000 mon.a (mon.0) 1787 : cluster [WRN] Health check update: Degraded data redundancy: 2127/92928 objects degraded (2.289%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:48 smithi192 bash[24608]: cluster 2024-09-13T23:00:48.174245+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-09-13T23:00:48.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:48.995 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:48.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:49.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:49.121 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:49.223 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:48 smithi178 bash[29889]: cluster 2024-09-13T23:00:46.984323+0000 mgr.y (mgr.14613) 858 : cluster [DBG] pgmap v1030: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 2127/92928 objects degraded (2.289%); 625 KiB/s, 55 objects/s recovering 2024-09-13T23:00:49.223 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:48 smithi178 bash[29889]: cluster 2024-09-13T23:00:48.168677+0000 mon.a (mon.0) 1787 : cluster [WRN] Health check update: Degraded data redundancy: 2127/92928 objects degraded (2.289%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:49.223 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:48 smithi178 bash[29889]: cluster 2024-09-13T23:00:48.174245+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-09-13T23:00:49.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:49.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:48 smithi178 bash[22840]: cluster 2024-09-13T23:00:46.984323+0000 mgr.y (mgr.14613) 858 : cluster [DBG] pgmap v1030: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 2127/92928 objects degraded (2.289%); 625 KiB/s, 55 objects/s recovering 2024-09-13T23:00:49.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:48 smithi178 bash[22840]: cluster 2024-09-13T23:00:48.168677+0000 mon.a (mon.0) 1787 : cluster [WRN] Health check update: Degraded data redundancy: 2127/92928 objects degraded (2.289%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:49.225 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:48 smithi178 bash[22840]: cluster 2024-09-13T23:00:48.174245+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-09-13T23:00:49.247 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:49.350 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:49.385 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:49.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:49.522 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:49.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:49.650 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:49.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:49.770 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:49.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:49.891 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:49.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:49.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:50.014 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:50.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:50.127 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:00:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:00:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:50.127 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:49 smithi178 bash[29889]: audit 2024-09-13T23:00:49.577384+0000 mon.b (mon.1) 288 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:50.127 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:49 smithi178 bash[22840]: audit 2024-09-13T23:00:49.577384+0000 mon.b (mon.1) 288 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:50.141 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:50.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:50.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:50.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:50.400 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:50.503 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:50.542 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:50.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:50.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:50 smithi192 bash[24608]: audit 2024-09-13T23:00:49.577384+0000 mon.b (mon.1) 288 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:00:50.665 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:50.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:50.788 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:50.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:50.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:50 smithi192 bash[24608]: cluster 2024-09-13T23:00:48.984963+0000 mgr.y (mgr.14613) 859 : cluster [DBG] pgmap v1032: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s; 2127/92046 objects degraded (2.311%); 0 B/s, 34 objects/s recovering 2024-09-13T23:00:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:50 smithi192 bash[24608]: audit 2024-09-13T23:00:49.834845+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:50 smithi192 bash[24608]: audit 2024-09-13T23:00:49.839766+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:50 smithi192 bash[24608]: audit 2024-09-13T23:00:50.125822+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:50 smithi192 bash[24608]: audit 2024-09-13T23:00:50.135647+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:50.915 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:50.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:51.018 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[29889]: cluster 2024-09-13T23:00:48.984963+0000 mgr.y (mgr.14613) 859 : cluster [DBG] pgmap v1032: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s; 2127/92046 objects degraded (2.311%); 0 B/s, 34 objects/s recovering 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[29889]: audit 2024-09-13T23:00:49.834845+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[29889]: audit 2024-09-13T23:00:49.839766+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[29889]: audit 2024-09-13T23:00:50.125822+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[29889]: audit 2024-09-13T23:00:50.135647+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[22840]: cluster 2024-09-13T23:00:48.984963+0000 mgr.y (mgr.14613) 859 : cluster [DBG] pgmap v1032: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s; 2127/92046 objects degraded (2.311%); 0 B/s, 34 objects/s recovering 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[22840]: audit 2024-09-13T23:00:49.834845+0000 mon.a (mon.0) 1789 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.029 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[22840]: audit 2024-09-13T23:00:49.839766+0000 mon.a (mon.0) 1790 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.030 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[22840]: audit 2024-09-13T23:00:50.125822+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.030 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:50 smithi178 bash[22840]: audit 2024-09-13T23:00:50.135647+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:51.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:51.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:51.181 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:51.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:51.308 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:51.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:51.436 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:51.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:51.580 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:51.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:51.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:51.843 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:51.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:51.963 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:51.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:51.983 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T23:00:51.983 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T23:00:51.983 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-09-13T23:00:51.983 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T23:00:52.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:51 smithi192 bash[24608]: audit 2024-09-13T23:00:50.925782+0000 mon.b (mon.1) 289 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:00:52.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:51 smithi192 bash[24608]: audit 2024-09-13T23:00:50.928596+0000 mon.b (mon.1) 290 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:00:52.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:51 smithi192 bash[24608]: audit 2024-09-13T23:00:50.938293+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:52.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:52.085 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:52.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:52.216 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:52.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:52.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:51 smithi178 bash[29889]: audit 2024-09-13T23:00:50.925782+0000 mon.b (mon.1) 289 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:00:52.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:51 smithi178 bash[29889]: audit 2024-09-13T23:00:50.928596+0000 mon.b (mon.1) 290 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:00:52.246 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:51 smithi178 bash[29889]: audit 2024-09-13T23:00:50.938293+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:52.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:51 smithi178 bash[22840]: audit 2024-09-13T23:00:50.925782+0000 mon.b (mon.1) 289 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:00:52.246 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:51 smithi178 bash[22840]: audit 2024-09-13T23:00:50.928596+0000 mon.b (mon.1) 290 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:00:52.247 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:51 smithi178 bash[22840]: audit 2024-09-13T23:00:50.938293+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:00:52.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:52.346 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:52.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:52.484 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:52.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:52.642 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:52.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:52.769 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:52.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:52.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:52.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:52.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T23:00:52.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:52 smithi178 bash[29889]: cluster 2024-09-13T23:00:50.985907+0000 mgr.y (mgr.14613) 860 : cluster [DBG] pgmap v1033: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 1874/79371 objects degraded (2.361%); 0 B/s, 65 objects/s recovering 2024-09-13T23:00:52.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:52 smithi178 bash[22840]: cluster 2024-09-13T23:00:50.985907+0000 mgr.y (mgr.14613) 860 : cluster [DBG] pgmap v1033: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 1874/79371 objects degraded (2.361%); 0 B/s, 65 objects/s recovering 2024-09-13T23:00:53.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:53.028 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:52 smithi192 bash[24608]: cluster 2024-09-13T23:00:50.985907+0000 mgr.y (mgr.14613) 860 : cluster [DBG] pgmap v1033: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 1874/79371 objects degraded (2.361%); 0 B/s, 65 objects/s recovering 2024-09-13T23:00:53.041 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:53.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:53.162 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:53.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:53.312 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:53.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:53.440 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:53.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:53.568 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:53.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:53.694 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:53.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:53.832 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:53.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:53.964 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:53.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:54.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:54.079 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:53 smithi178 bash[29889]: cluster 2024-09-13T23:00:53.170507+0000 mon.a (mon.0) 1794 : cluster [WRN] Health check update: Degraded data redundancy: 1874/79371 objects degraded (2.361%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:54.079 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:53 smithi178 bash[22840]: cluster 2024-09-13T23:00:53.170507+0000 mon.a (mon.0) 1794 : cluster [WRN] Health check update: Degraded data redundancy: 1874/79371 objects degraded (2.361%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:54.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:54.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:54.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:53 smithi192 bash[24608]: cluster 2024-09-13T23:00:53.170507+0000 mon.a (mon.0) 1794 : cluster [WRN] Health check update: Degraded data redundancy: 1874/79371 objects degraded (2.361%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:54.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:54.230 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:54.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:54.364 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:54.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:54.490 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:54.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:54.611 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:54.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:54.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:54.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:54.871 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:54.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:54 smithi192 bash[24608]: cluster 2024-09-13T23:00:52.986608+0000 mgr.y (mgr.14613) 861 : cluster [DBG] pgmap v1034: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s; 1874/78651 objects degraded (2.383%); 0 B/s, 31 objects/s recovering 2024-09-13T23:00:54.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:54.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:54.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:55.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:55.105 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:54 smithi178 bash[29889]: cluster 2024-09-13T23:00:52.986608+0000 mgr.y (mgr.14613) 861 : cluster [DBG] pgmap v1034: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s; 1874/78651 objects degraded (2.383%); 0 B/s, 31 objects/s recovering 2024-09-13T23:00:55.106 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:54 smithi178 bash[22840]: cluster 2024-09-13T23:00:52.986608+0000 mgr.y (mgr.14613) 861 : cluster [DBG] pgmap v1034: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 633 op/s; 1874/78651 objects degraded (2.383%); 0 B/s, 31 objects/s recovering 2024-09-13T23:00:55.146 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:55.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:55.280 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:55.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:55.406 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:55.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:55.534 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:55.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:55.661 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:55.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:55.790 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:55.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:55.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:55.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:56.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:56.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:00:56 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:00:56.039 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:55 smithi192 bash[24608]: cluster 2024-09-13T23:00:55.511125+0000 osd.3 (osd.3) 389 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:56.039 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:55 smithi192 bash[24608]: cluster 2024-09-13T23:00:55.511191+0000 osd.3 (osd.3) 390 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:56.054 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:56.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:56.175 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:56.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:55 smithi178 bash[29889]: cluster 2024-09-13T23:00:55.511125+0000 osd.3 (osd.3) 389 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:56.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:55 smithi178 bash[29889]: cluster 2024-09-13T23:00:55.511191+0000 osd.3 (osd.3) 390 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:56.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:55 smithi178 bash[22840]: cluster 2024-09-13T23:00:55.511125+0000 osd.3 (osd.3) 389 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:56.208 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:55 smithi178 bash[22840]: cluster 2024-09-13T23:00:55.511191+0000 osd.3 (osd.3) 390 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:56.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:56.301 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:56.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:56.430 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:56.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:56.558 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:00:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:00:56.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:56.686 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:56.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:56.827 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:56.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:56.956 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:56.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:57.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:57.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:56 smithi192 bash[24608]: cluster 2024-09-13T23:00:54.987617+0000 mgr.y (mgr.14613) 862 : cluster [DBG] pgmap v1035: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 893 op/s; 1580/68895 objects degraded (2.293%); 0 B/s, 58 objects/s recovering 2024-09-13T23:00:57.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:56 smithi192 bash[24608]: cluster 2024-09-13T23:00:55.859855+0000 mon.a (mon.0) 1795 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-09-13T23:00:57.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:56 smithi192 bash[24608]: audit 2024-09-13T23:00:56.014117+0000 mgr.y (mgr.14613) 863 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:57.081 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:57.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:56 smithi178 bash[29889]: cluster 2024-09-13T23:00:54.987617+0000 mgr.y (mgr.14613) 862 : cluster [DBG] pgmap v1035: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 893 op/s; 1580/68895 objects degraded (2.293%); 0 B/s, 58 objects/s recovering 2024-09-13T23:00:57.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:56 smithi178 bash[29889]: cluster 2024-09-13T23:00:55.859855+0000 mon.a (mon.0) 1795 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-09-13T23:00:57.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:56 smithi178 bash[29889]: audit 2024-09-13T23:00:56.014117+0000 mgr.y (mgr.14613) 863 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:56 smithi178 bash[22840]: cluster 2024-09-13T23:00:54.987617+0000 mgr.y (mgr.14613) 862 : cluster [DBG] pgmap v1035: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 988 B/s rd, 0 B/s wr, 893 op/s; 1580/68895 objects degraded (2.293%); 0 B/s, 58 objects/s recovering 2024-09-13T23:00:57.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:56 smithi178 bash[22840]: cluster 2024-09-13T23:00:55.859855+0000 mon.a (mon.0) 1795 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-09-13T23:00:57.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:56 smithi178 bash[22840]: audit 2024-09-13T23:00:56.014117+0000 mgr.y (mgr.14613) 863 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:00:57.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:57.208 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:57.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.342 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:57.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:57.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:57.464 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:57.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:57.589 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:57.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:57.711 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:57.790 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:00:57.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:57.841 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:57.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:57.965 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:57.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:58.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:58.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:57 smithi178 bash[29889]: cluster 2024-09-13T23:00:56.988233+0000 mgr.y (mgr.14613) 864 : cluster [DBG] pgmap v1037: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 942 op/s; 1580/68040 objects degraded (2.322%); 0 B/s, 62 objects/s recovering 2024-09-13T23:00:58.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:57 smithi178 bash[29889]: cluster 2024-09-13T23:00:57.383690+0000 osd.6 (osd.6) 312 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:58.080 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:57 smithi178 bash[29889]: cluster 2024-09-13T23:00:57.383748+0000 osd.6 (osd.6) 313 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:58.080 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:57 smithi178 bash[22840]: cluster 2024-09-13T23:00:56.988233+0000 mgr.y (mgr.14613) 864 : cluster [DBG] pgmap v1037: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 942 op/s; 1580/68040 objects degraded (2.322%); 0 B/s, 62 objects/s recovering 2024-09-13T23:00:58.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:57 smithi178 bash[22840]: cluster 2024-09-13T23:00:57.383690+0000 osd.6 (osd.6) 312 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:58.081 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:57 smithi178 bash[22840]: cluster 2024-09-13T23:00:57.383748+0000 osd.6 (osd.6) 313 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:58.097 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:00:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:00:58.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:57 smithi192 bash[24608]: cluster 2024-09-13T23:00:56.988233+0000 mgr.y (mgr.14613) 864 : cluster [DBG] pgmap v1037: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 942 op/s; 1580/68040 objects degraded (2.322%); 0 B/s, 62 objects/s recovering 2024-09-13T23:00:58.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:57 smithi192 bash[24608]: cluster 2024-09-13T23:00:57.383690+0000 osd.6 (osd.6) 312 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:00:58.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:57 smithi192 bash[24608]: cluster 2024-09-13T23:00:57.383748+0000 osd.6 (osd.6) 313 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:00:58.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:58.221 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:58.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:58.345 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:58.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:58.470 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:58.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:58.592 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:00:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:00:58.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:58.723 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:58.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:58.875 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:58.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:58.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:58.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:58 smithi192 bash[24608]: cluster 2024-09-13T23:00:57.866415+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-09-13T23:00:58.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:58 smithi192 bash[24608]: cluster 2024-09-13T23:00:58.171401+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check update: Degraded data redundancy: 1580/68040 objects degraded (2.322%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:59.005 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:59.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:59.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:58 smithi178 bash[29889]: cluster 2024-09-13T23:00:57.866415+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-09-13T23:00:59.120 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:58 smithi178 bash[29889]: cluster 2024-09-13T23:00:58.171401+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check update: Degraded data redundancy: 1580/68040 objects degraded (2.322%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:59.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:58 smithi178 bash[22840]: cluster 2024-09-13T23:00:57.866415+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-09-13T23:00:59.120 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:58 smithi178 bash[22840]: cluster 2024-09-13T23:00:58.171401+0000 mon.a (mon.0) 1797 : cluster [WRN] Health check update: Degraded data redundancy: 1580/68040 objects degraded (2.322%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:00:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:00:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:00:59.252 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T23:00:59.252 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T23:00:59.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:59.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:59.289 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T23:00:59.290 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T23:00:59.340 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T23:00:59.340 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T23:00:59.378 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T23:00:59.379 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T23:00:59.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:59.408 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:00:59.440 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T23:00:59.440 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T23:00:59.475 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T23:00:59.476 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T23:00:59.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:59.511 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T23:00:59.511 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T23:00:59.533 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:00:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:00:59.553 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T23:00:59.553 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T23:00:59.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:59.662 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:00:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:00:59.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:59.791 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:00:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:00:59.869 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T23:00:59.875 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T23:00:59.878 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:00:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:00:59.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:59.916 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:00:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:00:59.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:00.019 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:00.067 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:59 smithi192 bash[24608]: cluster 2024-09-13T23:00:58.989176+0000 mgr.y (mgr.14613) 865 : cluster [DBG] pgmap v1039: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 510 op/s; 1580/67128 objects degraded (2.354%); 0 B/s, 36 objects/s recovering 2024-09-13T23:01:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:59 smithi192 bash[24608]: audit 2024-09-13T23:00:59.195819+0000 mon.a (mon.0) 1798 : audit [DBG] from='client.? v1:172.21.15.178:0/2055148084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:59 smithi192 bash[24608]: audit 2024-09-13T23:00:59.536671+0000 mon.c (mon.2) 86 : audit [INF] from='client.? v1:172.21.15.178:0/2459806740' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:01:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:59 smithi192 bash[24608]: audit 2024-09-13T23:00:59.537140+0000 mon.a (mon.0) 1799 : audit [INF] from='client.25907 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:01:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:59 smithi192 bash[24608]: audit 2024-09-13T23:00:59.590983+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.178:0/2630384604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:00:59 smithi192 bash[24608]: audit 2024-09-13T23:00:59.591432+0000 mon.a (mon.0) 1800 : audit [INF] from='client.25789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:00.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[29889]: cluster 2024-09-13T23:00:58.989176+0000 mgr.y (mgr.14613) 865 : cluster [DBG] pgmap v1039: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 510 op/s; 1580/67128 objects degraded (2.354%); 0 B/s, 36 objects/s recovering 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[29889]: audit 2024-09-13T23:00:59.195819+0000 mon.a (mon.0) 1798 : audit [DBG] from='client.? v1:172.21.15.178:0/2055148084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[29889]: audit 2024-09-13T23:00:59.536671+0000 mon.c (mon.2) 86 : audit [INF] from='client.? v1:172.21.15.178:0/2459806740' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[29889]: audit 2024-09-13T23:00:59.537140+0000 mon.a (mon.0) 1799 : audit [INF] from='client.25907 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[29889]: audit 2024-09-13T23:00:59.590983+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.178:0/2630384604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[29889]: audit 2024-09-13T23:00:59.591432+0000 mon.a (mon.0) 1800 : audit [INF] from='client.25789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[22840]: cluster 2024-09-13T23:00:58.989176+0000 mgr.y (mgr.14613) 865 : cluster [DBG] pgmap v1039: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 510 op/s; 1580/67128 objects degraded (2.354%); 0 B/s, 36 objects/s recovering 2024-09-13T23:01:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[22840]: audit 2024-09-13T23:00:59.195819+0000 mon.a (mon.0) 1798 : audit [DBG] from='client.? v1:172.21.15.178:0/2055148084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[22840]: audit 2024-09-13T23:00:59.536671+0000 mon.c (mon.2) 86 : audit [INF] from='client.? v1:172.21.15.178:0/2459806740' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:01:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[22840]: audit 2024-09-13T23:00:59.537140+0000 mon.a (mon.0) 1799 : audit [INF] from='client.25907 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:01:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[22840]: audit 2024-09-13T23:00:59.590983+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.178:0/2630384604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:00:59 smithi178 bash[22840]: audit 2024-09-13T23:00:59.591432+0000 mon.a (mon.0) 1800 : audit [INF] from='client.25789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:00.198 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:00.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:00.332 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:00.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:00.460 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:00.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:00.591 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:00.649 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:01:00.649 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:01:00.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:00.711 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:00.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:00.835 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:00.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:00.970 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:00.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:01.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:01.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:59.865337+0000 mon.a (mon.0) 1801 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:01:01.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:00 smithi192 bash[24608]: audit 2024-09-13T23:00:59.866554+0000 mon.a (mon.0) 1802 : audit [INF] from='client.25907 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:01:01.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:00 smithi192 bash[24608]: audit 2024-09-13T23:00:59.866707+0000 mon.a (mon.0) 1803 : audit [INF] from='client.25789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T23:01:01.083 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:00 smithi192 bash[24608]: cluster 2024-09-13T23:00:59.873673+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-09-13T23:01:01.093 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:01.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:01.216 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:59.865337+0000 mon.a (mon.0) 1801 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:01:01.216 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[29889]: audit 2024-09-13T23:00:59.866554+0000 mon.a (mon.0) 1802 : audit [INF] from='client.25907 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:01:01.216 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[29889]: audit 2024-09-13T23:00:59.866707+0000 mon.a (mon.0) 1803 : audit [INF] from='client.25789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T23:01:01.216 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[29889]: cluster 2024-09-13T23:00:59.873673+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-09-13T23:01:01.216 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:59.865337+0000 mon.a (mon.0) 1801 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:01:01.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[22840]: audit 2024-09-13T23:00:59.866554+0000 mon.a (mon.0) 1802 : audit [INF] from='client.25907 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:01:01.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[22840]: audit 2024-09-13T23:00:59.866707+0000 mon.a (mon.0) 1803 : audit [INF] from='client.25789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T23:01:01.217 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:00 smithi178 bash[22840]: cluster 2024-09-13T23:00:59.873673+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-09-13T23:01:01.219 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:01.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:01.344 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:01.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:01.480 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:01.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:01.613 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:01.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:01.738 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:01.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:01.867 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:01.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T23:01:01.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:01.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:01.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:02.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.131 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:02.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:02.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:00.693301+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.a scrub starts 2024-09-13T23:01:02.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:00.694233+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.a scrub ok 2024-09-13T23:01:02.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:00.989894+0000 mgr.y (mgr.14613) 866 : cluster [DBG] pgmap v1041: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 712 op/s; 1234/56073 objects degraded (2.201%); 0 B/s, 57 objects/s recovering 2024-09-13T23:01:02.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:00.693301+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.a scrub starts 2024-09-13T23:01:02.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:00.694233+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.a scrub ok 2024-09-13T23:01:02.207 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:00.989894+0000 mgr.y (mgr.14613) 866 : cluster [DBG] pgmap v1041: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 712 op/s; 1234/56073 objects degraded (2.201%); 0 B/s, 57 objects/s recovering 2024-09-13T23:01:02.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:00.693301+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.a scrub starts 2024-09-13T23:01:02.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:00.694233+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.a scrub ok 2024-09-13T23:01:02.209 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:00.989894+0000 mgr.y (mgr.14613) 866 : cluster [DBG] pgmap v1041: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 712 op/s; 1234/56073 objects degraded (2.201%); 0 B/s, 57 objects/s recovering 2024-09-13T23:01:02.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:02.256 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:02.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:02.387 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:02.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:02.525 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:02.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:02.655 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:02.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:02.778 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:02.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:02.902 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:02.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:03.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:03.024 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:03.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:03.151 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:03.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:02 smithi192 bash[24608]: cluster 2024-09-13T23:01:01.481940+0000 osd.5 (osd.5) 451 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:01:03.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:02 smithi192 bash[24608]: cluster 2024-09-13T23:01:01.487797+0000 osd.5 (osd.5) 452 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:01:03.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:02 smithi192 bash[24608]: cluster 2024-09-13T23:01:01.745708+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.4 scrub starts 2024-09-13T23:01:03.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:02 smithi192 bash[24608]: cluster 2024-09-13T23:01:01.749335+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.4 scrub ok 2024-09-13T23:01:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[22840]: cluster 2024-09-13T23:01:01.481940+0000 osd.5 (osd.5) 451 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:01:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[22840]: cluster 2024-09-13T23:01:01.487797+0000 osd.5 (osd.5) 452 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:01:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[22840]: cluster 2024-09-13T23:01:01.745708+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.4 scrub starts 2024-09-13T23:01:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[22840]: cluster 2024-09-13T23:01:01.749335+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.4 scrub ok 2024-09-13T23:01:03.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[29889]: cluster 2024-09-13T23:01:01.481940+0000 osd.5 (osd.5) 451 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:01:03.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[29889]: cluster 2024-09-13T23:01:01.487797+0000 osd.5 (osd.5) 452 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:01:03.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[29889]: cluster 2024-09-13T23:01:01.745708+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.4 scrub starts 2024-09-13T23:01:03.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:02 smithi178 bash[29889]: cluster 2024-09-13T23:01:01.749335+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.4 scrub ok 2024-09-13T23:01:03.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:03.286 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:03.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:03.414 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:03.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:03.548 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:03.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:03.678 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:03.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:03.809 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:03.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T23:01:03.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:03.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T23:01:03.997 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:03.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:04.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:04.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:02.445122+0000 osd.5 (osd.5) 453 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:04.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:02.445186+0000 osd.5 (osd.5) 454 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:04.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:02.623144+0000 osd.2 (osd.2) 299 : cluster [DBG] 3.19 scrub starts 2024-09-13T23:01:04.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:02.630608+0000 osd.2 (osd.2) 300 : cluster [DBG] 3.19 scrub ok 2024-09-13T23:01:04.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:02.893990+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-09-13T23:01:04.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:02.990291+0000 mgr.y (mgr.14613) 867 : cluster [DBG] pgmap v1043: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.1 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 705 op/s; 1234/55353 objects degraded (2.229%); 0 B/s, 57 objects/s recovering 2024-09-13T23:01:04.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:03 smithi192 bash[24608]: cluster 2024-09-13T23:01:03.173877+0000 mon.a (mon.0) 1806 : cluster [WRN] Health check update: Degraded data redundancy: 1234/56073 objects degraded (2.201%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:04.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:02.445122+0000 osd.5 (osd.5) 453 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:04.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:02.445186+0000 osd.5 (osd.5) 454 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:04.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:02.623144+0000 osd.2 (osd.2) 299 : cluster [DBG] 3.19 scrub starts 2024-09-13T23:01:04.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:02.630608+0000 osd.2 (osd.2) 300 : cluster [DBG] 3.19 scrub ok 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:02.893990+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:02.990291+0000 mgr.y (mgr.14613) 867 : cluster [DBG] pgmap v1043: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.1 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 705 op/s; 1234/55353 objects degraded (2.229%); 0 B/s, 57 objects/s recovering 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[29889]: cluster 2024-09-13T23:01:03.173877+0000 mon.a (mon.0) 1806 : cluster [WRN] Health check update: Degraded data redundancy: 1234/56073 objects degraded (2.201%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:02.445122+0000 osd.5 (osd.5) 453 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:02.445186+0000 osd.5 (osd.5) 454 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:02.623144+0000 osd.2 (osd.2) 299 : cluster [DBG] 3.19 scrub starts 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:02.630608+0000 osd.2 (osd.2) 300 : cluster [DBG] 3.19 scrub ok 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:02.893990+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-09-13T23:01:04.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:02.990291+0000 mgr.y (mgr.14613) 867 : cluster [DBG] pgmap v1043: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1.1 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 705 op/s; 1234/55353 objects degraded (2.229%); 0 B/s, 57 objects/s recovering 2024-09-13T23:01:04.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:03 smithi178 bash[22840]: cluster 2024-09-13T23:01:03.173877+0000 mon.a (mon.0) 1806 : cluster [WRN] Health check update: Degraded data redundancy: 1234/56073 objects degraded (2.201%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:04.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:04.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:04.393 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:04.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:04.528 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:04.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:04.657 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:04.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:04.791 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:04.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:04.925 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:04.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:05.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:05.037 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[29889]: cluster 2024-09-13T23:01:03.480365+0000 osd.5 (osd.5) 455 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[29889]: cluster 2024-09-13T23:01:03.483538+0000 osd.5 (osd.5) 456 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[29889]: cluster 2024-09-13T23:01:03.776154+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[29889]: cluster 2024-09-13T23:01:03.778496+0000 osd.1 (osd.1) 407 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[29889]: audit 2024-09-13T23:01:04.595278+0000 mon.b (mon.1) 292 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[22840]: cluster 2024-09-13T23:01:03.480365+0000 osd.5 (osd.5) 455 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[22840]: cluster 2024-09-13T23:01:03.483538+0000 osd.5 (osd.5) 456 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[22840]: cluster 2024-09-13T23:01:03.776154+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[22840]: cluster 2024-09-13T23:01:03.778496+0000 osd.1 (osd.1) 407 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:01:05.038 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:04 smithi178 bash[22840]: audit 2024-09-13T23:01:04.595278+0000 mon.b (mon.1) 292 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.068 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:05.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:05.169 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:05.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:05.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:05.333 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:05.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:04 smithi192 bash[24608]: cluster 2024-09-13T23:01:03.480365+0000 osd.5 (osd.5) 455 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:01:05.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:04 smithi192 bash[24608]: cluster 2024-09-13T23:01:03.483538+0000 osd.5 (osd.5) 456 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:01:05.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:04 smithi192 bash[24608]: cluster 2024-09-13T23:01:03.776154+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:01:05.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:04 smithi192 bash[24608]: cluster 2024-09-13T23:01:03.778496+0000 osd.1 (osd.1) 407 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:01:05.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:04 smithi192 bash[24608]: audit 2024-09-13T23:01:04.595278+0000 mon.b (mon.1) 292 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:05.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:05.472 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:05.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:05.604 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:05.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:05.737 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:05.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:05.866 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:05.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:05.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:06.003 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:06.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T23:01:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:04.467653+0000 osd.5 (osd.5) 457 : cluster [DBG] 2.12 scrub starts 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:04.468633+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub ok 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:04.813940+0000 osd.1 (osd.1) 408 : cluster [DBG] 6.1d scrub starts 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:04.818925+0000 osd.1 (osd.1) 409 : cluster [DBG] 6.1d scrub ok 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:04.991600+0000 mgr.y (mgr.14613) 868 : cluster [DBG] pgmap v1044: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 909 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.12k op/s; 727/44199 objects degraded (1.645%); 0 B/s, 119 objects/s recovering 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:05.191967+0000 osd.7 (osd.7) 420 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:01:06.021 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:05 smithi192 bash[24608]: cluster 2024-09-13T23:01:05.196322+0000 osd.7 (osd.7) 421 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:01:06.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:06.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:04.467653+0000 osd.5 (osd.5) 457 : cluster [DBG] 2.12 scrub starts 2024-09-13T23:01:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:04.468633+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub ok 2024-09-13T23:01:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:04.813940+0000 osd.1 (osd.1) 408 : cluster [DBG] 6.1d scrub starts 2024-09-13T23:01:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:04.818925+0000 osd.1 (osd.1) 409 : cluster [DBG] 6.1d scrub ok 2024-09-13T23:01:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:04.991600+0000 mgr.y (mgr.14613) 868 : cluster [DBG] pgmap v1044: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 909 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.12k op/s; 727/44199 objects degraded (1.645%); 0 B/s, 119 objects/s recovering 2024-09-13T23:01:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:05.191967+0000 osd.7 (osd.7) 420 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:01:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[29889]: cluster 2024-09-13T23:01:05.196322+0000 osd.7 (osd.7) 421 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:01:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:04.467653+0000 osd.5 (osd.5) 457 : cluster [DBG] 2.12 scrub starts 2024-09-13T23:01:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:04.468633+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub ok 2024-09-13T23:01:06.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:04.813940+0000 osd.1 (osd.1) 408 : cluster [DBG] 6.1d scrub starts 2024-09-13T23:01:06.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:04.818925+0000 osd.1 (osd.1) 409 : cluster [DBG] 6.1d scrub ok 2024-09-13T23:01:06.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:04.991600+0000 mgr.y (mgr.14613) 868 : cluster [DBG] pgmap v1044: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 909 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.12k op/s; 727/44199 objects degraded (1.645%); 0 B/s, 119 objects/s recovering 2024-09-13T23:01:06.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:05.191967+0000 osd.7 (osd.7) 420 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:01:06.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:05 smithi178 bash[22840]: cluster 2024-09-13T23:01:05.196322+0000 osd.7 (osd.7) 421 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:01:06.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:06.273 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:06.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:06.403 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:06.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:01:06.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:06.530 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:06.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:06.656 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:06.749 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:06.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:06.795 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:06.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:06.935 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:06.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:07.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[29889]: cluster 2024-09-13T23:01:05.439531+0000 osd.2 (osd.2) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[29889]: cluster 2024-09-13T23:01:05.439591+0000 osd.2 (osd.2) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[29889]: cluster 2024-09-13T23:01:05.931400+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[29889]: audit 2024-09-13T23:01:06.022136+0000 mgr.y (mgr.14613) 869 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[29889]: cluster 2024-09-13T23:01:06.195040+0000 osd.7 (osd.7) 422 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[29889]: cluster 2024-09-13T23:01:06.196059+0000 osd.7 (osd.7) 423 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[22840]: cluster 2024-09-13T23:01:05.439531+0000 osd.2 (osd.2) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[22840]: cluster 2024-09-13T23:01:05.439591+0000 osd.2 (osd.2) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[22840]: cluster 2024-09-13T23:01:05.931400+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-09-13T23:01:07.049 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[22840]: audit 2024-09-13T23:01:06.022136+0000 mgr.y (mgr.14613) 869 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:07.050 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[22840]: cluster 2024-09-13T23:01:06.195040+0000 osd.7 (osd.7) 422 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:01:07.050 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:06 smithi178 bash[22840]: cluster 2024-09-13T23:01:06.196059+0000 osd.7 (osd.7) 423 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:01:07.066 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:07.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[24608]: cluster 2024-09-13T23:01:05.439531+0000 osd.2 (osd.2) 301 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[24608]: cluster 2024-09-13T23:01:05.439591+0000 osd.2 (osd.2) 302 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[24608]: cluster 2024-09-13T23:01:05.931400+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-09-13T23:01:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[24608]: audit 2024-09-13T23:01:06.022136+0000 mgr.y (mgr.14613) 869 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[24608]: cluster 2024-09-13T23:01:06.195040+0000 osd.7 (osd.7) 422 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:01:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:06 smithi192 bash[24608]: cluster 2024-09-13T23:01:06.196059+0000 osd.7 (osd.7) 423 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:01:07.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:07.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.214 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:07.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:07.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:07.339 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:07.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:07.466 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:07.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:07.591 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:07.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:07.713 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:07.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:07.838 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:07.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:07.962 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:07.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:08.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:08.085 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:08.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:08.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:07 smithi178 bash[29889]: cluster 2024-09-13T23:01:06.493167+0000 osd.5 (osd.5) 459 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:01:08.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:07 smithi178 bash[29889]: cluster 2024-09-13T23:01:06.499301+0000 osd.5 (osd.5) 460 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:01:08.198 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:07 smithi178 bash[29889]: cluster 2024-09-13T23:01:06.992252+0000 mgr.y (mgr.14613) 870 : cluster [DBG] pgmap v1046: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 890 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 718 B/s rd, 0 B/s wr, 635 op/s; 727/43296 objects degraded (1.679%); 0 B/s, 71 objects/s recovering 2024-09-13T23:01:08.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:07 smithi178 bash[22840]: cluster 2024-09-13T23:01:06.493167+0000 osd.5 (osd.5) 459 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:01:08.198 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:07 smithi178 bash[22840]: cluster 2024-09-13T23:01:06.499301+0000 osd.5 (osd.5) 460 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:01:08.199 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:07 smithi178 bash[22840]: cluster 2024-09-13T23:01:06.992252+0000 mgr.y (mgr.14613) 870 : cluster [DBG] pgmap v1046: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 890 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 718 B/s rd, 0 B/s wr, 635 op/s; 727/43296 objects degraded (1.679%); 0 B/s, 71 objects/s recovering 2024-09-13T23:01:08.214 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:08.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:08.349 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:08.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:08 smithi192 bash[24608]: cluster 2024-09-13T23:01:06.493167+0000 osd.5 (osd.5) 459 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:01:08.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:08 smithi192 bash[24608]: cluster 2024-09-13T23:01:06.499301+0000 osd.5 (osd.5) 460 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:01:08.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:08 smithi192 bash[24608]: cluster 2024-09-13T23:01:06.992252+0000 mgr.y (mgr.14613) 870 : cluster [DBG] pgmap v1046: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 890 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 718 B/s rd, 0 B/s wr, 635 op/s; 727/43296 objects degraded (1.679%); 0 B/s, 71 objects/s recovering 2024-09-13T23:01:08.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:08.488 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:08.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:08.615 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:08.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:08.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:08.740 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:08.777 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:08.780 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:08.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:08.867 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:08.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:08.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:09.000 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:09.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:09.114 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:08 smithi192 bash[24608]: cluster 2024-09-13T23:01:08.175333+0000 mon.a (mon.0) 1808 : cluster [WRN] Health check update: Degraded data redundancy: 727/43296 objects degraded (1.679%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:09.114 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:08 smithi192 bash[24608]: cluster 2024-09-13T23:01:08.230971+0000 osd.7 (osd.7) 424 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:01:09.114 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:08 smithi192 bash[24608]: cluster 2024-09-13T23:01:08.231842+0000 osd.7 (osd.7) 425 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:01:09.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:08 smithi178 bash[29889]: cluster 2024-09-13T23:01:08.175333+0000 mon.a (mon.0) 1808 : cluster [WRN] Health check update: Degraded data redundancy: 727/43296 objects degraded (1.679%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:08 smithi178 bash[29889]: cluster 2024-09-13T23:01:08.230971+0000 osd.7 (osd.7) 424 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:01:09.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:08 smithi178 bash[29889]: cluster 2024-09-13T23:01:08.231842+0000 osd.7 (osd.7) 425 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:01:09.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:08 smithi178 bash[22840]: cluster 2024-09-13T23:01:08.175333+0000 mon.a (mon.0) 1808 : cluster [WRN] Health check update: Degraded data redundancy: 727/43296 objects degraded (1.679%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:09.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:08 smithi178 bash[22840]: cluster 2024-09-13T23:01:08.230971+0000 osd.7 (osd.7) 424 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:01:09.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:08 smithi178 bash[22840]: cluster 2024-09-13T23:01:08.231842+0000 osd.7 (osd.7) 425 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:01:09.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:09.256 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:09.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:09.380 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:09.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:09.510 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:09.614 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:01:09.615 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:01:09.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:09.638 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:09.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:09.766 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:09.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:09.893 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:09.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:09.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:10.007 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:01:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:01:10.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[29889]: cluster 2024-09-13T23:01:08.580375+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:01:10.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[29889]: cluster 2024-09-13T23:01:08.586779+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:01:10.008 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[29889]: cluster 2024-09-13T23:01:08.992837+0000 mgr.y (mgr.14613) 871 : cluster [DBG] pgmap v1047: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 870 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 727/42348 objects degraded (1.717%); 0 B/s, 63 objects/s recovering 2024-09-13T23:01:10.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[22840]: cluster 2024-09-13T23:01:08.580375+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:01:10.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[22840]: cluster 2024-09-13T23:01:08.586779+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:01:10.008 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:09 smithi178 bash[22840]: cluster 2024-09-13T23:01:08.992837+0000 mgr.y (mgr.14613) 871 : cluster [DBG] pgmap v1047: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 870 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 727/42348 objects degraded (1.717%); 0 B/s, 63 objects/s recovering 2024-09-13T23:01:10.036 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:10.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:10.147 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:09 smithi192 bash[24608]: cluster 2024-09-13T23:01:08.580375+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:01:10.147 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:09 smithi192 bash[24608]: cluster 2024-09-13T23:01:08.586779+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:01:10.148 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:09 smithi192 bash[24608]: cluster 2024-09-13T23:01:08.992837+0000 mgr.y (mgr.14613) 871 : cluster [DBG] pgmap v1047: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 870 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 727/42348 objects degraded (1.717%); 0 B/s, 63 objects/s recovering 2024-09-13T23:01:10.158 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:10.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:10.296 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:10.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:10.422 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:10.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:10.550 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:10.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:10.687 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:10.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:10.810 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:10.813 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:10.915 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:10.933 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:10.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:11.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:11.070 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:11.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:10 smithi192 bash[24608]: cluster 2024-09-13T23:01:09.273728+0000 osd.7 (osd.7) 426 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:01:11.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:10 smithi192 bash[24608]: cluster 2024-09-13T23:01:09.277583+0000 osd.7 (osd.7) 427 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:01:11.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:10 smithi192 bash[24608]: cluster 2024-09-13T23:01:09.457883+0000 osd.5 (osd.5) 461 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:01:11.118 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:10 smithi192 bash[24608]: cluster 2024-09-13T23:01:09.461406+0000 osd.5 (osd.5) 462 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:01:11.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:10 smithi192 bash[24608]: cluster 2024-09-13T23:01:09.809586+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.d scrub starts 2024-09-13T23:01:11.119 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:10 smithi192 bash[24608]: cluster 2024-09-13T23:01:09.811157+0000 osd.1 (osd.1) 411 : cluster [DBG] 2.d scrub ok 2024-09-13T23:01:11.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T23:01:11.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:11.194 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:11.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:11.306 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[29889]: cluster 2024-09-13T23:01:09.273728+0000 osd.7 (osd.7) 426 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:01:11.306 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[29889]: cluster 2024-09-13T23:01:09.277583+0000 osd.7 (osd.7) 427 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:01:11.306 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[29889]: cluster 2024-09-13T23:01:09.457883+0000 osd.5 (osd.5) 461 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:01:11.306 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[29889]: cluster 2024-09-13T23:01:09.461406+0000 osd.5 (osd.5) 462 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:01:11.306 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[29889]: cluster 2024-09-13T23:01:09.809586+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.d scrub starts 2024-09-13T23:01:11.306 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[29889]: cluster 2024-09-13T23:01:09.811157+0000 osd.1 (osd.1) 411 : cluster [DBG] 2.d scrub ok 2024-09-13T23:01:11.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[22840]: cluster 2024-09-13T23:01:09.273728+0000 osd.7 (osd.7) 426 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:01:11.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[22840]: cluster 2024-09-13T23:01:09.277583+0000 osd.7 (osd.7) 427 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:01:11.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[22840]: cluster 2024-09-13T23:01:09.457883+0000 osd.5 (osd.5) 461 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:01:11.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[22840]: cluster 2024-09-13T23:01:09.461406+0000 osd.5 (osd.5) 462 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:01:11.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[22840]: cluster 2024-09-13T23:01:09.809586+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.d scrub starts 2024-09-13T23:01:11.307 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:10 smithi178 bash[22840]: cluster 2024-09-13T23:01:09.811157+0000 osd.1 (osd.1) 411 : cluster [DBG] 2.d scrub ok 2024-09-13T23:01:11.318 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:11.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:11.446 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:11.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:11.574 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:11.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:11.705 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:11.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:11.836 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:11.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:11.967 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:11.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:12.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:12.092 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:11 smithi192 bash[24608]: cluster 2024-09-13T23:01:10.317708+0000 osd.7 (osd.7) 428 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:01:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:11 smithi192 bash[24608]: cluster 2024-09-13T23:01:10.320669+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:01:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:11 smithi192 bash[24608]: cluster 2024-09-13T23:01:10.859044+0000 osd.1 (osd.1) 412 : cluster [DBG] 11.e scrub starts 2024-09-13T23:01:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:11 smithi192 bash[24608]: cluster 2024-09-13T23:01:10.976903+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-09-13T23:01:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:11 smithi192 bash[24608]: cluster 2024-09-13T23:01:10.993339+0000 mgr.y (mgr.14613) 872 : cluster [DBG] pgmap v1049: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 612 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 158/29949 objects degraded (0.528%); 0 B/s, 134 objects/s recovering 2024-09-13T23:01:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:11 smithi192 bash[24608]: cluster 2024-09-13T23:01:11.114558+0000 osd.1 (osd.1) 413 : cluster [DBG] 11.e scrub ok 2024-09-13T23:01:12.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:12.221 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:12.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:12.350 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:12.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:12.478 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:12.529 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:11 smithi178 bash[29889]: cluster 2024-09-13T23:01:10.317708+0000 osd.7 (osd.7) 428 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:01:12.529 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:11 smithi178 bash[29889]: cluster 2024-09-13T23:01:10.320669+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:01:12.529 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:11 smithi178 bash[29889]: cluster 2024-09-13T23:01:10.859044+0000 osd.1 (osd.1) 412 : cluster [DBG] 11.e scrub starts 2024-09-13T23:01:12.529 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:11 smithi178 bash[29889]: cluster 2024-09-13T23:01:10.976903+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-09-13T23:01:12.529 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:11 smithi178 bash[29889]: cluster 2024-09-13T23:01:10.993339+0000 mgr.y (mgr.14613) 872 : cluster [DBG] pgmap v1049: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 612 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 158/29949 objects degraded (0.528%); 0 B/s, 134 objects/s recovering 2024-09-13T23:01:12.530 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:11 smithi178 bash[29889]: cluster 2024-09-13T23:01:11.114558+0000 osd.1 (osd.1) 413 : cluster [DBG] 11.e scrub ok 2024-09-13T23:01:12.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:12.607 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:12.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:12.737 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:12.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:12.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:10.317708+0000 osd.7 (osd.7) 428 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:01:12.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:10.320669+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:01:12.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:10.859044+0000 osd.1 (osd.1) 412 : cluster [DBG] 11.e scrub starts 2024-09-13T23:01:12.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:10.976903+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-09-13T23:01:12.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:10.993339+0000 mgr.y (mgr.14613) 872 : cluster [DBG] pgmap v1049: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 612 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 158/29949 objects degraded (0.528%); 0 B/s, 134 objects/s recovering 2024-09-13T23:01:12.852 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:11.114558+0000 osd.1 (osd.1) 413 : cluster [DBG] 11.e scrub ok 2024-09-13T23:01:12.875 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:12.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:12.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:13.007 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:13.110 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:13.140 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:13.175 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:11.266450+0000 osd.7 (osd.7) 430 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:01:13.176 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:11.267883+0000 osd.7 (osd.7) 431 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:01:13.176 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:11.833677+0000 osd.1 (osd.1) 414 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:01:13.176 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:11.835770+0000 osd.1 (osd.1) 415 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:01:13.176 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:12 smithi178 bash[22840]: cluster 2024-09-13T23:01:12.003470+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-09-13T23:01:13.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:13.263 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:13.268 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:11.266450+0000 osd.7 (osd.7) 430 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:01:13.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:11.267883+0000 osd.7 (osd.7) 431 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:01:13.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:11.833677+0000 osd.1 (osd.1) 414 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:01:13.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:11.835770+0000 osd.1 (osd.1) 415 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:01:13.269 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:12.003470+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-09-13T23:01:13.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:13.387 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:12 smithi192 bash[24608]: cluster 2024-09-13T23:01:11.266450+0000 osd.7 (osd.7) 430 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:01:13.407 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:12 smithi192 bash[24608]: cluster 2024-09-13T23:01:11.267883+0000 osd.7 (osd.7) 431 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:01:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:12 smithi192 bash[24608]: cluster 2024-09-13T23:01:11.833677+0000 osd.1 (osd.1) 414 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:01:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:12 smithi192 bash[24608]: cluster 2024-09-13T23:01:11.835770+0000 osd.1 (osd.1) 415 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:01:13.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:12 smithi192 bash[24608]: cluster 2024-09-13T23:01:12.003470+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-09-13T23:01:13.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:13.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:13.625 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:13.652 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:13.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:13.778 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:13.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:13.912 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:13.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:14.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.041 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:14.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:14.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:12.292224+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:01:14.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:12.294452+0000 osd.7 (osd.7) 433 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:12.889185+0000 osd.1 (osd.1) 416 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:12.898999+0000 osd.1 (osd.1) 417 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:12.993851+0000 mgr.y (mgr.14613) 873 : cluster [DBG] pgmap v1051: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 597 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 705 op/s; 158/29250 objects degraded (0.540%); 0 B/s, 80 objects/s recovering 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:13.177068+0000 mon.a (mon.0) 1811 : cluster [WRN] Health check update: Degraded data redundancy: 158/29949 objects degraded (0.528%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:13.181557+0000 osd.3 (osd.3) 391 : cluster [DBG] 2.19 scrub starts 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:13.185133+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:13 smithi192 bash[24608]: cluster 2024-09-13T23:01:13.190688+0000 osd.3 (osd.3) 392 : cluster [DBG] 2.19 scrub ok 2024-09-13T23:01:14.158 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:12.292224+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:12.294452+0000 osd.7 (osd.7) 433 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:12.889185+0000 osd.1 (osd.1) 416 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:12.898999+0000 osd.1 (osd.1) 417 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:12.993851+0000 mgr.y (mgr.14613) 873 : cluster [DBG] pgmap v1051: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 597 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 705 op/s; 158/29250 objects degraded (0.540%); 0 B/s, 80 objects/s recovering 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:13.177068+0000 mon.a (mon.0) 1811 : cluster [WRN] Health check update: Degraded data redundancy: 158/29949 objects degraded (0.528%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:13.181557+0000 osd.3 (osd.3) 391 : cluster [DBG] 2.19 scrub starts 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:13.185133+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[29889]: cluster 2024-09-13T23:01:13.190688+0000 osd.3 (osd.3) 392 : cluster [DBG] 2.19 scrub ok 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:12.292224+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:01:14.159 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:12.294452+0000 osd.7 (osd.7) 433 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:12.889185+0000 osd.1 (osd.1) 416 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:12.898999+0000 osd.1 (osd.1) 417 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:12.993851+0000 mgr.y (mgr.14613) 873 : cluster [DBG] pgmap v1051: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 597 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 705 op/s; 158/29250 objects degraded (0.540%); 0 B/s, 80 objects/s recovering 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:13.177068+0000 mon.a (mon.0) 1811 : cluster [WRN] Health check update: Degraded data redundancy: 158/29949 objects degraded (0.528%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:13.181557+0000 osd.3 (osd.3) 391 : cluster [DBG] 2.19 scrub starts 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:13.185133+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-09-13T23:01:14.160 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:13 smithi178 bash[22840]: cluster 2024-09-13T23:01:13.190688+0000 osd.3 (osd.3) 392 : cluster [DBG] 2.19 scrub ok 2024-09-13T23:01:14.176 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:14.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:14.336 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:14.439 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.475 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:14.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:14.609 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:14.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.744 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:14.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:14.865 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:14.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.996 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:14.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:15.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:15.128 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:15.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:14 smithi192 bash[24608]: cluster 2024-09-13T23:01:13.453705+0000 osd.5 (osd.5) 463 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:01:15.157 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:14 smithi192 bash[24608]: cluster 2024-09-13T23:01:13.457321+0000 osd.5 (osd.5) 464 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:01:15.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:15.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:14 smithi178 bash[29889]: cluster 2024-09-13T23:01:13.453705+0000 osd.5 (osd.5) 463 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:01:15.241 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:14 smithi178 bash[29889]: cluster 2024-09-13T23:01:13.457321+0000 osd.5 (osd.5) 464 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:01:15.241 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:14 smithi178 bash[22840]: cluster 2024-09-13T23:01:13.453705+0000 osd.5 (osd.5) 463 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:01:15.241 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:14 smithi178 bash[22840]: cluster 2024-09-13T23:01:13.457321+0000 osd.5 (osd.5) 464 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:01:15.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:15.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:15.406 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:15.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:15.535 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:15.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:15.668 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:15.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:15.797 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:15.897 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:15.902 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:15.923 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:15.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:16.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:16.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:01:16.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.347808+0000 osd.7 (osd.7) 434 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:01:16.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.349023+0000 osd.7 (osd.7) 435 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:01:16.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.435277+0000 osd.5 (osd.5) 465 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:01:16.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.440331+0000 osd.5 (osd.5) 466 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:01:16.034 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.560229+0000 osd.6 (osd.6) 314 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:01:16.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.563118+0000 osd.6 (osd.6) 315 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:01:16.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.923137+0000 osd.0 (osd.0) 411 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:01:16.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.927606+0000 osd.0 (osd.0) 412 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:01:16.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:14.994790+0000 mgr.y (mgr.14613) 874 : cluster [DBG] pgmap v1053: 158 pgs: 158 active+clean; 380 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.31k op/s; 0 B/s, 121 objects/s recovering 2024-09-13T23:01:16.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:15.141276+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:01:16.035 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:16 smithi192 bash[24608]: cluster 2024-09-13T23:01:15.143830+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:01:16.049 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:16.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:16.174 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.347808+0000 osd.7 (osd.7) 434 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.349023+0000 osd.7 (osd.7) 435 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.435277+0000 osd.5 (osd.5) 465 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.440331+0000 osd.5 (osd.5) 466 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.560229+0000 osd.6 (osd.6) 314 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.563118+0000 osd.6 (osd.6) 315 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:01:16.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.923137+0000 osd.0 (osd.0) 411 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.927606+0000 osd.0 (osd.0) 412 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:14.994790+0000 mgr.y (mgr.14613) 874 : cluster [DBG] pgmap v1053: 158 pgs: 158 active+clean; 380 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.31k op/s; 0 B/s, 121 objects/s recovering 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:15.141276+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[29889]: cluster 2024-09-13T23:01:15.143830+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.347808+0000 osd.7 (osd.7) 434 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.349023+0000 osd.7 (osd.7) 435 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.435277+0000 osd.5 (osd.5) 465 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.440331+0000 osd.5 (osd.5) 466 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.560229+0000 osd.6 (osd.6) 314 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:01:16.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.563118+0000 osd.6 (osd.6) 315 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:01:16.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.923137+0000 osd.0 (osd.0) 411 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:01:16.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.927606+0000 osd.0 (osd.0) 412 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:01:16.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:14.994790+0000 mgr.y (mgr.14613) 874 : cluster [DBG] pgmap v1053: 158 pgs: 158 active+clean; 380 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.31k op/s; 0 B/s, 121 objects/s recovering 2024-09-13T23:01:16.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:15.141276+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:01:16.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:16 smithi178 bash[22840]: cluster 2024-09-13T23:01:15.143830+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:01:16.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:16.301 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:16.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.433 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:16.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:16.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:16.569 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:16.674 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:16.692 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:16.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:16.821 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:16.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:16.950 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:16.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:17.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:17.082 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:17.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:17.220 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:17.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:17.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:17 smithi178 bash[29889]: cluster 2024-09-13T23:01:15.992405+0000 mon.a (mon.0) 1813 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 158/29250 objects degraded (0.540%), 1 pg degraded, 1 pg undersized) 2024-09-13T23:01:17.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:17 smithi178 bash[29889]: audit 2024-09-13T23:01:16.030019+0000 mgr.y (mgr.14613) 875 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:17.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:17 smithi178 bash[22840]: cluster 2024-09-13T23:01:15.992405+0000 mon.a (mon.0) 1813 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 158/29250 objects degraded (0.540%), 1 pg degraded, 1 pg undersized) 2024-09-13T23:01:17.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:17 smithi178 bash[22840]: audit 2024-09-13T23:01:16.030019+0000 mgr.y (mgr.14613) 875 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.361 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:17.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:17.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:17.488 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:17.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:17.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:17 smithi192 bash[24608]: cluster 2024-09-13T23:01:15.992405+0000 mon.a (mon.0) 1813 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 158/29250 objects degraded (0.540%), 1 pg degraded, 1 pg undersized) 2024-09-13T23:01:17.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:17 smithi192 bash[24608]: audit 2024-09-13T23:01:16.030019+0000 mgr.y (mgr.14613) 875 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:17.617 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:17.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:17.741 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:17.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:17.883 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:17.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:17.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:18.023 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:18.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:18.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:18 smithi178 bash[29889]: cluster 2024-09-13T23:01:16.436611+0000 osd.5 (osd.5) 467 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:01:18.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:18 smithi178 bash[29889]: cluster 2024-09-13T23:01:16.437594+0000 osd.5 (osd.5) 468 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:01:18.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:18 smithi178 bash[29889]: cluster 2024-09-13T23:01:16.995657+0000 mgr.y (mgr.14613) 876 : cluster [DBG] pgmap v1054: 158 pgs: 158 active+clean; 362 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 850 B/s rd, 0 B/s wr, 665 op/s; 0 B/s, 26 objects/s recovering 2024-09-13T23:01:18.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:18 smithi178 bash[22840]: cluster 2024-09-13T23:01:16.436611+0000 osd.5 (osd.5) 467 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:01:18.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:18 smithi178 bash[22840]: cluster 2024-09-13T23:01:16.437594+0000 osd.5 (osd.5) 468 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:01:18.137 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:18 smithi178 bash[22840]: cluster 2024-09-13T23:01:16.995657+0000 mgr.y (mgr.14613) 876 : cluster [DBG] pgmap v1054: 158 pgs: 158 active+clean; 362 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 850 B/s rd, 0 B/s wr, 665 op/s; 0 B/s, 26 objects/s recovering 2024-09-13T23:01:18.160 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:18.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:18.311 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:18.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:18 smithi192 bash[24608]: cluster 2024-09-13T23:01:16.436611+0000 osd.5 (osd.5) 467 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:01:18.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:18 smithi192 bash[24608]: cluster 2024-09-13T23:01:16.437594+0000 osd.5 (osd.5) 468 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:01:18.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:18 smithi192 bash[24608]: cluster 2024-09-13T23:01:16.995657+0000 mgr.y (mgr.14613) 876 : cluster [DBG] pgmap v1054: 158 pgs: 158 active+clean; 362 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 850 B/s rd, 0 B/s wr, 665 op/s; 0 B/s, 26 objects/s recovering 2024-09-13T23:01:18.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:18.442 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:18.445 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T23:01:18.445 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T23:01:18.480 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T23:01:18.481 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T23:01:18.536 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T23:01:18.536 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T23:01:18.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:18.546 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:01:18.546 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:01:18.578 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T23:01:18.579 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T23:01:18.580 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:18.603 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T23:01:18.604 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T23:01:18.628 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T23:01:18.629 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T23:01:18.656 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T23:01:18.656 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T23:01:18.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:18.692 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T23:01:18.693 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T23:01:18.715 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:18.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:18.842 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:18.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:18.970 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:18.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:19.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:19.116 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:19.203 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T23:01:19.210 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T23:01:19.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:19.249 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:19.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:19.382 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:19.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:17.309500+0000 osd.7 (osd.7) 436 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:01:19.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:17.315395+0000 osd.7 (osd.7) 437 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:17.839449+0000 osd.1 (osd.1) 418 : cluster [DBG] 6.e scrub starts 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:17.845948+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.e scrub ok 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.128380+0000 osd.3 (osd.3) 395 : cluster [DBG] 3.b scrub starts 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.132742+0000 osd.3 (osd.3) 396 : cluster [DBG] 3.b scrub ok 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.187520+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: audit 2024-09-13T23:01:18.367034+0000 mon.a (mon.0) 1815 : audit [DBG] from='client.? v1:172.21.15.178:0/2800246516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: audit 2024-09-13T23:01:18.568903+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.178:0/2795025305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:01:19.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: audit 2024-09-13T23:01:18.570048+0000 mon.a (mon.0) 1816 : audit [INF] from='client.25792 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:01:19.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[22840]: audit 2024-09-13T23:01:18.603764+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.178:0/2072866010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:19.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:17.309500+0000 osd.7 (osd.7) 436 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:01:19.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:17.315395+0000 osd.7 (osd.7) 437 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:01:19.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:17.839449+0000 osd.1 (osd.1) 418 : cluster [DBG] 6.e scrub starts 2024-09-13T23:01:19.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:17.845948+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.e scrub ok 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.128380+0000 osd.3 (osd.3) 395 : cluster [DBG] 3.b scrub starts 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.132742+0000 osd.3 (osd.3) 396 : cluster [DBG] 3.b scrub ok 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.187520+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: audit 2024-09-13T23:01:18.367034+0000 mon.a (mon.0) 1815 : audit [DBG] from='client.? v1:172.21.15.178:0/2800246516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: audit 2024-09-13T23:01:18.568903+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.178:0/2795025305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: audit 2024-09-13T23:01:18.570048+0000 mon.a (mon.0) 1816 : audit [INF] from='client.25792 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:01:19.433 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[29889]: audit 2024-09-13T23:01:18.603764+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.178:0/2072866010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:19.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:17.309500+0000 osd.7 (osd.7) 436 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:17.315395+0000 osd.7 (osd.7) 437 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:17.839449+0000 osd.1 (osd.1) 418 : cluster [DBG] 6.e scrub starts 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:17.845948+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.e scrub ok 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.128380+0000 osd.3 (osd.3) 395 : cluster [DBG] 3.b scrub starts 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.132742+0000 osd.3 (osd.3) 396 : cluster [DBG] 3.b scrub ok 2024-09-13T23:01:19.496 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.187520+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-09-13T23:01:19.497 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: audit 2024-09-13T23:01:18.367034+0000 mon.a (mon.0) 1815 : audit [DBG] from='client.? v1:172.21.15.178:0/2800246516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:19.497 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: audit 2024-09-13T23:01:18.568903+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.178:0/2795025305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:01:19.497 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: audit 2024-09-13T23:01:18.570048+0000 mon.a (mon.0) 1816 : audit [INF] from='client.25792 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:01:19.497 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:19 smithi192 bash[24608]: audit 2024-09-13T23:01:18.603764+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.178:0/2072866010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:19.514 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:19.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T23:01:19.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:19.684 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:19.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:19.812 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:19.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:19.937 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:19.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:20.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:20.062 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:20.166 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:20.177 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:01:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:01:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:01:20.191 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:20.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:20.305 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.428722+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.e scrub starts 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.436646+0000 osd.5 (osd.5) 470 : cluster [DBG] 5.e scrub ok 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.537100+0000 osd.2 (osd.2) 305 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.539107+0000 osd.2 (osd.2) 306 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:18.996330+0000 mgr.y (mgr.14613) 877 : cluster [DBG] pgmap v1056: 158 pgs: 158 active+clean; 343 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 729 B/s rd, 0 B/s wr, 581 op/s; 0 B/s, 22 objects/s recovering 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.119633+0000 osd.3 (osd.3) 397 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.136402+0000 osd.3 (osd.3) 398 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.202491+0000 mon.a (mon.0) 1818 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.202536+0000 mon.a (mon.0) 1819 : cluster [INF] Cluster is now healthy 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: audit 2024-09-13T23:01:19.204047+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25792 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: audit 2024-09-13T23:01:19.204181+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? v1:172.21.15.178:0/2072866010' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T23:01:20.306 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.207446+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-09-13T23:01:20.307 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: audit 2024-09-13T23:01:19.619196+0000 mon.a (mon.0) 1823 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:01:20.307 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:20 smithi192 bash[24608]: audit 2024-09-13T23:01:19.623416+0000 mon.b (mon.1) 294 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:20.318 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:20.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.428722+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.e scrub starts 2024-09-13T23:01:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.436646+0000 osd.5 (osd.5) 470 : cluster [DBG] 5.e scrub ok 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.537100+0000 osd.2 (osd.2) 305 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.539107+0000 osd.2 (osd.2) 306 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:18.996330+0000 mgr.y (mgr.14613) 877 : cluster [DBG] pgmap v1056: 158 pgs: 158 active+clean; 343 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 729 B/s rd, 0 B/s wr, 581 op/s; 0 B/s, 22 objects/s recovering 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.119633+0000 osd.3 (osd.3) 397 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.136402+0000 osd.3 (osd.3) 398 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.202491+0000 mon.a (mon.0) 1818 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.202536+0000 mon.a (mon.0) 1819 : cluster [INF] Cluster is now healthy 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: audit 2024-09-13T23:01:19.204047+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25792 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: audit 2024-09-13T23:01:19.204181+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? v1:172.21.15.178:0/2072866010' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.207446+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: audit 2024-09-13T23:01:19.619196+0000 mon.a (mon.0) 1823 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:01:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[29889]: audit 2024-09-13T23:01:19.623416+0000 mon.b (mon.1) 294 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.428722+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.e scrub starts 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.436646+0000 osd.5 (osd.5) 470 : cluster [DBG] 5.e scrub ok 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.537100+0000 osd.2 (osd.2) 305 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.539107+0000 osd.2 (osd.2) 306 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:18.996330+0000 mgr.y (mgr.14613) 877 : cluster [DBG] pgmap v1056: 158 pgs: 158 active+clean; 343 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 729 B/s rd, 0 B/s wr, 581 op/s; 0 B/s, 22 objects/s recovering 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.119633+0000 osd.3 (osd.3) 397 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.136402+0000 osd.3 (osd.3) 398 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.202491+0000 mon.a (mon.0) 1818 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.202536+0000 mon.a (mon.0) 1819 : cluster [INF] Cluster is now healthy 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: audit 2024-09-13T23:01:19.204047+0000 mon.a (mon.0) 1820 : audit [INF] from='client.25792 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: audit 2024-09-13T23:01:19.204181+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? v1:172.21.15.178:0/2072866010' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T23:01:20.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.207446+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-09-13T23:01:20.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: audit 2024-09-13T23:01:19.619196+0000 mon.a (mon.0) 1823 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:01:20.433 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:20 smithi178 bash[22840]: audit 2024-09-13T23:01:19.623416+0000 mon.b (mon.1) 294 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:20.449 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:20.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:20.581 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:20.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:20.715 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:20.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:20.839 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:20.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:20.975 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:20.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:21.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:21.116 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:21.189 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Removed 85027 objects 2024-09-13T23:01:21.189 INFO:tasks.radosbench.radosbench.0.smithi178.stdout:Clean up completed and total clean up time :103.442 2024-09-13T23:01:21.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.241 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:21.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:21.367 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:21.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.480 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:21 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.546335+0000 osd.6 (osd.6) 316 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:01:21.480 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:21 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.547727+0000 osd.6 (osd.6) 317 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:01:21.480 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:21 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.926942+0000 osd.1 (osd.1) 420 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:21.480 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:21 smithi192 bash[24608]: cluster 2024-09-13T23:01:19.927004+0000 osd.1 (osd.1) 421 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:21.480 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:21 smithi192 bash[24608]: cluster 2024-09-13T23:01:20.629756+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-09-13T23:01:21.495 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:21.503 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.546335+0000 osd.6 (osd.6) 316 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:01:21.503 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.547727+0000 osd.6 (osd.6) 317 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:01:21.503 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.926942+0000 osd.1 (osd.1) 420 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:21.503 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[29889]: cluster 2024-09-13T23:01:19.927004+0000 osd.1 (osd.1) 421 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:21.503 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[29889]: cluster 2024-09-13T23:01:20.629756+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-09-13T23:01:21.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.546335+0000 osd.6 (osd.6) 316 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:01:21.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.547727+0000 osd.6 (osd.6) 317 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:01:21.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.926942+0000 osd.1 (osd.1) 420 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:21.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[22840]: cluster 2024-09-13T23:01:19.927004+0000 osd.1 (osd.1) 421 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:21.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:21 smithi178 bash[22840]: cluster 2024-09-13T23:01:20.629756+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-09-13T23:01:21.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.625 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:21.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:21.756 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:21.819 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-13T23:01:21.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-13T23:01:21.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.887 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:21.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:21.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:22.017 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:22.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:22.146 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:22.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:22.261 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:22 smithi192 bash[24608]: cluster 2024-09-13T23:01:20.579657+0000 osd.6 (osd.6) 318 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:01:22.261 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:22 smithi192 bash[24608]: cluster 2024-09-13T23:01:20.581283+0000 osd.6 (osd.6) 319 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:01:22.261 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:22 smithi192 bash[24608]: cluster 2024-09-13T23:01:20.851139+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:01:22.261 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:22 smithi192 bash[24608]: cluster 2024-09-13T23:01:20.851975+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:01:22.261 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:22 smithi192 bash[24608]: cluster 2024-09-13T23:01:20.996761+0000 mgr.y (mgr.14613) 878 : cluster [DBG] pgmap v1059: 158 pgs: 158 active+clean; 91 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 769 op/s 2024-09-13T23:01:22.277 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:22.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:22.409 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:22.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[29889]: cluster 2024-09-13T23:01:20.579657+0000 osd.6 (osd.6) 318 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:01:22.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[29889]: cluster 2024-09-13T23:01:20.581283+0000 osd.6 (osd.6) 319 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:01:22.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[29889]: cluster 2024-09-13T23:01:20.851139+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[29889]: cluster 2024-09-13T23:01:20.851975+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[29889]: cluster 2024-09-13T23:01:20.996761+0000 mgr.y (mgr.14613) 878 : cluster [DBG] pgmap v1059: 158 pgs: 158 active+clean; 91 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 769 op/s 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[22840]: cluster 2024-09-13T23:01:20.579657+0000 osd.6 (osd.6) 318 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[22840]: cluster 2024-09-13T23:01:20.581283+0000 osd.6 (osd.6) 319 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[22840]: cluster 2024-09-13T23:01:20.851139+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[22840]: cluster 2024-09-13T23:01:20.851975+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:01:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:22 smithi178 bash[22840]: cluster 2024-09-13T23:01:20.996761+0000 mgr.y (mgr.14613) 878 : cluster [DBG] pgmap v1059: 158 pgs: 158 active+clean; 91 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 769 op/s 2024-09-13T23:01:22.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:22.539 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:22.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:22.669 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:22.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:22.799 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:22.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:22.932 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:22.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:23.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:23.062 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:23.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:23.191 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:23.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.308113+0000 osd.7 (osd.7) 438 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.309605+0000 osd.7 (osd.7) 439 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.394430+0000 osd.5 (osd.5) 471 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.396217+0000 osd.5 (osd.5) 472 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.582750+0000 osd.6 (osd.6) 320 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.585082+0000 osd.6 (osd.6) 321 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.844718+0000 osd.1 (osd.1) 424 : cluster [DBG] 11.c deep-scrub starts 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.854181+0000 osd.1 (osd.1) 425 : cluster [DBG] 11.c deep-scrub ok 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.886269+0000 osd.0 (osd.0) 413 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:01:23.299 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[22840]: cluster 2024-09-13T23:01:21.888388+0000 osd.0 (osd.0) 414 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:01:23.300 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.308113+0000 osd.7 (osd.7) 438 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:01:23.300 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.309605+0000 osd.7 (osd.7) 439 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:01:23.300 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.394430+0000 osd.5 (osd.5) 471 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:01:23.300 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.396217+0000 osd.5 (osd.5) 472 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:01:23.300 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.582750+0000 osd.6 (osd.6) 320 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:01:23.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.585082+0000 osd.6 (osd.6) 321 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:01:23.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.844718+0000 osd.1 (osd.1) 424 : cluster [DBG] 11.c deep-scrub starts 2024-09-13T23:01:23.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.854181+0000 osd.1 (osd.1) 425 : cluster [DBG] 11.c deep-scrub ok 2024-09-13T23:01:23.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.886269+0000 osd.0 (osd.0) 413 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:01:23.301 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:23 smithi178 bash[29889]: cluster 2024-09-13T23:01:21.888388+0000 osd.0 (osd.0) 414 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:01:23.319 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:23.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T23:01:23.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:23.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.308113+0000 osd.7 (osd.7) 438 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:01:23.435 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.309605+0000 osd.7 (osd.7) 439 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.394430+0000 osd.5 (osd.5) 471 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.396217+0000 osd.5 (osd.5) 472 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.582750+0000 osd.6 (osd.6) 320 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.585082+0000 osd.6 (osd.6) 321 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.844718+0000 osd.1 (osd.1) 424 : cluster [DBG] 11.c deep-scrub starts 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.854181+0000 osd.1 (osd.1) 425 : cluster [DBG] 11.c deep-scrub ok 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.886269+0000 osd.0 (osd.0) 413 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:01:23.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:23 smithi192 bash[24608]: cluster 2024-09-13T23:01:21.888388+0000 osd.0 (osd.0) 414 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:01:23.451 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:23.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:23.576 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:23.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:23.704 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:23.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:23.833 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:23.936 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:23.963 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:23.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:24.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:24.093 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:24.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T23:01:24.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.336352+0000 osd.7 (osd.7) 440 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.338160+0000 osd.7 (osd.7) 441 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.537272+0000 osd.2 (osd.2) 307 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.538974+0000 osd.2 (osd.2) 308 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.576268+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.578409+0000 osd.6 (osd.6) 323 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.822583+0000 osd.1 (osd.1) 426 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.826806+0000 osd.1 (osd.1) 427 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:01:24.206 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.897908+0000 osd.0 (osd.0) 415 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:01:24.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.901948+0000 osd.0 (osd.0) 416 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:01:24.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:22.997570+0000 mgr.y (mgr.14613) 879 : cluster [DBG] pgmap v1060: 158 pgs: 158 active+clean; 77 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 761 op/s 2024-09-13T23:01:24.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.189922+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-09-13T23:01:24.207 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.524921+0000 osd.2 (osd.2) 309 : cluster [DBG] 3.3 deep-scrub starts 2024-09-13T23:01:24.224 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:24.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:24.353 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:24.438 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:24.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:24.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.336352+0000 osd.7 (osd.7) 440 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:01:24.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.338160+0000 osd.7 (osd.7) 441 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:01:24.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.537272+0000 osd.2 (osd.2) 307 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:01:24.465 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.538974+0000 osd.2 (osd.2) 308 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.576268+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.578409+0000 osd.6 (osd.6) 323 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.822583+0000 osd.1 (osd.1) 426 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.826806+0000 osd.1 (osd.1) 427 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.897908+0000 osd.0 (osd.0) 415 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.901948+0000 osd.0 (osd.0) 416 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:22.997570+0000 mgr.y (mgr.14613) 879 : cluster [DBG] pgmap v1060: 158 pgs: 158 active+clean; 77 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 761 op/s 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.189922+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-09-13T23:01:24.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:24 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.524921+0000 osd.2 (osd.2) 309 : cluster [DBG] 3.3 deep-scrub starts 2024-09-13T23:01:24.477 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:24.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:24.590 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.336352+0000 osd.7 (osd.7) 440 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:01:24.590 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.338160+0000 osd.7 (osd.7) 441 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:01:24.590 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.537272+0000 osd.2 (osd.2) 307 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:01:24.590 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.538974+0000 osd.2 (osd.2) 308 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:01:24.590 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.576268+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:01:24.590 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.578409+0000 osd.6 (osd.6) 323 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.822583+0000 osd.1 (osd.1) 426 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.826806+0000 osd.1 (osd.1) 427 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.897908+0000 osd.0 (osd.0) 415 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.901948+0000 osd.0 (osd.0) 416 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:22.997570+0000 mgr.y (mgr.14613) 879 : cluster [DBG] pgmap v1060: 158 pgs: 158 active+clean; 77 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 761 op/s 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.189922+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-09-13T23:01:24.591 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:24 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.524921+0000 osd.2 (osd.2) 309 : cluster [DBG] 3.3 deep-scrub starts 2024-09-13T23:01:24.605 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:24.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:24.737 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:24.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:24.870 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:24.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:24.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:25.001 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:25.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:25.132 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:25.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:25.263 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:25.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T23:01:25.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:25.390 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:25.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:25.503 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.311249+0000 osd.7 (osd.7) 442 : cluster [DBG] 5.f scrub starts 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.313458+0000 osd.7 (osd.7) 443 : cluster [DBG] 5.f scrub ok 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.406332+0000 osd.5 (osd.5) 473 : cluster [DBG] 6.d scrub starts 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.410370+0000 osd.5 (osd.5) 474 : cluster [DBG] 6.d scrub ok 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.540218+0000 osd.2 (osd.2) 310 : cluster [DBG] 3.3 deep-scrub ok 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.581908+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:23.585071+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.259574+0000 osd.7 (osd.7) 444 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.259686+0000 osd.7 (osd.7) 445 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.311249+0000 osd.7 (osd.7) 442 : cluster [DBG] 5.f scrub starts 2024-09-13T23:01:25.504 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.313458+0000 osd.7 (osd.7) 443 : cluster [DBG] 5.f scrub ok 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.406332+0000 osd.5 (osd.5) 473 : cluster [DBG] 6.d scrub starts 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.410370+0000 osd.5 (osd.5) 474 : cluster [DBG] 6.d scrub ok 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.540218+0000 osd.2 (osd.2) 310 : cluster [DBG] 3.3 deep-scrub ok 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.581908+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:23.585071+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.259574+0000 osd.7 (osd.7) 444 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:25.505 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:25 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.259686+0000 osd.7 (osd.7) 445 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:25.515 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:25.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.311249+0000 osd.7 (osd.7) 442 : cluster [DBG] 5.f scrub starts 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.313458+0000 osd.7 (osd.7) 443 : cluster [DBG] 5.f scrub ok 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.406332+0000 osd.5 (osd.5) 473 : cluster [DBG] 6.d scrub starts 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.410370+0000 osd.5 (osd.5) 474 : cluster [DBG] 6.d scrub ok 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.540218+0000 osd.2 (osd.2) 310 : cluster [DBG] 3.3 deep-scrub ok 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.581908+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:23.585071+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:01:25.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.259574+0000 osd.7 (osd.7) 444 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:25.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:25 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.259686+0000 osd.7 (osd.7) 445 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:25.645 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:25.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:25.775 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:25.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:25.904 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:25.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:26.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:26.034 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:26.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:26.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:26.165 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:26.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.443333+0000 osd.5 (osd.5) 475 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.444992+0000 osd.5 (osd.5) 476 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.557787+0000 osd.2 (osd.2) 311 : cluster [DBG] 11.d scrub starts 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.569294+0000 osd.2 (osd.2) 312 : cluster [DBG] 11.d scrub ok 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:24.999244+0000 mgr.y (mgr.14613) 880 : cluster [DBG] pgmap v1062: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 914 op/s 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:25.220018+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:25.500374+0000 osd.4 (osd.4) 280 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:26.279 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[29889]: cluster 2024-09-13T23:01:25.500477+0000 osd.4 (osd.4) 281 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.443333+0000 osd.5 (osd.5) 475 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.444992+0000 osd.5 (osd.5) 476 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.557787+0000 osd.2 (osd.2) 311 : cluster [DBG] 11.d scrub starts 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.569294+0000 osd.2 (osd.2) 312 : cluster [DBG] 11.d scrub ok 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:24.999244+0000 mgr.y (mgr.14613) 880 : cluster [DBG] pgmap v1062: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 914 op/s 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:25.220018+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:25.500374+0000 osd.4 (osd.4) 280 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:26.280 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:26 smithi178 bash[22840]: cluster 2024-09-13T23:01:25.500477+0000 osd.4 (osd.4) 281 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:26.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:01:26.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.443333+0000 osd.5 (osd.5) 475 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:01:26.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.444992+0000 osd.5 (osd.5) 476 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:01:26.295 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.557787+0000 osd.2 (osd.2) 311 : cluster [DBG] 11.d scrub starts 2024-09-13T23:01:26.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.569294+0000 osd.2 (osd.2) 312 : cluster [DBG] 11.d scrub ok 2024-09-13T23:01:26.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:24.999244+0000 mgr.y (mgr.14613) 880 : cluster [DBG] pgmap v1062: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 914 op/s 2024-09-13T23:01:26.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:25.220018+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-09-13T23:01:26.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:25.500374+0000 osd.4 (osd.4) 280 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:01:26.296 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:26 smithi192 bash[24608]: cluster 2024-09-13T23:01:25.500477+0000 osd.4 (osd.4) 281 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:01:26.296 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:26.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:26.422 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:26.469 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T23:01:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-09-13T23:01:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-09-13T23:01:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-09-13T23:01:26.470 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T23:01:26.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:26.546 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:26.610 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:26.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:26.670 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:26.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:26.797 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:26.900 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:26.928 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:26.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:27.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:27.058 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:27.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:27.188 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:27.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:27.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:27 smithi192 bash[24608]: audit 2024-09-13T23:01:26.038107+0000 mgr.y (mgr.14613) 881 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:27.302 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:27 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.222291+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-09-13T23:01:27.303 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:27 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.263988+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.a scrub starts 2024-09-13T23:01:27.303 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:27 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.266516+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.a scrub ok 2024-09-13T23:01:27.317 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:27.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T23:01:27.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:27.447 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:27.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:27.562 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[29889]: audit 2024-09-13T23:01:26.038107+0000 mgr.y (mgr.14613) 881 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:27.562 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.222291+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-09-13T23:01:27.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.263988+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.a scrub starts 2024-09-13T23:01:27.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.266516+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.a scrub ok 2024-09-13T23:01:27.563 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[22840]: audit 2024-09-13T23:01:26.038107+0000 mgr.y (mgr.14613) 881 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:27.563 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.222291+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-09-13T23:01:27.563 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.263988+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.a scrub starts 2024-09-13T23:01:27.564 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:27 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.266516+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.a scrub ok 2024-09-13T23:01:27.579 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:27.628 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:27.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:27.702 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:27.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:27.833 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:27.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:27.964 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:27.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:28.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:28.095 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:28.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:28.222 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:28.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:28.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.317202+0000 osd.7 (osd.7) 446 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.320259+0000 osd.7 (osd.7) 447 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.606245+0000 osd.2 (osd.2) 313 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.610081+0000 osd.2 (osd.2) 314 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.953079+0000 osd.0 (osd.0) 417 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:26.955043+0000 osd.0 (osd.0) 418 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.000177+0000 mgr.y (mgr.14613) 882 : cluster [DBG] pgmap v1065: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 243 op/s 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.317202+0000 osd.7 (osd.7) 446 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.320259+0000 osd.7 (osd.7) 447 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.606245+0000 osd.2 (osd.2) 313 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.610081+0000 osd.2 (osd.2) 314 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.953079+0000 osd.0 (osd.0) 417 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:01:28.338 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:26.955043+0000 osd.0 (osd.0) 418 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:01:28.339 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:28 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.000177+0000 mgr.y (mgr.14613) 882 : cluster [DBG] pgmap v1065: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 243 op/s 2024-09-13T23:01:28.353 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:28.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:28.483 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:28.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:28.598 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.317202+0000 osd.7 (osd.7) 446 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:01:28.599 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.320259+0000 osd.7 (osd.7) 447 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:01:28.599 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.606245+0000 osd.2 (osd.2) 313 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:01:28.599 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.610081+0000 osd.2 (osd.2) 314 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:01:28.599 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.953079+0000 osd.0 (osd.0) 417 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:01:28.599 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:26.955043+0000 osd.0 (osd.0) 418 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:01:28.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:28 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.000177+0000 mgr.y (mgr.14613) 882 : cluster [DBG] pgmap v1065: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 243 op/s 2024-09-13T23:01:28.615 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:28.640 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:28.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:28.745 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:28.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:28.875 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:28.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:28.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:29.006 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:29.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:29.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:29.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.511562+0000 osd.5 (osd.5) 477 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.514417+0000 osd.5 (osd.5) 478 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.628260+0000 osd.2 (osd.2) 315 : cluster [DBG] 2.1 deep-scrub starts 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.631206+0000 osd.2 (osd.2) 316 : cluster [DBG] 2.1 deep-scrub ok 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.806980+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.810007+0000 osd.1 (osd.1) 429 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.918973+0000 osd.0 (osd.0) 419 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:01:29.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[29889]: cluster 2024-09-13T23:01:27.921309+0000 osd.0 (osd.0) 420 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.511562+0000 osd.5 (osd.5) 477 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.514417+0000 osd.5 (osd.5) 478 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.628260+0000 osd.2 (osd.2) 315 : cluster [DBG] 2.1 deep-scrub starts 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.631206+0000 osd.2 (osd.2) 316 : cluster [DBG] 2.1 deep-scrub ok 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.806980+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.810007+0000 osd.1 (osd.1) 429 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.918973+0000 osd.0 (osd.0) 419 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:01:29.385 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[22840]: cluster 2024-09-13T23:01:27.921309+0000 osd.0 (osd.0) 420 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:01:29.399 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:29.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:29.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.511562+0000 osd.5 (osd.5) 477 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:01:29.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.514417+0000 osd.5 (osd.5) 478 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:01:29.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.628260+0000 osd.2 (osd.2) 315 : cluster [DBG] 2.1 deep-scrub starts 2024-09-13T23:01:29.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.631206+0000 osd.2 (osd.2) 316 : cluster [DBG] 2.1 deep-scrub ok 2024-09-13T23:01:29.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.806980+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:01:29.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.810007+0000 osd.1 (osd.1) 429 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:01:29.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.918973+0000 osd.0 (osd.0) 419 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:01:29.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:29 smithi192 bash[24608]: cluster 2024-09-13T23:01:27.921309+0000 osd.0 (osd.0) 420 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:01:29.526 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:29.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:29.657 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:29.665 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:29.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:29.785 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:29.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:29.914 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:29.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:30.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:30.043 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:30.106 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:01:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:01:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:01:30.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:30.171 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:30.274 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:30.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:30.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:30 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.001187+0000 mgr.y (mgr.14613) 883 : cluster [DBG] pgmap v1066: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 203 op/s 2024-09-13T23:01:30.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:30 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.236225+0000 osd.3 (osd.3) 401 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:01:30.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:30 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.240232+0000 osd.3 (osd.3) 402 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:01:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:30 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.001187+0000 mgr.y (mgr.14613) 883 : cluster [DBG] pgmap v1066: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 203 op/s 2024-09-13T23:01:30.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:30 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.236225+0000 osd.3 (osd.3) 401 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:01:30.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:30 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.240232+0000 osd.3 (osd.3) 402 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:01:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:30 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.001187+0000 mgr.y (mgr.14613) 883 : cluster [DBG] pgmap v1066: 158 pgs: 158 active+clean; 583 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 203 op/s 2024-09-13T23:01:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:30 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.236225+0000 osd.3 (osd.3) 401 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:01:30.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:30 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.240232+0000 osd.3 (osd.3) 402 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:01:30.433 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:30.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:30.563 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:30.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:30.693 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:30.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:30.824 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:30.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:30.954 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:30.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:31.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:31.079 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:31.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:31.202 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:31.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:31.334 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:31.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.341308+0000 osd.7 (osd.7) 448 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.345340+0000 osd.7 (osd.7) 449 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.455644+0000 osd.5 (osd.5) 479 : cluster [DBG] 3.c scrub starts 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.459730+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.c scrub ok 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.622818+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.625032+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.898510+0000 osd.1 (osd.1) 430 : cluster [DBG] 11.0 deep-scrub starts 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:29.909787+0000 osd.1 (osd.1) 431 : cluster [DBG] 11.0 deep-scrub ok 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.224656+0000 osd.3 (osd.3) 403 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.226990+0000 osd.3 (osd.3) 404 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.341308+0000 osd.7 (osd.7) 448 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T23:01:31.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.345340+0000 osd.7 (osd.7) 449 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.455644+0000 osd.5 (osd.5) 479 : cluster [DBG] 3.c scrub starts 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.459730+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.c scrub ok 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.622818+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.625032+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.898510+0000 osd.1 (osd.1) 430 : cluster [DBG] 11.0 deep-scrub starts 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:29.909787+0000 osd.1 (osd.1) 431 : cluster [DBG] 11.0 deep-scrub ok 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.224656+0000 osd.3 (osd.3) 403 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:01:31.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:31 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.226990+0000 osd.3 (osd.3) 404 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:01:31.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:31.464 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:31.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:31.595 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.341308+0000 osd.7 (osd.7) 448 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T23:01:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.345340+0000 osd.7 (osd.7) 449 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T23:01:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.455644+0000 osd.5 (osd.5) 479 : cluster [DBG] 3.c scrub starts 2024-09-13T23:01:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.459730+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.c scrub ok 2024-09-13T23:01:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.622818+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.c scrub starts 2024-09-13T23:01:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.625032+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.c scrub ok 2024-09-13T23:01:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.898510+0000 osd.1 (osd.1) 430 : cluster [DBG] 11.0 deep-scrub starts 2024-09-13T23:01:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:29.909787+0000 osd.1 (osd.1) 431 : cluster [DBG] 11.0 deep-scrub ok 2024-09-13T23:01:31.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.224656+0000 osd.3 (osd.3) 403 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:01:31.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:31 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.226990+0000 osd.3 (osd.3) 404 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:01:31.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:31.725 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:31.828 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:31.855 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:31.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:31.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:32.089 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:32.117 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:32.191 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:32.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:32.246 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:32.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:32.374 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.316580+0000 osd.7 (osd.7) 450 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.318712+0000 osd.7 (osd.7) 451 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.647138+0000 osd.2 (osd.2) 319 : cluster [DBG] 5.b scrub starts 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.648980+0000 osd.2 (osd.2) 320 : cluster [DBG] 5.b scrub ok 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.884720+0000 osd.1 (osd.1) 432 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:30.886948+0000 osd.1 (osd.1) 433 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.002625+0000 mgr.y (mgr.14613) 884 : cluster [DBG] pgmap v1067: 158 pgs: 158 active+clean; 583 KiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 156 op/s 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.006965+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:01:32.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.009168+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.177239+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.178434+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.316580+0000 osd.7 (osd.7) 450 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.318712+0000 osd.7 (osd.7) 451 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.647138+0000 osd.2 (osd.2) 319 : cluster [DBG] 5.b scrub starts 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.648980+0000 osd.2 (osd.2) 320 : cluster [DBG] 5.b scrub ok 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.884720+0000 osd.1 (osd.1) 432 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:30.886948+0000 osd.1 (osd.1) 433 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.002625+0000 mgr.y (mgr.14613) 884 : cluster [DBG] pgmap v1067: 158 pgs: 158 active+clean; 583 KiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 156 op/s 2024-09-13T23:01:32.390 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.006965+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:01:32.391 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.009168+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:01:32.391 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.177239+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:01:32.391 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:32 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.178434+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:01:32.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:32.505 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:32.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:32.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.316580+0000 osd.7 (osd.7) 450 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:01:32.620 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.318712+0000 osd.7 (osd.7) 451 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.647138+0000 osd.2 (osd.2) 319 : cluster [DBG] 5.b scrub starts 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.648980+0000 osd.2 (osd.2) 320 : cluster [DBG] 5.b scrub ok 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.884720+0000 osd.1 (osd.1) 432 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:30.886948+0000 osd.1 (osd.1) 433 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.002625+0000 mgr.y (mgr.14613) 884 : cluster [DBG] pgmap v1067: 158 pgs: 158 active+clean; 583 KiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 156 op/s 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.006965+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.009168+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.177239+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:01:32.621 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:32 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.178434+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:01:32.634 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:32.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:32.764 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:32.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:32.896 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:32.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:32.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:33.027 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:33.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:33.158 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:33.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:33.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.602302+0000 osd.2 (osd.2) 321 : cluster [DBG] 1.5 deep-scrub starts 2024-09-13T23:01:33.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[22840]: cluster 2024-09-13T23:01:31.604107+0000 osd.2 (osd.2) 322 : cluster [DBG] 1.5 deep-scrub ok 2024-09-13T23:01:33.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[22840]: cluster 2024-09-13T23:01:32.176459+0000 osd.3 (osd.3) 407 : cluster [DBG] 5.17 scrub starts 2024-09-13T23:01:33.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[22840]: cluster 2024-09-13T23:01:32.178919+0000 osd.3 (osd.3) 408 : cluster [DBG] 5.17 scrub ok 2024-09-13T23:01:33.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[22840]: audit 2024-09-13T23:01:32.706268+0000 mon.b (mon.1) 295 : audit [INF] from='client.? v1:172.21.15.178:0/11924787' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T23:01:33.265 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[22840]: audit 2024-09-13T23:01:32.706737+0000 mon.a (mon.0) 1828 : audit [INF] from='client.25807 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T23:01:33.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.602302+0000 osd.2 (osd.2) 321 : cluster [DBG] 1.5 deep-scrub starts 2024-09-13T23:01:33.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[29889]: cluster 2024-09-13T23:01:31.604107+0000 osd.2 (osd.2) 322 : cluster [DBG] 1.5 deep-scrub ok 2024-09-13T23:01:33.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[29889]: cluster 2024-09-13T23:01:32.176459+0000 osd.3 (osd.3) 407 : cluster [DBG] 5.17 scrub starts 2024-09-13T23:01:33.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[29889]: cluster 2024-09-13T23:01:32.178919+0000 osd.3 (osd.3) 408 : cluster [DBG] 5.17 scrub ok 2024-09-13T23:01:33.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[29889]: audit 2024-09-13T23:01:32.706268+0000 mon.b (mon.1) 295 : audit [INF] from='client.? v1:172.21.15.178:0/11924787' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T23:01:33.266 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:33 smithi178 bash[29889]: audit 2024-09-13T23:01:32.706737+0000 mon.a (mon.0) 1828 : audit [INF] from='client.25807 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T23:01:33.272 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'unique_pool_4' removed 2024-09-13T23:01:33.284 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:33.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:33.408 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:33.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:33.542 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:33.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:33.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:33 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.602302+0000 osd.2 (osd.2) 321 : cluster [DBG] 1.5 deep-scrub starts 2024-09-13T23:01:33.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:33 smithi192 bash[24608]: cluster 2024-09-13T23:01:31.604107+0000 osd.2 (osd.2) 322 : cluster [DBG] 1.5 deep-scrub ok 2024-09-13T23:01:33.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:33 smithi192 bash[24608]: cluster 2024-09-13T23:01:32.176459+0000 osd.3 (osd.3) 407 : cluster [DBG] 5.17 scrub starts 2024-09-13T23:01:33.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:33 smithi192 bash[24608]: cluster 2024-09-13T23:01:32.178919+0000 osd.3 (osd.3) 408 : cluster [DBG] 5.17 scrub ok 2024-09-13T23:01:33.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:33 smithi192 bash[24608]: audit 2024-09-13T23:01:32.706268+0000 mon.b (mon.1) 295 : audit [INF] from='client.? v1:172.21.15.178:0/11924787' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T23:01:33.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:33 smithi192 bash[24608]: audit 2024-09-13T23:01:32.706737+0000 mon.a (mon.0) 1828 : audit [INF] from='client.25807 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-13T23:01:33.701 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0 2024-09-13T23:01:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.0' 2024-09-13T23:01:33.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177306 10.e 10:771d6b3d:::benchmark_data_smithi178_111521_object66570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.301669+0000", 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.50691239100001, 2024-09-13T23:01:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.045613065, 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177306 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301669+0000", 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301669+0000", 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301671+0000", 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-09-13T23:01:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301678+0000", 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7789999999999998e-06 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301693+0000", 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5551999999999999e-05 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301701+0000", 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0050000000000004e-06 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301768+0000", 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7111000000000002e-05 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301776+0000", 2024-09-13T23:01:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0560000000000005e-06 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.323444+0000", 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.021667702 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.323651+0000", 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000206527 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.324252+0000", 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000601546 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.341049+0000", 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016796993999999999 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.341223+0000", 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017398000000000001 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347168+0000", 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059451870000000002 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347243+0000", 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4627000000000002e-05 2024-09-13T23:01:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347282+0000", 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8729000000000001e-05 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177307 10.e 10:72a3b04f:::benchmark_data_smithi178_111521_object66571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.301720+0000", 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.50686082300001, 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0505944739999999, 2024-09-13T23:01:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177307 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301720+0000", 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301720+0000", 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301721+0000", 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3900000000000004e-07 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301724+0000", 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.818e-06 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301731+0000", 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.063e-06 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301735+0000", 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5299999999999998e-06 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301814+0000", 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8314999999999998e-05 2024-09-13T23:01:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.301827+0000", 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3484e-05 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.341292+0000", 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0394650320000001 2024-09-13T23:01:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.341424+0000", 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013176400000000001 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.341529+0000", 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104515 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347150+0000", 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056211229999999996 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347367+0000", 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000217209 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352000+0000", 2024-09-13T23:01:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046328569999999998 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352086+0000", 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5736999999999996e-05 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352315+0000", 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022918799999999999 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177325 10.e 10:79159b8c:::benchmark_data_smithi178_111521_object66589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.322823+0000", 2024-09-13T23:01:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.485757605, 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0296663260000001, 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177325 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322823+0000", 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322823+0000", 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322826+0000", 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4399999999999999e-06 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322833+0000", 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3089999999999998e-06 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322846+0000", 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2991e-05 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322855+0000", 2024-09-13T23:01:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8140000000000007e-06 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322943+0000", 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8221999999999997e-05 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.322956+0000", 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2757e-05 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.343259+0000", 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0203030930000001 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.343388+0000", 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012872400000000001 2024-09-13T23:01:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.343481+0000", 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3345999999999997e-05 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.351723+0000", 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082417940000000002 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.351814+0000", 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0958000000000006e-05 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352390+0000", 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00057574899999999997 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352443+0000", 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3248000000000002e-05 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352490+0000", 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6881000000000001e-05 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177386 10.e 10:7b212f41:::benchmark_data_smithi178_111521_object66650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.392241+0000", 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.41634040500003, 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96037671499999999, 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177386 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.392241+0000", 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.392241+0000", 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.392243+0000", 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2069999999999998e-06 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.392250+0000", 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7880000000000001e-06 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.392266+0000", 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6175e-05 2024-09-13T23:01:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.392274+0000", 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3559999999999993e-06 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394650+0000", 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023761659999999999 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394660+0000", 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0111e-05 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.345004+0000", 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95034316900000004 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.345179+0000", 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017509900000000001 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.345386+0000", 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020684699999999999 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.351729+0000", 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063431889999999999 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.351871+0000", 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014199899999999999 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352541+0000", 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067014199999999998 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352583+0000", 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2132999999999998e-05 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352617+0000", 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4334e-05 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177387 10.e 10:76be60f2:::benchmark_data_smithi178_111521_object66651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.394269+0000", 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.41431240200001, 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95868055100000005, 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177387 2024-09-13T23:01:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394269+0000", 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394269+0000", 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394270+0000", 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.184e-06 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394274+0000", 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4220000000000002e-06 2024-09-13T23:01:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394282+0000", 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7940000000000003e-06 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394288+0000", 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9359999999999996e-06 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394686+0000", 2024-09-13T23:01:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000398133 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.394693+0000", 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6980000000000001e-06 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347408+0000", 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95271494099999998 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347548+0000", 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014070599999999999 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.347735+0000", 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018638300000000001 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352505+0000", 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004770069 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352660+0000", 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015529699999999999 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352856+0000", 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000195663 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352909+0000", 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3204000000000001e-05 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352949+0000", 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0120999999999998e-05 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177402 10.e 10:7ffd0a9e:::benchmark_data_smithi178_111521_object66666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.416147+0000", 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.392434106, 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93698223800000002, 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177402 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416147+0000", 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416147+0000", 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416149+0000", 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3010000000000001e-06 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416155+0000", 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-09-13T23:01:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416175+0000", 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9748000000000001e-05 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416181+0000", 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7360000000000004e-06 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416246+0000", 2024-09-13T23:01:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4788000000000004e-05 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.416259+0000", 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2262999999999999e-05 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.349723+0000", 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93346406699999995 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.349847+0000", 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000124428 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.350076+0000", 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022917400000000001 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352626+0000", 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025496469999999999 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.352998+0000", 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037221299999999998 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.353063+0000", 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5197000000000006e-05 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.353092+0000", 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8626e-05 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.353129+0000", 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7320999999999997e-05 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177460 10.e 10:70b1bfef:::benchmark_data_smithi178_111521_object66724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.478665+0000", 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.32991611800003, 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.888435325, 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177460 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478665+0000", 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478665+0000", 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478668+0000", 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.836e-06 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478674+0000", 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6170000000000003e-06 2024-09-13T23:01:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478688+0000", 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3574000000000001e-05 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478696+0000", 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9789999999999993e-06 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478749+0000", 2024-09-13T23:01:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3297000000000001e-05 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.478756+0000", 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7530000000000004e-06 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.353206+0000", 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87444952600000003 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.353395+0000", 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018914100000000001 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.353505+0000", 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000110525 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.363559+0000", 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010054307 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.366900+0000", 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003340113 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.366967+0000", 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7422000000000006e-05 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.367027+0000", 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9551e-05 2024-09-13T23:01:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.367100+0000", 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3683999999999995e-05 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177469 10.e 10:795b7d64:::benchmark_data_smithi178_111521_object66733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.487334+0000", 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.32124721600002, 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88438218700000004, 2024-09-13T23:01:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177469 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487334+0000", 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487334+0000", 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487336+0000", 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.824e-06 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487340+0000", 2024-09-13T23:01:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7439999999999998e-06 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487350+0000", 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7399999999999999e-06 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487356+0000", 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5539999999999997e-06 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487401+0000", 2024-09-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5785999999999998e-05 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.487408+0000", 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1739999999999997e-06 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.358515+0000", 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87110716799999999 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.358664+0000", 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014963300000000001 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.358807+0000", 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000142173 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.367116+0000", 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0083097899999999992 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.367157+0000", 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0564e-05 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.371597+0000", 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004439681 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.371659+0000", 2024-09-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.198e-05 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.371716+0000", 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7376000000000001e-05 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:177478 10.e 10:775baf7f:::benchmark_data_smithi178_111521_object66742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:56:22.497990+0000", 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 311.310590972, 2024-09-13T23:01:33.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88354499900000005, 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 177478 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.497990+0000", 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.497990+0000", 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.497992+0000", 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9479999999999998e-06 2024-09-13T23:01:33.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.497998+0000", 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8139999999999997e-06 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.498008+0000", 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0594000000000001e-05 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.498016+0000", 2024-09-13T23:01:33.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.763e-06 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.498065+0000", 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8687999999999999e-05 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:22.498073+0000", 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5210000000000002e-06 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.361072+0000", 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86299840400000005 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.361201+0000", 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012967100000000001 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.361346+0000", 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000144571 2024-09-13T23:01:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.367320+0000", 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059742019999999996 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.367337+0000", 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6626e-05 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.381419+0000", 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014081978 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.381481+0000", 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1754999999999994e-05 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:56:23.381535+0000", 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4463999999999999e-05 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:209556 10.6 10:6e1699fe:::benchmark_data_smithi178_111521_object98820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:57:04.230096+0000", 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 269.57848552199999, 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57538203799999998, 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 209556 2024-09-13T23:01:33.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230096+0000", 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230096+0000", 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230097+0000", 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2619999999999999e-06 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230101+0000", 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3379999999999998e-06 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230110+0000", 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7150000000000004e-06 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230115+0000", 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0540000000000002e-06 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230162+0000", 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6623999999999999e-05 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230314+0000", 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000152953 2024-09-13T23:01:33.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230390+0000", 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5692000000000006e-05 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.236123+0000", 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057324070000000001 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.245230+0000", 2024-09-13T23:01:33.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0091071220000000005 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.805379+0000", 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56014917200000003 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.805417+0000", 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8538999999999998e-05 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.805478+0000", 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.016e-05 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:209558 10.6 10:67276ba2:::benchmark_data_smithi178_111521_object98822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:57:04.230127+0000", 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 269.57845412500001, 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57711978500000005, 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.889 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 209558 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230127+0000", 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230127+0000", 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230128+0000", 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7700000000000004e-07 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230131+0000", 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.225e-06 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230137+0000", 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.93e-06 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.230143+0000", 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9970000000000004e-06 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.232276+0000", 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021334259999999999 2024-09-13T23:01:33.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.232351+0000", 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5310999999999998e-05 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.232434+0000", 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2835000000000002e-05 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.236130+0000", 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003695261 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.246964+0000", 2024-09-13T23:01:33.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010834173000000001 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.807135+0000", 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56017154599999996 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.807200+0000", 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4366e-05 2024-09-13T23:01:33.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.807247+0000", 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7037999999999999e-05 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25538.0:209564 10.6 10:680f00bd:::benchmark_data_smithi178_111521_object98828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T22:57:04.233543+0000", 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 269.57503834900001, 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57390310600000005, 2024-09-13T23:01:33.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25538", 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/344510238", 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 209564 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.233543+0000", 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.233543+0000", 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.233544+0000", 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.327e-06 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.233548+0000", 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1760000000000003e-06 2024-09-13T23:01:33.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.233561+0000", 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2405999999999999e-05 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.233567+0000", 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7109999999999998e-06 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.234240+0000", 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067246500000000002 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.234366+0000", 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000126358 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.234492+0000", 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012573599999999999 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.236228+0000", 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017358359999999999 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.247115+0000", 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010887272999999999 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.807323+0000", 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56020777600000005 2024-09-13T23:01:33.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.807394+0000", 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1372999999999996e-05 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T22:57:04.807446+0000", 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1669000000000003e-05 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15909.0:103619 11.7 11:eba2daa8:::benchmark_data_smithi178_128208_object18589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:00:01.073506+0000", 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 92.735075500999997, 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.48720821399999997, 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15909", 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 103619 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073506+0000", 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073506+0000", 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073509+0000", 2024-09-13T23:01:33.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5209999999999998e-06 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073517+0000", 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0479999999999999e-06 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073538+0000", 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1331e-05 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073547+0000", 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7439999999999996e-06 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073618+0000", 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0555999999999993e-05 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073763+0000", 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001449 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.073907+0000", 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000144356 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.078955+0000", 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050477530000000003 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.082909+0000", 2024-09-13T23:01:33.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039546479999999998 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.560503+0000", 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.477593139 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.560619+0000", 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001165 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.560714+0000", 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4717999999999997e-05 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15909.0:103623 11.7 11:e60ebaf7:::benchmark_data_smithi178_128208_object18593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:00:01.076115+0000", 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 92.732466058, 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.48487291199999999, 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15909", 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 103623 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076115+0000", 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076115+0000", 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076117+0000", 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3319999999999999e-06 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076127+0000", 2024-09-13T23:01:33.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3689999999999996e-06 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076141+0000", 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4671999999999999e-05 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076149+0000", 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1019999999999997e-06 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076308+0000", 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015871599999999999 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076439+0000", 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013098 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.076574+0000", 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013474400000000001 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.079254+0000", 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002680095 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.085738+0000", 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0064844849999999999 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.560810+0000", 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47507181500000001 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.560911+0000", 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100729 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:01.560988+0000", 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6872999999999994e-05 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.915 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15909.0:135766 11.7 11:e5214ba9:::benchmark_data_smithi178_128208_object50736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:00:40.690049+0000", 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 53.118531615999999, 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.263619363, 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15909", 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 135766 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690049+0000", 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690049+0000", 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690051+0000", 2024-09-13T23:01:33.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9999999999999999e-06 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690056+0000", 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3529999999999997e-06 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690066+0000", 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0546e-05 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690072+0000", 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5890000000000002e-06 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690124+0000", 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1864999999999999e-05 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690208+0000", 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4633000000000001e-05 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.690280+0000", 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1973999999999999e-05 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.692100+0000", 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001819361 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.696730+0000", 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046300539999999998 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.953592+0000", 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25686261999999999 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.953632+0000", 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9208000000000001e-05 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:40.953669+0000", 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7160000000000003e-05 2024-09-13T23:01:33.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15909.0:144255 11.5 e456/450 11:a0951100:::benchmark_data_smithi178_128208_object59225:head v 456'8926, mlcod=456'8924)", 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:00:49.690089+0000", 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 44.118491931000001, 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65478650800000004, 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690089+0000", 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690089+0000", 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690091+0000", 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8470000000000001e-06 2024-09-13T23:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690101+0000", 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0207000000000001e-05 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690114+0000", 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2526e-05 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690122+0000", 2024-09-13T23:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0010000000000001e-06 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690190+0000", 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7811999999999994e-05 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.690210+0000", 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0477000000000001e-05 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.692585+0000", 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023746240000000001 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.344876+0000", 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.652291014 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15909.0:144256 11.8 e456/439 11:1cd9571c:::benchmark_data_smithi178_128208_object59226:head v 456'9041, mlcod=456'9037)", 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:00:49.691825+0000", 2024-09-13T23:01:33.926 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 44.116756168999999, 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65683694100000001, 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691825+0000", 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691825+0000", 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691826+0000", 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5489999999999999e-06 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691834+0000", 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5299999999999999e-06 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691845+0000", 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1226000000000001e-05 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691853+0000", 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691915+0000", 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2892999999999996e-05 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.691934+0000", 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8187000000000001e-05 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.348637+0000", 2024-09-13T23:01:33.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65670381499999997 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.348662+0000", 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4328000000000002e-05 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15909.0:144263 11.8 e456/439 11:1b46a59c:::benchmark_data_smithi178_128208_object59233:head v 456'9042, mlcod=456'9037)", 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:00:49.701958+0000", 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 44.106622838, 2024-09-13T23:01:33.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64671972099999997, 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.701958+0000", 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.701958+0000", 2024-09-13T23:01:33.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.701960+0000", 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.734e-06 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:49.701969+0000", 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2189999999999993e-06 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.344870+0000", 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64290107799999996 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.344879+0000", 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4130000000000006e-06 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.345113+0000", 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000234439 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.345172+0000", 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9043999999999999e-05 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.348665+0000", 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034927909999999999 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:00:50.348678+0000", 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3003000000000001e-05 2024-09-13T23:01:33.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15909.0:169484 11.7 11:e1053931:::benchmark_data_smithi178_128208_object84454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:01:20.172873+0000", 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.635708032, 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43852347699999999, 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15909", 2024-09-13T23:01:33.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 169484 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.172873+0000", 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.172873+0000", 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.172883+0000", 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0067e-05 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.172895+0000", 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.204e-05 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.172928+0000", 2024-09-13T23:01:33.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2818999999999999e-05 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.172948+0000", 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9910000000000001e-05 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.173123+0000", 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174964 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.173439+0000", 2024-09-13T23:01:33.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031594499999999998 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.173799+0000", 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036055000000000001 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.177729+0000", 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003929409 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.181005+0000", 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032763390000000001 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.611308+0000", 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43030282600000003 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.611356+0000", 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8291000000000002e-05 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.611396+0000", 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0317e-05 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15909.0:169491 11.7 11:e8a4948b:::benchmark_data_smithi178_128208_object84461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-09-13T23:01:33.942 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-09-13T23:01:20.182846+0000", 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.625734692, 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42866939300000001, 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15909", 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.178:0/3893845195", 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 169491 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-09-13T23:01:33.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.182846+0000", 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.182846+0000", 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.182850+0000", 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.163e-06 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.182859+0000", 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9369999999999999e-06 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.182885+0000", 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5236e-05 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.182899+0000", 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4019000000000001e-05 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.183050+0000", 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015095500000000001 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.183432+0000", 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000381837 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.183699+0000", 2024-09-13T23:01:33.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000267545 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.188940+0000", 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005241084 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.189259+0000", 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031904500000000001 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.611443+0000", 2024-09-13T23:01:33.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42218417400000002 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.611483+0000", 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9162999999999998e-05 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-09-13T23:01:20.611516+0000", 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3235000000000003e-05 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-09-13T23:01:33.949 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-09-13T23:01:33.949 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T23:01:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:33.950 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:33.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:34.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:34.078 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:34.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:34.207 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6 2024-09-13T23:01:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.6' 2024-09-13T23:01:34.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:34.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:32.327952+0000 osd.7 (osd.7) 452 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:01:34.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:32.330270+0000 osd.7 (osd.7) 453 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:01:34.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.001401+0000 osd.0 (osd.0) 423 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.003635+0000 mgr.y (mgr.14613) 885 : cluster [DBG] pgmap v1068: 158 pgs: 158 active+clean; 583 KiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.006318+0000 osd.0 (osd.0) 424 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.169516+0000 osd.3 (osd.3) 409 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.171226+0000 osd.3 (osd.3) 410 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: audit 2024-09-13T23:01:33.267727+0000 mon.a (mon.0) 1829 : audit [INF] from='client.25807 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.271859+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.298434+0000 osd.7 (osd.7) 454 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:01:34.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:34 smithi192 bash[24608]: cluster 2024-09-13T23:01:33.301468+0000 osd.7 (osd.7) 455 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:01:34.338 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7 2024-09-13T23:01:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.7' 2024-09-13T23:01:34.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:34.469 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:34.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T23:01:34.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:34.618 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:34.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:32.327952+0000 osd.7 (osd.7) 452 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:32.330270+0000 osd.7 (osd.7) 453 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.001401+0000 osd.0 (osd.0) 423 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.003635+0000 mgr.y (mgr.14613) 885 : cluster [DBG] pgmap v1068: 158 pgs: 158 active+clean; 583 KiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.006318+0000 osd.0 (osd.0) 424 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.169516+0000 osd.3 (osd.3) 409 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.171226+0000 osd.3 (osd.3) 410 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: audit 2024-09-13T23:01:33.267727+0000 mon.a (mon.0) 1829 : audit [INF] from='client.25807 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.271859+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.298434+0000 osd.7 (osd.7) 454 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:01:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[22840]: cluster 2024-09-13T23:01:33.301468+0000 osd.7 (osd.7) 455 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:32.327952+0000 osd.7 (osd.7) 452 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:32.330270+0000 osd.7 (osd.7) 453 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.001401+0000 osd.0 (osd.0) 423 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.003635+0000 mgr.y (mgr.14613) 885 : cluster [DBG] pgmap v1068: 158 pgs: 158 active+clean; 583 KiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.006318+0000 osd.0 (osd.0) 424 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.169516+0000 osd.3 (osd.3) 409 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.171226+0000 osd.3 (osd.3) 410 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: audit 2024-09-13T23:01:33.267727+0000 mon.a (mon.0) 1829 : audit [INF] from='client.25807 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.271859+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.298434+0000 osd.7 (osd.7) 454 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:01:34.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:34 smithi178 bash[29889]: cluster 2024-09-13T23:01:33.301468+0000 osd.7 (osd.7) 455 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:01:34.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:34.743 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5 2024-09-13T23:01:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.5' 2024-09-13T23:01:34.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:34.870 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:34.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:34.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:35.002 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3 2024-09-13T23:01:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:35.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.3' 2024-09-13T23:01:35.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:35.133 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:35.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:35.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:35.262 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2 2024-09-13T23:01:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:35.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.2' 2024-09-13T23:01:35.282 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T23:01:35.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:35.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[29889]: cluster 2024-09-13T23:01:34.120941+0000 osd.3 (osd.3) 411 : cluster [DBG] 2.e scrub starts 2024-09-13T23:01:35.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[29889]: cluster 2024-09-13T23:01:34.122721+0000 osd.3 (osd.3) 412 : cluster [DBG] 2.e scrub ok 2024-09-13T23:01:35.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[29889]: audit 2024-09-13T23:01:34.468735+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? v1:172.21.15.178:0/1992633770' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[29889]: audit 2024-09-13T23:01:34.641709+0000 mon.b (mon.1) 296 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.252593+0000 osd.7 (osd.7) 456 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.255245+0000 osd.7 (osd.7) 457 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[22840]: cluster 2024-09-13T23:01:34.120941+0000 osd.3 (osd.3) 411 : cluster [DBG] 2.e scrub starts 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[22840]: cluster 2024-09-13T23:01:34.122721+0000 osd.3 (osd.3) 412 : cluster [DBG] 2.e scrub ok 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[22840]: audit 2024-09-13T23:01:34.468735+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? v1:172.21.15.178:0/1992633770' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[22840]: audit 2024-09-13T23:01:34.641709+0000 mon.b (mon.1) 296 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.252593+0000 osd.7 (osd.7) 456 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:01:35.376 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:35 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.255245+0000 osd.7 (osd.7) 457 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:01:35.388 INFO:teuthology.orchestra.run.smithi178.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1 2024-09-13T23:01:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:35.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi178 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.1' 2024-09-13T23:01:35.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:35.521 INFO:teuthology.orchestra.run.smithi192.stderr:Error response from daemon: Cannot kill container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4 2024-09-13T23:01:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:01:35.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd.4' 2024-09-13T23:01:35.543 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T23:01:35.554 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T23:01:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:35 smithi192 bash[24608]: cluster 2024-09-13T23:01:34.120941+0000 osd.3 (osd.3) 411 : cluster [DBG] 2.e scrub starts 2024-09-13T23:01:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:35 smithi192 bash[24608]: cluster 2024-09-13T23:01:34.122721+0000 osd.3 (osd.3) 412 : cluster [DBG] 2.e scrub ok 2024-09-13T23:01:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:35 smithi192 bash[24608]: audit 2024-09-13T23:01:34.468735+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? v1:172.21.15.178:0/1992633770' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:01:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:35 smithi192 bash[24608]: audit 2024-09-13T23:01:34.641709+0000 mon.b (mon.1) 296 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:35 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.252593+0000 osd.7 (osd.7) 456 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:01:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:35 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.255245+0000 osd.7 (osd.7) 457 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:01:36.293 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is set 2024-09-13T23:01:36.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: cluster 2024-09-13T23:01:34.912716+0000 osd.0 (osd.0) 425 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: cluster 2024-09-13T23:01:34.915509+0000 osd.0 (osd.0) 426 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.004707+0000 mgr.y (mgr.14613) 886 : cluster [DBG] pgmap v1070: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: audit 2024-09-13T23:01:35.283348+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? v1:172.21.15.178:0/1992633770' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:01:36.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.287340+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-09-13T23:01:36.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: audit 2024-09-13T23:01:35.994949+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? v1:172.21.15.178:0/225923957' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:01:36.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: cluster 2024-09-13T23:01:36.234325+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T23:01:36.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:36 smithi192 bash[24608]: cluster 2024-09-13T23:01:36.239703+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T23:01:36.436 INFO:teuthology.orchestra.run.smithi178.stdout:osd.0: {} 2024-09-13T23:01:36.436 INFO:teuthology.orchestra.run.smithi178.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T23:01:36.454 INFO:teuthology.orchestra.run.smithi178.stdout:osd.1: {} 2024-09-13T23:01:36.454 INFO:teuthology.orchestra.run.smithi178.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T23:01:36.471 INFO:teuthology.orchestra.run.smithi178.stdout:osd.2: {} 2024-09-13T23:01:36.471 INFO:teuthology.orchestra.run.smithi178.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T23:01:36.488 INFO:teuthology.orchestra.run.smithi178.stdout:osd.3: {} 2024-09-13T23:01:36.489 INFO:teuthology.orchestra.run.smithi178.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T23:01:36.508 INFO:teuthology.orchestra.run.smithi178.stdout:osd.4: {} 2024-09-13T23:01:36.509 INFO:teuthology.orchestra.run.smithi178.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T23:01:36.530 INFO:teuthology.orchestra.run.smithi178.stdout:osd.5: {} 2024-09-13T23:01:36.530 INFO:teuthology.orchestra.run.smithi178.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T23:01:36.550 INFO:teuthology.orchestra.run.smithi178.stdout:osd.6: {} 2024-09-13T23:01:36.551 INFO:teuthology.orchestra.run.smithi178.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T23:01:36.571 INFO:teuthology.orchestra.run.smithi178.stdout:osd.7: {} 2024-09-13T23:01:36.572 INFO:teuthology.orchestra.run.smithi178.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: cluster 2024-09-13T23:01:34.912716+0000 osd.0 (osd.0) 425 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: cluster 2024-09-13T23:01:34.915509+0000 osd.0 (osd.0) 426 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.004707+0000 mgr.y (mgr.14613) 886 : cluster [DBG] pgmap v1070: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: audit 2024-09-13T23:01:35.283348+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? v1:172.21.15.178:0/1992633770' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.287340+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: audit 2024-09-13T23:01:35.994949+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? v1:172.21.15.178:0/225923957' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: cluster 2024-09-13T23:01:36.234325+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T23:01:36.601 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[22840]: cluster 2024-09-13T23:01:36.239703+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T23:01:36.602 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: cluster 2024-09-13T23:01:34.912716+0000 osd.0 (osd.0) 425 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:01:36.602 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: cluster 2024-09-13T23:01:34.915509+0000 osd.0 (osd.0) 426 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:01:36.602 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.004707+0000 mgr.y (mgr.14613) 886 : cluster [DBG] pgmap v1070: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:01:36.602 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: audit 2024-09-13T23:01:35.283348+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? v1:172.21.15.178:0/1992633770' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:01:36.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.287340+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-09-13T23:01:36.603 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: audit 2024-09-13T23:01:35.994949+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? v1:172.21.15.178:0/225923957' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-13T23:01:36.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: cluster 2024-09-13T23:01:36.234325+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.f deep-scrub starts 2024-09-13T23:01:36.604 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:36 smithi178 bash[29889]: cluster 2024-09-13T23:01:36.239703+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.f deep-scrub ok 2024-09-13T23:01:37.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.453735+0000 osd.5 (osd.5) 481 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.457129+0000 osd.5 (osd.5) 482 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.583415+0000 osd.6 (osd.6) 326 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.587014+0000 osd.6 (osd.6) 327 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.663476+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.665971+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.902092+0000 osd.1 (osd.1) 434 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.903962+0000 osd.1 (osd.1) 435 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.960171+0000 osd.0 (osd.0) 427 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:35.962507+0000 osd.0 (osd.0) 428 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: audit 2024-09-13T23:01:36.049217+0000 mgr.y (mgr.14613) 887 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:37.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:36.284322+0000 mon.a (mon.0) 1835 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: audit 2024-09-13T23:01:36.293758+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? v1:172.21.15.178:0/225923957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: cluster 2024-09-13T23:01:36.300270+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: audit 2024-09-13T23:01:36.420287+0000 mon.a (mon.0) 1838 : audit [DBG] from='client.? v1:172.21.15.178:0/1512555126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: audit 2024-09-13T23:01:36.453823+0000 mon.b (mon.1) 297 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[29889]: audit 2024-09-13T23:01:36.454238+0000 mon.a (mon.0) 1839 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.453735+0000 osd.5 (osd.5) 481 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.457129+0000 osd.5 (osd.5) 482 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.583415+0000 osd.6 (osd.6) 326 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.587014+0000 osd.6 (osd.6) 327 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.663476+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:01:37.405 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.665971+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.902092+0000 osd.1 (osd.1) 434 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.903962+0000 osd.1 (osd.1) 435 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.960171+0000 osd.0 (osd.0) 427 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:35.962507+0000 osd.0 (osd.0) 428 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: audit 2024-09-13T23:01:36.049217+0000 mgr.y (mgr.14613) 887 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:36.284322+0000 mon.a (mon.0) 1835 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: audit 2024-09-13T23:01:36.293758+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? v1:172.21.15.178:0/225923957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: cluster 2024-09-13T23:01:36.300270+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: audit 2024-09-13T23:01:36.420287+0000 mon.a (mon.0) 1838 : audit [DBG] from='client.? v1:172.21.15.178:0/1512555126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: audit 2024-09-13T23:01:36.453823+0000 mon.b (mon.1) 297 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-09-13T23:01:37.406 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:37 smithi178 bash[22840]: audit 2024-09-13T23:01:36.454238+0000 mon.a (mon.0) 1839 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-09-13T23:01:37.550 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T23:01:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.453735+0000 osd.5 (osd.5) 481 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:01:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.457129+0000 osd.5 (osd.5) 482 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:01:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.583415+0000 osd.6 (osd.6) 326 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:01:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.587014+0000 osd.6 (osd.6) 327 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.663476+0000 osd.2 (osd.2) 323 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.665971+0000 osd.2 (osd.2) 324 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.902092+0000 osd.1 (osd.1) 434 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.903962+0000 osd.1 (osd.1) 435 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.960171+0000 osd.0 (osd.0) 427 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:35.962507+0000 osd.0 (osd.0) 428 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: audit 2024-09-13T23:01:36.049217+0000 mgr.y (mgr.14613) 887 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:36.284322+0000 mon.a (mon.0) 1835 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: audit 2024-09-13T23:01:36.293758+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? v1:172.21.15.178:0/225923957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: cluster 2024-09-13T23:01:36.300270+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: audit 2024-09-13T23:01:36.420287+0000 mon.a (mon.0) 1838 : audit [DBG] from='client.? v1:172.21.15.178:0/1512555126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: audit 2024-09-13T23:01:36.453823+0000 mon.b (mon.1) 297 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-09-13T23:01:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:37 smithi192 bash[24608]: audit 2024-09-13T23:01:36.454238+0000 mon.a (mon.0) 1839 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-09-13T23:01:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:38 smithi192 bash[24608]: cluster 2024-09-13T23:01:36.945971+0000 osd.1 (osd.1) 436 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:01:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:38 smithi192 bash[24608]: cluster 2024-09-13T23:01:36.948679+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:01:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:38 smithi192 bash[24608]: cluster 2024-09-13T23:01:37.005599+0000 mgr.y (mgr.14613) 888 : cluster [DBG] pgmap v1073: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T23:01:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:38 smithi192 bash[24608]: audit 2024-09-13T23:01:37.297326+0000 mon.a (mon.0) 1840 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]': finished 2024-09-13T23:01:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:38 smithi192 bash[24608]: cluster 2024-09-13T23:01:37.303167+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-09-13T23:01:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[22840]: cluster 2024-09-13T23:01:36.945971+0000 osd.1 (osd.1) 436 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:01:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[22840]: cluster 2024-09-13T23:01:36.948679+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:01:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[22840]: cluster 2024-09-13T23:01:37.005599+0000 mgr.y (mgr.14613) 888 : cluster [DBG] pgmap v1073: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[22840]: audit 2024-09-13T23:01:37.297326+0000 mon.a (mon.0) 1840 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]': finished 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[22840]: cluster 2024-09-13T23:01:37.303167+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[29889]: cluster 2024-09-13T23:01:36.945971+0000 osd.1 (osd.1) 436 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[29889]: cluster 2024-09-13T23:01:36.948679+0000 osd.1 (osd.1) 437 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[29889]: cluster 2024-09-13T23:01:37.005599+0000 mgr.y (mgr.14613) 888 : cluster [DBG] pgmap v1073: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[29889]: audit 2024-09-13T23:01:37.297326+0000 mon.a (mon.0) 1840 : audit [INF] from='mgr.14613 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]': finished 2024-09-13T23:01:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:38 smithi178 bash[29889]: cluster 2024-09-13T23:01:37.303167+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-09-13T23:01:39.357 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:39 smithi192 bash[24608]: cluster 2024-09-13T23:01:38.315094+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-09-13T23:01:39.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:39 smithi178 bash[29889]: cluster 2024-09-13T23:01:38.315094+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-09-13T23:01:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:39 smithi178 bash[22840]: cluster 2024-09-13T23:01:38.315094+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-09-13T23:01:40.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:01:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:01:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:01:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:40 smithi192 bash[24608]: cluster 2024-09-13T23:01:39.006616+0000 mgr.y (mgr.14613) 889 : cluster [DBG] pgmap v1076: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-13T23:01:40.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:40 smithi178 bash[29889]: cluster 2024-09-13T23:01:39.006616+0000 mgr.y (mgr.14613) 889 : cluster [DBG] pgmap v1076: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-13T23:01:40.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:40 smithi178 bash[22840]: cluster 2024-09-13T23:01:39.006616+0000 mgr.y (mgr.14613) 889 : cluster [DBG] pgmap v1076: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-13T23:01:40.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T23:01:40.887 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:01:40.887 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:01:41.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T23:01:42.353 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:42.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:42 smithi192 bash[24608]: cluster 2024-09-13T23:01:41.008093+0000 mgr.y (mgr.14613) 890 : cluster [DBG] pgmap v1077: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 894 B/s rd, 0 op/s 2024-09-13T23:01:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:42 smithi178 bash[29889]: cluster 2024-09-13T23:01:41.008093+0000 mgr.y (mgr.14613) 890 : cluster [DBG] pgmap v1077: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 894 B/s rd, 0 op/s 2024-09-13T23:01:42.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:42 smithi178 bash[22840]: cluster 2024-09-13T23:01:41.008093+0000 mgr.y (mgr.14613) 890 : cluster [DBG] pgmap v1077: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 894 B/s rd, 0 op/s 2024-09-13T23:01:44.335 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[29889]: cluster 2024-09-13T23:01:43.008992+0000 mgr.y (mgr.14613) 891 : cluster [DBG] pgmap v1078: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 763 B/s rd, 0 op/s 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[29889]: cluster 2024-09-13T23:01:43.193158+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[29889]: audit 2024-09-13T23:01:44.066788+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.178:0/3888225024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[29889]: audit 2024-09-13T23:01:44.067363+0000 mon.a (mon.0) 1844 : audit [INF] from='client.25955 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[22840]: cluster 2024-09-13T23:01:43.008992+0000 mgr.y (mgr.14613) 891 : cluster [DBG] pgmap v1078: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 763 B/s rd, 0 op/s 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[22840]: cluster 2024-09-13T23:01:43.193158+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[22840]: audit 2024-09-13T23:01:44.066788+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.178:0/3888225024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:01:44.559 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:44 smithi178 bash[22840]: audit 2024-09-13T23:01:44.067363+0000 mon.a (mon.0) 1844 : audit [INF] from='client.25955 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:01:44.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:44 smithi192 bash[24608]: cluster 2024-09-13T23:01:43.008992+0000 mgr.y (mgr.14613) 891 : cluster [DBG] pgmap v1078: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 763 B/s rd, 0 op/s 2024-09-13T23:01:44.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:44 smithi192 bash[24608]: cluster 2024-09-13T23:01:43.193158+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-09-13T23:01:44.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:44 smithi192 bash[24608]: audit 2024-09-13T23:01:44.066788+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.178:0/3888225024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:01:44.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:44 smithi192 bash[24608]: audit 2024-09-13T23:01:44.067363+0000 mon.a (mon.0) 1844 : audit [INF] from='client.25955 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:01:45.184 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T23:01:45.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T23:01:45.581 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:45 smithi192 bash[24608]: audit 2024-09-13T23:01:44.328058+0000 mon.a (mon.0) 1845 : audit [INF] from='client.25955 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:01:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:45 smithi192 bash[24608]: cluster 2024-09-13T23:01:44.336270+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-09-13T23:01:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:45 smithi178 bash[29889]: audit 2024-09-13T23:01:44.328058+0000 mon.a (mon.0) 1845 : audit [INF] from='client.25955 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:01:45.674 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:45 smithi178 bash[29889]: cluster 2024-09-13T23:01:44.336270+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-09-13T23:01:45.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:45 smithi178 bash[22840]: audit 2024-09-13T23:01:44.328058+0000 mon.a (mon.0) 1845 : audit [INF] from='client.25955 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:01:45.675 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:45 smithi178 bash[22840]: cluster 2024-09-13T23:01:44.336270+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-09-13T23:01:46.343 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:01:46 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:01:46.365 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:46 smithi192 bash[24608]: cluster 2024-09-13T23:01:45.010059+0000 mgr.y (mgr.14613) 892 : cluster [DBG] pgmap v1081: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:01:46.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:46 smithi178 bash[22840]: cluster 2024-09-13T23:01:45.010059+0000 mgr.y (mgr.14613) 892 : cluster [DBG] pgmap v1081: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:01:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:46 smithi178 bash[29889]: cluster 2024-09-13T23:01:45.010059+0000 mgr.y (mgr.14613) 892 : cluster [DBG] pgmap v1081: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:01:47.604 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:47 smithi192 bash[24608]: audit 2024-09-13T23:01:46.060561+0000 mgr.y (mgr.14613) 893 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:47.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:47 smithi178 bash[22840]: audit 2024-09-13T23:01:46.060561+0000 mgr.y (mgr.14613) 893 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:47.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:47 smithi178 bash[29889]: audit 2024-09-13T23:01:46.060561+0000 mgr.y (mgr.14613) 893 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:47.742 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:01:47.742 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:01:48.191 INFO:teuthology.orchestra.run.smithi178.stderr:noscrub is unset 2024-09-13T23:01:48.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T23:01:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:48 smithi192 bash[24608]: cluster 2024-09-13T23:01:47.010973+0000 mgr.y (mgr.14613) 894 : cluster [DBG] pgmap v1082: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:01:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:48 smithi192 bash[24608]: audit 2024-09-13T23:01:47.369495+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? v1:172.21.15.178:0/4146031611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:48 smithi192 bash[24608]: cluster 2024-09-13T23:01:48.189932+0000 mon.a (mon.0) 1848 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-13T23:01:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:48 smithi192 bash[24608]: cluster 2024-09-13T23:01:48.189973+0000 mon.a (mon.0) 1849 : cluster [INF] Cluster is now healthy 2024-09-13T23:01:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:48 smithi192 bash[24608]: audit 2024-09-13T23:01:48.192516+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? v1:172.21.15.178:0/4146031611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T23:01:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:48 smithi192 bash[24608]: cluster 2024-09-13T23:01:48.199781+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-09-13T23:01:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[29889]: cluster 2024-09-13T23:01:47.010973+0000 mgr.y (mgr.14613) 894 : cluster [DBG] pgmap v1082: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:01:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[29889]: audit 2024-09-13T23:01:47.369495+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? v1:172.21.15.178:0/4146031611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[29889]: cluster 2024-09-13T23:01:48.189932+0000 mon.a (mon.0) 1848 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[29889]: cluster 2024-09-13T23:01:48.189973+0000 mon.a (mon.0) 1849 : cluster [INF] Cluster is now healthy 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[29889]: audit 2024-09-13T23:01:48.192516+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? v1:172.21.15.178:0/4146031611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[29889]: cluster 2024-09-13T23:01:48.199781+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[22840]: cluster 2024-09-13T23:01:47.010973+0000 mgr.y (mgr.14613) 894 : cluster [DBG] pgmap v1082: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[22840]: audit 2024-09-13T23:01:47.369495+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? v1:172.21.15.178:0/4146031611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[22840]: cluster 2024-09-13T23:01:48.189932+0000 mon.a (mon.0) 1848 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[22840]: cluster 2024-09-13T23:01:48.189973+0000 mon.a (mon.0) 1849 : cluster [INF] Cluster is now healthy 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[22840]: audit 2024-09-13T23:01:48.192516+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? v1:172.21.15.178:0/4146031611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T23:01:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:48 smithi178 bash[22840]: cluster 2024-09-13T23:01:48.199781+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-09-13T23:01:49.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T23:01:50.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:01:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:01:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:01:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:50 smithi192 bash[24608]: cluster 2024-09-13T23:01:48.864969+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:01:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:50 smithi192 bash[24608]: cluster 2024-09-13T23:01:48.867307+0000 osd.1 (osd.1) 439 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:01:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:50 smithi192 bash[24608]: cluster 2024-09-13T23:01:49.011855+0000 mgr.y (mgr.14613) 895 : cluster [DBG] pgmap v1084: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:01:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:50 smithi192 bash[24608]: cluster 2024-09-13T23:01:49.634978+0000 osd.5 (osd.5) 483 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:01:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:50 smithi192 bash[24608]: cluster 2024-09-13T23:01:49.636892+0000 osd.5 (osd.5) 484 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:01:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:50 smithi192 bash[24608]: audit 2024-09-13T23:01:49.670446+0000 mon.b (mon.1) 298 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[29889]: cluster 2024-09-13T23:01:48.864969+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:01:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[29889]: cluster 2024-09-13T23:01:48.867307+0000 osd.1 (osd.1) 439 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:01:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[29889]: cluster 2024-09-13T23:01:49.011855+0000 mgr.y (mgr.14613) 895 : cluster [DBG] pgmap v1084: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[29889]: cluster 2024-09-13T23:01:49.634978+0000 osd.5 (osd.5) 483 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[29889]: cluster 2024-09-13T23:01:49.636892+0000 osd.5 (osd.5) 484 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[29889]: audit 2024-09-13T23:01:49.670446+0000 mon.b (mon.1) 298 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[22840]: cluster 2024-09-13T23:01:48.864969+0000 osd.1 (osd.1) 438 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[22840]: cluster 2024-09-13T23:01:48.867307+0000 osd.1 (osd.1) 439 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[22840]: cluster 2024-09-13T23:01:49.011855+0000 mgr.y (mgr.14613) 895 : cluster [DBG] pgmap v1084: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[22840]: cluster 2024-09-13T23:01:49.634978+0000 osd.5 (osd.5) 483 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[22840]: cluster 2024-09-13T23:01:49.636892+0000 osd.5 (osd.5) 484 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:01:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:50 smithi178 bash[22840]: audit 2024-09-13T23:01:49.670446+0000 mon.b (mon.1) 298 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:01:50.724 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:51.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:51 smithi192 bash[24608]: cluster 2024-09-13T23:01:50.627400+0000 osd.5 (osd.5) 485 : cluster [DBG] 1.a scrub starts 2024-09-13T23:01:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:51 smithi192 bash[24608]: cluster 2024-09-13T23:01:50.630376+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.a scrub ok 2024-09-13T23:01:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:51 smithi192 bash[24608]: audit 2024-09-13T23:01:51.074828+0000 mon.b (mon.1) 299 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:01:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:51 smithi178 bash[29889]: cluster 2024-09-13T23:01:50.627400+0000 osd.5 (osd.5) 485 : cluster [DBG] 1.a scrub starts 2024-09-13T23:01:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:51 smithi178 bash[29889]: cluster 2024-09-13T23:01:50.630376+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.a scrub ok 2024-09-13T23:01:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:51 smithi178 bash[29889]: audit 2024-09-13T23:01:51.074828+0000 mon.b (mon.1) 299 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:01:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:51 smithi178 bash[22840]: cluster 2024-09-13T23:01:50.627400+0000 osd.5 (osd.5) 485 : cluster [DBG] 1.a scrub starts 2024-09-13T23:01:51.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:51 smithi178 bash[22840]: cluster 2024-09-13T23:01:50.630376+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.a scrub ok 2024-09-13T23:01:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:51 smithi178 bash[22840]: audit 2024-09-13T23:01:51.074828+0000 mon.b (mon.1) 299 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:50.947571+0000 osd.1 (osd.1) 440 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:50.952981+0000 osd.1 (osd.1) 441 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.013300+0000 mgr.y (mgr.14613) 896 : cluster [DBG] pgmap v1085: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.660298+0000 osd.5 (osd.5) 487 : cluster [DBG] 6.a scrub starts 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.662244+0000 osd.5 (osd.5) 488 : cluster [DBG] 6.a scrub ok 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: audit 2024-09-13T23:01:51.844357+0000 mon.b (mon.1) 300 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:01:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: audit 2024-09-13T23:01:51.845809+0000 mon.b (mon.1) 301 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:01:52.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: audit 2024-09-13T23:01:51.853888+0000 mon.a (mon.0) 1852 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:01:52.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:52.079084+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.d scrub starts 2024-09-13T23:01:52.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:52 smithi192 bash[24608]: cluster 2024-09-13T23:01:52.082085+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.d scrub ok 2024-09-13T23:01:52.681 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T23:01:52.684 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T23:01:52.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:50.947571+0000 osd.1 (osd.1) 440 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:01:52.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:50.952981+0000 osd.1 (osd.1) 441 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:01:52.685 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.013300+0000 mgr.y (mgr.14613) 896 : cluster [DBG] pgmap v1085: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.660298+0000 osd.5 (osd.5) 487 : cluster [DBG] 6.a scrub starts 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.662244+0000 osd.5 (osd.5) 488 : cluster [DBG] 6.a scrub ok 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: audit 2024-09-13T23:01:51.844357+0000 mon.b (mon.1) 300 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: audit 2024-09-13T23:01:51.845809+0000 mon.b (mon.1) 301 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: audit 2024-09-13T23:01:51.853888+0000 mon.a (mon.0) 1852 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:52.079084+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.d scrub starts 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[29889]: cluster 2024-09-13T23:01:52.082085+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.d scrub ok 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:50.947571+0000 osd.1 (osd.1) 440 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:50.952981+0000 osd.1 (osd.1) 441 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.013300+0000 mgr.y (mgr.14613) 896 : cluster [DBG] pgmap v1085: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T23:01:52.686 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.660298+0000 osd.5 (osd.5) 487 : cluster [DBG] 6.a scrub starts 2024-09-13T23:01:52.687 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.662244+0000 osd.5 (osd.5) 488 : cluster [DBG] 6.a scrub ok 2024-09-13T23:01:52.687 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: audit 2024-09-13T23:01:51.844357+0000 mon.b (mon.1) 300 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:01:52.687 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: audit 2024-09-13T23:01:51.845809+0000 mon.b (mon.1) 301 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:01:52.687 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: audit 2024-09-13T23:01:51.853888+0000 mon.a (mon.0) 1852 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:01:52.687 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:52.079084+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.d scrub starts 2024-09-13T23:01:52.687 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:52 smithi178 bash[22840]: cluster 2024-09-13T23:01:52.082085+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.d scrub ok 2024-09-13T23:01:53.207 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:01:53.534 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":1085,"stamp":"2024-09-13T23:01:51.012159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3012,"num_read_kb":2764,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":9059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2629,"ondisk_log_size":2629,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":430,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5478364,"kb_used_data":195528,"kb_used_omap":10,"kb_used_meta":5282677,"kb_avail":650737700,"statfs":{"total":671965249536,"available":666355404800,"internally_reserved":0,"allocated":200220672,"data_stored":196930638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5409461355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.820749"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":879,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196678+0000","last_change":"2024-09-13T22:58:42.763562+0000","last_active":"2024-09-13T23:01:48.196678+0000","last_peered":"2024-09-13T23:01:48.196678+0000","last_clean":"2024-09-13T23:01:48.196678+0000","last_became_active":"2024-09-13T22:58:42.763279+0000","last_became_peered":"2024-09-13T22:58:42.763279+0000","last_unstale":"2024-09-13T23:01:48.196678+0000","last_undegraded":"2024-09-13T23:01:48.196678+0000","last_fullsized":"2024-09-13T23:01:48.196678+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.827330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.827330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093516+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":933,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.334202+0000","last_change":"2024-09-13T22:58:42.740974+0000","last_active":"2024-09-13T23:01:44.334202+0000","last_peered":"2024-09-13T23:01:44.334202+0000","last_clean":"2024-09-13T23:01:44.334202+0000","last_became_active":"2024-09-13T22:58:42.740638+0000","last_became_peered":"2024-09-13T22:58:42.740638+0000","last_unstale":"2024-09-13T23:01:44.334202+0000","last_undegraded":"2024-09-13T23:01:44.334202+0000","last_fullsized":"2024-09-13T23:01:44.334202+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:47.259089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:22.232806+0000","last_clean_scrub_stamp":"2024-09-13T22:57:47.259089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":972,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197592+0000","last_change":"2024-09-13T22:58:42.783796+0000","last_active":"2024-09-13T23:01:48.197592+0000","last_peered":"2024-09-13T23:01:48.197592+0000","last_clean":"2024-09-13T23:01:48.197592+0000","last_became_active":"2024-09-13T22:58:42.747843+0000","last_became_peered":"2024-09-13T22:58:42.747843+0000","last_unstale":"2024-09-13T23:01:48.197592+0000","last_undegraded":"2024-09-13T23:01:48.197592+0000","last_fullsized":"2024-09-13T23:01:48.197592+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'25","last_scrub_stamp":"2024-09-13T22:57:46.573001+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.573001+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":146,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.460793+0000","last_change":"2024-09-13T23:01:22.578614+0000","last_active":"2024-09-13T23:01:44.460793+0000","last_peered":"2024-09-13T23:01:44.460793+0000","last_clean":"2024-09-13T23:01:44.460793+0000","last_became_active":"2024-09-13T22:58:42.745341+0000","last_became_peered":"2024-09-13T22:58:42.745341+0000","last_unstale":"2024-09-13T23:01:44.460793+0000","last_undegraded":"2024-09-13T23:01:44.460793+0000","last_fullsized":"2024-09-13T23:01:44.460793+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:22.578421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.578421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:45.233250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":882,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197614+0000","last_change":"2024-09-13T22:56:28.054932+0000","last_active":"2024-09-13T23:01:48.197614+0000","last_peered":"2024-09-13T23:01:48.197614+0000","last_clean":"2024-09-13T23:01:48.197614+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T23:01:48.197614+0000","last_undegraded":"2024-09-13T23:01:48.197614+0000","last_fullsized":"2024-09-13T23:01:48.197614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:28.051743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:56:28.051743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":949,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333692+0000","last_change":"2024-09-13T23:01:22.902178+0000","last_active":"2024-09-13T23:01:44.333692+0000","last_peered":"2024-09-13T23:01:44.333692+0000","last_clean":"2024-09-13T23:01:44.333692+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T23:01:44.333692+0000","last_undegraded":"2024-09-13T23:01:44.333692+0000","last_fullsized":"2024-09-13T23:01:44.333692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:01:22.901958+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.901958+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:39.278580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":888,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501058+0000","last_change":"2024-09-13T23:01:20.852046+0000","last_active":"2024-09-13T23:01:48.501058+0000","last_peered":"2024-09-13T23:01:48.501058+0000","last_clean":"2024-09-13T23:01:48.501058+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T23:01:48.501058+0000","last_undegraded":"2024-09-13T23:01:48.501058+0000","last_fullsized":"2024-09-13T23:01:48.501058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:20.851978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T23:01:20.851978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.169703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":406,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.576915+0000","last_change":"2024-09-13T23:01:24.445111+0000","last_active":"2024-09-13T23:01:48.576915+0000","last_peered":"2024-09-13T23:01:48.576915+0000","last_clean":"2024-09-13T23:01:48.576915+0000","last_became_active":"2024-09-13T22:55:38.143783+0000","last_became_peered":"2024-09-13T22:55:38.143783+0000","last_unstale":"2024-09-13T23:01:48.576915+0000","last_undegraded":"2024-09-13T23:01:48.576915+0000","last_fullsized":"2024-09-13T23:01:48.576915+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:24.444998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T23:01:24.444998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:27.272769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":898,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.251012+0000","last_change":"2024-09-13T23:01:12.294658+0000","last_active":"2024-09-13T23:01:45.251012+0000","last_peered":"2024-09-13T23:01:45.251012+0000","last_clean":"2024-09-13T23:01:45.251012+0000","last_became_active":"2024-09-13T22:58:42.770668+0000","last_became_peered":"2024-09-13T22:58:42.770668+0000","last_unstale":"2024-09-13T23:01:45.251012+0000","last_undegraded":"2024-09-13T23:01:45.251012+0000","last_fullsized":"2024-09-13T23:01:45.251012+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:12.294461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:12.294461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.302281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":1078,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577497+0000","last_change":"2024-09-13T23:01:09.461488+0000","last_active":"2024-09-13T23:01:48.577497+0000","last_peered":"2024-09-13T23:01:48.577497+0000","last_clean":"2024-09-13T23:01:48.577497+0000","last_became_active":"2024-09-13T22:58:42.728658+0000","last_became_peered":"2024-09-13T22:58:42.728658+0000","last_unstale":"2024-09-13T23:01:48.577497+0000","last_undegraded":"2024-09-13T23:01:48.577497+0000","last_fullsized":"2024-09-13T23:01:48.577497+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:01:09.461410+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T23:01:09.461410+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:20.730973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":888,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248460+0000","last_change":"2024-09-13T23:01:10.320782+0000","last_active":"2024-09-13T23:01:45.248460+0000","last_peered":"2024-09-13T23:01:45.248460+0000","last_clean":"2024-09-13T23:01:45.248460+0000","last_became_active":"2024-09-13T22:55:38.151599+0000","last_became_peered":"2024-09-13T22:55:38.151599+0000","last_unstale":"2024-09-13T23:01:45.248460+0000","last_undegraded":"2024-09-13T23:01:45.248460+0000","last_fullsized":"2024-09-13T23:01:45.248460+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:10.320675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T23:01:10.320675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.237394+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":884,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577293+0000","last_change":"2024-09-13T22:58:14.162918+0000","last_active":"2024-09-13T23:01:48.577293+0000","last_peered":"2024-09-13T23:01:48.577293+0000","last_clean":"2024-09-13T23:01:48.577293+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T23:01:48.577293+0000","last_undegraded":"2024-09-13T23:01:48.577293+0000","last_fullsized":"2024-09-13T23:01:48.577293+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:14.162730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.162730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":886,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249593+0000","last_change":"2024-09-13T23:01:08.231928+0000","last_active":"2024-09-13T23:01:45.249593+0000","last_peered":"2024-09-13T23:01:45.249593+0000","last_clean":"2024-09-13T23:01:45.249593+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T23:01:45.249593+0000","last_undegraded":"2024-09-13T23:01:45.249593+0000","last_fullsized":"2024-09-13T23:01:45.249593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:08.231846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:08.231846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:18.625381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"397'24","reported_seq":458,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249549+0000","last_change":"2024-09-13T22:58:42.802871+0000","last_active":"2024-09-13T23:01:45.249549+0000","last_peered":"2024-09-13T23:01:45.249549+0000","last_clean":"2024-09-13T23:01:45.249549+0000","last_became_active":"2024-09-13T22:58:42.768934+0000","last_became_peered":"2024-09-13T22:58:42.768934+0000","last_unstale":"2024-09-13T23:01:45.249549+0000","last_undegraded":"2024-09-13T23:01:45.249549+0000","last_fullsized":"2024-09-13T23:01:45.249549+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T22:58:19.363461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:19.363461+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":888,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577358+0000","last_change":"2024-09-13T22:58:09.016326+0000","last_active":"2024-09-13T23:01:48.577358+0000","last_peered":"2024-09-13T23:01:48.577358+0000","last_clean":"2024-09-13T23:01:48.577358+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T23:01:48.577358+0000","last_undegraded":"2024-09-13T23:01:48.577358+0000","last_fullsized":"2024-09-13T23:01:48.577358+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:09.016219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.016219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":891,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.331872+0000","last_change":"2024-09-13T23:01:27.921506+0000","last_active":"2024-09-13T23:01:44.331872+0000","last_peered":"2024-09-13T23:01:44.331872+0000","last_clean":"2024-09-13T23:01:44.331872+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T23:01:44.331872+0000","last_undegraded":"2024-09-13T23:01:44.331872+0000","last_fullsized":"2024-09-13T23:01:44.331872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:27.921318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.921318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.142579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":951,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577382+0000","last_change":"2024-09-13T23:01:06.499430+0000","last_active":"2024-09-13T23:01:48.577382+0000","last_peered":"2024-09-13T23:01:48.577382+0000","last_clean":"2024-09-13T23:01:48.577382+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T23:01:48.577382+0000","last_undegraded":"2024-09-13T23:01:48.577382+0000","last_fullsized":"2024-09-13T23:01:48.577382+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:06.499317+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:06.499317+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:35.638341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":158,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.461164+0000","last_change":"2024-09-13T22:58:42.778291+0000","last_active":"2024-09-13T23:01:44.461164+0000","last_peered":"2024-09-13T23:01:44.461164+0000","last_clean":"2024-09-13T23:01:44.461164+0000","last_became_active":"2024-09-13T22:58:42.749141+0000","last_became_peered":"2024-09-13T22:58:42.749141+0000","last_unstale":"2024-09-13T23:01:44.461164+0000","last_undegraded":"2024-09-13T23:01:44.461164+0000","last_fullsized":"2024-09-13T23:01:44.461164+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'44","last_scrub_stamp":"2024-09-13T22:57:48.143715+0000","last_deep_scrub":"228'44","last_deep_scrub_stamp":"2024-09-13T22:54:34.302937+0000","last_clean_scrub_stamp":"2024-09-13T22:57:48.143715+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":897,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249573+0000","last_change":"2024-09-13T22:58:16.344650+0000","last_active":"2024-09-13T23:01:45.249573+0000","last_peered":"2024-09-13T23:01:45.249573+0000","last_clean":"2024-09-13T23:01:45.249573+0000","last_became_active":"2024-09-13T22:55:38.140796+0000","last_became_peered":"2024-09-13T22:55:38.140796+0000","last_unstale":"2024-09-13T23:01:45.249573+0000","last_undegraded":"2024-09-13T23:01:45.249573+0000","last_fullsized":"2024-09-13T23:01:45.249573+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:16.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:58:16.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":919,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502055+0000","last_change":"2024-09-13T22:58:42.743933+0000","last_active":"2024-09-13T23:01:48.502055+0000","last_peered":"2024-09-13T23:01:48.502055+0000","last_clean":"2024-09-13T23:01:48.502055+0000","last_became_active":"2024-09-13T22:58:42.743564+0000","last_became_peered":"2024-09-13T22:58:42.743564+0000","last_unstale":"2024-09-13T23:01:48.502055+0000","last_undegraded":"2024-09-13T23:01:48.502055+0000","last_fullsized":"2024-09-13T23:01:48.502055+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.123464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":814,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197823+0000","last_change":"2024-09-13T22:58:42.748052+0000","last_active":"2024-09-13T23:01:48.197823+0000","last_peered":"2024-09-13T23:01:48.197823+0000","last_clean":"2024-09-13T23:01:48.197823+0000","last_became_active":"2024-09-13T22:58:42.747455+0000","last_became_peered":"2024-09-13T22:58:42.747455+0000","last_unstale":"2024-09-13T23:01:48.197823+0000","last_undegraded":"2024-09-13T23:01:48.197823+0000","last_fullsized":"2024-09-13T23:01:48.197823+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:58.481749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:58.481749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093087+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":1158,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248877+0000","last_change":"2024-09-13T22:58:02.221879+0000","last_active":"2024-09-13T23:01:45.248877+0000","last_peered":"2024-09-13T23:01:45.248877+0000","last_clean":"2024-09-13T23:01:45.248877+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T23:01:45.248877+0000","last_undegraded":"2024-09-13T23:01:45.248877+0000","last_fullsized":"2024-09-13T23:01:45.248877+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:02.221654+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.221654+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":970,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196944+0000","last_change":"2024-09-13T23:01:18.133842+0000","last_active":"2024-09-13T23:01:48.196944+0000","last_peered":"2024-09-13T23:01:48.196944+0000","last_clean":"2024-09-13T23:01:48.196944+0000","last_became_active":"2024-09-13T22:55:38.095029+0000","last_became_peered":"2024-09-13T22:55:38.095029+0000","last_unstale":"2024-09-13T23:01:48.196944+0000","last_undegraded":"2024-09-13T23:01:48.196944+0000","last_fullsized":"2024-09-13T23:01:48.196944+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:01:18.132748+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T23:01:18.132748+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:34.930897+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501835+0000","last_change":"2024-09-13T23:01:00.694314+0000","last_active":"2024-09-13T23:01:48.501835+0000","last_peered":"2024-09-13T23:01:48.501835+0000","last_clean":"2024-09-13T23:01:48.501835+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T23:01:48.501835+0000","last_undegraded":"2024-09-13T23:01:48.501835+0000","last_fullsized":"2024-09-13T23:01:48.501835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:00.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:00.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:25.230076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":643,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.250317+0000","last_change":"2024-09-13T22:57:06.962170+0000","last_active":"2024-09-13T23:01:45.250317+0000","last_peered":"2024-09-13T23:01:45.250317+0000","last_clean":"2024-09-13T23:01:45.250317+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T23:01:45.250317+0000","last_undegraded":"2024-09-13T23:01:45.250317+0000","last_fullsized":"2024-09-13T23:01:45.250317+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'12","last_scrub_stamp":"2024-09-13T22:57:06.962075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:06.962075+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:51.093865+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":406,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502429+0000","last_change":"2024-09-13T23:01:17.846052+0000","last_active":"2024-09-13T23:01:48.502429+0000","last_peered":"2024-09-13T23:01:48.502429+0000","last_clean":"2024-09-13T23:01:48.502429+0000","last_became_active":"2024-09-13T22:55:38.103505+0000","last_became_peered":"2024-09-13T22:55:38.103505+0000","last_unstale":"2024-09-13T23:01:48.502429+0000","last_undegraded":"2024-09-13T23:01:48.502429+0000","last_fullsized":"2024-09-13T23:01:48.502429+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:17.845954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:17.845954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:25.741237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":969,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.576913+0000","last_change":"2024-09-13T22:58:45.828688+0000","last_active":"2024-09-13T23:01:48.576913+0000","last_peered":"2024-09-13T23:01:48.576913+0000","last_clean":"2024-09-13T23:01:48.576913+0000","last_became_active":"2024-09-13T22:58:45.828166+0000","last_became_peered":"2024-09-13T22:58:45.828166+0000","last_unstale":"2024-09-13T23:01:48.576913+0000","last_undegraded":"2024-09-13T23:01:48.576913+0000","last_fullsized":"2024-09-13T23:01:48.576913+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:12.165912+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.165912+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609224+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"397'39","reported_seq":938,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196384+0000","last_change":"2024-09-13T22:55:29.816311+0000","last_active":"2024-09-13T23:01:48.196384+0000","last_peered":"2024-09-13T23:01:48.196384+0000","last_clean":"2024-09-13T23:01:48.196384+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T23:01:48.196384+0000","last_undegraded":"2024-09-13T23:01:48.196384+0000","last_fullsized":"2024-09-13T23:01:48.196384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'27","last_scrub_stamp":"2024-09-13T22:55:29.816212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:55:29.816212+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.092990+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":906,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502456+0000","last_change":"2024-09-13T23:01:12.899149+0000","last_active":"2024-09-13T23:01:48.502456+0000","last_peered":"2024-09-13T23:01:48.502456+0000","last_clean":"2024-09-13T23:01:48.502456+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T23:01:48.502456+0000","last_undegraded":"2024-09-13T23:01:48.502456+0000","last_fullsized":"2024-09-13T23:01:48.502456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:12.899007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:12.899007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.957521+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"351'15","reported_seq":614,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578084+0000","last_change":"2024-09-13T23:01:18.436735+0000","last_active":"2024-09-13T23:01:48.578084+0000","last_peered":"2024-09-13T23:01:48.578084+0000","last_clean":"2024-09-13T23:01:48.578084+0000","last_became_active":"2024-09-13T22:55:38.106061+0000","last_became_peered":"2024-09-13T22:55:38.106061+0000","last_unstale":"2024-09-13T23:01:48.578084+0000","last_undegraded":"2024-09-13T23:01:48.578084+0000","last_fullsized":"2024-09-13T23:01:48.578084+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'15","last_scrub_stamp":"2024-09-13T23:01:18.436650+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T23:01:18.436650+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:44.225039+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":885,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578144+0000","last_change":"2024-09-13T23:01:23.410605+0000","last_active":"2024-09-13T23:01:48.578144+0000","last_peered":"2024-09-13T23:01:48.578144+0000","last_clean":"2024-09-13T23:01:48.578144+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T23:01:48.578144+0000","last_undegraded":"2024-09-13T23:01:48.578144+0000","last_fullsized":"2024-09-13T23:01:48.578144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:23.410379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.410379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.175055+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":498,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.429050+0000","last_change":"2024-09-13T23:01:26.610248+0000","last_active":"2024-09-13T23:01:49.429050+0000","last_peered":"2024-09-13T23:01:49.429050+0000","last_clean":"2024-09-13T23:01:49.429050+0000","last_became_active":"2024-09-13T22:55:38.124814+0000","last_became_peered":"2024-09-13T22:55:38.124814+0000","last_unstale":"2024-09-13T23:01:49.429050+0000","last_undegraded":"2024-09-13T23:01:49.429050+0000","last_fullsized":"2024-09-13T23:01:49.429050+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T23:01:26.610091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.610091+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:49.319333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":880,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248486+0000","last_change":"2024-09-13T22:57:51.208483+0000","last_active":"2024-09-13T23:01:45.248486+0000","last_peered":"2024-09-13T23:01:45.248486+0000","last_clean":"2024-09-13T23:01:45.248486+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T23:01:45.248486+0000","last_undegraded":"2024-09-13T23:01:45.248486+0000","last_fullsized":"2024-09-13T23:01:45.248486+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.208413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.208413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":714,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.250358+0000","last_change":"2024-09-13T23:01:23.313660+0000","last_active":"2024-09-13T23:01:45.250358+0000","last_peered":"2024-09-13T23:01:45.250358+0000","last_clean":"2024-09-13T23:01:45.250358+0000","last_became_active":"2024-09-13T22:58:42.746002+0000","last_became_peered":"2024-09-13T22:58:42.746002+0000","last_unstale":"2024-09-13T23:01:45.250358+0000","last_undegraded":"2024-09-13T23:01:45.250358+0000","last_fullsized":"2024-09-13T23:01:45.250358+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:23.313470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.313470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:43.384132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":880,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196473+0000","last_change":"2024-09-13T22:58:42.746884+0000","last_active":"2024-09-13T23:01:48.196473+0000","last_peered":"2024-09-13T23:01:48.196473+0000","last_clean":"2024-09-13T23:01:48.196473+0000","last_became_active":"2024-09-13T22:58:42.746385+0000","last_became_peered":"2024-09-13T22:58:42.746385+0000","last_unstale":"2024-09-13T23:01:48.196473+0000","last_undegraded":"2024-09-13T23:01:48.196473+0000","last_fullsized":"2024-09-13T23:01:48.196473+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.718269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093484+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":933,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196366+0000","last_change":"2024-09-13T22:58:18.833271+0000","last_active":"2024-09-13T23:01:48.196366+0000","last_peered":"2024-09-13T23:01:48.196366+0000","last_clean":"2024-09-13T23:01:48.196366+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T23:01:48.196366+0000","last_undegraded":"2024-09-13T23:01:48.196366+0000","last_fullsized":"2024-09-13T23:01:48.196366+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:18.833192+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T22:58:18.833192+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":1015,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333533+0000","last_change":"2024-09-13T23:01:34.915626+0000","last_active":"2024-09-13T23:01:44.333533+0000","last_peered":"2024-09-13T23:01:44.333533+0000","last_clean":"2024-09-13T23:01:44.333533+0000","last_became_active":"2024-09-13T22:55:38.135796+0000","last_became_peered":"2024-09-13T22:55:38.135796+0000","last_unstale":"2024-09-13T23:01:44.333533+0000","last_undegraded":"2024-09-13T23:01:44.333533+0000","last_fullsized":"2024-09-13T23:01:44.333533+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'36","last_scrub_stamp":"2024-09-13T23:01:34.915514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:34.915514+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:52.880988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.462843+0000","last_change":"2024-09-13T22:58:42.748161+0000","last_active":"2024-09-13T23:01:44.462843+0000","last_peered":"2024-09-13T23:01:44.462843+0000","last_clean":"2024-09-13T23:01:44.462843+0000","last_became_active":"2024-09-13T22:58:42.747048+0000","last_became_peered":"2024-09-13T22:58:42.747048+0000","last_unstale":"2024-09-13T23:01:44.462843+0000","last_undegraded":"2024-09-13T23:01:44.462843+0000","last_fullsized":"2024-09-13T23:01:44.462843+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:04.243067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:04.243067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":717,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577559+0000","last_change":"2024-09-13T22:58:42.755687+0000","last_active":"2024-09-13T23:01:48.577559+0000","last_peered":"2024-09-13T23:01:48.577559+0000","last_clean":"2024-09-13T23:01:48.577559+0000","last_became_active":"2024-09-13T22:58:42.721344+0000","last_became_peered":"2024-09-13T22:58:42.721344+0000","last_unstale":"2024-09-13T23:01:48.577559+0000","last_undegraded":"2024-09-13T23:01:48.577559+0000","last_fullsized":"2024-09-13T23:01:48.577559+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:58:11.043740+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:58:11.043740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"351'14","reported_seq":1205,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.195712+0000","last_change":"2024-09-13T22:58:02.750352+0000","last_active":"2024-09-13T23:01:48.195712+0000","last_peered":"2024-09-13T23:01:48.195712+0000","last_clean":"2024-09-13T23:01:48.195712+0000","last_became_active":"2024-09-13T22:55:38.087590+0000","last_became_peered":"2024-09-13T22:55:38.087590+0000","last_unstale":"2024-09-13T23:01:48.195712+0000","last_undegraded":"2024-09-13T23:01:48.195712+0000","last_fullsized":"2024-09-13T23:01:48.195712+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'14","last_scrub_stamp":"2024-09-13T22:58:02.750208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.750208+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":880,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248540+0000","last_change":"2024-09-13T22:58:42.771114+0000","last_active":"2024-09-13T23:01:45.248540+0000","last_peered":"2024-09-13T23:01:45.248540+0000","last_clean":"2024-09-13T23:01:45.248540+0000","last_became_active":"2024-09-13T22:58:42.770701+0000","last_became_peered":"2024-09-13T22:58:42.770701+0000","last_unstale":"2024-09-13T23:01:45.248540+0000","last_undegraded":"2024-09-13T23:01:45.248540+0000","last_fullsized":"2024-09-13T23:01:45.248540+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.267276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.267276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":767,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.429183+0000","last_change":"2024-09-13T23:01:31.604209+0000","last_active":"2024-09-13T23:01:49.429183+0000","last_peered":"2024-09-13T23:01:49.429183+0000","last_clean":"2024-09-13T23:01:49.429183+0000","last_became_active":"2024-09-13T22:58:45.843410+0000","last_became_peered":"2024-09-13T22:58:45.843410+0000","last_unstale":"2024-09-13T23:01:49.429183+0000","last_undegraded":"2024-09-13T23:01:49.429183+0000","last_fullsized":"2024-09-13T23:01:49.429183+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:31.604112+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:01:31.604112+0000","last_clean_scrub_stamp":"2024-09-13T23:01:31.604112+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:38.154528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":921,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.198146+0000","last_change":"2024-09-13T22:56:59.524045+0000","last_active":"2024-09-13T23:01:48.198146+0000","last_peered":"2024-09-13T23:01:48.198146+0000","last_clean":"2024-09-13T23:01:48.198146+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T23:01:48.198146+0000","last_undegraded":"2024-09-13T23:01:48.198146+0000","last_fullsized":"2024-09-13T23:01:48.198146+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:56:59.523938+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:56:59.523938+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093154+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":958,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502706+0000","last_change":"2024-09-13T23:01:27.810222+0000","last_active":"2024-09-13T23:01:48.502706+0000","last_peered":"2024-09-13T23:01:48.502706+0000","last_clean":"2024-09-13T23:01:48.502706+0000","last_became_active":"2024-09-13T22:58:42.742912+0000","last_became_peered":"2024-09-13T22:58:42.742912+0000","last_unstale":"2024-09-13T23:01:48.502706+0000","last_undegraded":"2024-09-13T23:01:48.502706+0000","last_fullsized":"2024-09-13T23:01:48.502706+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T23:01:27.810020+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.810020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:45.023945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":2309,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.686211+0000","last_change":"2024-09-13T22:57:55.678130+0000","last_active":"2024-09-13T23:01:48.686211+0000","last_peered":"2024-09-13T23:01:48.686211+0000","last_clean":"2024-09-13T23:01:48.686211+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T23:01:48.686211+0000","last_undegraded":"2024-09-13T23:01:48.686211+0000","last_fullsized":"2024-09-13T23:01:48.686211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T22:57:55.678034+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.678034+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1420,"num_read_kb":1415,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":396,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.195868+0000","last_change":"2024-09-13T22:57:49.593322+0000","last_active":"2024-09-13T23:01:48.195868+0000","last_peered":"2024-09-13T23:01:48.195868+0000","last_clean":"2024-09-13T23:01:48.195868+0000","last_became_active":"2024-09-13T22:55:38.124054+0000","last_became_peered":"2024-09-13T22:55:38.124054+0000","last_unstale":"2024-09-13T23:01:48.195868+0000","last_undegraded":"2024-09-13T23:01:48.195868+0000","last_fullsized":"2024-09-13T23:01:48.195868+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.593224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.593224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":401,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196745+0000","last_change":"2024-09-13T23:01:19.136514+0000","last_active":"2024-09-13T23:01:48.196745+0000","last_peered":"2024-09-13T23:01:48.196745+0000","last_clean":"2024-09-13T23:01:48.196745+0000","last_became_active":"2024-09-13T22:58:42.747444+0000","last_became_peered":"2024-09-13T22:58:42.747444+0000","last_unstale":"2024-09-13T23:01:48.196745+0000","last_undegraded":"2024-09-13T23:01:48.196745+0000","last_fullsized":"2024-09-13T23:01:48.196745+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:19.136408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T23:01:19.136408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:38.147672+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":586,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.462867+0000","last_change":"2024-09-13T23:01:19.547842+0000","last_active":"2024-09-13T23:01:44.462867+0000","last_peered":"2024-09-13T23:01:44.462867+0000","last_clean":"2024-09-13T23:01:44.462867+0000","last_became_active":"2024-09-13T22:58:45.829511+0000","last_became_peered":"2024-09-13T22:58:45.829511+0000","last_unstale":"2024-09-13T23:01:44.462867+0000","last_undegraded":"2024-09-13T23:01:44.462867+0000","last_fullsized":"2024-09-13T23:01:44.462867+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:19.547734+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:19.547734+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:49.347580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1280,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502251+0000","last_change":"2024-09-13T22:58:09.054534+0000","last_active":"2024-09-13T23:01:48.502251+0000","last_peered":"2024-09-13T23:01:48.502251+0000","last_clean":"2024-09-13T23:01:48.502251+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T23:01:48.502251+0000","last_undegraded":"2024-09-13T23:01:48.502251+0000","last_fullsized":"2024-09-13T23:01:48.502251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'64","last_scrub_stamp":"2024-09-13T22:58:09.054428+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:55:36.587996+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.054428+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":898,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.251242+0000","last_change":"2024-09-13T22:58:07.245450+0000","last_active":"2024-09-13T23:01:45.251242+0000","last_peered":"2024-09-13T23:01:45.251242+0000","last_clean":"2024-09-13T23:01:45.251242+0000","last_became_active":"2024-09-13T22:55:38.145203+0000","last_became_peered":"2024-09-13T22:55:38.145203+0000","last_unstale":"2024-09-13T23:01:45.251242+0000","last_undegraded":"2024-09-13T23:01:45.251242+0000","last_fullsized":"2024-09-13T23:01:45.251242+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.245370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.245370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":138,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.461047+0000","last_change":"2024-09-13T22:58:42.741243+0000","last_active":"2024-09-13T23:01:44.461047+0000","last_peered":"2024-09-13T23:01:44.461047+0000","last_clean":"2024-09-13T23:01:44.461047+0000","last_became_active":"2024-09-13T22:58:42.740862+0000","last_became_peered":"2024-09-13T22:58:42.740862+0000","last_unstale":"2024-09-13T23:01:44.461047+0000","last_undegraded":"2024-09-13T23:01:44.461047+0000","last_fullsized":"2024-09-13T23:01:44.461047+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.310671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.310671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501986+0000","last_change":"2024-09-13T23:01:03.778607+0000","last_active":"2024-09-13T23:01:48.501986+0000","last_peered":"2024-09-13T23:01:48.501986+0000","last_clean":"2024-09-13T23:01:48.501986+0000","last_became_active":"2024-09-13T22:58:42.742606+0000","last_became_peered":"2024-09-13T22:58:42.742606+0000","last_unstale":"2024-09-13T23:01:48.501986+0000","last_undegraded":"2024-09-13T23:01:48.501986+0000","last_fullsized":"2024-09-13T23:01:48.501986+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:03.778503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:03.778503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:21.469814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":589,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.463133+0000","last_change":"2024-09-13T23:01:20.581391+0000","last_active":"2024-09-13T23:01:44.463133+0000","last_peered":"2024-09-13T23:01:44.463133+0000","last_clean":"2024-09-13T23:01:44.463133+0000","last_became_active":"2024-09-13T22:58:45.832840+0000","last_became_peered":"2024-09-13T22:58:45.832840+0000","last_unstale":"2024-09-13T23:01:44.463133+0000","last_undegraded":"2024-09-13T23:01:44.463133+0000","last_fullsized":"2024-09-13T23:01:44.463133+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:20.581288+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:21.295886+0000","last_clean_scrub_stamp":"2024-09-13T23:01:20.581288+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:26.252840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1250,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.579293+0000","last_change":"2024-09-13T22:58:09.995562+0000","last_active":"2024-09-13T23:01:48.579293+0000","last_peered":"2024-09-13T23:01:48.579293+0000","last_clean":"2024-09-13T23:01:48.579293+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T23:01:48.579293+0000","last_undegraded":"2024-09-13T23:01:48.579293+0000","last_fullsized":"2024-09-13T23:01:48.579293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'48","last_scrub_stamp":"2024-09-13T22:58:09.995445+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.995445+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":915,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502323+0000","last_change":"2024-09-13T23:01:01.749507+0000","last_active":"2024-09-13T23:01:48.502323+0000","last_peered":"2024-09-13T23:01:48.502323+0000","last_clean":"2024-09-13T23:01:48.502323+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T23:01:48.502323+0000","last_undegraded":"2024-09-13T23:01:48.502323+0000","last_fullsized":"2024-09-13T23:01:48.502323+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T23:01:01.749341+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T22:54:26.175576+0000","last_clean_scrub_stamp":"2024-09-13T23:01:01.749341+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:30.434533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":934,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501662+0000","last_change":"2024-09-13T22:58:06.134146+0000","last_active":"2024-09-13T23:01:48.501662+0000","last_peered":"2024-09-13T23:01:48.501662+0000","last_clean":"2024-09-13T23:01:48.501662+0000","last_became_active":"2024-09-13T22:55:38.114687+0000","last_became_peered":"2024-09-13T22:55:38.114687+0000","last_unstale":"2024-09-13T23:01:48.501662+0000","last_undegraded":"2024-09-13T23:01:48.501662+0000","last_fullsized":"2024-09-13T23:01:48.501662+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T22:58:06.133998+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T22:55:21.433994+0000","last_clean_scrub_stamp":"2024-09-13T22:58:06.133998+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"414'14","reported_seq":1218,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.331944+0000","last_change":"2024-09-13T22:58:42.782423+0000","last_active":"2024-09-13T23:01:44.331944+0000","last_peered":"2024-09-13T23:01:44.331944+0000","last_clean":"2024-09-13T23:01:44.331944+0000","last_became_active":"2024-09-13T22:58:42.740603+0000","last_became_peered":"2024-09-13T22:58:42.740603+0000","last_unstale":"2024-09-13T23:01:44.331944+0000","last_undegraded":"2024-09-13T23:01:44.331944+0000","last_fullsized":"2024-09-13T23:01:44.331944+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'13","last_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_deep_scrub":"358'13","last_deep_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_clean_scrub_stamp":"2024-09-13T22:58:05.208554+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":885,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.331524+0000","last_change":"2024-09-13T22:57:56.259013+0000","last_active":"2024-09-13T23:01:44.331524+0000","last_peered":"2024-09-13T23:01:44.331524+0000","last_clean":"2024-09-13T23:01:44.331524+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T23:01:44.331524+0000","last_undegraded":"2024-09-13T23:01:44.331524+0000","last_fullsized":"2024-09-13T23:01:44.331524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:56.258783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:56.258783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":943,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.576805+0000","last_change":"2024-09-13T22:57:57.050275+0000","last_active":"2024-09-13T23:01:48.576805+0000","last_peered":"2024-09-13T23:01:48.576805+0000","last_clean":"2024-09-13T23:01:48.576805+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T23:01:48.576805+0000","last_undegraded":"2024-09-13T23:01:48.576805+0000","last_fullsized":"2024-09-13T23:01:48.576805+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:57.050042+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.050042+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609438+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"468'47","reported_seq":266,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.428528+0000","last_change":"2024-09-13T23:01:23.540614+0000","last_active":"2024-09-13T23:01:49.428528+0000","last_peered":"2024-09-13T23:01:49.428528+0000","last_clean":"2024-09-13T23:01:49.428528+0000","last_became_active":"2024-09-13T22:58:42.733200+0000","last_became_peered":"2024-09-13T22:58:42.733200+0000","last_unstale":"2024-09-13T23:01:49.428528+0000","last_undegraded":"2024-09-13T23:01:49.428528+0000","last_fullsized":"2024-09-13T23:01:49.428528+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'47","last_scrub_stamp":"2024-09-13T23:01:23.540241+0000","last_deep_scrub":"468'47","last_deep_scrub_stamp":"2024-09-13T23:01:23.540241+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.540241+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:44.220471+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":918,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578351+0000","last_change":"2024-09-13T22:58:42.742254+0000","last_active":"2024-09-13T23:01:48.578351+0000","last_peered":"2024-09-13T23:01:48.578351+0000","last_clean":"2024-09-13T23:01:48.578351+0000","last_became_active":"2024-09-13T22:58:42.741601+0000","last_became_peered":"2024-09-13T22:58:42.741601+0000","last_unstale":"2024-09-13T23:01:48.578351+0000","last_undegraded":"2024-09-13T23:01:48.578351+0000","last_fullsized":"2024-09-13T23:01:48.578351+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:17.058474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:58:17.058474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":927,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333171+0000","last_change":"2024-09-13T23:01:26.955132+0000","last_active":"2024-09-13T23:01:44.333171+0000","last_peered":"2024-09-13T23:01:44.333171+0000","last_clean":"2024-09-13T23:01:44.333171+0000","last_became_active":"2024-09-13T22:58:42.740401+0000","last_became_peered":"2024-09-13T22:58:42.740401+0000","last_unstale":"2024-09-13T23:01:44.333171+0000","last_undegraded":"2024-09-13T23:01:44.333171+0000","last_fullsized":"2024-09-13T23:01:44.333171+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:26.955048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.955048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:30.495780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":899,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197435+0000","last_change":"2024-09-13T22:58:42.746942+0000","last_active":"2024-09-13T23:01:48.197435+0000","last_peered":"2024-09-13T23:01:48.197435+0000","last_clean":"2024-09-13T23:01:48.197435+0000","last_became_active":"2024-09-13T22:58:42.746572+0000","last_became_peered":"2024-09-13T22:58:42.746572+0000","last_unstale":"2024-09-13T23:01:48.197435+0000","last_undegraded":"2024-09-13T23:01:48.197435+0000","last_fullsized":"2024-09-13T23:01:48.197435+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:56.539208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:56.539208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093056+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1193,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248803+0000","last_change":"2024-09-13T23:01:32.330495+0000","last_active":"2024-09-13T23:01:45.248803+0000","last_peered":"2024-09-13T23:01:45.248803+0000","last_clean":"2024-09-13T23:01:45.248803+0000","last_became_active":"2024-09-13T22:55:41.103393+0000","last_became_peered":"2024-09-13T22:55:41.103393+0000","last_unstale":"2024-09-13T23:01:45.248803+0000","last_undegraded":"2024-09-13T23:01:45.248803+0000","last_fullsized":"2024-09-13T23:01:45.248803+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:32.330280+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:32.330280+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:35.531503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":253,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.576683+0000","last_change":"2024-09-13T23:01:13.457411+0000","last_active":"2024-09-13T23:01:48.576683+0000","last_peered":"2024-09-13T23:01:48.576683+0000","last_clean":"2024-09-13T23:01:48.576683+0000","last_became_active":"2024-09-13T22:58:42.727647+0000","last_became_peered":"2024-09-13T22:58:42.727647+0000","last_unstale":"2024-09-13T23:01:48.576683+0000","last_undegraded":"2024-09-13T23:01:48.576683+0000","last_fullsized":"2024-09-13T23:01:48.576683+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-09-13T23:01:13.457325+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T23:01:13.457325+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:14.464305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"470'2","reported_seq":948,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.428780+0000","last_change":"2024-09-13T23:01:27.631319+0000","last_active":"2024-09-13T23:01:49.428780+0000","last_peered":"2024-09-13T23:01:49.428780+0000","last_clean":"2024-09-13T23:01:49.428780+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T23:01:49.428780+0000","last_undegraded":"2024-09-13T23:01:49.428780+0000","last_fullsized":"2024-09-13T23:01:49.428780+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'2","last_scrub_stamp":"2024-09-13T23:01:27.631212+0000","last_deep_scrub":"470'2","last_deep_scrub_stamp":"2024-09-13T23:01:27.631212+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.631212+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:34.588613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":931,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.429440+0000","last_change":"2024-09-13T23:01:35.666130+0000","last_active":"2024-09-13T23:01:49.429440+0000","last_peered":"2024-09-13T23:01:49.429440+0000","last_clean":"2024-09-13T23:01:49.429440+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T23:01:49.429440+0000","last_undegraded":"2024-09-13T23:01:49.429440+0000","last_fullsized":"2024-09-13T23:01:49.429440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.665979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.665979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:50.400063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":888,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249889+0000","last_change":"2024-09-13T23:01:22.338274+0000","last_active":"2024-09-13T23:01:45.249889+0000","last_peered":"2024-09-13T23:01:45.249889+0000","last_clean":"2024-09-13T23:01:45.249889+0000","last_became_active":"2024-09-13T22:58:42.764039+0000","last_became_peered":"2024-09-13T22:58:42.764039+0000","last_unstale":"2024-09-13T23:01:45.249889+0000","last_undegraded":"2024-09-13T23:01:45.249889+0000","last_fullsized":"2024-09-13T23:01:45.249889+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:22.338166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.338166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:50.408989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":995,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501105+0000","last_change":"2024-09-13T23:01:11.835987+0000","last_active":"2024-09-13T23:01:48.501105+0000","last_peered":"2024-09-13T23:01:48.501105+0000","last_clean":"2024-09-13T23:01:48.501105+0000","last_became_active":"2024-09-13T22:58:45.843305+0000","last_became_peered":"2024-09-13T22:58:45.843305+0000","last_unstale":"2024-09-13T23:01:48.501105+0000","last_undegraded":"2024-09-13T23:01:48.501105+0000","last_fullsized":"2024-09-13T23:01:48.501105+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:11.835784+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T23:01:11.835784+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:12.085238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"397'42","reported_seq":1030,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.332322+0000","last_change":"2024-09-13T22:58:15.305788+0000","last_active":"2024-09-13T23:01:44.332322+0000","last_peered":"2024-09-13T23:01:44.332322+0000","last_clean":"2024-09-13T23:01:44.332322+0000","last_became_active":"2024-09-13T22:55:38.098695+0000","last_became_peered":"2024-09-13T22:55:38.098695+0000","last_unstale":"2024-09-13T23:01:44.332322+0000","last_undegraded":"2024-09-13T23:01:44.332322+0000","last_fullsized":"2024-09-13T23:01:44.332322+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-09-13T22:58:15.305297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.305297+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816893+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":663,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.247908+0000","last_change":"2024-09-13T23:01:17.315498+0000","last_active":"2024-09-13T23:01:45.247908+0000","last_peered":"2024-09-13T23:01:45.247908+0000","last_clean":"2024-09-13T23:01:45.247908+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T23:01:45.247908+0000","last_undegraded":"2024-09-13T23:01:45.247908+0000","last_fullsized":"2024-09-13T23:01:45.247908+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:17.315399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T23:01:17.315399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:43.589754+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578302+0000","last_change":"2024-09-13T23:01:21.396408+0000","last_active":"2024-09-13T23:01:48.578302+0000","last_peered":"2024-09-13T23:01:48.578302+0000","last_clean":"2024-09-13T23:01:48.578302+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T23:01:48.578302+0000","last_undegraded":"2024-09-13T23:01:48.578302+0000","last_fullsized":"2024-09-13T23:01:48.578302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:21.396226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.396226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:46.302469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":894,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501748+0000","last_change":"2024-09-13T23:01:30.887160+0000","last_active":"2024-09-13T23:01:48.501748+0000","last_peered":"2024-09-13T23:01:48.501748+0000","last_clean":"2024-09-13T23:01:48.501748+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T23:01:48.501748+0000","last_undegraded":"2024-09-13T23:01:48.501748+0000","last_fullsized":"2024-09-13T23:01:48.501748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.886958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:18.659096+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.886958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:51.664086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":958,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197245+0000","last_change":"2024-09-13T22:58:42.787694+0000","last_active":"2024-09-13T23:01:48.197245+0000","last_peered":"2024-09-13T23:01:48.197245+0000","last_clean":"2024-09-13T23:01:48.197245+0000","last_became_active":"2024-09-13T22:58:42.746819+0000","last_became_peered":"2024-09-13T22:58:42.746819+0000","last_unstale":"2024-09-13T23:01:48.197245+0000","last_undegraded":"2024-09-13T23:01:48.197245+0000","last_fullsized":"2024-09-13T23:01:48.197245+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:58:09.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.777653+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093317+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":906,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577565+0000","last_change":"2024-09-13T23:01:16.437663+0000","last_active":"2024-09-13T23:01:48.577565+0000","last_peered":"2024-09-13T23:01:48.577565+0000","last_clean":"2024-09-13T23:01:48.577565+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T23:01:48.577565+0000","last_undegraded":"2024-09-13T23:01:48.577565+0000","last_fullsized":"2024-09-13T23:01:48.577565+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:16.437598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:16.437598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:17.327827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1177,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248775+0000","last_change":"2024-09-13T23:01:21.309692+0000","last_active":"2024-09-13T23:01:45.248775+0000","last_peered":"2024-09-13T23:01:45.248775+0000","last_clean":"2024-09-13T23:01:45.248775+0000","last_became_active":"2024-09-13T22:58:45.829938+0000","last_became_peered":"2024-09-13T22:58:45.829938+0000","last_unstale":"2024-09-13T23:01:45.248775+0000","last_undegraded":"2024-09-13T23:01:45.248775+0000","last_fullsized":"2024-09-13T23:01:45.248775+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:21.309609+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.309609+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:35.236445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":880,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249922+0000","last_change":"2024-09-13T23:01:14.349135+0000","last_active":"2024-09-13T23:01:45.249922+0000","last_peered":"2024-09-13T23:01:45.249922+0000","last_clean":"2024-09-13T23:01:45.249922+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T23:01:45.249922+0000","last_undegraded":"2024-09-13T23:01:45.249922+0000","last_fullsized":"2024-09-13T23:01:45.249922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:14.349029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.349029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:27.002108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":912,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578231+0000","last_change":"2024-09-13T23:01:03.483621+0000","last_active":"2024-09-13T23:01:48.578231+0000","last_peered":"2024-09-13T23:01:48.578231+0000","last_clean":"2024-09-13T23:01:48.578231+0000","last_became_active":"2024-09-13T22:55:38.101749+0000","last_became_peered":"2024-09-13T22:55:38.101749+0000","last_unstale":"2024-09-13T23:01:48.578231+0000","last_undegraded":"2024-09-13T23:01:48.578231+0000","last_fullsized":"2024-09-13T23:01:48.578231+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:03.483542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:03.483542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:11.483888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":993,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.250766+0000","last_change":"2024-09-13T22:58:42.803636+0000","last_active":"2024-09-13T23:01:45.250766+0000","last_peered":"2024-09-13T23:01:45.250766+0000","last_clean":"2024-09-13T23:01:45.250766+0000","last_became_active":"2024-09-13T22:58:42.764099+0000","last_became_peered":"2024-09-13T22:58:42.764099+0000","last_unstale":"2024-09-13T23:01:45.250766+0000","last_undegraded":"2024-09-13T23:01:45.250766+0000","last_fullsized":"2024-09-13T23:01:45.250766+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-09-13T22:58:14.318902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.318902+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":938,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.427673+0000","last_change":"2024-09-13T23:01:29.625234+0000","last_active":"2024-09-13T23:01:49.427673+0000","last_peered":"2024-09-13T23:01:49.427673+0000","last_clean":"2024-09-13T23:01:49.427673+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T23:01:49.427673+0000","last_undegraded":"2024-09-13T23:01:49.427673+0000","last_fullsized":"2024-09-13T23:01:49.427673+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:29.625043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.625043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:53.981960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":922,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.428760+0000","last_change":"2024-09-13T23:01:30.649247+0000","last_active":"2024-09-13T23:01:49.428760+0000","last_peered":"2024-09-13T23:01:49.428760+0000","last_clean":"2024-09-13T23:01:49.428760+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T23:01:49.428760+0000","last_undegraded":"2024-09-13T23:01:49.428760+0000","last_fullsized":"2024-09-13T23:01:49.428760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.648989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.648989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.411813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":868,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248826+0000","last_change":"2024-09-13T23:01:06.196129+0000","last_active":"2024-09-13T23:01:45.248826+0000","last_peered":"2024-09-13T23:01:45.248826+0000","last_clean":"2024-09-13T23:01:45.248826+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T23:01:45.248826+0000","last_undegraded":"2024-09-13T23:01:45.248826+0000","last_fullsized":"2024-09-13T23:01:45.248826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:06.196062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T23:01:06.196062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:08.228449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"397'30","reported_seq":982,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577779+0000","last_change":"2024-09-13T23:01:29.459955+0000","last_active":"2024-09-13T23:01:48.577779+0000","last_peered":"2024-09-13T23:01:48.577779+0000","last_clean":"2024-09-13T23:01:48.577779+0000","last_became_active":"2024-09-13T22:58:42.731142+0000","last_became_peered":"2024-09-13T22:58:42.731142+0000","last_unstale":"2024-09-13T23:01:48.577779+0000","last_undegraded":"2024-09-13T23:01:48.577779+0000","last_fullsized":"2024-09-13T23:01:48.577779+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:01:29.459740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.459740+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:48.107858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":923,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502270+0000","last_change":"2024-09-13T23:01:09.811260+0000","last_active":"2024-09-13T23:01:48.502270+0000","last_peered":"2024-09-13T23:01:48.502270+0000","last_clean":"2024-09-13T23:01:48.502270+0000","last_became_active":"2024-09-13T22:55:38.093381+0000","last_became_peered":"2024-09-13T22:55:38.093381+0000","last_unstale":"2024-09-13T23:01:48.502270+0000","last_undegraded":"2024-09-13T23:01:48.502270+0000","last_fullsized":"2024-09-13T23:01:48.502270+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:09.811162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:09.811162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:20.896122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":617,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196480+0000","last_change":"2024-09-13T23:01:26.266715+0000","last_active":"2024-09-13T23:01:48.196480+0000","last_peered":"2024-09-13T23:01:48.196480+0000","last_clean":"2024-09-13T23:01:48.196480+0000","last_became_active":"2024-09-13T22:55:38.098515+0000","last_became_peered":"2024-09-13T22:55:38.098515+0000","last_unstale":"2024-09-13T23:01:48.196480+0000","last_undegraded":"2024-09-13T23:01:48.196480+0000","last_fullsized":"2024-09-13T23:01:48.196480+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:26.266526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.266526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:38.051187+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":893,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.332176+0000","last_change":"2024-09-13T22:58:16.384185+0000","last_active":"2024-09-13T23:01:44.332176+0000","last_peered":"2024-09-13T23:01:44.332176+0000","last_clean":"2024-09-13T23:01:44.332176+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T23:01:44.332176+0000","last_undegraded":"2024-09-13T23:01:44.332176+0000","last_fullsized":"2024-09-13T23:01:44.332176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:16.383923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:58:16.383923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816798+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":1017,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.250801+0000","last_change":"2024-09-13T23:01:36.239941+0000","last_active":"2024-09-13T23:01:45.250801+0000","last_peered":"2024-09-13T23:01:45.250801+0000","last_clean":"2024-09-13T23:01:45.250801+0000","last_became_active":"2024-09-13T22:55:38.151658+0000","last_became_peered":"2024-09-13T22:55:38.151658+0000","last_unstale":"2024-09-13T23:01:45.250801+0000","last_undegraded":"2024-09-13T23:01:45.250801+0000","last_fullsized":"2024-09-13T23:01:45.250801+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:01:36.239714+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-09-13T23:01:36.239714+0000","last_clean_scrub_stamp":"2024-09-13T23:01:36.239714+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:46.968037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":479,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197651+0000","last_change":"2024-09-13T23:01:34.122869+0000","last_active":"2024-09-13T23:01:48.197651+0000","last_peered":"2024-09-13T23:01:48.197651+0000","last_clean":"2024-09-13T23:01:48.197651+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T23:01:48.197651+0000","last_undegraded":"2024-09-13T23:01:48.197651+0000","last_fullsized":"2024-09-13T23:01:48.197651+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:34.122727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:32.054441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:34.122727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.135029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"414'16","reported_seq":1233,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248466+0000","last_change":"2024-09-13T23:01:09.277696+0000","last_active":"2024-09-13T23:01:45.248466+0000","last_peered":"2024-09-13T23:01:45.248466+0000","last_clean":"2024-09-13T23:01:45.248466+0000","last_became_active":"2024-09-13T22:58:42.758407+0000","last_became_peered":"2024-09-13T22:58:42.758407+0000","last_unstale":"2024-09-13T23:01:45.248466+0000","last_undegraded":"2024-09-13T23:01:45.248466+0000","last_fullsized":"2024-09-13T23:01:45.248466+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'16","last_scrub_stamp":"2024-09-13T23:01:09.277590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:09.277590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:14.772303+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577315+0000","last_change":"2024-09-13T22:58:42.734657+0000","last_active":"2024-09-13T23:01:48.577315+0000","last_peered":"2024-09-13T23:01:48.577315+0000","last_clean":"2024-09-13T23:01:48.577315+0000","last_became_active":"2024-09-13T22:58:42.732057+0000","last_became_peered":"2024-09-13T22:58:42.732057+0000","last_unstale":"2024-09-13T23:01:48.577315+0000","last_undegraded":"2024-09-13T23:01:48.577315+0000","last_fullsized":"2024-09-13T23:01:48.577315+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:03.020207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T22:58:03.020207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"397'27","reported_seq":980,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.250617+0000","last_change":"2024-09-13T22:57:53.254550+0000","last_active":"2024-09-13T23:01:45.250617+0000","last_peered":"2024-09-13T23:01:45.250617+0000","last_clean":"2024-09-13T23:01:45.250617+0000","last_became_active":"2024-09-13T22:55:38.151005+0000","last_became_peered":"2024-09-13T22:55:38.151005+0000","last_unstale":"2024-09-13T23:01:45.250617+0000","last_undegraded":"2024-09-13T23:01:45.250617+0000","last_fullsized":"2024-09-13T23:01:45.250617+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T22:57:53.254254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:57:53.254254+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":520,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.334038+0000","last_change":"2024-09-13T22:58:01.252621+0000","last_active":"2024-09-13T23:01:44.334038+0000","last_peered":"2024-09-13T23:01:44.334038+0000","last_clean":"2024-09-13T23:01:44.334038+0000","last_became_active":"2024-09-13T22:55:38.136788+0000","last_became_peered":"2024-09-13T22:55:38.136788+0000","last_unstale":"2024-09-13T23:01:44.334038+0000","last_undegraded":"2024-09-13T23:01:44.334038+0000","last_fullsized":"2024-09-13T23:01:44.334038+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T22:58:01.252317+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T22:40:29.676505+0000","last_clean_scrub_stamp":"2024-09-13T22:58:01.252317+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":487,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.332136+0000","last_change":"2024-09-13T23:01:35.962699+0000","last_active":"2024-09-13T23:01:44.332136+0000","last_peered":"2024-09-13T23:01:44.332136+0000","last_clean":"2024-09-13T23:01:44.332136+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T23:01:44.332136+0000","last_undegraded":"2024-09-13T23:01:44.332136+0000","last_fullsized":"2024-09-13T23:01:44.332136+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.962519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:35.962519+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.962519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:48.660516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":876,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196687+0000","last_change":"2024-09-13T22:57:50.594198+0000","last_active":"2024-09-13T23:01:48.196687+0000","last_peered":"2024-09-13T23:01:48.196687+0000","last_clean":"2024-09-13T23:01:48.196687+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T23:01:48.196687+0000","last_undegraded":"2024-09-13T23:01:48.196687+0000","last_fullsized":"2024-09-13T23:01:48.196687+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.594120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.594120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":1021,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248042+0000","last_change":"2024-09-13T23:01:29.345584+0000","last_active":"2024-09-13T23:01:45.248042+0000","last_peered":"2024-09-13T23:01:45.248042+0000","last_clean":"2024-09-13T23:01:45.248042+0000","last_became_active":"2024-09-13T22:58:42.758655+0000","last_became_peered":"2024-09-13T22:58:42.758655+0000","last_unstale":"2024-09-13T23:01:45.248042+0000","last_undegraded":"2024-09-13T23:01:45.248042+0000","last_fullsized":"2024-09-13T23:01:45.248042+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:01:29.345354+0000","last_deep_scrub":"397'32","last_deep_scrub_stamp":"2024-09-13T23:01:29.345354+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.345354+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:57.161111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":930,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.427574+0000","last_change":"2024-09-13T23:01:08.586865+0000","last_active":"2024-09-13T23:01:49.427574+0000","last_peered":"2024-09-13T23:01:49.427574+0000","last_clean":"2024-09-13T23:01:49.427574+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T23:01:49.427574+0000","last_undegraded":"2024-09-13T23:01:49.427574+0000","last_fullsized":"2024-09-13T23:01:49.427574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:08.586782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T23:01:08.586782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:15.017576+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":879,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197997+0000","last_change":"2024-09-13T23:01:32.179108+0000","last_active":"2024-09-13T23:01:48.197997+0000","last_peered":"2024-09-13T23:01:48.197997+0000","last_clean":"2024-09-13T23:01:48.197997+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T23:01:48.197997+0000","last_undegraded":"2024-09-13T23:01:48.197997+0000","last_fullsized":"2024-09-13T23:01:48.197997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:32.178929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:32.178929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:39.017030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":944,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.429005+0000","last_change":"2024-09-13T23:01:18.539224+0000","last_active":"2024-09-13T23:01:49.429005+0000","last_peered":"2024-09-13T23:01:49.429005+0000","last_clean":"2024-09-13T23:01:49.429005+0000","last_became_active":"2024-09-13T22:55:38.126360+0000","last_became_peered":"2024-09-13T22:55:38.126360+0000","last_unstale":"2024-09-13T23:01:49.429005+0000","last_undegraded":"2024-09-13T23:01:49.429005+0000","last_fullsized":"2024-09-13T23:01:49.429005+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:18.539114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T23:01:18.539114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.206035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":152,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.460534+0000","last_change":"2024-09-13T23:01:35.587143+0000","last_active":"2024-09-13T23:01:44.460534+0000","last_peered":"2024-09-13T23:01:44.460534+0000","last_clean":"2024-09-13T23:01:44.460534+0000","last_became_active":"2024-09-13T22:58:42.736178+0000","last_became_peered":"2024-09-13T22:58:42.736178+0000","last_unstale":"2024-09-13T23:01:44.460534+0000","last_undegraded":"2024-09-13T23:01:44.460534+0000","last_fullsized":"2024-09-13T23:01:44.460534+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T23:01:35.587020+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.587020+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.733540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":146,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.462742+0000","last_change":"2024-09-13T23:01:21.585285+0000","last_active":"2024-09-13T23:01:44.462742+0000","last_peered":"2024-09-13T23:01:44.462742+0000","last_clean":"2024-09-13T23:01:44.462742+0000","last_became_active":"2024-09-13T22:58:42.744948+0000","last_became_peered":"2024-09-13T22:58:42.744948+0000","last_unstale":"2024-09-13T23:01:44.462742+0000","last_undegraded":"2024-09-13T23:01:44.462742+0000","last_fullsized":"2024-09-13T23:01:44.462742+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:21.585092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.585092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:22.631485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":896,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578093+0000","last_change":"2024-09-13T23:01:35.457385+0000","last_active":"2024-09-13T23:01:48.578093+0000","last_peered":"2024-09-13T23:01:48.578093+0000","last_clean":"2024-09-13T23:01:48.578093+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T23:01:48.578093+0000","last_undegraded":"2024-09-13T23:01:48.578093+0000","last_fullsized":"2024-09-13T23:01:48.578093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.457138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.457138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:45.923641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":896,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249561+0000","last_change":"2024-09-13T22:58:42.769155+0000","last_active":"2024-09-13T23:01:45.249561+0000","last_peered":"2024-09-13T23:01:45.249561+0000","last_clean":"2024-09-13T23:01:45.249561+0000","last_became_active":"2024-09-13T22:58:42.768571+0000","last_became_peered":"2024-09-13T22:58:42.768571+0000","last_unstale":"2024-09-13T23:01:45.249561+0000","last_undegraded":"2024-09-13T23:01:45.249561+0000","last_fullsized":"2024-09-13T23:01:45.249561+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.349704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.349704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":1016,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249234+0000","last_change":"2024-09-13T23:01:26.320429+0000","last_active":"2024-09-13T23:01:45.249234+0000","last_peered":"2024-09-13T23:01:45.249234+0000","last_clean":"2024-09-13T23:01:45.249234+0000","last_became_active":"2024-09-13T22:58:42.759372+0000","last_became_peered":"2024-09-13T22:58:42.759372+0000","last_unstale":"2024-09-13T23:01:45.249234+0000","last_undegraded":"2024-09-13T23:01:45.249234+0000","last_fullsized":"2024-09-13T23:01:45.249234+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T23:01:26.320267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.320267+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.326421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":908,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577877+0000","last_change":"2024-09-13T23:01:04.468733+0000","last_active":"2024-09-13T23:01:48.577877+0000","last_peered":"2024-09-13T23:01:48.577877+0000","last_clean":"2024-09-13T23:01:48.577877+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T23:01:48.577877+0000","last_undegraded":"2024-09-13T23:01:48.577877+0000","last_fullsized":"2024-09-13T23:01:48.577877+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:04.468638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T23:01:04.468638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:11.277071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":1200,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578498+0000","last_change":"2024-09-13T23:01:14.440415+0000","last_active":"2024-09-13T23:01:48.578498+0000","last_peered":"2024-09-13T23:01:48.578498+0000","last_clean":"2024-09-13T23:01:48.578498+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T23:01:48.578498+0000","last_undegraded":"2024-09-13T23:01:48.578498+0000","last_fullsized":"2024-09-13T23:01:48.578498+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T23:01:14.440335+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T22:53:18.805172+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.440335+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:24.072999+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":891,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333085+0000","last_change":"2024-09-13T23:01:31.009368+0000","last_active":"2024-09-13T23:01:44.333085+0000","last_peered":"2024-09-13T23:01:44.333085+0000","last_clean":"2024-09-13T23:01:44.333085+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T23:01:44.333085+0000","last_undegraded":"2024-09-13T23:01:44.333085+0000","last_fullsized":"2024-09-13T23:01:44.333085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:31.009178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:31.009178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:43.912379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":934,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333267+0000","last_change":"2024-09-13T23:01:33.006568+0000","last_active":"2024-09-13T23:01:44.333267+0000","last_peered":"2024-09-13T23:01:44.333267+0000","last_clean":"2024-09-13T23:01:44.333267+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T23:01:44.333267+0000","last_undegraded":"2024-09-13T23:01:44.333267+0000","last_fullsized":"2024-09-13T23:01:44.333267+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:33.006332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:33.006332+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:39.870873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":922,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333661+0000","last_change":"2024-09-13T23:01:14.927758+0000","last_active":"2024-09-13T23:01:44.333661+0000","last_peered":"2024-09-13T23:01:44.333661+0000","last_clean":"2024-09-13T23:01:44.333661+0000","last_became_active":"2024-09-13T22:55:38.138035+0000","last_became_peered":"2024-09-13T22:55:38.138035+0000","last_unstale":"2024-09-13T23:01:44.333661+0000","last_undegraded":"2024-09-13T23:01:44.333661+0000","last_fullsized":"2024-09-13T23:01:44.333661+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:14.927611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.927611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:21.794756+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":1180,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196587+0000","last_change":"2024-09-13T22:58:14.754552+0000","last_active":"2024-09-13T23:01:48.196587+0000","last_peered":"2024-09-13T23:01:48.196587+0000","last_clean":"2024-09-13T23:01:48.196587+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T23:01:48.196587+0000","last_undegraded":"2024-09-13T23:01:48.196587+0000","last_fullsized":"2024-09-13T23:01:48.196587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:58:14.754430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.754430+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093420+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":414,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.429571+0000","last_change":"2024-09-13T23:01:22.539081+0000","last_active":"2024-09-13T23:01:49.429571+0000","last_peered":"2024-09-13T23:01:49.429571+0000","last_clean":"2024-09-13T23:01:49.429571+0000","last_became_active":"2024-09-13T22:55:38.098486+0000","last_became_peered":"2024-09-13T22:55:38.098486+0000","last_unstale":"2024-09-13T23:01:49.429571+0000","last_undegraded":"2024-09-13T23:01:49.429571+0000","last_fullsized":"2024-09-13T23:01:49.429571+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:22.538979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.538979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:29.121704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":965,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.250614+0000","last_change":"2024-09-13T23:01:11.267967+0000","last_active":"2024-09-13T23:01:45.250614+0000","last_peered":"2024-09-13T23:01:45.250614+0000","last_clean":"2024-09-13T23:01:45.250614+0000","last_became_active":"2024-09-13T22:55:38.144744+0000","last_became_peered":"2024-09-13T22:55:38.144744+0000","last_unstale":"2024-09-13T23:01:45.250614+0000","last_undegraded":"2024-09-13T23:01:45.250614+0000","last_fullsized":"2024-09-13T23:01:45.250614+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:11.267888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:11.267888+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.946659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":138,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.461525+0000","last_change":"2024-09-13T22:58:42.749921+0000","last_active":"2024-09-13T23:01:44.461525+0000","last_peered":"2024-09-13T23:01:44.461525+0000","last_clean":"2024-09-13T23:01:44.461525+0000","last_became_active":"2024-09-13T22:58:42.749062+0000","last_became_peered":"2024-09-13T22:58:42.749062+0000","last_unstale":"2024-09-13T23:01:44.461525+0000","last_undegraded":"2024-09-13T23:01:44.461525+0000","last_fullsized":"2024-09-13T23:01:44.461525+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:54.689197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.689197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":880,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196855+0000","last_change":"2024-09-13T23:01:31.178527+0000","last_active":"2024-09-13T23:01:48.196855+0000","last_peered":"2024-09-13T23:01:48.196855+0000","last_clean":"2024-09-13T23:01:48.196855+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T23:01:48.196855+0000","last_undegraded":"2024-09-13T23:01:48.196855+0000","last_fullsized":"2024-09-13T23:01:48.196855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:31.178438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T23:01:31.178438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:49.241914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":884,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.332376+0000","last_change":"2024-09-13T22:58:13.326329+0000","last_active":"2024-09-13T23:01:44.332376+0000","last_peered":"2024-09-13T23:01:44.332376+0000","last_clean":"2024-09-13T23:01:44.332376+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T23:01:44.332376+0000","last_undegraded":"2024-09-13T23:01:44.332376+0000","last_fullsized":"2024-09-13T23:01:44.332376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.326193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.326193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":520,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249589+0000","last_change":"2024-09-13T23:01:05.196417+0000","last_active":"2024-09-13T23:01:45.249589+0000","last_peered":"2024-09-13T23:01:45.249589+0000","last_clean":"2024-09-13T23:01:45.249589+0000","last_became_active":"2024-09-13T22:58:42.767781+0000","last_became_peered":"2024-09-13T22:58:42.767781+0000","last_unstale":"2024-09-13T23:01:45.249589+0000","last_undegraded":"2024-09-13T23:01:45.249589+0000","last_fullsized":"2024-09-13T23:01:45.249589+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:01:05.196330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:05.196330+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:18.398596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":904,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502117+0000","last_change":"2024-09-13T23:01:35.904091+0000","last_active":"2024-09-13T23:01:48.502117+0000","last_peered":"2024-09-13T23:01:48.502117+0000","last_clean":"2024-09-13T23:01:48.502117+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T23:01:48.502117+0000","last_undegraded":"2024-09-13T23:01:48.502117+0000","last_fullsized":"2024-09-13T23:01:48.502117+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.903970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.903970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:51.073022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":897,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.867455+0000","last_change":"2024-09-13T23:01:48.867508+0000","last_active":"2024-09-13T23:01:48.867455+0000","last_peered":"2024-09-13T23:01:48.867455+0000","last_clean":"2024-09-13T23:01:48.867455+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T23:01:48.867455+0000","last_undegraded":"2024-09-13T23:01:48.867455+0000","last_fullsized":"2024-09-13T23:01:48.867455+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:48.867317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T23:01:48.867317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:50.173590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":868,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.195664+0000","last_change":"2024-09-13T22:57:07.530820+0000","last_active":"2024-09-13T23:01:48.195664+0000","last_peered":"2024-09-13T23:01:48.195664+0000","last_clean":"2024-09-13T23:01:48.195664+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T23:01:48.195664+0000","last_undegraded":"2024-09-13T23:01:48.195664+0000","last_fullsized":"2024-09-13T23:01:48.195664+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.530747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.530747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"397'18","reported_seq":926,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333947+0000","last_change":"2024-09-13T22:58:14.333753+0000","last_active":"2024-09-13T23:01:44.333947+0000","last_peered":"2024-09-13T23:01:44.333947+0000","last_clean":"2024-09-13T23:01:44.333947+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T23:01:44.333947+0000","last_undegraded":"2024-09-13T23:01:44.333947+0000","last_fullsized":"2024-09-13T23:01:44.333947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'18","last_scrub_stamp":"2024-09-13T22:58:14.333490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.333490+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":908,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.579015+0000","last_change":"2024-09-13T22:58:42.727064+0000","last_active":"2024-09-13T23:01:48.579015+0000","last_peered":"2024-09-13T23:01:48.579015+0000","last_clean":"2024-09-13T23:01:48.579015+0000","last_became_active":"2024-09-13T22:58:42.726569+0000","last_became_peered":"2024-09-13T22:58:42.726569+0000","last_unstale":"2024-09-13T23:01:48.579015+0000","last_undegraded":"2024-09-13T23:01:48.579015+0000","last_fullsized":"2024-09-13T23:01:48.579015+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:00.019958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:00.019958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.609171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":138,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.460924+0000","last_change":"2024-09-13T22:58:42.752271+0000","last_active":"2024-09-13T23:01:44.460924+0000","last_peered":"2024-09-13T23:01:44.460924+0000","last_clean":"2024-09-13T23:01:44.460924+0000","last_became_active":"2024-09-13T22:58:42.748403+0000","last_became_peered":"2024-09-13T22:58:42.748403+0000","last_unstale":"2024-09-13T23:01:44.460924+0000","last_undegraded":"2024-09-13T23:01:44.460924+0000","last_fullsized":"2024-09-13T23:01:44.460924+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.200989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.200989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":893,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.248304+0000","last_change":"2024-09-13T23:01:30.318915+0000","last_active":"2024-09-13T23:01:45.248304+0000","last_peered":"2024-09-13T23:01:45.248304+0000","last_clean":"2024-09-13T23:01:45.248304+0000","last_became_active":"2024-09-13T22:55:38.135406+0000","last_became_peered":"2024-09-13T22:55:38.135406+0000","last_unstale":"2024-09-13T23:01:45.248304+0000","last_undegraded":"2024-09-13T23:01:45.248304+0000","last_fullsized":"2024-09-13T23:01:45.248304+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.318722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.318722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:52.783710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":934,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.578381+0000","last_change":"2024-09-13T23:01:27.514568+0000","last_active":"2024-09-13T23:01:48.578381+0000","last_peered":"2024-09-13T23:01:48.578381+0000","last_clean":"2024-09-13T23:01:48.578381+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T23:01:48.578381+0000","last_undegraded":"2024-09-13T23:01:48.578381+0000","last_fullsized":"2024-09-13T23:01:48.578381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:27.514424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.514424+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:33.132445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":138,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.462769+0000","last_change":"2024-09-13T22:58:42.742636+0000","last_active":"2024-09-13T23:01:44.462769+0000","last_peered":"2024-09-13T23:01:44.462769+0000","last_clean":"2024-09-13T23:01:44.462769+0000","last_became_active":"2024-09-13T22:58:42.742229+0000","last_became_peered":"2024-09-13T22:58:42.742229+0000","last_unstale":"2024-09-13T23:01:44.462769+0000","last_undegraded":"2024-09-13T23:01:44.462769+0000","last_fullsized":"2024-09-13T23:01:44.462769+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.134261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.134261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":915,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.247923+0000","last_change":"2024-09-13T23:01:38.567264+0000","last_active":"2024-09-13T23:01:45.247923+0000","last_peered":"2024-09-13T23:01:45.247923+0000","last_clean":"2024-09-13T23:01:45.247923+0000","last_became_active":"2024-09-13T23:01:38.566551+0000","last_became_peered":"2024-09-13T23:01:38.566551+0000","last_unstale":"2024-09-13T23:01:45.247923+0000","last_undegraded":"2024-09-13T23:01:45.247923+0000","last_fullsized":"2024-09-13T23:01:45.247923+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:33.301478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:33.301478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:33.702750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":888,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196908+0000","last_change":"2024-09-13T22:58:42.749489+0000","last_active":"2024-09-13T23:01:48.196908+0000","last_peered":"2024-09-13T23:01:48.196908+0000","last_clean":"2024-09-13T23:01:48.196908+0000","last_became_active":"2024-09-13T22:58:42.749118+0000","last_became_peered":"2024-09-13T22:58:42.749118+0000","last_unstale":"2024-09-13T23:01:48.196908+0000","last_undegraded":"2024-09-13T23:01:48.196908+0000","last_fullsized":"2024-09-13T23:01:48.196908+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:59.732688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:57:59.732688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:49.093351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":877,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:45.249940+0000","last_change":"2024-09-13T23:01:35.255461+0000","last_active":"2024-09-13T23:01:45.249940+0000","last_peered":"2024-09-13T23:01:45.249940+0000","last_clean":"2024-09-13T23:01:45.249940+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T23:01:45.249940+0000","last_undegraded":"2024-09-13T23:01:45.249940+0000","last_fullsized":"2024-09-13T23:01:45.249940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T23:01:35.255256+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.255256+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.068806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":248,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:49.427859+0000","last_change":"2024-09-13T23:01:02.630716+0000","last_active":"2024-09-13T23:01:49.427859+0000","last_peered":"2024-09-13T23:01:49.427859+0000","last_clean":"2024-09-13T23:01:49.427859+0000","last_became_active":"2024-09-13T22:58:42.740500+0000","last_became_peered":"2024-09-13T22:58:42.740500+0000","last_unstale":"2024-09-13T23:01:49.427859+0000","last_undegraded":"2024-09-13T23:01:49.427859+0000","last_fullsized":"2024-09-13T23:01:49.427859+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:01:02.630615+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T23:01:02.630615+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:31.545469+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":934,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.577826+0000","last_change":"2024-09-13T23:01:01.488015+0000","last_active":"2024-09-13T23:01:48.577826+0000","last_peered":"2024-09-13T23:01:48.577826+0000","last_clean":"2024-09-13T23:01:48.577826+0000","last_became_active":"2024-09-13T22:58:42.746984+0000","last_became_peered":"2024-09-13T22:58:42.746984+0000","last_unstale":"2024-09-13T23:01:48.577826+0000","last_undegraded":"2024-09-13T23:01:48.577826+0000","last_fullsized":"2024-09-13T23:01:48.577826+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:01.487808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T23:01:01.487808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:21.618995+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"414'17","reported_seq":188,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.463018+0000","last_change":"2024-09-13T23:01:14.563397+0000","last_active":"2024-09-13T23:01:44.463018+0000","last_peered":"2024-09-13T23:01:44.463018+0000","last_clean":"2024-09-13T23:01:44.463018+0000","last_became_active":"2024-09-13T22:58:42.743192+0000","last_became_peered":"2024-09-13T22:58:42.743192+0000","last_unstale":"2024-09-13T23:01:44.463018+0000","last_undegraded":"2024-09-13T23:01:44.463018+0000","last_fullsized":"2024-09-13T23:01:44.463018+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'17","last_scrub_stamp":"2024-09-13T23:01:14.563131+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.563131+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.035856+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":910,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501652+0000","last_change":"2024-09-13T23:01:04.819037+0000","last_active":"2024-09-13T23:01:48.501652+0000","last_peered":"2024-09-13T23:01:48.501652+0000","last_clean":"2024-09-13T23:01:48.501652+0000","last_became_active":"2024-09-13T22:55:38.096392+0000","last_became_peered":"2024-09-13T22:55:38.096392+0000","last_unstale":"2024-09-13T23:01:48.501652+0000","last_undegraded":"2024-09-13T23:01:48.501652+0000","last_fullsized":"2024-09-13T23:01:48.501652+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:04.818930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:04.818930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:20.543539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":933,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196137+0000","last_change":"2024-09-13T23:01:29.240502+0000","last_active":"2024-09-13T23:01:48.196137+0000","last_peered":"2024-09-13T23:01:48.196137+0000","last_clean":"2024-09-13T23:01:48.196137+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T23:01:48.196137+0000","last_undegraded":"2024-09-13T23:01:48.196137+0000","last_fullsized":"2024-09-13T23:01:48.196137+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:29.240246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.240246+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.497246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":900,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196157+0000","last_change":"2024-09-13T23:01:13.190802+0000","last_active":"2024-09-13T23:01:48.196157+0000","last_peered":"2024-09-13T23:01:48.196157+0000","last_clean":"2024-09-13T23:01:48.196157+0000","last_became_active":"2024-09-13T22:58:42.748914+0000","last_became_peered":"2024-09-13T22:58:42.748914+0000","last_unstale":"2024-09-13T23:01:48.196157+0000","last_undegraded":"2024-09-13T23:01:48.196157+0000","last_fullsized":"2024-09-13T23:01:48.196157+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T23:01:13.190695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:13.190695+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.162269+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":897,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.333296+0000","last_change":"2024-09-13T23:01:21.888588+0000","last_active":"2024-09-13T23:01:44.333296+0000","last_peered":"2024-09-13T23:01:44.333296+0000","last_clean":"2024-09-13T23:01:44.333296+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T23:01:44.333296+0000","last_undegraded":"2024-09-13T23:01:44.333296+0000","last_fullsized":"2024-09-13T23:01:44.333296+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:21.888398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.888398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.335531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":138,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.463166+0000","last_change":"2024-09-13T22:58:42.750451+0000","last_active":"2024-09-13T23:01:44.463166+0000","last_peered":"2024-09-13T23:01:44.463166+0000","last_clean":"2024-09-13T23:01:44.463166+0000","last_became_active":"2024-09-13T22:58:42.748790+0000","last_became_peered":"2024-09-13T22:58:42.748790+0000","last_unstale":"2024-09-13T23:01:44.463166+0000","last_undegraded":"2024-09-13T23:01:44.463166+0000","last_fullsized":"2024-09-13T23:01:44.463166+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.025258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.025258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":146,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.462753+0000","last_change":"2024-09-13T23:01:23.585275+0000","last_active":"2024-09-13T23:01:44.462753+0000","last_peered":"2024-09-13T23:01:44.462753+0000","last_clean":"2024-09-13T23:01:44.462753+0000","last_became_active":"2024-09-13T22:58:42.746020+0000","last_became_peered":"2024-09-13T22:58:42.746020+0000","last_unstale":"2024-09-13T23:01:44.462753+0000","last_undegraded":"2024-09-13T23:01:44.462753+0000","last_fullsized":"2024-09-13T23:01:44.462753+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:23.585081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.585081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.925002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":953,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T23:01:44.332277+0000","last_change":"2024-09-13T22:58:12.352800+0000","last_active":"2024-09-13T23:01:44.332277+0000","last_peered":"2024-09-13T23:01:44.332277+0000","last_clean":"2024-09-13T23:01:44.332277+0000","last_became_active":"2024-09-13T22:55:38.138369+0000","last_became_peered":"2024-09-13T22:55:38.138369+0000","last_unstale":"2024-09-13T23:01:44.332277+0000","last_undegraded":"2024-09-13T23:01:44.332277+0000","last_fullsized":"2024-09-13T23:01:44.332277+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'11","last_scrub_stamp":"2024-09-13T22:58:12.352675+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.352675+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:01:47.816847+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":914,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.502008+0000","last_change":"2024-09-13T23:01:36.948975+0000","last_active":"2024-09-13T23:01:48.502008+0000","last_peered":"2024-09-13T23:01:48.502008+0000","last_clean":"2024-09-13T23:01:48.502008+0000","last_became_active":"2024-09-13T22:55:38.094235+0000","last_became_peered":"2024-09-13T22:55:38.094235+0000","last_unstale":"2024-09-13T23:01:48.502008+0000","last_undegraded":"2024-09-13T23:01:48.502008+0000","last_fullsized":"2024-09-13T23:01:48.502008+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:36.948688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:17.313617+0000","last_clean_scrub_stamp":"2024-09-13T23:01:36.948688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:02.234273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":898,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197977+0000","last_change":"2024-09-13T23:01:15.143950+0000","last_active":"2024-09-13T23:01:48.197977+0000","last_peered":"2024-09-13T23:01:48.197977+0000","last_clean":"2024-09-13T23:01:48.197977+0000","last_became_active":"2024-09-13T22:58:42.748249+0000","last_became_peered":"2024-09-13T22:58:42.748249+0000","last_unstale":"2024-09-13T23:01:48.197977+0000","last_undegraded":"2024-09-13T23:01:48.197977+0000","last_fullsized":"2024-09-13T23:01:48.197977+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T23:01:15.143836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:15.143836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.403472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":899,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.197039+0000","last_change":"2024-09-13T23:01:33.171325+0000","last_active":"2024-09-13T23:01:48.197039+0000","last_peered":"2024-09-13T23:01:48.197039+0000","last_clean":"2024-09-13T23:01:48.197039+0000","last_became_active":"2024-09-13T22:58:42.769318+0000","last_became_peered":"2024-09-13T22:58:42.769318+0000","last_unstale":"2024-09-13T23:01:48.197039+0000","last_undegraded":"2024-09-13T23:01:48.197039+0000","last_fullsized":"2024-09-13T23:01:48.197039+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:33.171231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:33.171231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.663204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":742,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.501305+0000","last_change":"2024-09-13T23:01:22.827073+0000","last_active":"2024-09-13T23:01:48.501305+0000","last_peered":"2024-09-13T23:01:48.501305+0000","last_clean":"2024-09-13T23:01:48.501305+0000","last_became_active":"2024-09-13T22:55:38.099550+0000","last_became_peered":"2024-09-13T22:55:38.099550+0000","last_unstale":"2024-09-13T23:01:48.501305+0000","last_undegraded":"2024-09-13T23:01:48.501305+0000","last_fullsized":"2024-09-13T23:01:48.501305+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:01:22.826820+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.826820+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.286016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":390,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T23:01:48.196573+0000","last_change":"2024-09-13T23:01:30.227188+0000","last_active":"2024-09-13T23:01:48.196573+0000","last_peered":"2024-09-13T23:01:48.196573+0000","last_clean":"2024-09-13T23:01:48.196573+0000","last_became_active":"2024-09-13T22:55:38.133425+0000","last_became_peered":"2024-09-13T22:55:38.133425+0000","last_unstale":"2024-09-13T23:01:48.196573+0000","last_undegraded":"2024-09-13T23:01:48.196573+0000","last_fullsized":"2024-09-13T23:01:48.196573+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.227003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.227003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:32.997184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1420,"num_read_kb":1415,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1555,"num_read_kb":1312,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":432,"num_bytes_recovered":8470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":12866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1063,"ondisk_log_size":1063,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809130,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722448,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93022704,"statfs":{"total":95995035648,"available":95255248896,"internally_reserved":0,"allocated":1306624,"data_stored":829614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136181,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762364,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982788,"statfs":{"total":95995035648,"available":95214374912,"internally_reserved":0,"allocated":954368,"data_stored":499722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528640,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668556,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93076596,"statfs":{"total":95995035648,"available":95310434304,"internally_reserved":0,"allocated":1564672,"data_stored":1097000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888394,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084628,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889288,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92855864,"statfs":{"total":95995035648,"available":95084404736,"internally_reserved":0,"allocated":1560576,"data_stored":1097375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378976,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614096,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93131056,"statfs":{"total":95995035648,"available":95366201344,"internally_reserved":0,"allocated":1241088,"data_stored":770072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575210,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999524,"kb_used_data":188112,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92745628,"statfs":{"total":95995035648,"available":94971523072,"internally_reserved":0,"allocated":192626688,"data_stored":192135373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738740,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822088,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92923064,"statfs":{"total":95995035648,"available":95153217536,"internally_reserved":0,"allocated":966656,"data_stored":501482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T23:01:53.541 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T23:01:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:53 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.952865+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub starts 2024-09-13T23:01:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:53 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.954853+0000 osd.1 (osd.1) 443 : cluster [DBG] 5.c scrub ok 2024-09-13T23:01:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:53 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.979758+0000 osd.3 (osd.3) 413 : cluster [DBG] 3.8 scrub starts 2024-09-13T23:01:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:53 smithi192 bash[24608]: cluster 2024-09-13T23:01:51.982949+0000 osd.3 (osd.3) 414 : cluster [DBG] 3.8 scrub ok 2024-09-13T23:01:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:53 smithi192 bash[24608]: audit 2024-09-13T23:01:52.648040+0000 mgr.y (mgr.14613) 897 : audit [DBG] from='client.16065 v1:172.21.15.178:0/9839544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:01:53.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.952865+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub starts 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.954853+0000 osd.1 (osd.1) 443 : cluster [DBG] 5.c scrub ok 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.979758+0000 osd.3 (osd.3) 413 : cluster [DBG] 3.8 scrub starts 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[29889]: cluster 2024-09-13T23:01:51.982949+0000 osd.3 (osd.3) 414 : cluster [DBG] 3.8 scrub ok 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[29889]: audit 2024-09-13T23:01:52.648040+0000 mgr.y (mgr.14613) 897 : audit [DBG] from='client.16065 v1:172.21.15.178:0/9839544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.952865+0000 osd.1 (osd.1) 442 : cluster [DBG] 5.c scrub starts 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.954853+0000 osd.1 (osd.1) 443 : cluster [DBG] 5.c scrub ok 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.979758+0000 osd.3 (osd.3) 413 : cluster [DBG] 3.8 scrub starts 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[22840]: cluster 2024-09-13T23:01:51.982949+0000 osd.3 (osd.3) 414 : cluster [DBG] 3.8 scrub ok 2024-09-13T23:01:53.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:53 smithi178 bash[22840]: audit 2024-09-13T23:01:52.648040+0000 mgr.y (mgr.14613) 897 : audit [DBG] from='client.16065 v1:172.21.15.178:0/9839544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:01:53.950 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:54.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:54 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.000950+0000 osd.1 (osd.1) 444 : cluster [DBG] 4.2 scrub starts 2024-09-13T23:01:54.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:54 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.004141+0000 osd.1 (osd.1) 445 : cluster [DBG] 4.2 scrub ok 2024-09-13T23:01:54.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:54 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.014182+0000 mgr.y (mgr.14613) 898 : cluster [DBG] pgmap v1086: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:01:54.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:54 smithi192 bash[24608]: cluster 2024-09-13T23:01:54.093464+0000 osd.7 (osd.7) 462 : cluster [DBG] 3.e scrub starts 2024-09-13T23:01:54.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:54 smithi192 bash[24608]: cluster 2024-09-13T23:01:54.097584+0000 osd.7 (osd.7) 463 : cluster [DBG] 3.e scrub ok 2024-09-13T23:01:54.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.000950+0000 osd.1 (osd.1) 444 : cluster [DBG] 4.2 scrub starts 2024-09-13T23:01:54.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.004141+0000 osd.1 (osd.1) 445 : cluster [DBG] 4.2 scrub ok 2024-09-13T23:01:54.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.014182+0000 mgr.y (mgr.14613) 898 : cluster [DBG] pgmap v1086: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:01:54.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[22840]: cluster 2024-09-13T23:01:54.093464+0000 osd.7 (osd.7) 462 : cluster [DBG] 3.e scrub starts 2024-09-13T23:01:54.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[22840]: cluster 2024-09-13T23:01:54.097584+0000 osd.7 (osd.7) 463 : cluster [DBG] 3.e scrub ok 2024-09-13T23:01:54.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.000950+0000 osd.1 (osd.1) 444 : cluster [DBG] 4.2 scrub starts 2024-09-13T23:01:54.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.004141+0000 osd.1 (osd.1) 445 : cluster [DBG] 4.2 scrub ok 2024-09-13T23:01:54.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.014182+0000 mgr.y (mgr.14613) 898 : cluster [DBG] pgmap v1086: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:01:54.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[29889]: cluster 2024-09-13T23:01:54.093464+0000 osd.7 (osd.7) 462 : cluster [DBG] 3.e scrub starts 2024-09-13T23:01:54.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:54 smithi178 bash[29889]: cluster 2024-09-13T23:01:54.097584+0000 osd.7 (osd.7) 463 : cluster [DBG] 3.e scrub ok 2024-09-13T23:01:54.769 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:01:54.769 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:01:55.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T23:01:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:55 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.792225+0000 osd.0 (osd.0) 429 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T23:01:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:55 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.795008+0000 osd.0 (osd.0) 430 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T23:01:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:55 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.959354+0000 osd.3 (osd.3) 415 : cluster [DBG] 2.1e scrub starts 2024-09-13T23:01:55.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:55 smithi192 bash[24608]: cluster 2024-09-13T23:01:53.960888+0000 osd.3 (osd.3) 416 : cluster [DBG] 2.1e scrub ok 2024-09-13T23:01:55.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.792225+0000 osd.0 (osd.0) 429 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.795008+0000 osd.0 (osd.0) 430 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.959354+0000 osd.3 (osd.3) 415 : cluster [DBG] 2.1e scrub starts 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[29889]: cluster 2024-09-13T23:01:53.960888+0000 osd.3 (osd.3) 416 : cluster [DBG] 2.1e scrub ok 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.792225+0000 osd.0 (osd.0) 429 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.795008+0000 osd.0 (osd.0) 430 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.959354+0000 osd.3 (osd.3) 415 : cluster [DBG] 2.1e scrub starts 2024-09-13T23:01:55.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:55 smithi178 bash[22840]: cluster 2024-09-13T23:01:53.960888+0000 osd.3 (osd.3) 416 : cluster [DBG] 2.1e scrub ok 2024-09-13T23:01:56.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:01:56 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:01:56.410 INFO:teuthology.orchestra.run.smithi178.stderr:nodeep-scrub is unset 2024-09-13T23:01:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:56 smithi192 bash[24608]: cluster 2024-09-13T23:01:55.015617+0000 mgr.y (mgr.14613) 899 : cluster [DBG] pgmap v1087: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:01:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:56 smithi192 bash[24608]: audit 2024-09-13T23:01:55.724135+0000 mon.c (mon.2) 88 : audit [INF] from='client.? v1:172.21.15.178:0/300513603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:56 smithi192 bash[24608]: audit 2024-09-13T23:01:55.724699+0000 mon.a (mon.0) 1853 : audit [INF] from='client.25973 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:56 smithi178 bash[29889]: cluster 2024-09-13T23:01:55.015617+0000 mgr.y (mgr.14613) 899 : cluster [DBG] pgmap v1087: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:01:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:56 smithi178 bash[29889]: audit 2024-09-13T23:01:55.724135+0000 mon.c (mon.2) 88 : audit [INF] from='client.? v1:172.21.15.178:0/300513603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:56 smithi178 bash[29889]: audit 2024-09-13T23:01:55.724699+0000 mon.a (mon.0) 1853 : audit [INF] from='client.25973 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:56.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:56 smithi178 bash[22840]: cluster 2024-09-13T23:01:55.015617+0000 mgr.y (mgr.14613) 899 : cluster [DBG] pgmap v1087: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:01:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:56 smithi178 bash[22840]: audit 2024-09-13T23:01:55.724135+0000 mon.c (mon.2) 88 : audit [INF] from='client.? v1:172.21.15.178:0/300513603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:56 smithi178 bash[22840]: audit 2024-09-13T23:01:55.724699+0000 mon.a (mon.0) 1853 : audit [INF] from='client.25973 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-13T23:01:57.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:55.754443+0000 osd.0 (osd.0) 431 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T23:01:57.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:55.756386+0000 osd.0 (osd.0) 432 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T23:01:57.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: audit 2024-09-13T23:01:56.071882+0000 mgr.y (mgr.14613) 900 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:56.306856+0000 osd.6 (osd.6) 328 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:56.309131+0000 osd.6 (osd.6) 329 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: audit 2024-09-13T23:01:56.405766+0000 mon.a (mon.0) 1854 : audit [INF] from='client.25973 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:56.412770+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:56.627927+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[22840]: cluster 2024-09-13T23:01:56.630760+0000 osd.5 (osd.5) 490 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:55.754443+0000 osd.0 (osd.0) 431 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:55.756386+0000 osd.0 (osd.0) 432 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: audit 2024-09-13T23:01:56.071882+0000 mgr.y (mgr.14613) 900 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:57.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:56.306856+0000 osd.6 (osd.6) 328 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:01:57.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:56.309131+0000 osd.6 (osd.6) 329 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:01:57.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: audit 2024-09-13T23:01:56.405766+0000 mon.a (mon.0) 1854 : audit [INF] from='client.25973 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T23:01:57.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:56.412770+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-09-13T23:01:57.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:56.627927+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:01:57.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:57 smithi178 bash[29889]: cluster 2024-09-13T23:01:56.630760+0000 osd.5 (osd.5) 490 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:01:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:55.754443+0000 osd.0 (osd.0) 431 : cluster [DBG] 2.1f deep-scrub starts 2024-09-13T23:01:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:55.756386+0000 osd.0 (osd.0) 432 : cluster [DBG] 2.1f deep-scrub ok 2024-09-13T23:01:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: audit 2024-09-13T23:01:56.071882+0000 mgr.y (mgr.14613) 900 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:01:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:56.306856+0000 osd.6 (osd.6) 328 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:01:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:56.309131+0000 osd.6 (osd.6) 329 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:01:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: audit 2024-09-13T23:01:56.405766+0000 mon.a (mon.0) 1854 : audit [INF] from='client.25973 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T23:01:57.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:56.412770+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-09-13T23:01:57.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:56.627927+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:01:57.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:57 smithi192 bash[24608]: cluster 2024-09-13T23:01:56.630760+0000 osd.5 (osd.5) 490 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:01:58.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:58 smithi178 bash[29889]: cluster 2024-09-13T23:01:57.016419+0000 mgr.y (mgr.14613) 901 : cluster [DBG] pgmap v1089: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:01:58.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:58 smithi178 bash[29889]: cluster 2024-09-13T23:01:58.202470+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-09-13T23:01:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:58 smithi178 bash[22840]: cluster 2024-09-13T23:01:57.016419+0000 mgr.y (mgr.14613) 901 : cluster [DBG] pgmap v1089: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:01:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:58 smithi178 bash[22840]: cluster 2024-09-13T23:01:58.202470+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-09-13T23:01:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:58 smithi192 bash[24608]: cluster 2024-09-13T23:01:57.016419+0000 mgr.y (mgr.14613) 901 : cluster [DBG] pgmap v1089: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:01:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:58 smithi192 bash[24608]: cluster 2024-09-13T23:01:58.202470+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-09-13T23:01:59.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:57.749983+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:57.752452+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:57.939944+0000 osd.3 (osd.3) 417 : cluster [DBG] 6.6 deep-scrub starts 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:57.942995+0000 osd.3 (osd.3) 418 : cluster [DBG] 6.6 deep-scrub ok 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:58.365908+0000 osd.6 (osd.6) 330 : cluster [DBG] 3.a scrub starts 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:58.369489+0000 osd.6 (osd.6) 331 : cluster [DBG] 3.a scrub ok 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:58.613786+0000 osd.5 (osd.5) 491 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[29889]: cluster 2024-09-13T23:01:58.616319+0000 osd.5 (osd.5) 492 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:57.749983+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:57.752452+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:01:59.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:57.939944+0000 osd.3 (osd.3) 417 : cluster [DBG] 6.6 deep-scrub starts 2024-09-13T23:01:59.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:57.942995+0000 osd.3 (osd.3) 418 : cluster [DBG] 6.6 deep-scrub ok 2024-09-13T23:01:59.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:58.365908+0000 osd.6 (osd.6) 330 : cluster [DBG] 3.a scrub starts 2024-09-13T23:01:59.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:58.369489+0000 osd.6 (osd.6) 331 : cluster [DBG] 3.a scrub ok 2024-09-13T23:01:59.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:58.613786+0000 osd.5 (osd.5) 491 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:01:59.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[22840]: cluster 2024-09-13T23:01:58.616319+0000 osd.5 (osd.5) 492 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:01:59.771 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:01:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:57.749983+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:01:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:57.752452+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:01:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:57.939944+0000 osd.3 (osd.3) 417 : cluster [DBG] 6.6 deep-scrub starts 2024-09-13T23:01:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:57.942995+0000 osd.3 (osd.3) 418 : cluster [DBG] 6.6 deep-scrub ok 2024-09-13T23:01:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:58.365908+0000 osd.6 (osd.6) 330 : cluster [DBG] 3.a scrub starts 2024-09-13T23:01:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:58.369489+0000 osd.6 (osd.6) 331 : cluster [DBG] 3.a scrub ok 2024-09-13T23:01:59.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:58.613786+0000 osd.5 (osd.5) 491 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:01:59.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:01:59 smithi192 bash[24608]: cluster 2024-09-13T23:01:58.616319+0000 osd.5 (osd.5) 492 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:02:00.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:01:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:01:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:02:00.359 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:02:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:00 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.017313+0000 mgr.y (mgr.14613) 902 : cluster [DBG] pgmap v1091: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:00 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.571010+0000 osd.5 (osd.5) 493 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T23:02:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:00 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.573394+0000 osd.5 (osd.5) 494 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T23:02:00.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:00 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.017313+0000 mgr.y (mgr.14613) 902 : cluster [DBG] pgmap v1091: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:00.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:00 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.571010+0000 osd.5 (osd.5) 493 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T23:02:00.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:00 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.573394+0000 osd.5 (osd.5) 494 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T23:02:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:00 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.017313+0000 mgr.y (mgr.14613) 902 : cluster [DBG] pgmap v1091: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:00 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.571010+0000 osd.5 (osd.5) 493 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T23:02:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:00 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.573394+0000 osd.5 (osd.5) 494 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T23:02:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.671966+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.672061+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.738748+0000 osd.0 (osd.0) 435 : cluster [DBG] 6.0 deep-scrub starts 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.741149+0000 osd.0 (osd.0) 436 : cluster [DBG] 6.0 deep-scrub ok 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.910428+0000 osd.3 (osd.3) 419 : cluster [DBG] 6.f scrub starts 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:01:59.912735+0000 osd.3 (osd.3) 420 : cluster [DBG] 6.f scrub ok 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:00.322960+0000 osd.6 (osd.6) 334 : cluster [DBG] 2.14 scrub starts 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:00.325247+0000 osd.6 (osd.6) 335 : cluster [DBG] 2.14 scrub ok 2024-09-13T23:02:01.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:00.437126+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:01.207051+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:01.210364+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.671966+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.672061+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.738748+0000 osd.0 (osd.0) 435 : cluster [DBG] 6.0 deep-scrub starts 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.741149+0000 osd.0 (osd.0) 436 : cluster [DBG] 6.0 deep-scrub ok 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.910428+0000 osd.3 (osd.3) 419 : cluster [DBG] 6.f scrub starts 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:01:59.912735+0000 osd.3 (osd.3) 420 : cluster [DBG] 6.f scrub ok 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:00.322960+0000 osd.6 (osd.6) 334 : cluster [DBG] 2.14 scrub starts 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:00.325247+0000 osd.6 (osd.6) 335 : cluster [DBG] 2.14 scrub ok 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:00.437126+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-09-13T23:02:01.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:01.207051+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:02:01.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:01.210364+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:02:01.721 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T23:02:01.723 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T23:02:01.726 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:02:01.726 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:02:01.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.671966+0000 osd.6 (osd.6) 332 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:01.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.672061+0000 osd.6 (osd.6) 333 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:01.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.738748+0000 osd.0 (osd.0) 435 : cluster [DBG] 6.0 deep-scrub starts 2024-09-13T23:02:01.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.741149+0000 osd.0 (osd.0) 436 : cluster [DBG] 6.0 deep-scrub ok 2024-09-13T23:02:01.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.910428+0000 osd.3 (osd.3) 419 : cluster [DBG] 6.f scrub starts 2024-09-13T23:02:01.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:01:59.912735+0000 osd.3 (osd.3) 420 : cluster [DBG] 6.f scrub ok 2024-09-13T23:02:01.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:00.322960+0000 osd.6 (osd.6) 334 : cluster [DBG] 2.14 scrub starts 2024-09-13T23:02:01.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:00.325247+0000 osd.6 (osd.6) 335 : cluster [DBG] 2.14 scrub ok 2024-09-13T23:02:01.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:00.437126+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-09-13T23:02:01.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:01.207051+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:02:01.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:01.210364+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:02:02.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T23:02:02.569 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":1093,"stamp":"2024-09-13T23:02:01.017670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3022,"num_read_kb":2774,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":9059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2629,"ondisk_log_size":2629,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":429,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5420944,"kb_used_data":138108,"kb_used_omap":10,"kb_used_meta":5282677,"kb_avail":650795120,"statfs":{"total":671965249536,"available":666414202880,"internally_reserved":0,"allocated":141422592,"data_stored":138047778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5409461355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003217"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231748+0000","last_change":"2024-09-13T22:58:42.763562+0000","last_active":"2024-09-13T23:01:58.231748+0000","last_peered":"2024-09-13T23:01:58.231748+0000","last_clean":"2024-09-13T23:01:58.231748+0000","last_became_active":"2024-09-13T22:58:42.763279+0000","last_became_peered":"2024-09-13T22:58:42.763279+0000","last_unstale":"2024-09-13T23:01:58.231748+0000","last_undegraded":"2024-09-13T23:01:58.231748+0000","last_fullsized":"2024-09-13T23:01:58.231748+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.827330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.827330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":945,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.623175+0000","last_change":"2024-09-13T23:01:55.756591+0000","last_active":"2024-09-13T23:01:56.623175+0000","last_peered":"2024-09-13T23:01:56.623175+0000","last_clean":"2024-09-13T23:01:56.623175+0000","last_became_active":"2024-09-13T22:58:42.740638+0000","last_became_peered":"2024-09-13T22:58:42.740638+0000","last_unstale":"2024-09-13T23:01:56.623175+0000","last_undegraded":"2024-09-13T23:01:56.623175+0000","last_fullsized":"2024-09-13T23:01:56.623175+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:55.756398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:55.756398+0000","last_clean_scrub_stamp":"2024-09-13T23:01:55.756398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:13.602533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":976,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235638+0000","last_change":"2024-09-13T22:58:42.783796+0000","last_active":"2024-09-13T23:01:58.235638+0000","last_peered":"2024-09-13T23:01:58.235638+0000","last_clean":"2024-09-13T23:01:58.235638+0000","last_became_active":"2024-09-13T22:58:42.747843+0000","last_became_peered":"2024-09-13T22:58:42.747843+0000","last_unstale":"2024-09-13T23:01:58.235638+0000","last_undegraded":"2024-09-13T23:01:58.235638+0000","last_fullsized":"2024-09-13T23:01:58.235638+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'25","last_scrub_stamp":"2024-09-13T22:57:46.573001+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T22:57:46.573001+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":152,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231729+0000","last_change":"2024-09-13T23:01:22.578614+0000","last_active":"2024-09-13T23:01:58.231729+0000","last_peered":"2024-09-13T23:01:58.231729+0000","last_clean":"2024-09-13T23:01:58.231729+0000","last_became_active":"2024-09-13T22:58:42.745341+0000","last_became_peered":"2024-09-13T22:58:42.745341+0000","last_unstale":"2024-09-13T23:01:58.231729+0000","last_undegraded":"2024-09-13T23:01:58.231729+0000","last_fullsized":"2024-09-13T23:01:58.231729+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:22.578421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:55.779975+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.578421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:45.233250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235677+0000","last_change":"2024-09-13T23:01:53.960976+0000","last_active":"2024-09-13T23:01:58.235677+0000","last_peered":"2024-09-13T23:01:58.235677+0000","last_clean":"2024-09-13T23:01:58.235677+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T23:01:58.235677+0000","last_undegraded":"2024-09-13T23:01:58.235677+0000","last_fullsized":"2024-09-13T23:01:58.235677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:53.960892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:53.960892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:08.367935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":953,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622674+0000","last_change":"2024-09-13T23:01:22.902178+0000","last_active":"2024-09-13T23:01:56.622674+0000","last_peered":"2024-09-13T23:01:56.622674+0000","last_clean":"2024-09-13T23:01:56.622674+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T23:01:56.622674+0000","last_undegraded":"2024-09-13T23:01:56.622674+0000","last_fullsized":"2024-09-13T23:01:56.622674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:01:22.901958+0000","last_deep_scrub":"102'16","last_deep_scrub_stamp":"2024-09-13T22:39:08.754231+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.901958+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:39.278580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234732+0000","last_change":"2024-09-13T23:01:20.852046+0000","last_active":"2024-09-13T23:01:58.234732+0000","last_peered":"2024-09-13T23:01:58.234732+0000","last_clean":"2024-09-13T23:01:58.234732+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T23:01:58.234732+0000","last_undegraded":"2024-09-13T23:01:58.234732+0000","last_fullsized":"2024-09-13T23:01:58.234732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:20.851978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T23:01:20.851978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.169703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":410,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236026+0000","last_change":"2024-09-13T23:01:24.445111+0000","last_active":"2024-09-13T23:01:58.236026+0000","last_peered":"2024-09-13T23:01:58.236026+0000","last_clean":"2024-09-13T23:01:58.236026+0000","last_became_active":"2024-09-13T22:55:38.143783+0000","last_became_peered":"2024-09-13T22:55:38.143783+0000","last_unstale":"2024-09-13T23:01:58.236026+0000","last_undegraded":"2024-09-13T23:01:58.236026+0000","last_fullsized":"2024-09-13T23:01:58.236026+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:24.444998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:53.957628+0000","last_clean_scrub_stamp":"2024-09-13T23:01:24.444998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:27.272769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":904,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235974+0000","last_change":"2024-09-13T23:01:12.294658+0000","last_active":"2024-09-13T23:01:58.235974+0000","last_peered":"2024-09-13T23:01:58.235974+0000","last_clean":"2024-09-13T23:01:58.235974+0000","last_became_active":"2024-09-13T22:58:42.770668+0000","last_became_peered":"2024-09-13T22:58:42.770668+0000","last_unstale":"2024-09-13T23:01:58.235974+0000","last_undegraded":"2024-09-13T23:01:58.235974+0000","last_fullsized":"2024-09-13T23:01:58.235974+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:12.294461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:12.294461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.302281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":1082,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233826+0000","last_change":"2024-09-13T23:01:09.461488+0000","last_active":"2024-09-13T23:01:58.233826+0000","last_peered":"2024-09-13T23:01:58.233826+0000","last_clean":"2024-09-13T23:01:58.233826+0000","last_became_active":"2024-09-13T22:58:42.728658+0000","last_became_peered":"2024-09-13T22:58:42.728658+0000","last_unstale":"2024-09-13T23:01:58.233826+0000","last_undegraded":"2024-09-13T23:01:58.233826+0000","last_fullsized":"2024-09-13T23:01:58.233826+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:01:09.461410+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T23:01:09.461410+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:20.730973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234260+0000","last_change":"2024-09-13T23:01:10.320782+0000","last_active":"2024-09-13T23:01:58.234260+0000","last_peered":"2024-09-13T23:01:58.234260+0000","last_clean":"2024-09-13T23:01:58.234260+0000","last_became_active":"2024-09-13T22:55:38.151599+0000","last_became_peered":"2024-09-13T22:55:38.151599+0000","last_unstale":"2024-09-13T23:01:58.234260+0000","last_undegraded":"2024-09-13T23:01:58.234260+0000","last_fullsized":"2024-09-13T23:01:58.234260+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:10.320675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T23:01:10.320675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.237394+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":896,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.616464+0000","last_change":"2024-09-13T23:01:58.616517+0000","last_active":"2024-09-13T23:01:58.616464+0000","last_peered":"2024-09-13T23:01:58.616464+0000","last_clean":"2024-09-13T23:01:58.616464+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T23:01:58.616464+0000","last_undegraded":"2024-09-13T23:01:58.616464+0000","last_fullsized":"2024-09-13T23:01:58.616464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:58.616332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T23:01:58.616332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:05.260207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235614+0000","last_change":"2024-09-13T23:01:08.231928+0000","last_active":"2024-09-13T23:01:58.235614+0000","last_peered":"2024-09-13T23:01:58.235614+0000","last_clean":"2024-09-13T23:01:58.235614+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T23:01:58.235614+0000","last_undegraded":"2024-09-13T23:01:58.235614+0000","last_fullsized":"2024-09-13T23:01:58.235614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:08.231846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:08.231846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:18.625381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"397'24","reported_seq":465,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234525+0000","last_change":"2024-09-13T22:58:42.802871+0000","last_active":"2024-09-13T23:01:58.234525+0000","last_peered":"2024-09-13T23:01:58.234525+0000","last_clean":"2024-09-13T23:01:58.234525+0000","last_became_active":"2024-09-13T22:58:42.768934+0000","last_became_peered":"2024-09-13T22:58:42.768934+0000","last_unstale":"2024-09-13T23:01:58.234525+0000","last_undegraded":"2024-09-13T23:01:58.234525+0000","last_fullsized":"2024-09-13T23:01:58.234525+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T22:58:19.363461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:19.363461+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234969+0000","last_change":"2024-09-13T23:01:56.631029+0000","last_active":"2024-09-13T23:01:58.234969+0000","last_peered":"2024-09-13T23:01:58.234969+0000","last_clean":"2024-09-13T23:01:58.234969+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T23:01:58.234969+0000","last_undegraded":"2024-09-13T23:01:58.234969+0000","last_fullsized":"2024-09-13T23:01:58.234969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:56.630769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T23:01:56.630769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:19.014811+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":895,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622288+0000","last_change":"2024-09-13T23:01:27.921506+0000","last_active":"2024-09-13T23:01:56.622288+0000","last_peered":"2024-09-13T23:01:56.622288+0000","last_clean":"2024-09-13T23:01:56.622288+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T23:01:56.622288+0000","last_undegraded":"2024-09-13T23:01:56.622288+0000","last_fullsized":"2024-09-13T23:01:56.622288+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:27.921318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.921318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.142579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":955,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230112+0000","last_change":"2024-09-13T23:01:06.499430+0000","last_active":"2024-09-13T23:01:58.230112+0000","last_peered":"2024-09-13T23:01:58.230112+0000","last_clean":"2024-09-13T23:01:58.230112+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T23:01:58.230112+0000","last_undegraded":"2024-09-13T23:01:58.230112+0000","last_fullsized":"2024-09-13T23:01:58.230112+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:06.499317+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:06.499317+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:35.638341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":172,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.369565+0000","last_change":"2024-09-13T23:01:58.369604+0000","last_active":"2024-09-13T23:01:58.369565+0000","last_peered":"2024-09-13T23:01:58.369565+0000","last_clean":"2024-09-13T23:01:58.369565+0000","last_became_active":"2024-09-13T22:58:42.749141+0000","last_became_peered":"2024-09-13T22:58:42.749141+0000","last_unstale":"2024-09-13T23:01:58.369565+0000","last_undegraded":"2024-09-13T23:01:58.369565+0000","last_fullsized":"2024-09-13T23:01:58.369565+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'60","last_scrub_stamp":"2024-09-13T23:01:58.369493+0000","last_deep_scrub":"228'44","last_deep_scrub_stamp":"2024-09-13T22:54:34.302937+0000","last_clean_scrub_stamp":"2024-09-13T23:01:58.369493+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:12.102598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237277+0000","last_change":"2024-09-13T22:58:16.344650+0000","last_active":"2024-09-13T23:01:58.237277+0000","last_peered":"2024-09-13T23:01:58.237277+0000","last_clean":"2024-09-13T23:01:58.237277+0000","last_became_active":"2024-09-13T22:55:38.140796+0000","last_became_peered":"2024-09-13T22:55:38.140796+0000","last_unstale":"2024-09-13T23:01:58.237277+0000","last_undegraded":"2024-09-13T23:01:58.237277+0000","last_fullsized":"2024-09-13T23:01:58.237277+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:16.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T22:58:16.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":931,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233070+0000","last_change":"2024-09-13T23:01:51.955080+0000","last_active":"2024-09-13T23:01:58.233070+0000","last_peered":"2024-09-13T23:01:58.233070+0000","last_clean":"2024-09-13T23:01:58.233070+0000","last_became_active":"2024-09-13T22:58:42.743564+0000","last_became_peered":"2024-09-13T22:58:42.743564+0000","last_unstale":"2024-09-13T23:01:58.233070+0000","last_undegraded":"2024-09-13T23:01:58.233070+0000","last_fullsized":"2024-09-13T23:01:58.233070+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:51.954863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_clean_scrub_stamp":"2024-09-13T23:01:51.954863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:17.843094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":818,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233840+0000","last_change":"2024-09-13T22:58:42.748052+0000","last_active":"2024-09-13T23:01:58.233840+0000","last_peered":"2024-09-13T23:01:58.233840+0000","last_clean":"2024-09-13T23:01:58.233840+0000","last_became_active":"2024-09-13T22:58:42.747455+0000","last_became_peered":"2024-09-13T22:58:42.747455+0000","last_unstale":"2024-09-13T23:01:58.233840+0000","last_undegraded":"2024-09-13T23:01:58.233840+0000","last_fullsized":"2024-09-13T23:01:58.233840+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:56:58.481749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:56:58.481749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":1164,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232688+0000","last_change":"2024-09-13T22:58:02.221879+0000","last_active":"2024-09-13T23:01:58.232688+0000","last_peered":"2024-09-13T23:01:58.232688+0000","last_clean":"2024-09-13T23:01:58.232688+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T23:01:58.232688+0000","last_undegraded":"2024-09-13T23:01:58.232688+0000","last_fullsized":"2024-09-13T23:01:58.232688+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:02.221654+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.221654+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":974,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232428+0000","last_change":"2024-09-13T23:01:18.133842+0000","last_active":"2024-09-13T23:01:58.232428+0000","last_peered":"2024-09-13T23:01:58.232428+0000","last_clean":"2024-09-13T23:01:58.232428+0000","last_became_active":"2024-09-13T22:55:38.095029+0000","last_became_peered":"2024-09-13T22:55:38.095029+0000","last_unstale":"2024-09-13T23:01:58.232428+0000","last_undegraded":"2024-09-13T23:01:58.232428+0000","last_fullsized":"2024-09-13T23:01:58.232428+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:01:18.132748+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T23:01:18.132748+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:34.930897+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234141+0000","last_change":"2024-09-13T23:01:00.694314+0000","last_active":"2024-09-13T23:01:58.234141+0000","last_peered":"2024-09-13T23:01:58.234141+0000","last_clean":"2024-09-13T23:01:58.234141+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T23:01:58.234141+0000","last_undegraded":"2024-09-13T23:01:58.234141+0000","last_fullsized":"2024-09-13T23:01:58.234141+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:00.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:00.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:25.230076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"258'12","reported_seq":659,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235176+0000","last_change":"2024-09-13T23:01:52.082294+0000","last_active":"2024-09-13T23:01:58.235176+0000","last_peered":"2024-09-13T23:01:58.235176+0000","last_clean":"2024-09-13T23:01:58.235176+0000","last_became_active":"2024-09-13T22:49:45.642242+0000","last_became_peered":"2024-09-13T22:49:45.642242+0000","last_unstale":"2024-09-13T23:01:58.235176+0000","last_undegraded":"2024-09-13T23:01:58.235176+0000","last_fullsized":"2024-09-13T23:01:58.235176+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'12","last_scrub_stamp":"2024-09-13T23:01:52.082095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:52.082095+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:13.634264+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":410,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231140+0000","last_change":"2024-09-13T23:01:17.846052+0000","last_active":"2024-09-13T23:01:58.231140+0000","last_peered":"2024-09-13T23:01:58.231140+0000","last_clean":"2024-09-13T23:01:58.231140+0000","last_became_active":"2024-09-13T22:55:38.103505+0000","last_became_peered":"2024-09-13T22:55:38.103505+0000","last_unstale":"2024-09-13T23:01:58.231140+0000","last_undegraded":"2024-09-13T23:01:58.231140+0000","last_fullsized":"2024-09-13T23:01:58.231140+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:17.845954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:17.845954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:25.741237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":981,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235755+0000","last_change":"2024-09-13T23:01:50.630718+0000","last_active":"2024-09-13T23:01:58.235755+0000","last_peered":"2024-09-13T23:01:58.235755+0000","last_clean":"2024-09-13T23:01:58.235755+0000","last_became_active":"2024-09-13T22:58:45.828166+0000","last_became_peered":"2024-09-13T22:58:45.828166+0000","last_unstale":"2024-09-13T23:01:58.235755+0000","last_undegraded":"2024-09-13T23:01:58.235755+0000","last_fullsized":"2024-09-13T23:01:58.235755+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:50.630398+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:50.630398+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:58.807559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"397'39","reported_seq":950,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231588+0000","last_change":"2024-09-13T23:01:51.983052+0000","last_active":"2024-09-13T23:01:58.231588+0000","last_peered":"2024-09-13T23:01:58.231588+0000","last_clean":"2024-09-13T23:01:58.231588+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T23:01:58.231588+0000","last_undegraded":"2024-09-13T23:01:58.231588+0000","last_fullsized":"2024-09-13T23:01:58.231588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'39","last_scrub_stamp":"2024-09-13T23:01:51.982954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:51.982954+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:14.975535+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235144+0000","last_change":"2024-09-13T23:01:12.899149+0000","last_active":"2024-09-13T23:01:58.235144+0000","last_peered":"2024-09-13T23:01:58.235144+0000","last_clean":"2024-09-13T23:01:58.235144+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T23:01:58.235144+0000","last_undegraded":"2024-09-13T23:01:58.235144+0000","last_fullsized":"2024-09-13T23:01:58.235144+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:12.899007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:12.899007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.957521+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"351'15","reported_seq":620,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231259+0000","last_change":"2024-09-13T23:01:18.436735+0000","last_active":"2024-09-13T23:01:58.231259+0000","last_peered":"2024-09-13T23:01:58.231259+0000","last_clean":"2024-09-13T23:01:58.231259+0000","last_became_active":"2024-09-13T22:55:38.106061+0000","last_became_peered":"2024-09-13T22:55:38.106061+0000","last_unstale":"2024-09-13T23:01:58.231259+0000","last_undegraded":"2024-09-13T23:01:58.231259+0000","last_fullsized":"2024-09-13T23:01:58.231259+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'15","last_scrub_stamp":"2024-09-13T23:01:18.436650+0000","last_deep_scrub":"102'12","last_deep_scrub_stamp":"2024-09-13T22:39:12.592134+0000","last_clean_scrub_stamp":"2024-09-13T23:01:18.436650+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:44.225039+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233327+0000","last_change":"2024-09-13T23:01:23.410605+0000","last_active":"2024-09-13T23:01:58.233327+0000","last_peered":"2024-09-13T23:01:58.233327+0000","last_clean":"2024-09-13T23:01:58.233327+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T23:01:58.233327+0000","last_undegraded":"2024-09-13T23:01:58.233327+0000","last_fullsized":"2024-09-13T23:01:58.233327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:23.410379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:53.003468+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.410379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.175055+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":502,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235571+0000","last_change":"2024-09-13T23:01:26.610248+0000","last_active":"2024-09-13T23:01:58.235571+0000","last_peered":"2024-09-13T23:01:58.235571+0000","last_clean":"2024-09-13T23:01:58.235571+0000","last_became_active":"2024-09-13T22:55:38.124814+0000","last_became_peered":"2024-09-13T22:55:38.124814+0000","last_unstale":"2024-09-13T23:01:58.235571+0000","last_undegraded":"2024-09-13T23:01:58.235571+0000","last_fullsized":"2024-09-13T23:01:58.235571+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T23:01:26.610091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.610091+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:49.319333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":886,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232075+0000","last_change":"2024-09-13T22:57:51.208483+0000","last_active":"2024-09-13T23:01:58.232075+0000","last_peered":"2024-09-13T23:01:58.232075+0000","last_clean":"2024-09-13T23:01:58.232075+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T23:01:58.232075+0000","last_undegraded":"2024-09-13T23:01:58.232075+0000","last_fullsized":"2024-09-13T23:01:58.232075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:51.208413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:51.208413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":720,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237334+0000","last_change":"2024-09-13T23:01:23.313660+0000","last_active":"2024-09-13T23:01:58.237334+0000","last_peered":"2024-09-13T23:01:58.237334+0000","last_clean":"2024-09-13T23:01:58.237334+0000","last_became_active":"2024-09-13T22:58:42.746002+0000","last_became_peered":"2024-09-13T22:58:42.746002+0000","last_unstale":"2024-09-13T23:01:58.237334+0000","last_undegraded":"2024-09-13T23:01:58.237334+0000","last_fullsized":"2024-09-13T23:01:58.237334+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:23.313470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:02.793958+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.313470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:43.384132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":884,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233286+0000","last_change":"2024-09-13T22:58:42.746884+0000","last_active":"2024-09-13T23:01:58.233286+0000","last_peered":"2024-09-13T23:01:58.233286+0000","last_clean":"2024-09-13T23:01:58.233286+0000","last_became_active":"2024-09-13T22:58:42.746385+0000","last_became_peered":"2024-09-13T22:58:42.746385+0000","last_unstale":"2024-09-13T23:01:58.233286+0000","last_undegraded":"2024-09-13T23:01:58.233286+0000","last_fullsized":"2024-09-13T23:01:58.233286+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.718269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":937,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231894+0000","last_change":"2024-09-13T22:58:18.833271+0000","last_active":"2024-09-13T23:01:58.231894+0000","last_peered":"2024-09-13T23:01:58.231894+0000","last_clean":"2024-09-13T23:01:58.231894+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T23:01:58.231894+0000","last_undegraded":"2024-09-13T23:01:58.231894+0000","last_fullsized":"2024-09-13T23:01:58.231894+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:58:18.833192+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T22:58:18.833192+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":1019,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622518+0000","last_change":"2024-09-13T23:01:34.915626+0000","last_active":"2024-09-13T23:01:56.622518+0000","last_peered":"2024-09-13T23:01:56.622518+0000","last_clean":"2024-09-13T23:01:56.622518+0000","last_became_active":"2024-09-13T22:55:38.135796+0000","last_became_peered":"2024-09-13T22:55:38.135796+0000","last_unstale":"2024-09-13T23:01:56.622518+0000","last_undegraded":"2024-09-13T23:01:56.622518+0000","last_fullsized":"2024-09-13T23:01:56.622518+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'36","last_scrub_stamp":"2024-09-13T23:01:34.915514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:34.915514+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:52.880988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236502+0000","last_change":"2024-09-13T23:01:56.309330+0000","last_active":"2024-09-13T23:01:58.236502+0000","last_peered":"2024-09-13T23:01:58.236502+0000","last_clean":"2024-09-13T23:01:58.236502+0000","last_became_active":"2024-09-13T22:58:42.747048+0000","last_became_peered":"2024-09-13T22:58:42.747048+0000","last_unstale":"2024-09-13T23:01:58.236502+0000","last_undegraded":"2024-09-13T23:01:58.236502+0000","last_fullsized":"2024-09-13T23:01:58.236502+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:56.309144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:56.309144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:18.792324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":722,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231774+0000","last_change":"2024-09-13T22:58:42.755687+0000","last_active":"2024-09-13T23:01:58.231774+0000","last_peered":"2024-09-13T23:01:58.231774+0000","last_clean":"2024-09-13T23:01:58.231774+0000","last_became_active":"2024-09-13T22:58:42.721344+0000","last_became_peered":"2024-09-13T22:58:42.721344+0000","last_unstale":"2024-09-13T23:01:58.231774+0000","last_undegraded":"2024-09-13T23:01:58.231774+0000","last_fullsized":"2024-09-13T23:01:58.231774+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T22:58:11.043740+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T22:46:07.018490+0000","last_clean_scrub_stamp":"2024-09-13T22:58:11.043740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"351'14","reported_seq":1212,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230515+0000","last_change":"2024-09-13T22:58:02.750352+0000","last_active":"2024-09-13T23:01:58.230515+0000","last_peered":"2024-09-13T23:01:58.230515+0000","last_clean":"2024-09-13T23:01:58.230515+0000","last_became_active":"2024-09-13T22:55:38.087590+0000","last_became_peered":"2024-09-13T22:55:38.087590+0000","last_unstale":"2024-09-13T23:01:58.230515+0000","last_undegraded":"2024-09-13T23:01:58.230515+0000","last_fullsized":"2024-09-13T23:01:58.230515+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'14","last_scrub_stamp":"2024-09-13T22:58:02.750208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:02.750208+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":886,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234564+0000","last_change":"2024-09-13T22:58:42.771114+0000","last_active":"2024-09-13T23:01:58.234564+0000","last_peered":"2024-09-13T23:01:58.234564+0000","last_clean":"2024-09-13T23:01:58.234564+0000","last_became_active":"2024-09-13T22:58:42.770701+0000","last_became_peered":"2024-09-13T22:58:42.770701+0000","last_unstale":"2024-09-13T23:01:58.234564+0000","last_undegraded":"2024-09-13T23:01:58.234564+0000","last_fullsized":"2024-09-13T23:01:58.234564+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.267276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.267276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":772,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236417+0000","last_change":"2024-09-13T23:01:31.604209+0000","last_active":"2024-09-13T23:01:58.236417+0000","last_peered":"2024-09-13T23:01:58.236417+0000","last_clean":"2024-09-13T23:01:58.236417+0000","last_became_active":"2024-09-13T22:58:45.843410+0000","last_became_peered":"2024-09-13T22:58:45.843410+0000","last_unstale":"2024-09-13T23:01:58.236417+0000","last_undegraded":"2024-09-13T23:01:58.236417+0000","last_fullsized":"2024-09-13T23:01:58.236417+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:31.604112+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:01:31.604112+0000","last_clean_scrub_stamp":"2024-09-13T23:01:31.604112+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:38.154528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":925,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234884+0000","last_change":"2024-09-13T22:56:59.524045+0000","last_active":"2024-09-13T23:01:58.234884+0000","last_peered":"2024-09-13T23:01:58.234884+0000","last_clean":"2024-09-13T23:01:58.234884+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T23:01:58.234884+0000","last_undegraded":"2024-09-13T23:01:58.234884+0000","last_fullsized":"2024-09-13T23:01:58.234884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'22","last_scrub_stamp":"2024-09-13T22:56:59.523938+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T22:56:59.523938+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":962,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231633+0000","last_change":"2024-09-13T23:01:27.810222+0000","last_active":"2024-09-13T23:01:58.231633+0000","last_peered":"2024-09-13T23:01:58.231633+0000","last_clean":"2024-09-13T23:01:58.231633+0000","last_became_active":"2024-09-13T22:58:42.742912+0000","last_became_peered":"2024-09-13T22:58:42.742912+0000","last_unstale":"2024-09-13T23:01:58.231633+0000","last_undegraded":"2024-09-13T23:01:58.231633+0000","last_fullsized":"2024-09-13T23:01:58.231633+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T23:01:27.810020+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.810020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:45.023945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":2323,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230977+0000","last_change":"2024-09-13T22:57:55.678130+0000","last_active":"2024-09-13T23:01:58.230977+0000","last_peered":"2024-09-13T23:01:58.230977+0000","last_clean":"2024-09-13T23:01:58.230977+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T23:01:58.230977+0000","last_undegraded":"2024-09-13T23:01:58.230977+0000","last_fullsized":"2024-09-13T23:01:58.230977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T22:57:55.678034+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T22:57:55.678034+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1430,"num_read_kb":1425,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":401,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230649+0000","last_change":"2024-09-13T22:57:49.593322+0000","last_active":"2024-09-13T23:01:58.230649+0000","last_peered":"2024-09-13T23:01:58.230649+0000","last_clean":"2024-09-13T23:01:58.230649+0000","last_became_active":"2024-09-13T22:55:38.124054+0000","last_became_peered":"2024-09-13T22:55:38.124054+0000","last_unstale":"2024-09-13T23:01:58.230649+0000","last_undegraded":"2024-09-13T23:01:58.230649+0000","last_fullsized":"2024-09-13T23:01:58.230649+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:49.593224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:57:49.593224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":406,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230598+0000","last_change":"2024-09-13T23:01:19.136514+0000","last_active":"2024-09-13T23:01:58.230598+0000","last_peered":"2024-09-13T23:01:58.230598+0000","last_clean":"2024-09-13T23:01:58.230598+0000","last_became_active":"2024-09-13T22:58:42.747444+0000","last_became_peered":"2024-09-13T22:58:42.747444+0000","last_unstale":"2024-09-13T23:01:58.230598+0000","last_undegraded":"2024-09-13T23:01:58.230598+0000","last_fullsized":"2024-09-13T23:01:58.230598+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:19.136408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T23:01:19.136408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:38.147672+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":592,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235603+0000","last_change":"2024-09-13T23:01:19.547842+0000","last_active":"2024-09-13T23:01:58.235603+0000","last_peered":"2024-09-13T23:01:58.235603+0000","last_clean":"2024-09-13T23:01:58.235603+0000","last_became_active":"2024-09-13T22:58:45.829511+0000","last_became_peered":"2024-09-13T22:58:45.829511+0000","last_unstale":"2024-09-13T23:01:58.235603+0000","last_undegraded":"2024-09-13T23:01:58.235603+0000","last_fullsized":"2024-09-13T23:01:58.235603+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:19.547734+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:19.547734+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:49.347580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1294,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233422+0000","last_change":"2024-09-13T23:01:50.953254+0000","last_active":"2024-09-13T23:01:58.233422+0000","last_peered":"2024-09-13T23:01:58.233422+0000","last_clean":"2024-09-13T23:01:58.233422+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T23:01:58.233422+0000","last_undegraded":"2024-09-13T23:01:58.233422+0000","last_fullsized":"2024-09-13T23:01:58.233422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'64","last_scrub_stamp":"2024-09-13T23:01:50.952992+0000","last_deep_scrub":"255'52","last_deep_scrub_stamp":"2024-09-13T22:55:36.587996+0000","last_clean_scrub_stamp":"2024-09-13T23:01:50.952992+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:17.321207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":904,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236208+0000","last_change":"2024-09-13T22:58:07.245450+0000","last_active":"2024-09-13T23:01:58.236208+0000","last_peered":"2024-09-13T23:01:58.236208+0000","last_clean":"2024-09-13T23:01:58.236208+0000","last_became_active":"2024-09-13T22:55:38.145203+0000","last_became_peered":"2024-09-13T22:55:38.145203+0000","last_unstale":"2024-09-13T23:01:58.236208+0000","last_undegraded":"2024-09-13T23:01:58.236208+0000","last_fullsized":"2024-09-13T23:01:58.236208+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:07.245370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:58:07.245370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":144,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233188+0000","last_change":"2024-09-13T22:58:42.741243+0000","last_active":"2024-09-13T23:01:58.233188+0000","last_peered":"2024-09-13T23:01:58.233188+0000","last_clean":"2024-09-13T23:01:58.233188+0000","last_became_active":"2024-09-13T22:58:42.740862+0000","last_became_peered":"2024-09-13T22:58:42.740862+0000","last_unstale":"2024-09-13T23:01:58.233188+0000","last_undegraded":"2024-09-13T23:01:58.233188+0000","last_fullsized":"2024-09-13T23:01:58.233188+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:10.310671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:29.115656+0000","last_clean_scrub_stamp":"2024-09-13T22:58:10.310671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234338+0000","last_change":"2024-09-13T23:01:03.778607+0000","last_active":"2024-09-13T23:01:58.234338+0000","last_peered":"2024-09-13T23:01:58.234338+0000","last_clean":"2024-09-13T23:01:58.234338+0000","last_became_active":"2024-09-13T22:58:42.742606+0000","last_became_peered":"2024-09-13T22:58:42.742606+0000","last_unstale":"2024-09-13T23:01:58.234338+0000","last_undegraded":"2024-09-13T23:01:58.234338+0000","last_fullsized":"2024-09-13T23:01:58.234338+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:03.778503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:03.778503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:21.469814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":595,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236789+0000","last_change":"2024-09-13T23:01:20.581391+0000","last_active":"2024-09-13T23:01:58.236789+0000","last_peered":"2024-09-13T23:01:58.236789+0000","last_clean":"2024-09-13T23:01:58.236789+0000","last_became_active":"2024-09-13T22:58:45.832840+0000","last_became_peered":"2024-09-13T22:58:45.832840+0000","last_unstale":"2024-09-13T23:01:58.236789+0000","last_undegraded":"2024-09-13T23:01:58.236789+0000","last_fullsized":"2024-09-13T23:01:58.236789+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:20.581288+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:21.295886+0000","last_clean_scrub_stamp":"2024-09-13T23:01:20.581288+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:26.252840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1256,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233986+0000","last_change":"2024-09-13T22:58:09.995562+0000","last_active":"2024-09-13T23:01:58.233986+0000","last_peered":"2024-09-13T23:01:58.233986+0000","last_clean":"2024-09-13T23:01:58.233986+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T23:01:58.233986+0000","last_undegraded":"2024-09-13T23:01:58.233986+0000","last_fullsized":"2024-09-13T23:01:58.233986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'48","last_scrub_stamp":"2024-09-13T22:58:09.995445+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.995445+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":919,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233568+0000","last_change":"2024-09-13T23:01:01.749507+0000","last_active":"2024-09-13T23:01:58.233568+0000","last_peered":"2024-09-13T23:01:58.233568+0000","last_clean":"2024-09-13T23:01:58.233568+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T23:01:58.233568+0000","last_undegraded":"2024-09-13T23:01:58.233568+0000","last_fullsized":"2024-09-13T23:01:58.233568+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T23:01:01.749341+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T22:54:26.175576+0000","last_clean_scrub_stamp":"2024-09-13T23:01:01.749341+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:30.434533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":946,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233725+0000","last_change":"2024-09-13T23:01:53.004372+0000","last_active":"2024-09-13T23:01:58.233725+0000","last_peered":"2024-09-13T23:01:58.233725+0000","last_clean":"2024-09-13T23:01:58.233725+0000","last_became_active":"2024-09-13T22:55:38.114687+0000","last_became_peered":"2024-09-13T22:55:38.114687+0000","last_unstale":"2024-09-13T23:01:58.233725+0000","last_undegraded":"2024-09-13T23:01:58.233725+0000","last_fullsized":"2024-09-13T23:01:58.233725+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T23:01:53.004155+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T22:55:21.433994+0000","last_clean_scrub_stamp":"2024-09-13T23:01:53.004155+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:14.850009+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"414'14","reported_seq":1224,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621535+0000","last_change":"2024-09-13T22:58:42.782423+0000","last_active":"2024-09-13T23:01:56.621535+0000","last_peered":"2024-09-13T23:01:56.621535+0000","last_clean":"2024-09-13T23:01:56.621535+0000","last_became_active":"2024-09-13T22:58:42.740603+0000","last_became_peered":"2024-09-13T22:58:42.740603+0000","last_unstale":"2024-09-13T23:01:56.621535+0000","last_undegraded":"2024-09-13T23:01:56.621535+0000","last_fullsized":"2024-09-13T23:01:56.621535+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'13","last_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_deep_scrub":"358'13","last_deep_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_clean_scrub_stamp":"2024-09-13T22:58:05.208554+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":890,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.620747+0000","last_change":"2024-09-13T22:57:56.259013+0000","last_active":"2024-09-13T23:01:56.620747+0000","last_peered":"2024-09-13T23:01:56.620747+0000","last_clean":"2024-09-13T23:01:56.620747+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T23:01:56.620747+0000","last_undegraded":"2024-09-13T23:01:56.620747+0000","last_fullsized":"2024-09-13T23:01:56.620747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:56.258783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:57:56.258783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":947,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230751+0000","last_change":"2024-09-13T22:57:57.050275+0000","last_active":"2024-09-13T23:01:58.230751+0000","last_peered":"2024-09-13T23:01:58.230751+0000","last_clean":"2024-09-13T23:01:58.230751+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T23:01:58.230751+0000","last_undegraded":"2024-09-13T23:01:58.230751+0000","last_fullsized":"2024-09-13T23:01:58.230751+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T22:57:57.050042+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:48:26.832692+0000","last_clean_scrub_stamp":"2024-09-13T22:57:57.050042+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"468'47","reported_seq":270,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234533+0000","last_change":"2024-09-13T23:01:23.540614+0000","last_active":"2024-09-13T23:01:58.234533+0000","last_peered":"2024-09-13T23:01:58.234533+0000","last_clean":"2024-09-13T23:01:58.234533+0000","last_became_active":"2024-09-13T22:58:42.733200+0000","last_became_peered":"2024-09-13T22:58:42.733200+0000","last_unstale":"2024-09-13T23:01:58.234533+0000","last_undegraded":"2024-09-13T23:01:58.234533+0000","last_fullsized":"2024-09-13T23:01:58.234533+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'47","last_scrub_stamp":"2024-09-13T23:01:23.540241+0000","last_deep_scrub":"468'47","last_deep_scrub_stamp":"2024-09-13T23:01:23.540241+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.540241+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:44.220471+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":922,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237308+0000","last_change":"2024-09-13T22:58:42.742254+0000","last_active":"2024-09-13T23:01:58.237308+0000","last_peered":"2024-09-13T23:01:58.237308+0000","last_clean":"2024-09-13T23:01:58.237308+0000","last_became_active":"2024-09-13T22:58:42.741601+0000","last_became_peered":"2024-09-13T22:58:42.741601+0000","last_unstale":"2024-09-13T23:01:58.237308+0000","last_undegraded":"2024-09-13T23:01:58.237308+0000","last_fullsized":"2024-09-13T23:01:58.237308+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:17.058474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.500054+0000","last_clean_scrub_stamp":"2024-09-13T22:58:17.058474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":931,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622466+0000","last_change":"2024-09-13T23:01:26.955132+0000","last_active":"2024-09-13T23:01:56.622466+0000","last_peered":"2024-09-13T23:01:56.622466+0000","last_clean":"2024-09-13T23:01:56.622466+0000","last_became_active":"2024-09-13T22:58:42.740401+0000","last_became_peered":"2024-09-13T22:58:42.740401+0000","last_unstale":"2024-09-13T23:01:56.622466+0000","last_undegraded":"2024-09-13T23:01:56.622466+0000","last_fullsized":"2024-09-13T23:01:56.622466+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:26.955048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.494388+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.955048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:30.495780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":911,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235394+0000","last_change":"2024-09-13T23:01:57.943202+0000","last_active":"2024-09-13T23:01:58.235394+0000","last_peered":"2024-09-13T23:01:58.235394+0000","last_clean":"2024-09-13T23:01:58.235394+0000","last_became_active":"2024-09-13T22:58:42.746572+0000","last_became_peered":"2024-09-13T22:58:42.746572+0000","last_unstale":"2024-09-13T23:01:58.235394+0000","last_undegraded":"2024-09-13T23:01:58.235394+0000","last_fullsized":"2024-09-13T23:01:58.235394+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:57.943005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:57.943005+0000","last_clean_scrub_stamp":"2024-09-13T23:01:57.943005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:07.672190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1199,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234713+0000","last_change":"2024-09-13T23:01:32.330495+0000","last_active":"2024-09-13T23:01:58.234713+0000","last_peered":"2024-09-13T23:01:58.234713+0000","last_clean":"2024-09-13T23:01:58.234713+0000","last_became_active":"2024-09-13T22:55:41.103393+0000","last_became_peered":"2024-09-13T22:55:41.103393+0000","last_unstale":"2024-09-13T23:01:58.234713+0000","last_undegraded":"2024-09-13T23:01:58.234713+0000","last_fullsized":"2024-09-13T23:01:58.234713+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:32.330280+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:32.330280+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:35.531503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":257,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.229996+0000","last_change":"2024-09-13T23:01:13.457411+0000","last_active":"2024-09-13T23:01:58.229996+0000","last_peered":"2024-09-13T23:01:58.229996+0000","last_clean":"2024-09-13T23:01:58.229996+0000","last_became_active":"2024-09-13T22:58:42.727647+0000","last_became_peered":"2024-09-13T22:58:42.727647+0000","last_unstale":"2024-09-13T23:01:58.229996+0000","last_undegraded":"2024-09-13T23:01:58.229996+0000","last_fullsized":"2024-09-13T23:01:58.229996+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-09-13T23:01:13.457325+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T23:01:13.457325+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:14.464305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"470'2","reported_seq":952,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235306+0000","last_change":"2024-09-13T23:01:27.631319+0000","last_active":"2024-09-13T23:01:58.235306+0000","last_peered":"2024-09-13T23:01:58.235306+0000","last_clean":"2024-09-13T23:01:58.235306+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T23:01:58.235306+0000","last_undegraded":"2024-09-13T23:01:58.235306+0000","last_fullsized":"2024-09-13T23:01:58.235306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'2","last_scrub_stamp":"2024-09-13T23:01:27.631212+0000","last_deep_scrub":"470'2","last_deep_scrub_stamp":"2024-09-13T23:01:27.631212+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.631212+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:34.588613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":935,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236877+0000","last_change":"2024-09-13T23:01:35.666130+0000","last_active":"2024-09-13T23:01:58.236877+0000","last_peered":"2024-09-13T23:01:58.236877+0000","last_clean":"2024-09-13T23:01:58.236877+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T23:01:58.236877+0000","last_undegraded":"2024-09-13T23:01:58.236877+0000","last_fullsized":"2024-09-13T23:01:58.236877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.665979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.665979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:50.400063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234912+0000","last_change":"2024-09-13T23:01:22.338274+0000","last_active":"2024-09-13T23:01:58.234912+0000","last_peered":"2024-09-13T23:01:58.234912+0000","last_clean":"2024-09-13T23:01:58.234912+0000","last_became_active":"2024-09-13T22:58:42.764039+0000","last_became_peered":"2024-09-13T22:58:42.764039+0000","last_unstale":"2024-09-13T23:01:58.234912+0000","last_undegraded":"2024-09-13T23:01:58.234912+0000","last_fullsized":"2024-09-13T23:01:58.234912+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:22.338166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.338166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:50.408989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":999,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232316+0000","last_change":"2024-09-13T23:01:11.835987+0000","last_active":"2024-09-13T23:01:58.232316+0000","last_peered":"2024-09-13T23:01:58.232316+0000","last_clean":"2024-09-13T23:01:58.232316+0000","last_became_active":"2024-09-13T22:58:45.843305+0000","last_became_peered":"2024-09-13T22:58:45.843305+0000","last_unstale":"2024-09-13T23:01:58.232316+0000","last_undegraded":"2024-09-13T23:01:58.232316+0000","last_fullsized":"2024-09-13T23:01:58.232316+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:11.835784+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T23:01:11.835784+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:12.085238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"397'42","reported_seq":1034,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621270+0000","last_change":"2024-09-13T22:58:15.305788+0000","last_active":"2024-09-13T23:01:56.621270+0000","last_peered":"2024-09-13T23:01:56.621270+0000","last_clean":"2024-09-13T23:01:56.621270+0000","last_became_active":"2024-09-13T22:55:38.098695+0000","last_became_peered":"2024-09-13T22:55:38.098695+0000","last_unstale":"2024-09-13T23:01:56.621270+0000","last_undegraded":"2024-09-13T23:01:56.621270+0000","last_fullsized":"2024-09-13T23:01:56.621270+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-09-13T22:58:15.305297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.305297+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":669,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230876+0000","last_change":"2024-09-13T23:01:17.315498+0000","last_active":"2024-09-13T23:01:58.230876+0000","last_peered":"2024-09-13T23:01:58.230876+0000","last_clean":"2024-09-13T23:01:58.230876+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T23:01:58.230876+0000","last_undegraded":"2024-09-13T23:01:58.230876+0000","last_fullsized":"2024-09-13T23:01:58.230876+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:17.315399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T23:01:17.315399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:43.589754+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232672+0000","last_change":"2024-09-13T23:01:21.396408+0000","last_active":"2024-09-13T23:01:58.232672+0000","last_peered":"2024-09-13T23:01:58.232672+0000","last_clean":"2024-09-13T23:01:58.232672+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T23:01:58.232672+0000","last_undegraded":"2024-09-13T23:01:58.232672+0000","last_fullsized":"2024-09-13T23:01:58.232672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:21.396226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.396226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:46.302469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":898,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232985+0000","last_change":"2024-09-13T23:01:30.887160+0000","last_active":"2024-09-13T23:01:58.232985+0000","last_peered":"2024-09-13T23:01:58.232985+0000","last_clean":"2024-09-13T23:01:58.232985+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T23:01:58.232985+0000","last_undegraded":"2024-09-13T23:01:58.232985+0000","last_fullsized":"2024-09-13T23:01:58.232985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.886958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:18.659096+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.886958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:51.664086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":962,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230956+0000","last_change":"2024-09-13T22:58:42.787694+0000","last_active":"2024-09-13T23:01:58.230956+0000","last_peered":"2024-09-13T23:01:58.230956+0000","last_clean":"2024-09-13T23:01:58.230956+0000","last_became_active":"2024-09-13T22:58:42.746819+0000","last_became_peered":"2024-09-13T22:58:42.746819+0000","last_unstale":"2024-09-13T23:01:58.230956+0000","last_undegraded":"2024-09-13T23:01:58.230956+0000","last_fullsized":"2024-09-13T23:01:58.230956+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T22:58:09.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:09.777653+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235482+0000","last_change":"2024-09-13T23:01:16.437663+0000","last_active":"2024-09-13T23:01:58.235482+0000","last_peered":"2024-09-13T23:01:58.235482+0000","last_clean":"2024-09-13T23:01:58.235482+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T23:01:58.235482+0000","last_undegraded":"2024-09-13T23:01:58.235482+0000","last_fullsized":"2024-09-13T23:01:58.235482+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:16.437598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:16.437598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:17.327827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1183,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232601+0000","last_change":"2024-09-13T23:01:21.309692+0000","last_active":"2024-09-13T23:01:58.232601+0000","last_peered":"2024-09-13T23:01:58.232601+0000","last_clean":"2024-09-13T23:01:58.232601+0000","last_became_active":"2024-09-13T22:58:45.829938+0000","last_became_peered":"2024-09-13T22:58:45.829938+0000","last_unstale":"2024-09-13T23:01:58.232601+0000","last_undegraded":"2024-09-13T23:01:58.232601+0000","last_fullsized":"2024-09-13T23:01:58.232601+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:01:21.309609+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.309609+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:35.236445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":886,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235846+0000","last_change":"2024-09-13T23:01:14.349135+0000","last_active":"2024-09-13T23:01:58.235846+0000","last_peered":"2024-09-13T23:01:58.235846+0000","last_clean":"2024-09-13T23:01:58.235846+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T23:01:58.235846+0000","last_undegraded":"2024-09-13T23:01:58.235846+0000","last_fullsized":"2024-09-13T23:01:58.235846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:14.349029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.349029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:27.002108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232620+0000","last_change":"2024-09-13T23:01:03.483621+0000","last_active":"2024-09-13T23:01:58.232620+0000","last_peered":"2024-09-13T23:01:58.232620+0000","last_clean":"2024-09-13T23:01:58.232620+0000","last_became_active":"2024-09-13T22:55:38.101749+0000","last_became_peered":"2024-09-13T22:55:38.101749+0000","last_unstale":"2024-09-13T23:01:58.232620+0000","last_undegraded":"2024-09-13T23:01:58.232620+0000","last_fullsized":"2024-09-13T23:01:58.232620+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:03.483542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:03.483542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:11.483888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":999,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235708+0000","last_change":"2024-09-13T22:58:42.803636+0000","last_active":"2024-09-13T23:01:58.235708+0000","last_peered":"2024-09-13T23:01:58.235708+0000","last_clean":"2024-09-13T23:01:58.235708+0000","last_became_active":"2024-09-13T22:58:42.764099+0000","last_became_peered":"2024-09-13T22:58:42.764099+0000","last_unstale":"2024-09-13T23:01:58.235708+0000","last_undegraded":"2024-09-13T23:01:58.235708+0000","last_fullsized":"2024-09-13T23:01:58.235708+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-09-13T22:58:14.318902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.318902+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":942,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.238379+0000","last_change":"2024-09-13T23:01:29.625234+0000","last_active":"2024-09-13T23:01:58.238379+0000","last_peered":"2024-09-13T23:01:58.238379+0000","last_clean":"2024-09-13T23:01:58.238379+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T23:01:58.238379+0000","last_undegraded":"2024-09-13T23:01:58.238379+0000","last_fullsized":"2024-09-13T23:01:58.238379+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:29.625043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.625043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:53.981960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":926,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236798+0000","last_change":"2024-09-13T23:01:30.649247+0000","last_active":"2024-09-13T23:01:58.236798+0000","last_peered":"2024-09-13T23:01:58.236798+0000","last_clean":"2024-09-13T23:01:58.236798+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T23:01:58.236798+0000","last_undegraded":"2024-09-13T23:01:58.236798+0000","last_fullsized":"2024-09-13T23:01:58.236798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.648989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.648989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.411813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":874,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232324+0000","last_change":"2024-09-13T23:01:06.196129+0000","last_active":"2024-09-13T23:01:58.232324+0000","last_peered":"2024-09-13T23:01:58.232324+0000","last_clean":"2024-09-13T23:01:58.232324+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T23:01:58.232324+0000","last_undegraded":"2024-09-13T23:01:58.232324+0000","last_fullsized":"2024-09-13T23:01:58.232324+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:06.196062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T23:01:06.196062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:08.228449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"397'30","reported_seq":986,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234273+0000","last_change":"2024-09-13T23:01:29.459955+0000","last_active":"2024-09-13T23:01:58.234273+0000","last_peered":"2024-09-13T23:01:58.234273+0000","last_clean":"2024-09-13T23:01:58.234273+0000","last_became_active":"2024-09-13T22:58:42.731142+0000","last_became_peered":"2024-09-13T22:58:42.731142+0000","last_unstale":"2024-09-13T23:01:58.234273+0000","last_undegraded":"2024-09-13T23:01:58.234273+0000","last_fullsized":"2024-09-13T23:01:58.234273+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:01:29.459740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.459740+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:48.107858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":927,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232579+0000","last_change":"2024-09-13T23:01:09.811260+0000","last_active":"2024-09-13T23:01:58.232579+0000","last_peered":"2024-09-13T23:01:58.232579+0000","last_clean":"2024-09-13T23:01:58.232579+0000","last_became_active":"2024-09-13T22:55:38.093381+0000","last_became_peered":"2024-09-13T22:55:38.093381+0000","last_unstale":"2024-09-13T23:01:58.232579+0000","last_undegraded":"2024-09-13T23:01:58.232579+0000","last_fullsized":"2024-09-13T23:01:58.232579+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:09.811162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:09.811162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:20.896122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":621,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230569+0000","last_change":"2024-09-13T23:01:26.266715+0000","last_active":"2024-09-13T23:01:58.230569+0000","last_peered":"2024-09-13T23:01:58.230569+0000","last_clean":"2024-09-13T23:01:58.230569+0000","last_became_active":"2024-09-13T22:55:38.098515+0000","last_became_peered":"2024-09-13T22:55:38.098515+0000","last_unstale":"2024-09-13T23:01:58.230569+0000","last_undegraded":"2024-09-13T23:01:58.230569+0000","last_fullsized":"2024-09-13T23:01:58.230569+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:26.266526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.266526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:38.051187+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":897,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621101+0000","last_change":"2024-09-13T22:58:16.384185+0000","last_active":"2024-09-13T23:01:56.621101+0000","last_peered":"2024-09-13T23:01:56.621101+0000","last_clean":"2024-09-13T23:01:56.621101+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T23:01:56.621101+0000","last_undegraded":"2024-09-13T23:01:56.621101+0000","last_fullsized":"2024-09-13T23:01:56.621101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:16.383923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:54.275361+0000","last_clean_scrub_stamp":"2024-09-13T22:58:16.383923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":1023,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237699+0000","last_change":"2024-09-13T23:01:36.239941+0000","last_active":"2024-09-13T23:01:58.237699+0000","last_peered":"2024-09-13T23:01:58.237699+0000","last_clean":"2024-09-13T23:01:58.237699+0000","last_became_active":"2024-09-13T22:55:38.151658+0000","last_became_peered":"2024-09-13T22:55:38.151658+0000","last_unstale":"2024-09-13T23:01:58.237699+0000","last_undegraded":"2024-09-13T23:01:58.237699+0000","last_fullsized":"2024-09-13T23:01:58.237699+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:01:36.239714+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-09-13T23:01:36.239714+0000","last_clean_scrub_stamp":"2024-09-13T23:01:36.239714+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:46.968037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":483,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235898+0000","last_change":"2024-09-13T23:01:34.122869+0000","last_active":"2024-09-13T23:01:58.235898+0000","last_peered":"2024-09-13T23:01:58.235898+0000","last_clean":"2024-09-13T23:01:58.235898+0000","last_became_active":"2024-09-13T22:49:45.645785+0000","last_became_peered":"2024-09-13T22:49:45.645785+0000","last_unstale":"2024-09-13T23:01:58.235898+0000","last_undegraded":"2024-09-13T23:01:58.235898+0000","last_fullsized":"2024-09-13T23:01:58.235898+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:34.122727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:32.054441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:34.122727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.135029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"414'16","reported_seq":1241,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231817+0000","last_change":"2024-09-13T23:01:09.277696+0000","last_active":"2024-09-13T23:01:58.231817+0000","last_peered":"2024-09-13T23:01:58.231817+0000","last_clean":"2024-09-13T23:01:58.231817+0000","last_became_active":"2024-09-13T22:58:42.758407+0000","last_became_peered":"2024-09-13T22:58:42.758407+0000","last_unstale":"2024-09-13T23:01:58.231817+0000","last_undegraded":"2024-09-13T23:01:58.231817+0000","last_fullsized":"2024-09-13T23:01:58.231817+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'16","last_scrub_stamp":"2024-09-13T23:01:09.277590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:09.277590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:14.772303+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236275+0000","last_change":"2024-09-13T23:01:51.662395+0000","last_active":"2024-09-13T23:01:58.236275+0000","last_peered":"2024-09-13T23:01:58.236275+0000","last_clean":"2024-09-13T23:01:58.236275+0000","last_became_active":"2024-09-13T22:58:42.732057+0000","last_became_peered":"2024-09-13T22:58:42.732057+0000","last_unstale":"2024-09-13T23:01:58.236275+0000","last_undegraded":"2024-09-13T23:01:58.236275+0000","last_fullsized":"2024-09-13T23:01:58.236275+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:51.662251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T23:01:51.662251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:06.375373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"397'27","reported_seq":994,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237271+0000","last_change":"2024-09-13T23:01:54.097804+0000","last_active":"2024-09-13T23:01:58.237271+0000","last_peered":"2024-09-13T23:01:58.237271+0000","last_clean":"2024-09-13T23:01:58.237271+0000","last_became_active":"2024-09-13T22:55:38.151005+0000","last_became_peered":"2024-09-13T22:55:38.151005+0000","last_unstale":"2024-09-13T23:01:58.237271+0000","last_undegraded":"2024-09-13T23:01:58.237271+0000","last_fullsized":"2024-09-13T23:01:58.237271+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T23:01:54.097594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:54.097594+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:09.451104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":532,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.623021+0000","last_change":"2024-09-13T23:01:53.795107+0000","last_active":"2024-09-13T23:01:56.623021+0000","last_peered":"2024-09-13T23:01:56.623021+0000","last_clean":"2024-09-13T23:01:56.623021+0000","last_became_active":"2024-09-13T22:55:38.136788+0000","last_became_peered":"2024-09-13T22:55:38.136788+0000","last_unstale":"2024-09-13T23:01:56.623021+0000","last_undegraded":"2024-09-13T23:01:56.623021+0000","last_fullsized":"2024-09-13T23:01:56.623021+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T23:01:53.795012+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T23:01:53.795012+0000","last_clean_scrub_stamp":"2024-09-13T23:01:53.795012+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:11.426723+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":491,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621147+0000","last_change":"2024-09-13T23:01:35.962699+0000","last_active":"2024-09-13T23:01:56.621147+0000","last_peered":"2024-09-13T23:01:56.621147+0000","last_clean":"2024-09-13T23:01:56.621147+0000","last_became_active":"2024-09-13T22:49:45.649591+0000","last_became_peered":"2024-09-13T22:49:45.649591+0000","last_unstale":"2024-09-13T23:01:56.621147+0000","last_undegraded":"2024-09-13T23:01:56.621147+0000","last_fullsized":"2024-09-13T23:01:56.621147+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.962519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:35.962519+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.962519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:48.660516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":880,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231911+0000","last_change":"2024-09-13T22:57:50.594198+0000","last_active":"2024-09-13T23:01:58.231911+0000","last_peered":"2024-09-13T23:01:58.231911+0000","last_clean":"2024-09-13T23:01:58.231911+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T23:01:58.231911+0000","last_undegraded":"2024-09-13T23:01:58.231911+0000","last_fullsized":"2024-09-13T23:01:58.231911+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:50.594120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:05.968767+0000","last_clean_scrub_stamp":"2024-09-13T22:57:50.594120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":1027,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231130+0000","last_change":"2024-09-13T23:01:29.345584+0000","last_active":"2024-09-13T23:01:58.231130+0000","last_peered":"2024-09-13T23:01:58.231130+0000","last_clean":"2024-09-13T23:01:58.231130+0000","last_became_active":"2024-09-13T22:58:42.758655+0000","last_became_peered":"2024-09-13T22:58:42.758655+0000","last_unstale":"2024-09-13T23:01:58.231130+0000","last_undegraded":"2024-09-13T23:01:58.231130+0000","last_fullsized":"2024-09-13T23:01:58.231130+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:01:29.345354+0000","last_deep_scrub":"397'32","last_deep_scrub_stamp":"2024-09-13T23:01:29.345354+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.345354+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:57.161111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":934,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234259+0000","last_change":"2024-09-13T23:01:08.586865+0000","last_active":"2024-09-13T23:01:58.234259+0000","last_peered":"2024-09-13T23:01:58.234259+0000","last_clean":"2024-09-13T23:01:58.234259+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T23:01:58.234259+0000","last_undegraded":"2024-09-13T23:01:58.234259+0000","last_fullsized":"2024-09-13T23:01:58.234259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:08.586782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:43.728107+0000","last_clean_scrub_stamp":"2024-09-13T23:01:08.586782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:15.017576+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234543+0000","last_change":"2024-09-13T23:01:32.179108+0000","last_active":"2024-09-13T23:01:58.234543+0000","last_peered":"2024-09-13T23:01:58.234543+0000","last_clean":"2024-09-13T23:01:58.234543+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T23:01:58.234543+0000","last_undegraded":"2024-09-13T23:01:58.234543+0000","last_fullsized":"2024-09-13T23:01:58.234543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:32.178929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:32.178929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:39.017030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":948,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236442+0000","last_change":"2024-09-13T23:01:18.539224+0000","last_active":"2024-09-13T23:01:58.236442+0000","last_peered":"2024-09-13T23:01:58.236442+0000","last_clean":"2024-09-13T23:01:58.236442+0000","last_became_active":"2024-09-13T22:55:38.126360+0000","last_became_peered":"2024-09-13T22:55:38.126360+0000","last_unstale":"2024-09-13T23:01:58.236442+0000","last_undegraded":"2024-09-13T23:01:58.236442+0000","last_fullsized":"2024-09-13T23:01:58.236442+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:18.539114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T23:01:18.539114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.206035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":158,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231314+0000","last_change":"2024-09-13T23:01:35.587143+0000","last_active":"2024-09-13T23:01:58.231314+0000","last_peered":"2024-09-13T23:01:58.231314+0000","last_clean":"2024-09-13T23:01:58.231314+0000","last_became_active":"2024-09-13T22:58:42.736178+0000","last_became_peered":"2024-09-13T22:58:42.736178+0000","last_unstale":"2024-09-13T23:01:58.231314+0000","last_undegraded":"2024-09-13T23:01:58.231314+0000","last_fullsized":"2024-09-13T23:01:58.231314+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T23:01:35.587020+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.587020+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.733540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":152,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233042+0000","last_change":"2024-09-13T23:01:21.585285+0000","last_active":"2024-09-13T23:01:58.233042+0000","last_peered":"2024-09-13T23:01:58.233042+0000","last_clean":"2024-09-13T23:01:58.233042+0000","last_became_active":"2024-09-13T22:58:42.744948+0000","last_became_peered":"2024-09-13T22:58:42.744948+0000","last_unstale":"2024-09-13T23:01:58.233042+0000","last_undegraded":"2024-09-13T23:01:58.233042+0000","last_fullsized":"2024-09-13T23:01:58.233042+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:21.585092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.585092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:22.631485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231540+0000","last_change":"2024-09-13T23:01:35.457385+0000","last_active":"2024-09-13T23:01:58.231540+0000","last_peered":"2024-09-13T23:01:58.231540+0000","last_clean":"2024-09-13T23:01:58.231540+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T23:01:58.231540+0000","last_undegraded":"2024-09-13T23:01:58.231540+0000","last_fullsized":"2024-09-13T23:01:58.231540+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.457138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.457138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:45.923641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234669+0000","last_change":"2024-09-13T22:58:42.769155+0000","last_active":"2024-09-13T23:01:58.234669+0000","last_peered":"2024-09-13T23:01:58.234669+0000","last_clean":"2024-09-13T23:01:58.234669+0000","last_became_active":"2024-09-13T22:58:42.768571+0000","last_became_peered":"2024-09-13T22:58:42.768571+0000","last_unstale":"2024-09-13T23:01:58.234669+0000","last_undegraded":"2024-09-13T23:01:58.234669+0000","last_fullsized":"2024-09-13T23:01:58.234669+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:15.349704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:15.349704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":1022,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236820+0000","last_change":"2024-09-13T23:01:26.320429+0000","last_active":"2024-09-13T23:01:58.236820+0000","last_peered":"2024-09-13T23:01:58.236820+0000","last_clean":"2024-09-13T23:01:58.236820+0000","last_became_active":"2024-09-13T22:58:42.759372+0000","last_became_peered":"2024-09-13T22:58:42.759372+0000","last_unstale":"2024-09-13T23:01:58.236820+0000","last_undegraded":"2024-09-13T23:01:58.236820+0000","last_fullsized":"2024-09-13T23:01:58.236820+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T23:01:26.320267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:26.320267+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.326421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":912,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237074+0000","last_change":"2024-09-13T23:01:04.468733+0000","last_active":"2024-09-13T23:01:58.237074+0000","last_peered":"2024-09-13T23:01:58.237074+0000","last_clean":"2024-09-13T23:01:58.237074+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T23:01:58.237074+0000","last_undegraded":"2024-09-13T23:01:58.237074+0000","last_fullsized":"2024-09-13T23:01:58.237074+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:04.468638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:52.950367+0000","last_clean_scrub_stamp":"2024-09-13T23:01:04.468638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:11.277071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":1207,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233612+0000","last_change":"2024-09-13T23:01:14.440415+0000","last_active":"2024-09-13T23:01:58.233612+0000","last_peered":"2024-09-13T23:01:58.233612+0000","last_clean":"2024-09-13T23:01:58.233612+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T23:01:58.233612+0000","last_undegraded":"2024-09-13T23:01:58.233612+0000","last_fullsized":"2024-09-13T23:01:58.233612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T23:01:14.440335+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T22:53:18.805172+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.440335+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:24.072999+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":895,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622115+0000","last_change":"2024-09-13T23:01:31.009368+0000","last_active":"2024-09-13T23:01:56.622115+0000","last_peered":"2024-09-13T23:01:56.622115+0000","last_clean":"2024-09-13T23:01:56.622115+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T23:01:56.622115+0000","last_undegraded":"2024-09-13T23:01:56.622115+0000","last_fullsized":"2024-09-13T23:01:56.622115+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:31.009178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:31.009178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:43.912379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":938,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621542+0000","last_change":"2024-09-13T23:01:33.006568+0000","last_active":"2024-09-13T23:01:56.621542+0000","last_peered":"2024-09-13T23:01:56.621542+0000","last_clean":"2024-09-13T23:01:56.621542+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T23:01:56.621542+0000","last_undegraded":"2024-09-13T23:01:56.621542+0000","last_fullsized":"2024-09-13T23:01:56.621542+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:33.006332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:33.006332+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:39.870873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":926,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622114+0000","last_change":"2024-09-13T23:01:14.927758+0000","last_active":"2024-09-13T23:01:56.622114+0000","last_peered":"2024-09-13T23:01:56.622114+0000","last_clean":"2024-09-13T23:01:56.622114+0000","last_became_active":"2024-09-13T22:55:38.138035+0000","last_became_peered":"2024-09-13T22:55:38.138035+0000","last_unstale":"2024-09-13T23:01:56.622114+0000","last_undegraded":"2024-09-13T23:01:56.622114+0000","last_fullsized":"2024-09-13T23:01:56.622114+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:14.927611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.927611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:21.794756+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":1186,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233823+0000","last_change":"2024-09-13T22:58:14.754552+0000","last_active":"2024-09-13T23:01:58.233823+0000","last_peered":"2024-09-13T23:01:58.233823+0000","last_clean":"2024-09-13T23:01:58.233823+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T23:01:58.233823+0000","last_undegraded":"2024-09-13T23:01:58.233823+0000","last_fullsized":"2024-09-13T23:01:58.233823+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T22:58:14.754430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.754430+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":418,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.237464+0000","last_change":"2024-09-13T23:01:22.539081+0000","last_active":"2024-09-13T23:01:58.237464+0000","last_peered":"2024-09-13T23:01:58.237464+0000","last_clean":"2024-09-13T23:01:58.237464+0000","last_became_active":"2024-09-13T22:55:38.098486+0000","last_became_peered":"2024-09-13T22:55:38.098486+0000","last_unstale":"2024-09-13T23:01:58.237464+0000","last_undegraded":"2024-09-13T23:01:58.237464+0000","last_fullsized":"2024-09-13T23:01:58.237464+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:22.538979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.538979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:29.121704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":971,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235439+0000","last_change":"2024-09-13T23:01:11.267967+0000","last_active":"2024-09-13T23:01:58.235439+0000","last_peered":"2024-09-13T23:01:58.235439+0000","last_clean":"2024-09-13T23:01:58.235439+0000","last_became_active":"2024-09-13T22:55:38.144744+0000","last_became_peered":"2024-09-13T22:55:38.144744+0000","last_unstale":"2024-09-13T23:01:58.235439+0000","last_undegraded":"2024-09-13T23:01:58.235439+0000","last_fullsized":"2024-09-13T23:01:58.235439+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:11.267888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:11.267888+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.946659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":144,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234395+0000","last_change":"2024-09-13T22:58:42.749921+0000","last_active":"2024-09-13T23:01:58.234395+0000","last_peered":"2024-09-13T23:01:58.234395+0000","last_clean":"2024-09-13T23:01:58.234395+0000","last_became_active":"2024-09-13T22:58:42.749062+0000","last_became_peered":"2024-09-13T22:58:42.749062+0000","last_unstale":"2024-09-13T23:01:58.234395+0000","last_undegraded":"2024-09-13T23:01:58.234395+0000","last_fullsized":"2024-09-13T23:01:58.234395+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:54.689197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T22:57:54.689197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":884,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232177+0000","last_change":"2024-09-13T23:01:31.178527+0000","last_active":"2024-09-13T23:01:58.232177+0000","last_peered":"2024-09-13T23:01:58.232177+0000","last_clean":"2024-09-13T23:01:58.232177+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T23:01:58.232177+0000","last_undegraded":"2024-09-13T23:01:58.232177+0000","last_fullsized":"2024-09-13T23:01:58.232177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:31.178438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:29.136366+0000","last_clean_scrub_stamp":"2024-09-13T23:01:31.178438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:49.241914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":888,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621450+0000","last_change":"2024-09-13T22:58:13.326329+0000","last_active":"2024-09-13T23:01:56.621450+0000","last_peered":"2024-09-13T23:01:56.621450+0000","last_clean":"2024-09-13T23:01:56.621450+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T23:01:56.621450+0000","last_undegraded":"2024-09-13T23:01:56.621450+0000","last_fullsized":"2024-09-13T23:01:56.621450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.326193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.326193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":526,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234264+0000","last_change":"2024-09-13T23:01:05.196417+0000","last_active":"2024-09-13T23:01:58.234264+0000","last_peered":"2024-09-13T23:01:58.234264+0000","last_clean":"2024-09-13T23:01:58.234264+0000","last_became_active":"2024-09-13T22:58:42.767781+0000","last_became_peered":"2024-09-13T22:58:42.767781+0000","last_unstale":"2024-09-13T23:01:58.234264+0000","last_undegraded":"2024-09-13T23:01:58.234264+0000","last_fullsized":"2024-09-13T23:01:58.234264+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:01:05.196330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:05.196330+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:18.398596+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":909,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232332+0000","last_change":"2024-09-13T23:01:35.904091+0000","last_active":"2024-09-13T23:01:58.232332+0000","last_peered":"2024-09-13T23:01:58.232332+0000","last_clean":"2024-09-13T23:01:58.232332+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T23:01:58.232332+0000","last_undegraded":"2024-09-13T23:01:58.232332+0000","last_fullsized":"2024-09-13T23:01:58.232332+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:35.903970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.903970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:51.073022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232981+0000","last_change":"2024-09-13T23:01:48.867508+0000","last_active":"2024-09-13T23:01:58.232981+0000","last_peered":"2024-09-13T23:01:58.232981+0000","last_clean":"2024-09-13T23:01:58.232981+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T23:01:58.232981+0000","last_undegraded":"2024-09-13T23:01:58.232981+0000","last_fullsized":"2024-09-13T23:01:58.232981+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:48.867317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:00.830738+0000","last_clean_scrub_stamp":"2024-09-13T23:01:48.867317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:50.173590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":872,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230378+0000","last_change":"2024-09-13T22:57:07.530820+0000","last_active":"2024-09-13T23:01:58.230378+0000","last_peered":"2024-09-13T23:01:58.230378+0000","last_clean":"2024-09-13T23:01:58.230378+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T23:01:58.230378+0000","last_undegraded":"2024-09-13T23:01:58.230378+0000","last_fullsized":"2024-09-13T23:01:58.230378+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:07.530747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T22:57:07.530747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"397'18","reported_seq":931,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.623128+0000","last_change":"2024-09-13T22:58:14.333753+0000","last_active":"2024-09-13T23:01:56.623128+0000","last_peered":"2024-09-13T23:01:56.623128+0000","last_clean":"2024-09-13T23:01:56.623128+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T23:01:56.623128+0000","last_undegraded":"2024-09-13T23:01:56.623128+0000","last_fullsized":"2024-09-13T23:01:56.623128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'18","last_scrub_stamp":"2024-09-13T22:58:14.333490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T22:58:14.333490+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":921,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232355+0000","last_change":"2024-09-13T23:01:49.637038+0000","last_active":"2024-09-13T23:01:58.232355+0000","last_peered":"2024-09-13T23:01:58.232355+0000","last_clean":"2024-09-13T23:01:58.232355+0000","last_became_active":"2024-09-13T22:58:42.726569+0000","last_became_peered":"2024-09-13T22:58:42.726569+0000","last_unstale":"2024-09-13T23:01:58.232355+0000","last_undegraded":"2024-09-13T23:01:58.232355+0000","last_fullsized":"2024-09-13T23:01:58.232355+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:49.636897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T23:01:49.636897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:18.181395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":144,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232649+0000","last_change":"2024-09-13T22:58:42.752271+0000","last_active":"2024-09-13T23:01:58.232649+0000","last_peered":"2024-09-13T23:01:58.232649+0000","last_clean":"2024-09-13T23:01:58.232649+0000","last_became_active":"2024-09-13T22:58:42.748403+0000","last_became_peered":"2024-09-13T22:58:42.748403+0000","last_unstale":"2024-09-13T23:01:58.232649+0000","last_undegraded":"2024-09-13T23:01:58.232649+0000","last_fullsized":"2024-09-13T23:01:58.232649+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.200989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.829631+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.200989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":899,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233058+0000","last_change":"2024-09-13T23:01:30.318915+0000","last_active":"2024-09-13T23:01:58.233058+0000","last_peered":"2024-09-13T23:01:58.233058+0000","last_clean":"2024-09-13T23:01:58.233058+0000","last_became_active":"2024-09-13T22:55:38.135406+0000","last_became_peered":"2024-09-13T22:55:38.135406+0000","last_unstale":"2024-09-13T23:01:58.233058+0000","last_undegraded":"2024-09-13T23:01:58.233058+0000","last_fullsized":"2024-09-13T23:01:58.233058+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.318722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.318722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:52.783710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":939,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231954+0000","last_change":"2024-09-13T23:01:27.514568+0000","last_active":"2024-09-13T23:01:58.231954+0000","last_peered":"2024-09-13T23:01:58.231954+0000","last_clean":"2024-09-13T23:01:58.231954+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T23:01:58.231954+0000","last_undegraded":"2024-09-13T23:01:58.231954+0000","last_fullsized":"2024-09-13T23:01:58.231954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:27.514424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:27.514424+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:33.132445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":144,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236246+0000","last_change":"2024-09-13T22:58:42.742636+0000","last_active":"2024-09-13T23:01:58.236246+0000","last_peered":"2024-09-13T23:01:58.236246+0000","last_clean":"2024-09-13T23:01:58.236246+0000","last_became_active":"2024-09-13T22:58:42.742229+0000","last_became_peered":"2024-09-13T22:58:42.742229+0000","last_unstale":"2024-09-13T23:01:58.236246+0000","last_undegraded":"2024-09-13T23:01:58.236246+0000","last_fullsized":"2024-09-13T23:01:58.236246+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:13.134261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T22:58:13.134261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":921,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231154+0000","last_change":"2024-09-13T23:01:38.567264+0000","last_active":"2024-09-13T23:01:58.231154+0000","last_peered":"2024-09-13T23:01:58.231154+0000","last_clean":"2024-09-13T23:01:58.231154+0000","last_became_active":"2024-09-13T23:01:38.566551+0000","last_became_peered":"2024-09-13T23:01:38.566551+0000","last_unstale":"2024-09-13T23:01:58.231154+0000","last_undegraded":"2024-09-13T23:01:58.231154+0000","last_fullsized":"2024-09-13T23:01:58.231154+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:33.301478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:01:33.301478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:33.702750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232299+0000","last_change":"2024-09-13T22:58:42.749489+0000","last_active":"2024-09-13T23:01:58.232299+0000","last_peered":"2024-09-13T23:01:58.232299+0000","last_clean":"2024-09-13T23:01:58.232299+0000","last_became_active":"2024-09-13T22:58:42.749118+0000","last_became_peered":"2024-09-13T22:58:42.749118+0000","last_unstale":"2024-09-13T23:01:58.232299+0000","last_undegraded":"2024-09-13T23:01:58.232299+0000","last_fullsized":"2024-09-13T23:01:58.232299+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:57:59.732688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T22:57:59.732688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236096+0000","last_change":"2024-09-13T23:01:35.255461+0000","last_active":"2024-09-13T23:01:58.236096+0000","last_peered":"2024-09-13T23:01:58.236096+0000","last_clean":"2024-09-13T23:01:58.236096+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T23:01:58.236096+0000","last_undegraded":"2024-09-13T23:01:58.236096+0000","last_fullsized":"2024-09-13T23:01:58.236096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T23:01:35.255256+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_clean_scrub_stamp":"2024-09-13T23:01:35.255256+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.068806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":252,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234498+0000","last_change":"2024-09-13T23:01:02.630716+0000","last_active":"2024-09-13T23:01:58.234498+0000","last_peered":"2024-09-13T23:01:58.234498+0000","last_clean":"2024-09-13T23:01:58.234498+0000","last_became_active":"2024-09-13T22:58:42.740500+0000","last_became_peered":"2024-09-13T22:58:42.740500+0000","last_unstale":"2024-09-13T23:01:58.234498+0000","last_undegraded":"2024-09-13T23:01:58.234498+0000","last_fullsized":"2024-09-13T23:01:58.234498+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:01:02.630615+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-13T22:40:21.070300+0000","last_clean_scrub_stamp":"2024-09-13T23:01:02.630615+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:31.545469+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":939,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.230450+0000","last_change":"2024-09-13T23:01:01.488015+0000","last_active":"2024-09-13T23:01:58.230450+0000","last_peered":"2024-09-13T23:01:58.230450+0000","last_clean":"2024-09-13T23:01:58.230450+0000","last_became_active":"2024-09-13T22:58:42.746984+0000","last_became_peered":"2024-09-13T22:58:42.746984+0000","last_unstale":"2024-09-13T23:01:58.230450+0000","last_undegraded":"2024-09-13T23:01:58.230450+0000","last_fullsized":"2024-09-13T23:01:58.230450+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:01.487808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T23:01:01.487808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:21.618995+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"414'17","reported_seq":196,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.236504+0000","last_change":"2024-09-13T23:01:14.563397+0000","last_active":"2024-09-13T23:01:58.236504+0000","last_peered":"2024-09-13T23:01:58.236504+0000","last_clean":"2024-09-13T23:01:58.236504+0000","last_became_active":"2024-09-13T22:58:42.743192+0000","last_became_peered":"2024-09-13T22:58:42.743192+0000","last_unstale":"2024-09-13T23:01:58.236504+0000","last_undegraded":"2024-09-13T23:01:58.236504+0000","last_fullsized":"2024-09-13T23:01:58.236504+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'17","last_scrub_stamp":"2024-09-13T23:01:14.563131+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T23:01:14.563131+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.035856+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":914,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231347+0000","last_change":"2024-09-13T23:01:04.819037+0000","last_active":"2024-09-13T23:01:58.231347+0000","last_peered":"2024-09-13T23:01:58.231347+0000","last_clean":"2024-09-13T23:01:58.231347+0000","last_became_active":"2024-09-13T22:55:38.096392+0000","last_became_peered":"2024-09-13T22:55:38.096392+0000","last_unstale":"2024-09-13T23:01:58.231347+0000","last_undegraded":"2024-09-13T23:01:58.231347+0000","last_fullsized":"2024-09-13T23:01:58.231347+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:04.818930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:04.818930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:20.543539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":938,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231424+0000","last_change":"2024-09-13T23:01:29.240502+0000","last_active":"2024-09-13T23:01:58.231424+0000","last_peered":"2024-09-13T23:01:58.231424+0000","last_clean":"2024-09-13T23:01:58.231424+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T23:01:58.231424+0000","last_undegraded":"2024-09-13T23:01:58.231424+0000","last_fullsized":"2024-09-13T23:01:58.231424+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:01:29.240246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:01:29.240246+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:36.497246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"75'1","reported_seq":905,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.231279+0000","last_change":"2024-09-13T23:01:13.190802+0000","last_active":"2024-09-13T23:01:58.231279+0000","last_peered":"2024-09-13T23:01:58.231279+0000","last_clean":"2024-09-13T23:01:58.231279+0000","last_became_active":"2024-09-13T22:58:42.748914+0000","last_became_peered":"2024-09-13T22:58:42.748914+0000","last_unstale":"2024-09-13T23:01:58.231279+0000","last_undegraded":"2024-09-13T23:01:58.231279+0000","last_fullsized":"2024-09-13T23:01:58.231279+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-13T23:01:13.190695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:13.190695+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.162269+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":902,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.622226+0000","last_change":"2024-09-13T23:01:21.888588+0000","last_active":"2024-09-13T23:01:56.622226+0000","last_peered":"2024-09-13T23:01:56.622226+0000","last_clean":"2024-09-13T23:01:56.622226+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T23:01:56.622226+0000","last_undegraded":"2024-09-13T23:01:56.622226+0000","last_fullsized":"2024-09-13T23:01:56.622226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:21.888398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T23:01:21.888398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:41.335531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":144,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.235869+0000","last_change":"2024-09-13T22:58:42.750451+0000","last_active":"2024-09-13T23:01:58.235869+0000","last_peered":"2024-09-13T23:01:58.235869+0000","last_clean":"2024-09-13T23:01:58.235869+0000","last_became_active":"2024-09-13T22:58:42.748790+0000","last_became_peered":"2024-09-13T22:58:42.748790+0000","last_unstale":"2024-09-13T23:01:58.235869+0000","last_undegraded":"2024-09-13T23:01:58.235869+0000","last_fullsized":"2024-09-13T23:01:58.235869+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T22:58:08.025258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T22:58:08.025258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":152,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233877+0000","last_change":"2024-09-13T23:01:23.585275+0000","last_active":"2024-09-13T23:01:58.233877+0000","last_peered":"2024-09-13T23:01:58.233877+0000","last_clean":"2024-09-13T23:01:58.233877+0000","last_became_active":"2024-09-13T22:58:42.746020+0000","last_became_peered":"2024-09-13T22:58:42.746020+0000","last_unstale":"2024-09-13T23:01:58.233877+0000","last_undegraded":"2024-09-13T23:01:58.233877+0000","last_fullsized":"2024-09-13T23:01:58.233877+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:23.585081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:23.585081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:23.925002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":957,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T23:01:56.621927+0000","last_change":"2024-09-13T22:58:12.352800+0000","last_active":"2024-09-13T23:01:56.621927+0000","last_peered":"2024-09-13T23:01:56.621927+0000","last_clean":"2024-09-13T23:01:56.621927+0000","last_became_active":"2024-09-13T22:55:38.138369+0000","last_became_peered":"2024-09-13T22:55:38.138369+0000","last_unstale":"2024-09-13T23:01:56.621927+0000","last_undegraded":"2024-09-13T23:01:56.621927+0000","last_fullsized":"2024-09-13T23:01:56.621927+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'11","last_scrub_stamp":"2024-09-13T22:58:12.352675+0000","last_deep_scrub":"100'7","last_deep_scrub_stamp":"2024-09-13T22:38:58.572643+0000","last_clean_scrub_stamp":"2024-09-13T22:58:12.352675+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":918,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232079+0000","last_change":"2024-09-13T23:01:36.948975+0000","last_active":"2024-09-13T23:01:58.232079+0000","last_peered":"2024-09-13T23:01:58.232079+0000","last_clean":"2024-09-13T23:01:58.232079+0000","last_became_active":"2024-09-13T22:55:38.094235+0000","last_became_peered":"2024-09-13T22:55:38.094235+0000","last_unstale":"2024-09-13T23:01:58.232079+0000","last_undegraded":"2024-09-13T23:01:58.232079+0000","last_fullsized":"2024-09-13T23:01:58.232079+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:36.948688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:17.313617+0000","last_clean_scrub_stamp":"2024-09-13T23:01:36.948688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:03:02.234273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"82'1","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.234272+0000","last_change":"2024-09-13T23:01:15.143950+0000","last_active":"2024-09-13T23:01:58.234272+0000","last_peered":"2024-09-13T23:01:58.234272+0000","last_clean":"2024-09-13T23:01:58.234272+0000","last_became_active":"2024-09-13T22:58:42.748249+0000","last_became_peered":"2024-09-13T22:58:42.748249+0000","last_unstale":"2024-09-13T23:01:58.234272+0000","last_undegraded":"2024-09-13T23:01:58.234272+0000","last_fullsized":"2024-09-13T23:01:58.234272+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'1","last_scrub_stamp":"2024-09-13T23:01:15.143836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:01:15.143836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:37.403472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.232719+0000","last_change":"2024-09-13T23:01:33.171325+0000","last_active":"2024-09-13T23:01:58.232719+0000","last_peered":"2024-09-13T23:01:58.232719+0000","last_clean":"2024-09-13T23:01:58.232719+0000","last_became_active":"2024-09-13T22:58:42.769318+0000","last_became_peered":"2024-09-13T22:58:42.769318+0000","last_unstale":"2024-09-13T23:01:58.232719+0000","last_undegraded":"2024-09-13T23:01:58.232719+0000","last_fullsized":"2024-09-13T23:01:58.232719+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:33.171231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:01:33.171231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.663204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":746,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233268+0000","last_change":"2024-09-13T23:01:22.827073+0000","last_active":"2024-09-13T23:01:58.233268+0000","last_peered":"2024-09-13T23:01:58.233268+0000","last_clean":"2024-09-13T23:01:58.233268+0000","last_became_active":"2024-09-13T22:55:38.099550+0000","last_became_peered":"2024-09-13T22:55:38.099550+0000","last_unstale":"2024-09-13T23:01:58.233268+0000","last_undegraded":"2024-09-13T23:01:58.233268+0000","last_fullsized":"2024-09-13T23:01:58.233268+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:01:22.826820+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T23:01:22.826820+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:40.286016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":394,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T23:01:58.233565+0000","last_change":"2024-09-13T23:01:30.227188+0000","last_active":"2024-09-13T23:01:58.233565+0000","last_peered":"2024-09-13T23:01:58.233565+0000","last_clean":"2024-09-13T23:01:58.233565+0000","last_became_active":"2024-09-13T22:55:38.133425+0000","last_became_peered":"2024-09-13T22:55:38.133425+0000","last_unstale":"2024-09-13T23:01:58.233565+0000","last_undegraded":"2024-09-13T23:01:58.233565+0000","last_fullsized":"2024-09-13T23:01:58.233565+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:01:30.227003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T23:01:30.227003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:02:32.997184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1430,"num_read_kb":1425,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1555,"num_read_kb":1312,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":432,"num_bytes_recovered":8470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":12866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1063,"ondisk_log_size":1063,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809132,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722488,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93022664,"statfs":{"total":95995035648,"available":95255207936,"internally_reserved":0,"allocated":1343488,"data_stored":849522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136183,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762400,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982752,"statfs":{"total":95995035648,"available":95214338048,"internally_reserved":0,"allocated":991232,"data_stored":519630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528642,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668580,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93076572,"statfs":{"total":95995035648,"available":95310409728,"internally_reserved":0,"allocated":1589248,"data_stored":1110272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":146028888396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084630,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889316,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92855836,"statfs":{"total":95995035648,"available":95084376064,"internally_reserved":0,"allocated":1585152,"data_stored":1110647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378978,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614120,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93131032,"statfs":{"total":95995035648,"available":95366176768,"internally_reserved":0,"allocated":1265664,"data_stored":783344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":941948,"kb_used_data":130536,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92803204,"statfs":{"total":95995035648,"available":95030480896,"internally_reserved":0,"allocated":133668864,"data_stored":133166245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738742,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822092,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92923060,"statfs":{"total":95995035648,"available":95153213440,"internally_reserved":0,"allocated":978944,"data_stored":508118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T23:02:02.575 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T23:02:02.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T23:02:02.575 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T23:02:02.575 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T23:02:02.575 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T23:02:02.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T23:02:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:02 smithi192 bash[24608]: cluster 2024-09-13T23:02:01.018132+0000 mgr.y (mgr.14613) 903 : cluster [DBG] pgmap v1093: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:02 smithi192 bash[24608]: audit 2024-09-13T23:02:01.704188+0000 mgr.y (mgr.14613) 904 : audit [DBG] from='client.16077 v1:172.21.15.178:0/732579690' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:02:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:02 smithi192 bash[24608]: cluster 2024-09-13T23:02:02.195687+0000 osd.7 (osd.7) 466 : cluster [DBG] 2.5 deep-scrub starts 2024-09-13T23:02:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:02 smithi192 bash[24608]: cluster 2024-09-13T23:02:02.197329+0000 osd.7 (osd.7) 467 : cluster [DBG] 2.5 deep-scrub ok 2024-09-13T23:02:02.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[29889]: cluster 2024-09-13T23:02:01.018132+0000 mgr.y (mgr.14613) 903 : cluster [DBG] pgmap v1093: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:02.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[29889]: audit 2024-09-13T23:02:01.704188+0000 mgr.y (mgr.14613) 904 : audit [DBG] from='client.16077 v1:172.21.15.178:0/732579690' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:02:02.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[29889]: cluster 2024-09-13T23:02:02.195687+0000 osd.7 (osd.7) 466 : cluster [DBG] 2.5 deep-scrub starts 2024-09-13T23:02:02.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[29889]: cluster 2024-09-13T23:02:02.197329+0000 osd.7 (osd.7) 467 : cluster [DBG] 2.5 deep-scrub ok 2024-09-13T23:02:02.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[22840]: cluster 2024-09-13T23:02:01.018132+0000 mgr.y (mgr.14613) 903 : cluster [DBG] pgmap v1093: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:02.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[22840]: audit 2024-09-13T23:02:01.704188+0000 mgr.y (mgr.14613) 904 : audit [DBG] from='client.16077 v1:172.21.15.178:0/732579690' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:02:02.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[22840]: cluster 2024-09-13T23:02:02.195687+0000 osd.7 (osd.7) 466 : cluster [DBG] 2.5 deep-scrub starts 2024-09-13T23:02:02.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:02 smithi178 bash[22840]: cluster 2024-09-13T23:02:02.197329+0000 osd.7 (osd.7) 467 : cluster [DBG] 2.5 deep-scrub ok 2024-09-13T23:02:04.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:04 smithi192 bash[24608]: cluster 2024-09-13T23:02:02.557909+0000 osd.5 (osd.5) 495 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T23:02:04.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:04 smithi192 bash[24608]: cluster 2024-09-13T23:02:02.559685+0000 osd.5 (osd.5) 496 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T23:02:04.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:04 smithi192 bash[24608]: cluster 2024-09-13T23:02:03.019076+0000 mgr.y (mgr.14613) 905 : cluster [DBG] pgmap v1094: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 774 B/s rd, 0 op/s 2024-09-13T23:02:04.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:04 smithi192 bash[24608]: cluster 2024-09-13T23:02:03.205007+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-09-13T23:02:04.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[29889]: cluster 2024-09-13T23:02:02.557909+0000 osd.5 (osd.5) 495 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T23:02:04.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[29889]: cluster 2024-09-13T23:02:02.559685+0000 osd.5 (osd.5) 496 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T23:02:04.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[29889]: cluster 2024-09-13T23:02:03.019076+0000 mgr.y (mgr.14613) 905 : cluster [DBG] pgmap v1094: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 774 B/s rd, 0 op/s 2024-09-13T23:02:04.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[29889]: cluster 2024-09-13T23:02:03.205007+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-09-13T23:02:04.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[22840]: cluster 2024-09-13T23:02:02.557909+0000 osd.5 (osd.5) 495 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T23:02:04.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[22840]: cluster 2024-09-13T23:02:02.559685+0000 osd.5 (osd.5) 496 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T23:02:04.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[22840]: cluster 2024-09-13T23:02:03.019076+0000 mgr.y (mgr.14613) 905 : cluster [DBG] pgmap v1094: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 774 B/s rd, 0 op/s 2024-09-13T23:02:04.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:04 smithi178 bash[22840]: cluster 2024-09-13T23:02:03.205007+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-09-13T23:02:05.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:05 smithi192 bash[24608]: cluster 2024-09-13T23:02:03.531723+0000 osd.5 (osd.5) 497 : cluster [DBG] 4.1 scrub starts 2024-09-13T23:02:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:05 smithi192 bash[24608]: cluster 2024-09-13T23:02:03.534552+0000 osd.5 (osd.5) 498 : cluster [DBG] 4.1 scrub ok 2024-09-13T23:02:05.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:05 smithi192 bash[24608]: audit 2024-09-13T23:02:04.698323+0000 mon.b (mon.1) 302 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:05.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:05 smithi178 bash[29889]: cluster 2024-09-13T23:02:03.531723+0000 osd.5 (osd.5) 497 : cluster [DBG] 4.1 scrub starts 2024-09-13T23:02:05.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:05 smithi178 bash[29889]: cluster 2024-09-13T23:02:03.534552+0000 osd.5 (osd.5) 498 : cluster [DBG] 4.1 scrub ok 2024-09-13T23:02:05.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:05 smithi178 bash[29889]: audit 2024-09-13T23:02:04.698323+0000 mon.b (mon.1) 302 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:05 smithi178 bash[22840]: cluster 2024-09-13T23:02:03.531723+0000 osd.5 (osd.5) 497 : cluster [DBG] 4.1 scrub starts 2024-09-13T23:02:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:05 smithi178 bash[22840]: cluster 2024-09-13T23:02:03.534552+0000 osd.5 (osd.5) 498 : cluster [DBG] 4.1 scrub ok 2024-09-13T23:02:05.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:05 smithi178 bash[22840]: audit 2024-09-13T23:02:04.698323+0000 mon.b (mon.1) 302 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:06.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:02:06 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:02:06.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:06 smithi192 bash[24608]: cluster 2024-09-13T23:02:04.708620+0000 osd.0 (osd.0) 437 : cluster [DBG] 6.9 scrub starts 2024-09-13T23:02:06.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:06 smithi192 bash[24608]: cluster 2024-09-13T23:02:04.710909+0000 osd.0 (osd.0) 438 : cluster [DBG] 6.9 scrub ok 2024-09-13T23:02:06.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:06 smithi192 bash[24608]: cluster 2024-09-13T23:02:05.020519+0000 mgr.y (mgr.14613) 906 : cluster [DBG] pgmap v1096: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:02:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:06 smithi178 bash[29889]: cluster 2024-09-13T23:02:04.708620+0000 osd.0 (osd.0) 437 : cluster [DBG] 6.9 scrub starts 2024-09-13T23:02:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:06 smithi178 bash[29889]: cluster 2024-09-13T23:02:04.710909+0000 osd.0 (osd.0) 438 : cluster [DBG] 6.9 scrub ok 2024-09-13T23:02:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:06 smithi178 bash[29889]: cluster 2024-09-13T23:02:05.020519+0000 mgr.y (mgr.14613) 906 : cluster [DBG] pgmap v1096: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:02:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:06 smithi178 bash[22840]: cluster 2024-09-13T23:02:04.708620+0000 osd.0 (osd.0) 437 : cluster [DBG] 6.9 scrub starts 2024-09-13T23:02:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:06 smithi178 bash[22840]: cluster 2024-09-13T23:02:04.710909+0000 osd.0 (osd.0) 438 : cluster [DBG] 6.9 scrub ok 2024-09-13T23:02:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:06 smithi178 bash[22840]: cluster 2024-09-13T23:02:05.020519+0000 mgr.y (mgr.14613) 906 : cluster [DBG] pgmap v1096: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:02:07.283 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:02:07.283 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:02:07.574 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:07 smithi192 bash[24608]: cluster 2024-09-13T23:02:05.489513+0000 osd.5 (osd.5) 499 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:02:07.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:07 smithi192 bash[24608]: cluster 2024-09-13T23:02:05.496742+0000 osd.5 (osd.5) 500 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:02:07.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:07 smithi192 bash[24608]: cluster 2024-09-13T23:02:05.673401+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:02:07.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:07 smithi192 bash[24608]: cluster 2024-09-13T23:02:05.676302+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:02:07.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:07 smithi192 bash[24608]: audit 2024-09-13T23:02:06.083204+0000 mgr.y (mgr.14613) 907 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[29889]: cluster 2024-09-13T23:02:05.489513+0000 osd.5 (osd.5) 499 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:02:07.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[29889]: cluster 2024-09-13T23:02:05.496742+0000 osd.5 (osd.5) 500 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[29889]: cluster 2024-09-13T23:02:05.673401+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[29889]: cluster 2024-09-13T23:02:05.676302+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[29889]: audit 2024-09-13T23:02:06.083204+0000 mgr.y (mgr.14613) 907 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[22840]: cluster 2024-09-13T23:02:05.489513+0000 osd.5 (osd.5) 499 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[22840]: cluster 2024-09-13T23:02:05.496742+0000 osd.5 (osd.5) 500 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[22840]: cluster 2024-09-13T23:02:05.673401+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[22840]: cluster 2024-09-13T23:02:05.676302+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:02:07.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:07 smithi178 bash[22840]: audit 2024-09-13T23:02:06.083204+0000 mgr.y (mgr.14613) 907 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:08.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:08 smithi192 bash[24608]: cluster 2024-09-13T23:02:07.021404+0000 mgr.y (mgr.14613) 908 : cluster [DBG] pgmap v1097: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:08.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:08 smithi192 bash[24608]: cluster 2024-09-13T23:02:07.195921+0000 osd.0 (osd.0) 441 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:08.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:08 smithi192 bash[24608]: cluster 2024-09-13T23:02:07.196068+0000 osd.0 (osd.0) 442 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:08.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:08 smithi192 bash[24608]: cluster 2024-09-13T23:02:07.227739+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-09-13T23:02:08.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:08 smithi192 bash[24608]: cluster 2024-09-13T23:02:08.207294+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-09-13T23:02:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[29889]: cluster 2024-09-13T23:02:07.021404+0000 mgr.y (mgr.14613) 908 : cluster [DBG] pgmap v1097: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[29889]: cluster 2024-09-13T23:02:07.195921+0000 osd.0 (osd.0) 441 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[29889]: cluster 2024-09-13T23:02:07.196068+0000 osd.0 (osd.0) 442 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[29889]: cluster 2024-09-13T23:02:07.227739+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[29889]: cluster 2024-09-13T23:02:08.207294+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[22840]: cluster 2024-09-13T23:02:07.021404+0000 mgr.y (mgr.14613) 908 : cluster [DBG] pgmap v1097: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[22840]: cluster 2024-09-13T23:02:07.195921+0000 osd.0 (osd.0) 441 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[22840]: cluster 2024-09-13T23:02:07.196068+0000 osd.0 (osd.0) 442 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[22840]: cluster 2024-09-13T23:02:07.227739+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-09-13T23:02:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:08 smithi178 bash[22840]: cluster 2024-09-13T23:02:08.207294+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-09-13T23:02:09.506 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:02:09.506 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:02:09.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:09 smithi192 bash[24608]: cluster 2024-09-13T23:02:07.710564+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:02:09.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:09 smithi192 bash[24608]: cluster 2024-09-13T23:02:07.713084+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:02:09.553 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-13T23:02:09.553 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "" 2024-09-13T23:02:09.553 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-13T23:02:09.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:09 smithi178 bash[22840]: cluster 2024-09-13T23:02:07.710564+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:02:09.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:09 smithi178 bash[22840]: cluster 2024-09-13T23:02:07.713084+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:02:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:09 smithi178 bash[29889]: cluster 2024-09-13T23:02:07.710564+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:02:09.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:09 smithi178 bash[29889]: cluster 2024-09-13T23:02:07.713084+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:02:10.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:02:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:02:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:02:10.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:10 smithi192 bash[24608]: cluster 2024-09-13T23:02:09.022281+0000 mgr.y (mgr.14613) 909 : cluster [DBG] pgmap v1100: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:10.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:10 smithi192 bash[24608]: cluster 2024-09-13T23:02:09.330365+0000 osd.6 (osd.6) 336 : cluster [DBG] 6.1e scrub starts 2024-09-13T23:02:10.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:10 smithi192 bash[24608]: cluster 2024-09-13T23:02:09.332273+0000 osd.6 (osd.6) 337 : cluster [DBG] 6.1e scrub ok 2024-09-13T23:02:10.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:10 smithi192 bash[24608]: cluster 2024-09-13T23:02:09.465502+0000 osd.3 (osd.3) 423 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:10.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:10 smithi192 bash[24608]: cluster 2024-09-13T23:02:09.465588+0000 osd.3 (osd.3) 424 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:10.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[22840]: cluster 2024-09-13T23:02:09.022281+0000 mgr.y (mgr.14613) 909 : cluster [DBG] pgmap v1100: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:10.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[22840]: cluster 2024-09-13T23:02:09.330365+0000 osd.6 (osd.6) 336 : cluster [DBG] 6.1e scrub starts 2024-09-13T23:02:10.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[22840]: cluster 2024-09-13T23:02:09.332273+0000 osd.6 (osd.6) 337 : cluster [DBG] 6.1e scrub ok 2024-09-13T23:02:10.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[22840]: cluster 2024-09-13T23:02:09.465502+0000 osd.3 (osd.3) 423 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:10.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[22840]: cluster 2024-09-13T23:02:09.465588+0000 osd.3 (osd.3) 424 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:10.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[29889]: cluster 2024-09-13T23:02:09.022281+0000 mgr.y (mgr.14613) 909 : cluster [DBG] pgmap v1100: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:10.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[29889]: cluster 2024-09-13T23:02:09.330365+0000 osd.6 (osd.6) 336 : cluster [DBG] 6.1e scrub starts 2024-09-13T23:02:10.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[29889]: cluster 2024-09-13T23:02:09.332273+0000 osd.6 (osd.6) 337 : cluster [DBG] 6.1e scrub ok 2024-09-13T23:02:10.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[29889]: cluster 2024-09-13T23:02:09.465502+0000 osd.3 (osd.3) 423 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:10.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:10 smithi178 bash[29889]: cluster 2024-09-13T23:02:09.465588+0000 osd.3 (osd.3) 424 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:10.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T23:02:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:11 smithi192 bash[24608]: cluster 2024-09-13T23:02:10.246079+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-09-13T23:02:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:11 smithi192 bash[24608]: cluster 2024-09-13T23:02:10.346690+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-13T23:02:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:11 smithi192 bash[24608]: cluster 2024-09-13T23:02:10.348679+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-13T23:02:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:11 smithi192 bash[24608]: cluster 2024-09-13T23:02:10.623101+0000 osd.3 (osd.3) 425 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:02:11.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:11 smithi192 bash[24608]: cluster 2024-09-13T23:02:10.629733+0000 osd.3 (osd.3) 426 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:02:11.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[22840]: cluster 2024-09-13T23:02:10.246079+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[22840]: cluster 2024-09-13T23:02:10.346690+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[22840]: cluster 2024-09-13T23:02:10.348679+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[22840]: cluster 2024-09-13T23:02:10.623101+0000 osd.3 (osd.3) 425 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[22840]: cluster 2024-09-13T23:02:10.629733+0000 osd.3 (osd.3) 426 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[29889]: cluster 2024-09-13T23:02:10.246079+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[29889]: cluster 2024-09-13T23:02:10.346690+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[29889]: cluster 2024-09-13T23:02:10.348679+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[29889]: cluster 2024-09-13T23:02:10.623101+0000 osd.3 (osd.3) 425 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:02:11.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:11 smithi178 bash[29889]: cluster 2024-09-13T23:02:10.629733+0000 osd.3 (osd.3) 426 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:02:12.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:12 smithi192 bash[24608]: cluster 2024-09-13T23:02:11.023004+0000 mgr.y (mgr.14613) 910 : cluster [DBG] pgmap v1102: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:12.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:12 smithi192 bash[24608]: cluster 2024-09-13T23:02:11.038852+0000 osd.7 (osd.7) 468 : cluster [DBG] 1.9 scrub starts 2024-09-13T23:02:12.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:12 smithi192 bash[24608]: cluster 2024-09-13T23:02:11.041604+0000 osd.7 (osd.7) 469 : cluster [DBG] 1.9 scrub ok 2024-09-13T23:02:12.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:12 smithi178 bash[29889]: cluster 2024-09-13T23:02:11.023004+0000 mgr.y (mgr.14613) 910 : cluster [DBG] pgmap v1102: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:12.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:12 smithi178 bash[29889]: cluster 2024-09-13T23:02:11.038852+0000 osd.7 (osd.7) 468 : cluster [DBG] 1.9 scrub starts 2024-09-13T23:02:12.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:12 smithi178 bash[29889]: cluster 2024-09-13T23:02:11.041604+0000 osd.7 (osd.7) 469 : cluster [DBG] 1.9 scrub ok 2024-09-13T23:02:12.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:12 smithi178 bash[22840]: cluster 2024-09-13T23:02:11.023004+0000 mgr.y (mgr.14613) 910 : cluster [DBG] pgmap v1102: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:12.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:12 smithi178 bash[22840]: cluster 2024-09-13T23:02:11.038852+0000 osd.7 (osd.7) 468 : cluster [DBG] 1.9 scrub starts 2024-09-13T23:02:12.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:12 smithi178 bash[22840]: cluster 2024-09-13T23:02:11.041604+0000 osd.7 (osd.7) 469 : cluster [DBG] 1.9 scrub ok 2024-09-13T23:02:13.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:13 smithi192 bash[24608]: cluster 2024-09-13T23:02:11.656260+0000 osd.0 (osd.0) 443 : cluster [DBG] 3.1 deep-scrub starts 2024-09-13T23:02:13.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:13 smithi192 bash[24608]: cluster 2024-09-13T23:02:11.668667+0000 osd.0 (osd.0) 444 : cluster [DBG] 3.1 deep-scrub ok 2024-09-13T23:02:13.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:13 smithi192 bash[24608]: cluster 2024-09-13T23:02:12.609385+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T23:02:13.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:13 smithi192 bash[24608]: cluster 2024-09-13T23:02:12.611707+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T23:02:13.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:13 smithi192 bash[24608]: cluster 2024-09-13T23:02:13.210634+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-09-13T23:02:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[29889]: cluster 2024-09-13T23:02:11.656260+0000 osd.0 (osd.0) 443 : cluster [DBG] 3.1 deep-scrub starts 2024-09-13T23:02:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[29889]: cluster 2024-09-13T23:02:11.668667+0000 osd.0 (osd.0) 444 : cluster [DBG] 3.1 deep-scrub ok 2024-09-13T23:02:13.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[29889]: cluster 2024-09-13T23:02:12.609385+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[29889]: cluster 2024-09-13T23:02:12.611707+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[29889]: cluster 2024-09-13T23:02:13.210634+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[22840]: cluster 2024-09-13T23:02:11.656260+0000 osd.0 (osd.0) 443 : cluster [DBG] 3.1 deep-scrub starts 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[22840]: cluster 2024-09-13T23:02:11.668667+0000 osd.0 (osd.0) 444 : cluster [DBG] 3.1 deep-scrub ok 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[22840]: cluster 2024-09-13T23:02:12.609385+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.b deep-scrub starts 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[22840]: cluster 2024-09-13T23:02:12.611707+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.b deep-scrub ok 2024-09-13T23:02:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:13 smithi178 bash[22840]: cluster 2024-09-13T23:02:13.210634+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-09-13T23:02:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:14 smithi192 bash[24608]: cluster 2024-09-13T23:02:12.560818+0000 osd.5 (osd.5) 501 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T23:02:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:14 smithi192 bash[24608]: cluster 2024-09-13T23:02:12.563100+0000 osd.5 (osd.5) 502 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T23:02:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:14 smithi192 bash[24608]: cluster 2024-09-13T23:02:13.023843+0000 mgr.y (mgr.14613) 911 : cluster [DBG] pgmap v1103: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:14.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:14 smithi178 bash[29889]: cluster 2024-09-13T23:02:12.560818+0000 osd.5 (osd.5) 501 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T23:02:14.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:14 smithi178 bash[29889]: cluster 2024-09-13T23:02:12.563100+0000 osd.5 (osd.5) 502 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T23:02:14.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:14 smithi178 bash[29889]: cluster 2024-09-13T23:02:13.023843+0000 mgr.y (mgr.14613) 911 : cluster [DBG] pgmap v1103: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:14.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:14 smithi178 bash[22840]: cluster 2024-09-13T23:02:12.560818+0000 osd.5 (osd.5) 501 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T23:02:14.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:14 smithi178 bash[22840]: cluster 2024-09-13T23:02:12.563100+0000 osd.5 (osd.5) 502 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T23:02:14.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:14 smithi178 bash[22840]: cluster 2024-09-13T23:02:13.023843+0000 mgr.y (mgr.14613) 911 : cluster [DBG] pgmap v1103: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:02:15.578 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T23:02:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:15 smithi192 bash[24608]: cluster 2024-09-13T23:02:13.710343+0000 osd.1 (osd.1) 446 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:02:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:15 smithi192 bash[24608]: cluster 2024-09-13T23:02:13.713200+0000 osd.1 (osd.1) 447 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:02:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:15 smithi192 bash[24608]: cluster 2024-09-13T23:02:14.278923+0000 osd.6 (osd.6) 340 : cluster [DBG] 5.2 scrub starts 2024-09-13T23:02:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:15 smithi192 bash[24608]: cluster 2024-09-13T23:02:14.281109+0000 osd.6 (osd.6) 341 : cluster [DBG] 5.2 scrub ok 2024-09-13T23:02:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[29889]: cluster 2024-09-13T23:02:13.710343+0000 osd.1 (osd.1) 446 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:02:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[29889]: cluster 2024-09-13T23:02:13.713200+0000 osd.1 (osd.1) 447 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:02:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[29889]: cluster 2024-09-13T23:02:14.278923+0000 osd.6 (osd.6) 340 : cluster [DBG] 5.2 scrub starts 2024-09-13T23:02:15.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[29889]: cluster 2024-09-13T23:02:14.281109+0000 osd.6 (osd.6) 341 : cluster [DBG] 5.2 scrub ok 2024-09-13T23:02:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[22840]: cluster 2024-09-13T23:02:13.710343+0000 osd.1 (osd.1) 446 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:02:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[22840]: cluster 2024-09-13T23:02:13.713200+0000 osd.1 (osd.1) 447 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:02:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[22840]: cluster 2024-09-13T23:02:14.278923+0000 osd.6 (osd.6) 340 : cluster [DBG] 5.2 scrub starts 2024-09-13T23:02:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:15 smithi178 bash[22840]: cluster 2024-09-13T23:02:14.281109+0000 osd.6 (osd.6) 341 : cluster [DBG] 5.2 scrub ok 2024-09-13T23:02:15.807 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T23:02:15.807 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T23:02:15.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T23:02:16.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:02:16 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:02:16.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:16 smithi192 bash[24608]: cluster 2024-09-13T23:02:15.025263+0000 mgr.y (mgr.14613) 912 : cluster [DBG] pgmap v1105: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:02:16.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:16 smithi192 bash[24608]: cluster 2024-09-13T23:02:16.048731+0000 osd.7 (osd.7) 470 : cluster [DBG] 2.b scrub starts 2024-09-13T23:02:16.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:16 smithi192 bash[24608]: cluster 2024-09-13T23:02:16.050841+0000 osd.7 (osd.7) 471 : cluster [DBG] 2.b scrub ok 2024-09-13T23:02:16.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:16 smithi178 bash[29889]: cluster 2024-09-13T23:02:15.025263+0000 mgr.y (mgr.14613) 912 : cluster [DBG] pgmap v1105: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:02:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:16 smithi178 bash[29889]: cluster 2024-09-13T23:02:16.048731+0000 osd.7 (osd.7) 470 : cluster [DBG] 2.b scrub starts 2024-09-13T23:02:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:16 smithi178 bash[29889]: cluster 2024-09-13T23:02:16.050841+0000 osd.7 (osd.7) 471 : cluster [DBG] 2.b scrub ok 2024-09-13T23:02:16.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:16 smithi178 bash[22840]: cluster 2024-09-13T23:02:15.025263+0000 mgr.y (mgr.14613) 912 : cluster [DBG] pgmap v1105: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:02:16.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:16 smithi178 bash[22840]: cluster 2024-09-13T23:02:16.048731+0000 osd.7 (osd.7) 470 : cluster [DBG] 2.b scrub starts 2024-09-13T23:02:16.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:16 smithi178 bash[22840]: cluster 2024-09-13T23:02:16.050841+0000 osd.7 (osd.7) 471 : cluster [DBG] 2.b scrub ok 2024-09-13T23:02:17.054 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-09-13T23:02:17.055 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T23:02:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:17 smithi192 bash[24608]: cluster 2024-09-13T23:02:15.693504+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:02:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:17 smithi192 bash[24608]: cluster 2024-09-13T23:02:15.696145+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:02:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:17 smithi192 bash[24608]: audit 2024-09-13T23:02:16.094035+0000 mgr.y (mgr.14613) 913 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:17 smithi178 bash[29889]: cluster 2024-09-13T23:02:15.693504+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:02:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:17 smithi178 bash[29889]: cluster 2024-09-13T23:02:15.696145+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:02:17.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:17 smithi178 bash[29889]: audit 2024-09-13T23:02:16.094035+0000 mgr.y (mgr.14613) 913 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:17 smithi178 bash[22840]: cluster 2024-09-13T23:02:15.693504+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:02:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:17 smithi178 bash[22840]: cluster 2024-09-13T23:02:15.696145+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:02:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:17 smithi178 bash[22840]: audit 2024-09-13T23:02:16.094035+0000 mgr.y (mgr.14613) 913 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:16.609929+0000 osd.5 (osd.5) 503 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:16.612461+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:16.656885+0000 osd.0 (osd.0) 445 : cluster [DBG] 3.17 deep-scrub starts 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:16.665428+0000 osd.0 (osd.0) 446 : cluster [DBG] 3.17 deep-scrub ok 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.025963+0000 mgr.y (mgr.14613) 914 : cluster [DBG] pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.189421+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:02:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.191465+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:02:18.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.353094+0000 osd.5 (osd.5) 505 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:18.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.353214+0000 osd.5 (osd.5) 506 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:18.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.604958+0000 osd.3 (osd.3) 431 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:02:18.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:18 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.608955+0000 osd.3 (osd.3) 432 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:02:18.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:16.609929+0000 osd.5 (osd.5) 503 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:16.612461+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:16.656885+0000 osd.0 (osd.0) 445 : cluster [DBG] 3.17 deep-scrub starts 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:16.665428+0000 osd.0 (osd.0) 446 : cluster [DBG] 3.17 deep-scrub ok 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.025963+0000 mgr.y (mgr.14613) 914 : cluster [DBG] pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.189421+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.191465+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.353094+0000 osd.5 (osd.5) 505 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.353214+0000 osd.5 (osd.5) 506 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.604958+0000 osd.3 (osd.3) 431 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.608955+0000 osd.3 (osd.3) 432 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:02:18.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:16.609929+0000 osd.5 (osd.5) 503 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:16.612461+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:16.656885+0000 osd.0 (osd.0) 445 : cluster [DBG] 3.17 deep-scrub starts 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:16.665428+0000 osd.0 (osd.0) 446 : cluster [DBG] 3.17 deep-scrub ok 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.025963+0000 mgr.y (mgr.14613) 914 : cluster [DBG] pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.189421+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.191465+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.353094+0000 osd.5 (osd.5) 505 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.353214+0000 osd.5 (osd.5) 506 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.604958+0000 osd.3 (osd.3) 431 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:02:18.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:18 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.608955+0000 osd.3 (osd.3) 432 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:02:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:19 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.638955+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:02:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:19 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.643778+0000 osd.5 (osd.5) 508 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:02:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:19 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.668247+0000 osd.0 (osd.0) 447 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:02:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:19 smithi192 bash[24608]: cluster 2024-09-13T23:02:17.670450+0000 osd.0 (osd.0) 448 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:02:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:19 smithi192 bash[24608]: cluster 2024-09-13T23:02:18.298641+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-09-13T23:02:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.638955+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:02:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.643778+0000 osd.5 (osd.5) 508 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:02:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.668247+0000 osd.0 (osd.0) 447 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:02:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[29889]: cluster 2024-09-13T23:02:17.670450+0000 osd.0 (osd.0) 448 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:02:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[29889]: cluster 2024-09-13T23:02:18.298641+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-09-13T23:02:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.638955+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:02:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.643778+0000 osd.5 (osd.5) 508 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:02:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.668247+0000 osd.0 (osd.0) 447 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:02:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[22840]: cluster 2024-09-13T23:02:17.670450+0000 osd.0 (osd.0) 448 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:02:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[22840]: cluster 2024-09-13T23:02:18.298641+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-09-13T23:02:20.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:02:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:02:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:02:20.603 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:02:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:20 smithi192 bash[24608]: cluster 2024-09-13T23:02:18.683865+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:02:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:20 smithi192 bash[24608]: cluster 2024-09-13T23:02:18.687214+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:02:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:20 smithi192 bash[24608]: cluster 2024-09-13T23:02:19.026848+0000 mgr.y (mgr.14613) 915 : cluster [DBG] pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:20 smithi192 bash[24608]: audit 2024-09-13T23:02:19.728675+0000 mon.b (mon.1) 303 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[29889]: cluster 2024-09-13T23:02:18.683865+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:02:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[29889]: cluster 2024-09-13T23:02:18.687214+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:02:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[29889]: cluster 2024-09-13T23:02:19.026848+0000 mgr.y (mgr.14613) 915 : cluster [DBG] pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[29889]: audit 2024-09-13T23:02:19.728675+0000 mon.b (mon.1) 303 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[22840]: cluster 2024-09-13T23:02:18.683865+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:02:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[22840]: cluster 2024-09-13T23:02:18.687214+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:02:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[22840]: cluster 2024-09-13T23:02:19.026848+0000 mgr.y (mgr.14613) 915 : cluster [DBG] pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:20 smithi178 bash[22840]: audit 2024-09-13T23:02:19.728675+0000 mon.b (mon.1) 303 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:21 smithi192 bash[24608]: cluster 2024-09-13T23:02:20.383592+0000 osd.2 (osd.2) 327 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:21 smithi192 bash[24608]: cluster 2024-09-13T23:02:20.383697+0000 osd.2 (osd.2) 328 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:21 smithi192 bash[24608]: cluster 2024-09-13T23:02:20.575270+0000 osd.3 (osd.3) 433 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:02:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:21 smithi192 bash[24608]: cluster 2024-09-13T23:02:20.578132+0000 osd.3 (osd.3) 434 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:02:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:21 smithi192 bash[24608]: cluster 2024-09-13T23:02:20.590974+0000 osd.1 (osd.1) 448 : cluster [DBG] 6.1d deep-scrub starts 2024-09-13T23:02:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:21 smithi192 bash[24608]: cluster 2024-09-13T23:02:20.593126+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1d deep-scrub ok 2024-09-13T23:02:21.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[29889]: cluster 2024-09-13T23:02:20.383592+0000 osd.2 (osd.2) 327 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[29889]: cluster 2024-09-13T23:02:20.383697+0000 osd.2 (osd.2) 328 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[29889]: cluster 2024-09-13T23:02:20.575270+0000 osd.3 (osd.3) 433 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[29889]: cluster 2024-09-13T23:02:20.578132+0000 osd.3 (osd.3) 434 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[29889]: cluster 2024-09-13T23:02:20.590974+0000 osd.1 (osd.1) 448 : cluster [DBG] 6.1d deep-scrub starts 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[29889]: cluster 2024-09-13T23:02:20.593126+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1d deep-scrub ok 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[22840]: cluster 2024-09-13T23:02:20.383592+0000 osd.2 (osd.2) 327 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[22840]: cluster 2024-09-13T23:02:20.383697+0000 osd.2 (osd.2) 328 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[22840]: cluster 2024-09-13T23:02:20.575270+0000 osd.3 (osd.3) 433 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[22840]: cluster 2024-09-13T23:02:20.578132+0000 osd.3 (osd.3) 434 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[22840]: cluster 2024-09-13T23:02:20.590974+0000 osd.1 (osd.1) 448 : cluster [DBG] 6.1d deep-scrub starts 2024-09-13T23:02:21.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:21 smithi178 bash[22840]: cluster 2024-09-13T23:02:20.593126+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1d deep-scrub ok 2024-09-13T23:02:22.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:22 smithi178 bash[29889]: cluster 2024-09-13T23:02:21.028203+0000 mgr.y (mgr.14613) 916 : cluster [DBG] pgmap v1109: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:22.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:22 smithi178 bash[29889]: cluster 2024-09-13T23:02:21.324601+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-09-13T23:02:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:22 smithi178 bash[22840]: cluster 2024-09-13T23:02:21.028203+0000 mgr.y (mgr.14613) 916 : cluster [DBG] pgmap v1109: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:22.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:22 smithi178 bash[22840]: cluster 2024-09-13T23:02:21.324601+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-09-13T23:02:22.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:22 smithi192 bash[24608]: cluster 2024-09-13T23:02:21.028203+0000 mgr.y (mgr.14613) 916 : cluster [DBG] pgmap v1109: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:22.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:22 smithi192 bash[24608]: cluster 2024-09-13T23:02:21.324601+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-09-13T23:02:22.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:22 smithi178 bash[22840]: debug 2024-09-13T23:02:22.433+0000 7f8b8e05d640 -1 mon.a@0(leader).osd e488 definitely_dead 0 2024-09-13T23:02:23.327 INFO:teuthology.orchestra.run.smithi178.stderr:marked in osd.4. 2024-09-13T23:02:23.657 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:23 smithi192 bash[24608]: audit 2024-09-13T23:02:22.436818+0000 mon.a (mon.0) 1865 : audit [DBG] from='client.? v1:172.21.15.178:0/587656906' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T23:02:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:23 smithi178 bash[29889]: audit 2024-09-13T23:02:22.436818+0000 mon.a (mon.0) 1865 : audit [DBG] from='client.? v1:172.21.15.178:0/587656906' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T23:02:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:23 smithi178 bash[22840]: audit 2024-09-13T23:02:22.436818+0000 mon.a (mon.0) 1865 : audit [DBG] from='client.? v1:172.21.15.178:0/587656906' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-13T23:02:24.217 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T23:02:24.217 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T23:02:24.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:22.625128+0000 osd.5 (osd.5) 509 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:22.627952+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:23.029036+0000 mgr.y (mgr.14613) 917 : cluster [DBG] pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: audit 2024-09-13T23:02:23.327853+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.178:0/587656906' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:23.335439+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:23.597625+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:23.599839+0000 osd.1 (osd.1) 451 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:02:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:23.630874+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:02:24.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:24 smithi192 bash[24608]: cluster 2024-09-13T23:02:23.634852+0000 osd.3 (osd.3) 436 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:22.625128+0000 osd.5 (osd.5) 509 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:22.627952+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:23.029036+0000 mgr.y (mgr.14613) 917 : cluster [DBG] pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: audit 2024-09-13T23:02:23.327853+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.178:0/587656906' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:23.335439+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:23.597625+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:23.599839+0000 osd.1 (osd.1) 451 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:23.630874+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:02:24.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[29889]: cluster 2024-09-13T23:02:23.634852+0000 osd.3 (osd.3) 436 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:22.625128+0000 osd.5 (osd.5) 509 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:22.627952+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:23.029036+0000 mgr.y (mgr.14613) 917 : cluster [DBG] pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: audit 2024-09-13T23:02:23.327853+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? v1:172.21.15.178:0/587656906' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:23.335439+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:23.597625+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:23.599839+0000 osd.1 (osd.1) 451 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:23.630874+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:02:24.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:24 smithi178 bash[22840]: cluster 2024-09-13T23:02:23.634852+0000 osd.3 (osd.3) 436 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:02:25.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:25 smithi192 bash[24608]: cluster 2024-09-13T23:02:24.338628+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2024-09-13T23:02:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:25 smithi178 bash[22840]: cluster 2024-09-13T23:02:24.338628+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2024-09-13T23:02:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:25 smithi178 bash[29889]: cluster 2024-09-13T23:02:24.338628+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2024-09-13T23:02:26.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:24.659156+0000 osd.2 (osd.2) 329 : cluster [DBG] 3.19 scrub starts 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:24.663851+0000 osd.2 (osd.2) 330 : cluster [DBG] 3.19 scrub ok 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:24.672749+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:24.674931+0000 osd.5 (osd.5) 512 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.030318+0000 mgr.y (mgr.14613) 918 : cluster [DBG] pgmap v1114: 142 pgs: 1 remapped+peering, 1 unknown, 39 peering, 101 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 47 B/s, 0 objects/s recovering 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.343286+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.360102+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.795923+0000 osd.4 (osd.4) 282 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.800193+0000 osd.4 (osd.4) 283 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:02:26.357 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:26.013778+0000 osd.7 (osd.7) 472 : cluster [DBG] 3.d scrub starts 2024-09-13T23:02:26.358 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:26 smithi192 bash[24608]: cluster 2024-09-13T23:02:26.018865+0000 osd.7 (osd.7) 473 : cluster [DBG] 3.d scrub ok 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:24.659156+0000 osd.2 (osd.2) 329 : cluster [DBG] 3.19 scrub starts 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:24.663851+0000 osd.2 (osd.2) 330 : cluster [DBG] 3.19 scrub ok 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:24.672749+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:24.674931+0000 osd.5 (osd.5) 512 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.030318+0000 mgr.y (mgr.14613) 918 : cluster [DBG] pgmap v1114: 142 pgs: 1 remapped+peering, 1 unknown, 39 peering, 101 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 47 B/s, 0 objects/s recovering 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.343286+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.360102+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.795923+0000 osd.4 (osd.4) 282 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:02:26.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.800193+0000 osd.4 (osd.4) 283 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:26.013778+0000 osd.7 (osd.7) 472 : cluster [DBG] 3.d scrub starts 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[29889]: cluster 2024-09-13T23:02:26.018865+0000 osd.7 (osd.7) 473 : cluster [DBG] 3.d scrub ok 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:24.659156+0000 osd.2 (osd.2) 329 : cluster [DBG] 3.19 scrub starts 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:24.663851+0000 osd.2 (osd.2) 330 : cluster [DBG] 3.19 scrub ok 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:24.672749+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:24.674931+0000 osd.5 (osd.5) 512 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.030318+0000 mgr.y (mgr.14613) 918 : cluster [DBG] pgmap v1114: 142 pgs: 1 remapped+peering, 1 unknown, 39 peering, 101 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 47 B/s, 0 objects/s recovering 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.343286+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.360102+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2024-09-13T23:02:26.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.795923+0000 osd.4 (osd.4) 282 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:02:26.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.800193+0000 osd.4 (osd.4) 283 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:02:26.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:26.013778+0000 osd.7 (osd.7) 472 : cluster [DBG] 3.d scrub starts 2024-09-13T23:02:26.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:26 smithi178 bash[22840]: cluster 2024-09-13T23:02:26.018865+0000 osd.7 (osd.7) 473 : cluster [DBG] 3.d scrub ok 2024-09-13T23:02:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:27 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.649759+0000 osd.5 (osd.5) 513 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:02:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:27 smithi192 bash[24608]: cluster 2024-09-13T23:02:25.654352+0000 osd.5 (osd.5) 514 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:02:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:27 smithi192 bash[24608]: audit 2024-09-13T23:02:26.105289+0000 mgr.y (mgr.14613) 919 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:27 smithi192 bash[24608]: cluster 2024-09-13T23:02:26.369775+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2024-09-13T23:02:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:27 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.026152+0000 osd.7 (osd.7) 474 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:02:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:27 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.028526+0000 osd.7 (osd.7) 475 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:02:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.649759+0000 osd.5 (osd.5) 513 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:02:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[29889]: cluster 2024-09-13T23:02:25.654352+0000 osd.5 (osd.5) 514 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[29889]: audit 2024-09-13T23:02:26.105289+0000 mgr.y (mgr.14613) 919 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[29889]: cluster 2024-09-13T23:02:26.369775+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.026152+0000 osd.7 (osd.7) 474 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.028526+0000 osd.7 (osd.7) 475 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.649759+0000 osd.5 (osd.5) 513 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[22840]: cluster 2024-09-13T23:02:25.654352+0000 osd.5 (osd.5) 514 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[22840]: audit 2024-09-13T23:02:26.105289+0000 mgr.y (mgr.14613) 919 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[22840]: cluster 2024-09-13T23:02:26.369775+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.026152+0000 osd.7 (osd.7) 474 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:02:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:27 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.028526+0000 osd.7 (osd.7) 475 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.030999+0000 mgr.y (mgr.14613) 920 : cluster [DBG] pgmap v1117: 142 pgs: 1 remapped+peering, 1 unknown, 39 peering, 101 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 98 B/s, 4 objects/s recovering 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.380993+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.580062+0000 osd.3 (osd.3) 437 : cluster [DBG] 5.0 scrub starts 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.583441+0000 osd.3 (osd.3) 438 : cluster [DBG] 5.0 scrub ok 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.849875+0000 osd.4 (osd.4) 284 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.852297+0000 osd.4 (osd.4) 285 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:02:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:28.068036+0000 osd.7 (osd.7) 476 : cluster [DBG] 6.3 scrub starts 2024-09-13T23:02:28.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:28 smithi192 bash[24608]: cluster 2024-09-13T23:02:28.070391+0000 osd.7 (osd.7) 477 : cluster [DBG] 6.3 scrub ok 2024-09-13T23:02:28.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.030999+0000 mgr.y (mgr.14613) 920 : cluster [DBG] pgmap v1117: 142 pgs: 1 remapped+peering, 1 unknown, 39 peering, 101 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 98 B/s, 4 objects/s recovering 2024-09-13T23:02:28.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.380993+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2024-09-13T23:02:28.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.580062+0000 osd.3 (osd.3) 437 : cluster [DBG] 5.0 scrub starts 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.583441+0000 osd.3 (osd.3) 438 : cluster [DBG] 5.0 scrub ok 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.849875+0000 osd.4 (osd.4) 284 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.852297+0000 osd.4 (osd.4) 285 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:28.068036+0000 osd.7 (osd.7) 476 : cluster [DBG] 6.3 scrub starts 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[29889]: cluster 2024-09-13T23:02:28.070391+0000 osd.7 (osd.7) 477 : cluster [DBG] 6.3 scrub ok 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.030999+0000 mgr.y (mgr.14613) 920 : cluster [DBG] pgmap v1117: 142 pgs: 1 remapped+peering, 1 unknown, 39 peering, 101 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 98 B/s, 4 objects/s recovering 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.380993+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.580062+0000 osd.3 (osd.3) 437 : cluster [DBG] 5.0 scrub starts 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.583441+0000 osd.3 (osd.3) 438 : cluster [DBG] 5.0 scrub ok 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.849875+0000 osd.4 (osd.4) 284 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:02:28.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.852297+0000 osd.4 (osd.4) 285 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:02:28.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:28.068036+0000 osd.7 (osd.7) 476 : cluster [DBG] 6.3 scrub starts 2024-09-13T23:02:28.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:28 smithi178 bash[22840]: cluster 2024-09-13T23:02:28.070391+0000 osd.7 (osd.7) 477 : cluster [DBG] 6.3 scrub ok 2024-09-13T23:02:29.056 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:02:29.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.644737+0000 osd.5 (osd.5) 515 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.647779+0000 osd.5 (osd.5) 516 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.722777+0000 osd.2 (osd.2) 331 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[29889]: cluster 2024-09-13T23:02:27.725403+0000 osd.2 (osd.2) 332 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[29889]: cluster 2024-09-13T23:02:28.805539+0000 osd.4 (osd.4) 286 : cluster [DBG] 5.e scrub starts 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[29889]: cluster 2024-09-13T23:02:28.808355+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.e scrub ok 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.644737+0000 osd.5 (osd.5) 515 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.647779+0000 osd.5 (osd.5) 516 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.722777+0000 osd.2 (osd.2) 331 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[22840]: cluster 2024-09-13T23:02:27.725403+0000 osd.2 (osd.2) 332 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[22840]: cluster 2024-09-13T23:02:28.805539+0000 osd.4 (osd.4) 286 : cluster [DBG] 5.e scrub starts 2024-09-13T23:02:29.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[22840]: cluster 2024-09-13T23:02:28.808355+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.e scrub ok 2024-09-13T23:02:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:29 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.644737+0000 osd.5 (osd.5) 515 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:02:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:29 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.647779+0000 osd.5 (osd.5) 516 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:02:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:29 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.722777+0000 osd.2 (osd.2) 331 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:02:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:29 smithi192 bash[24608]: cluster 2024-09-13T23:02:27.725403+0000 osd.2 (osd.2) 332 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:02:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:29 smithi192 bash[24608]: cluster 2024-09-13T23:02:28.805539+0000 osd.4 (osd.4) 286 : cluster [DBG] 5.e scrub starts 2024-09-13T23:02:29.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:29 smithi192 bash[24608]: cluster 2024-09-13T23:02:28.808355+0000 osd.4 (osd.4) 287 : cluster [DBG] 5.e scrub ok 2024-09-13T23:02:30.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:02:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:02:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:29.032036+0000 mgr.y (mgr.14613) 921 : cluster [DBG] pgmap v1119: 142 pgs: 1 remapped+peering, 20 peering, 121 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 267 B/s, 13 objects/s recovering 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:29.373974+0000 mon.a (mon.0) 1873 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:29.374014+0000 mon.a (mon.0) 1874 : cluster [INF] Cluster is now healthy 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:29.777424+0000 osd.4 (osd.4) 288 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:29.781851+0000 osd.4 (osd.4) 289 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:30.052679+0000 osd.7 (osd.7) 478 : cluster [DBG] 3.1d scrub starts 2024-09-13T23:02:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:30 smithi192 bash[24608]: cluster 2024-09-13T23:02:30.056807+0000 osd.7 (osd.7) 479 : cluster [DBG] 3.1d scrub ok 2024-09-13T23:02:30.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:29.032036+0000 mgr.y (mgr.14613) 921 : cluster [DBG] pgmap v1119: 142 pgs: 1 remapped+peering, 20 peering, 121 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 267 B/s, 13 objects/s recovering 2024-09-13T23:02:30.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:29.373974+0000 mon.a (mon.0) 1873 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:29.374014+0000 mon.a (mon.0) 1874 : cluster [INF] Cluster is now healthy 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:29.777424+0000 osd.4 (osd.4) 288 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:29.781851+0000 osd.4 (osd.4) 289 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:30.052679+0000 osd.7 (osd.7) 478 : cluster [DBG] 3.1d scrub starts 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[22840]: cluster 2024-09-13T23:02:30.056807+0000 osd.7 (osd.7) 479 : cluster [DBG] 3.1d scrub ok 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:29.032036+0000 mgr.y (mgr.14613) 921 : cluster [DBG] pgmap v1119: 142 pgs: 1 remapped+peering, 20 peering, 121 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 267 B/s, 13 objects/s recovering 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:29.373974+0000 mon.a (mon.0) 1873 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:29.374014+0000 mon.a (mon.0) 1874 : cluster [INF] Cluster is now healthy 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:29.777424+0000 osd.4 (osd.4) 288 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:02:30.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:29.781851+0000 osd.4 (osd.4) 289 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:02:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:30.052679+0000 osd.7 (osd.7) 478 : cluster [DBG] 3.1d scrub starts 2024-09-13T23:02:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:30 smithi178 bash[29889]: cluster 2024-09-13T23:02:30.056807+0000 osd.7 (osd.7) 479 : cluster [DBG] 3.1d scrub ok 2024-09-13T23:02:31.393 INFO:teuthology.orchestra.run.smithi178.stderr:set pool 1 pgp_num to 11 2024-09-13T23:02:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:31 smithi192 bash[24608]: cluster 2024-09-13T23:02:30.592136+0000 osd.1 (osd.1) 452 : cluster [DBG] 2.a scrub starts 2024-09-13T23:02:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:31 smithi192 bash[24608]: cluster 2024-09-13T23:02:30.594837+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.a scrub ok 2024-09-13T23:02:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:31 smithi192 bash[24608]: audit 2024-09-13T23:02:31.051121+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.178:0/4020095312' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-13T23:02:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:31 smithi192 bash[24608]: audit 2024-09-13T23:02:31.051533+0000 mon.a (mon.0) 1875 : audit [INF] from='client.25991 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-13T23:02:31.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[22840]: cluster 2024-09-13T23:02:30.592136+0000 osd.1 (osd.1) 452 : cluster [DBG] 2.a scrub starts 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[22840]: cluster 2024-09-13T23:02:30.594837+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.a scrub ok 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[22840]: audit 2024-09-13T23:02:31.051121+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.178:0/4020095312' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[22840]: audit 2024-09-13T23:02:31.051533+0000 mon.a (mon.0) 1875 : audit [INF] from='client.25991 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[29889]: cluster 2024-09-13T23:02:30.592136+0000 osd.1 (osd.1) 452 : cluster [DBG] 2.a scrub starts 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[29889]: cluster 2024-09-13T23:02:30.594837+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.a scrub ok 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[29889]: audit 2024-09-13T23:02:31.051121+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.178:0/4020095312' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-13T23:02:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:31 smithi178 bash[29889]: audit 2024-09-13T23:02:31.051533+0000 mon.a (mon.0) 1875 : audit [INF] from='client.25991 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-13T23:02:32.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: cluster 2024-09-13T23:02:31.033388+0000 mgr.y (mgr.14613) 922 : cluster [DBG] pgmap v1120: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 308 B/s, 14 objects/s recovering 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: cluster 2024-09-13T23:02:31.271237+0000 osd.6 (osd.6) 344 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: cluster 2024-09-13T23:02:31.273821+0000 osd.6 (osd.6) 345 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: audit 2024-09-13T23:02:31.386723+0000 mon.a (mon.0) 1876 : audit [INF] from='client.25991 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: cluster 2024-09-13T23:02:31.393986+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: cluster 2024-09-13T23:02:31.616124+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:02:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:32 smithi192 bash[24608]: cluster 2024-09-13T23:02:31.620115+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:02:32.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: cluster 2024-09-13T23:02:31.033388+0000 mgr.y (mgr.14613) 922 : cluster [DBG] pgmap v1120: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 308 B/s, 14 objects/s recovering 2024-09-13T23:02:32.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: cluster 2024-09-13T23:02:31.271237+0000 osd.6 (osd.6) 344 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: cluster 2024-09-13T23:02:31.273821+0000 osd.6 (osd.6) 345 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: audit 2024-09-13T23:02:31.386723+0000 mon.a (mon.0) 1876 : audit [INF] from='client.25991 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: cluster 2024-09-13T23:02:31.393986+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: cluster 2024-09-13T23:02:31.616124+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[29889]: cluster 2024-09-13T23:02:31.620115+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: cluster 2024-09-13T23:02:31.033388+0000 mgr.y (mgr.14613) 922 : cluster [DBG] pgmap v1120: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 308 B/s, 14 objects/s recovering 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: cluster 2024-09-13T23:02:31.271237+0000 osd.6 (osd.6) 344 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: cluster 2024-09-13T23:02:31.273821+0000 osd.6 (osd.6) 345 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: audit 2024-09-13T23:02:31.386723+0000 mon.a (mon.0) 1876 : audit [INF] from='client.25991 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-13T23:02:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: cluster 2024-09-13T23:02:31.393986+0000 mon.a (mon.0) 1877 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2024-09-13T23:02:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: cluster 2024-09-13T23:02:31.616124+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:02:32.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:32 smithi178 bash[22840]: cluster 2024-09-13T23:02:31.620115+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:02:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:33 smithi192 bash[24608]: cluster 2024-09-13T23:02:32.255925+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:02:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:33 smithi192 bash[24608]: cluster 2024-09-13T23:02:32.258315+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:02:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:33 smithi192 bash[24608]: cluster 2024-09-13T23:02:32.585717+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:02:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:33 smithi192 bash[24608]: cluster 2024-09-13T23:02:32.588257+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:02:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[29889]: cluster 2024-09-13T23:02:32.255925+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:02:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[29889]: cluster 2024-09-13T23:02:32.258315+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:02:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[29889]: cluster 2024-09-13T23:02:32.585717+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:02:33.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[29889]: cluster 2024-09-13T23:02:32.588257+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:02:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[22840]: cluster 2024-09-13T23:02:32.255925+0000 osd.6 (osd.6) 346 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:02:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[22840]: cluster 2024-09-13T23:02:32.258315+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:02:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[22840]: cluster 2024-09-13T23:02:32.585717+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:02:33.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:33 smithi178 bash[22840]: cluster 2024-09-13T23:02:32.588257+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:02:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[29889]: cluster 2024-09-13T23:02:32.647732+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.4 scrub starts 2024-09-13T23:02:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[29889]: cluster 2024-09-13T23:02:32.650703+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.4 scrub ok 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[29889]: cluster 2024-09-13T23:02:33.034267+0000 mgr.y (mgr.14613) 923 : cluster [DBG] pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 277 B/s, 13 objects/s recovering 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[29889]: cluster 2024-09-13T23:02:33.259589+0000 osd.6 (osd.6) 348 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[29889]: cluster 2024-09-13T23:02:33.262869+0000 osd.6 (osd.6) 349 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[22840]: cluster 2024-09-13T23:02:32.647732+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.4 scrub starts 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[22840]: cluster 2024-09-13T23:02:32.650703+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.4 scrub ok 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[22840]: cluster 2024-09-13T23:02:33.034267+0000 mgr.y (mgr.14613) 923 : cluster [DBG] pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 277 B/s, 13 objects/s recovering 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[22840]: cluster 2024-09-13T23:02:33.259589+0000 osd.6 (osd.6) 348 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:02:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:34 smithi178 bash[22840]: cluster 2024-09-13T23:02:33.262869+0000 osd.6 (osd.6) 349 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:02:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:34 smithi192 bash[24608]: cluster 2024-09-13T23:02:32.647732+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.4 scrub starts 2024-09-13T23:02:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:34 smithi192 bash[24608]: cluster 2024-09-13T23:02:32.650703+0000 osd.1 (osd.1) 457 : cluster [DBG] 2.4 scrub ok 2024-09-13T23:02:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:34 smithi192 bash[24608]: cluster 2024-09-13T23:02:33.034267+0000 mgr.y (mgr.14613) 923 : cluster [DBG] pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 277 B/s, 13 objects/s recovering 2024-09-13T23:02:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:34 smithi192 bash[24608]: cluster 2024-09-13T23:02:33.259589+0000 osd.6 (osd.6) 348 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:02:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:34 smithi192 bash[24608]: cluster 2024-09-13T23:02:33.262869+0000 osd.6 (osd.6) 349 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:02:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[29889]: cluster 2024-09-13T23:02:33.668956+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.d deep-scrub starts 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[29889]: cluster 2024-09-13T23:02:33.671169+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.d deep-scrub ok 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[29889]: cluster 2024-09-13T23:02:34.236133+0000 osd.6 (osd.6) 350 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[29889]: cluster 2024-09-13T23:02:34.238868+0000 osd.6 (osd.6) 351 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[29889]: audit 2024-09-13T23:02:34.759177+0000 mon.b (mon.1) 304 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[22840]: cluster 2024-09-13T23:02:33.668956+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.d deep-scrub starts 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[22840]: cluster 2024-09-13T23:02:33.671169+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.d deep-scrub ok 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[22840]: cluster 2024-09-13T23:02:34.236133+0000 osd.6 (osd.6) 350 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[22840]: cluster 2024-09-13T23:02:34.238868+0000 osd.6 (osd.6) 351 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:02:35.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:35 smithi178 bash[22840]: audit 2024-09-13T23:02:34.759177+0000 mon.b (mon.1) 304 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:35.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:35 smithi192 bash[24608]: cluster 2024-09-13T23:02:33.668956+0000 osd.1 (osd.1) 458 : cluster [DBG] 2.d deep-scrub starts 2024-09-13T23:02:35.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:35 smithi192 bash[24608]: cluster 2024-09-13T23:02:33.671169+0000 osd.1 (osd.1) 459 : cluster [DBG] 2.d deep-scrub ok 2024-09-13T23:02:35.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:35 smithi192 bash[24608]: cluster 2024-09-13T23:02:34.236133+0000 osd.6 (osd.6) 350 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:02:35.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:35 smithi192 bash[24608]: cluster 2024-09-13T23:02:34.238868+0000 osd.6 (osd.6) 351 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:02:35.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:35 smithi192 bash[24608]: audit 2024-09-13T23:02:34.759177+0000 mon.b (mon.1) 304 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:36.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:02:36 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:02:36.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:36 smithi178 bash[29889]: cluster 2024-09-13T23:02:35.035400+0000 mgr.y (mgr.14613) 924 : cluster [DBG] pgmap v1123: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 217 B/s, 9 objects/s recovering 2024-09-13T23:02:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:36 smithi178 bash[22840]: cluster 2024-09-13T23:02:35.035400+0000 mgr.y (mgr.14613) 924 : cluster [DBG] pgmap v1123: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 217 B/s, 9 objects/s recovering 2024-09-13T23:02:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:36 smithi192 bash[24608]: cluster 2024-09-13T23:02:35.035400+0000 mgr.y (mgr.14613) 924 : cluster [DBG] pgmap v1123: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 217 B/s, 9 objects/s recovering 2024-09-13T23:02:37.084 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:02:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:37 smithi178 bash[29889]: audit 2024-09-13T23:02:36.116661+0000 mgr.y (mgr.14613) 925 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:37 smithi178 bash[29889]: cluster 2024-09-13T23:02:36.612200+0000 osd.0 (osd.0) 449 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:02:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:37 smithi178 bash[29889]: cluster 2024-09-13T23:02:36.614537+0000 osd.0 (osd.0) 450 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:02:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:37 smithi178 bash[22840]: audit 2024-09-13T23:02:36.116661+0000 mgr.y (mgr.14613) 925 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:37 smithi178 bash[22840]: cluster 2024-09-13T23:02:36.612200+0000 osd.0 (osd.0) 449 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:02:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:37 smithi178 bash[22840]: cluster 2024-09-13T23:02:36.614537+0000 osd.0 (osd.0) 450 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:02:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:37 smithi192 bash[24608]: audit 2024-09-13T23:02:36.116661+0000 mgr.y (mgr.14613) 925 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:37 smithi192 bash[24608]: cluster 2024-09-13T23:02:36.612200+0000 osd.0 (osd.0) 449 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:02:37.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:37 smithi192 bash[24608]: cluster 2024-09-13T23:02:36.614537+0000 osd.0 (osd.0) 450 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:02:38.700 INFO:teuthology.orchestra.run.smithi178.stdout:mon.a: {} 2024-09-13T23:02:38.701 INFO:teuthology.orchestra.run.smithi178.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T23:02:38.714 INFO:teuthology.orchestra.run.smithi178.stdout:mon.b: {} 2024-09-13T23:02:38.715 INFO:teuthology.orchestra.run.smithi178.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T23:02:38.723 INFO:teuthology.orchestra.run.smithi178.stdout:mon.c: {} 2024-09-13T23:02:38.723 INFO:teuthology.orchestra.run.smithi178.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T23:02:38.733 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[22840]: cluster 2024-09-13T23:02:37.036174+0000 mgr.y (mgr.14613) 926 : cluster [DBG] pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 180 B/s, 7 objects/s recovering 2024-09-13T23:02:38.733 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[22840]: cluster 2024-09-13T23:02:37.571211+0000 osd.3 (osd.3) 441 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:02:38.733 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[22840]: cluster 2024-09-13T23:02:37.574363+0000 osd.3 (osd.3) 442 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:02:38.733 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[22840]: cluster 2024-09-13T23:02:37.624231+0000 osd.5 (osd.5) 517 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:02:38.733 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[22840]: cluster 2024-09-13T23:02:37.627560+0000 osd.5 (osd.5) 518 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:02:38.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[29889]: cluster 2024-09-13T23:02:37.036174+0000 mgr.y (mgr.14613) 926 : cluster [DBG] pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 180 B/s, 7 objects/s recovering 2024-09-13T23:02:38.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[29889]: cluster 2024-09-13T23:02:37.571211+0000 osd.3 (osd.3) 441 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:02:38.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[29889]: cluster 2024-09-13T23:02:37.574363+0000 osd.3 (osd.3) 442 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:02:38.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[29889]: cluster 2024-09-13T23:02:37.624231+0000 osd.5 (osd.5) 517 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:02:38.734 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:38 smithi178 bash[29889]: cluster 2024-09-13T23:02:37.627560+0000 osd.5 (osd.5) 518 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:02:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:38 smithi192 bash[24608]: cluster 2024-09-13T23:02:37.036174+0000 mgr.y (mgr.14613) 926 : cluster [DBG] pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 180 B/s, 7 objects/s recovering 2024-09-13T23:02:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:38 smithi192 bash[24608]: cluster 2024-09-13T23:02:37.571211+0000 osd.3 (osd.3) 441 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:02:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:38 smithi192 bash[24608]: cluster 2024-09-13T23:02:37.574363+0000 osd.3 (osd.3) 442 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:02:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:38 smithi192 bash[24608]: cluster 2024-09-13T23:02:37.624231+0000 osd.5 (osd.5) 517 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:02:38.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:38 smithi192 bash[24608]: cluster 2024-09-13T23:02:37.627560+0000 osd.5 (osd.5) 518 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:02:39.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T23:02:39.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[22840]: cluster 2024-09-13T23:02:38.626279+0000 osd.0 (osd.0) 451 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:02:39.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[22840]: cluster 2024-09-13T23:02:38.627787+0000 osd.0 (osd.0) 452 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:02:39.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[22840]: cluster 2024-09-13T23:02:38.666726+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:02:39.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[22840]: cluster 2024-09-13T23:02:38.670705+0000 osd.5 (osd.5) 520 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:02:39.809 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[22840]: audit 2024-09-13T23:02:38.691557+0000 mon.a (mon.0) 1878 : audit [DBG] from='client.? v1:172.21.15.178:0/2514523066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T23:02:39.809 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[29889]: cluster 2024-09-13T23:02:38.626279+0000 osd.0 (osd.0) 451 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:02:39.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[29889]: cluster 2024-09-13T23:02:38.627787+0000 osd.0 (osd.0) 452 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:02:39.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[29889]: cluster 2024-09-13T23:02:38.666726+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:02:39.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[29889]: cluster 2024-09-13T23:02:38.670705+0000 osd.5 (osd.5) 520 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:02:39.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[29889]: audit 2024-09-13T23:02:38.691557+0000 mon.a (mon.0) 1878 : audit [DBG] from='client.? v1:172.21.15.178:0/2514523066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T23:02:39.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:39 smithi192 bash[24608]: cluster 2024-09-13T23:02:38.626279+0000 osd.0 (osd.0) 451 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:02:39.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:39 smithi192 bash[24608]: cluster 2024-09-13T23:02:38.627787+0000 osd.0 (osd.0) 452 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:02:39.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:39 smithi192 bash[24608]: cluster 2024-09-13T23:02:38.666726+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:02:39.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:39 smithi192 bash[24608]: cluster 2024-09-13T23:02:38.670705+0000 osd.5 (osd.5) 520 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:02:39.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:39 smithi192 bash[24608]: audit 2024-09-13T23:02:38.691557+0000 mon.a (mon.0) 1878 : audit [DBG] from='client.? v1:172.21.15.178:0/2514523066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T23:02:39.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:02:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:02:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.037182+0000 mgr.y (mgr.14613) 927 : cluster [DBG] pgmap v1125: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 87 B/s, 4 objects/s recovering 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.143770+0000 osd.6 (osd.6) 352 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.146994+0000 osd.6 (osd.6) 353 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.369274+0000 osd.7 (osd.7) 480 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.369364+0000 osd.7 (osd.7) 481 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.448643+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2024-09-13T23:02:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.697768+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:02:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:40 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.701722+0000 osd.5 (osd.5) 522 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:02:40.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.037182+0000 mgr.y (mgr.14613) 927 : cluster [DBG] pgmap v1125: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 87 B/s, 4 objects/s recovering 2024-09-13T23:02:40.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.143770+0000 osd.6 (osd.6) 352 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.146994+0000 osd.6 (osd.6) 353 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.369274+0000 osd.7 (osd.7) 480 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.369364+0000 osd.7 (osd.7) 481 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.448643+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.697768+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.701722+0000 osd.5 (osd.5) 522 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.037182+0000 mgr.y (mgr.14613) 927 : cluster [DBG] pgmap v1125: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 87 B/s, 4 objects/s recovering 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.143770+0000 osd.6 (osd.6) 352 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.146994+0000 osd.6 (osd.6) 353 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:02:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.369274+0000 osd.7 (osd.7) 480 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:40.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.369364+0000 osd.7 (osd.7) 481 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:40.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.448643+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2024-09-13T23:02:40.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.697768+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:02:40.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:40 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.701722+0000 osd.5 (osd.5) 522 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:02:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:41 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.607127+0000 osd.2 (osd.2) 333 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:02:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:41 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.609529+0000 osd.2 (osd.2) 334 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:02:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:41 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.663641+0000 osd.0 (osd.0) 453 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:02:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:41 smithi192 bash[24608]: cluster 2024-09-13T23:02:39.665135+0000 osd.0 (osd.0) 454 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:02:41.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.607127+0000 osd.2 (osd.2) 333 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.609529+0000 osd.2 (osd.2) 334 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.663641+0000 osd.0 (osd.0) 453 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[29889]: cluster 2024-09-13T23:02:39.665135+0000 osd.0 (osd.0) 454 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.607127+0000 osd.2 (osd.2) 333 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.609529+0000 osd.2 (osd.2) 334 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.663641+0000 osd.0 (osd.0) 453 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:02:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:41 smithi178 bash[22840]: cluster 2024-09-13T23:02:39.665135+0000 osd.0 (osd.0) 454 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:02:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:42 smithi192 bash[24608]: cluster 2024-09-13T23:02:40.686471+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:02:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:42 smithi192 bash[24608]: cluster 2024-09-13T23:02:40.690282+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:02:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:42 smithi192 bash[24608]: cluster 2024-09-13T23:02:41.038314+0000 mgr.y (mgr.14613) 928 : cluster [DBG] pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:02:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:42 smithi178 bash[29889]: cluster 2024-09-13T23:02:40.686471+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:02:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:42 smithi178 bash[29889]: cluster 2024-09-13T23:02:40.690282+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:02:42.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:42 smithi178 bash[29889]: cluster 2024-09-13T23:02:41.038314+0000 mgr.y (mgr.14613) 928 : cluster [DBG] pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:02:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:42 smithi178 bash[22840]: cluster 2024-09-13T23:02:40.686471+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:02:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:42 smithi178 bash[22840]: cluster 2024-09-13T23:02:40.690282+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:02:42.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:42 smithi178 bash[22840]: cluster 2024-09-13T23:02:41.038314+0000 mgr.y (mgr.14613) 928 : cluster [DBG] pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:02:43.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:43 smithi192 bash[24608]: cluster 2024-09-13T23:02:41.669246+0000 osd.0 (osd.0) 457 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:02:43.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:43 smithi192 bash[24608]: cluster 2024-09-13T23:02:41.673043+0000 osd.0 (osd.0) 458 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:02:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:43 smithi178 bash[29889]: cluster 2024-09-13T23:02:41.669246+0000 osd.0 (osd.0) 457 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:02:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:43 smithi178 bash[29889]: cluster 2024-09-13T23:02:41.673043+0000 osd.0 (osd.0) 458 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:02:43.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:43 smithi178 bash[22840]: cluster 2024-09-13T23:02:41.669246+0000 osd.0 (osd.0) 457 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:02:43.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:43 smithi178 bash[22840]: cluster 2024-09-13T23:02:41.673043+0000 osd.0 (osd.0) 458 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:02:44.384 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:02:44.721 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:42.682877+0000 osd.2 (osd.2) 335 : cluster [DBG] 5.b scrub starts 2024-09-13T23:02:44.721 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:42.683184+0000 osd.1 (osd.1) 460 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:02:44.721 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:42.685493+0000 osd.2 (osd.2) 336 : cluster [DBG] 5.b scrub ok 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:42.685982+0000 osd.1 (osd.1) 461 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.039307+0000 mgr.y (mgr.14613) 929 : cluster [DBG] pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.534377+0000 osd.3 (osd.3) 443 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.537626+0000 osd.3 (osd.3) 444 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:42.682877+0000 osd.2 (osd.2) 335 : cluster [DBG] 5.b scrub starts 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:42.683184+0000 osd.1 (osd.1) 460 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:42.685493+0000 osd.2 (osd.2) 336 : cluster [DBG] 5.b scrub ok 2024-09-13T23:02:44.722 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:42.685982+0000 osd.1 (osd.1) 461 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:02:44.723 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.039307+0000 mgr.y (mgr.14613) 929 : cluster [DBG] pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:02:44.723 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.534377+0000 osd.3 (osd.3) 443 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:02:44.723 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:44 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.537626+0000 osd.3 (osd.3) 444 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:02:44.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:42.682877+0000 osd.2 (osd.2) 335 : cluster [DBG] 5.b scrub starts 2024-09-13T23:02:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:42.683184+0000 osd.1 (osd.1) 460 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:02:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:42.685493+0000 osd.2 (osd.2) 336 : cluster [DBG] 5.b scrub ok 2024-09-13T23:02:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:42.685982+0000 osd.1 (osd.1) 461 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:02:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.039307+0000 mgr.y (mgr.14613) 929 : cluster [DBG] pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:02:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.534377+0000 osd.3 (osd.3) 443 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:02:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:44 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.537626+0000 osd.3 (osd.3) 444 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:02:45.837 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.682914+0000 osd.1 (osd.1) 462 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.685302+0000 osd.1 (osd.1) 463 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.716557+0000 osd.0 (osd.0) 459 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:43.718733+0000 osd.0 (osd.0) 460 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:44.754378+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.e scrub starts 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:44.756800+0000 osd.4 (osd.4) 291 : cluster [DBG] 6.e scrub ok 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.015950+0000 osd.7 (osd.7) 482 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:02:45.838 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.018375+0000 osd.7 (osd.7) 483 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.682914+0000 osd.1 (osd.1) 462 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.685302+0000 osd.1 (osd.1) 463 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.716557+0000 osd.0 (osd.0) 459 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:43.718733+0000 osd.0 (osd.0) 460 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:44.754378+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.e scrub starts 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:44.756800+0000 osd.4 (osd.4) 291 : cluster [DBG] 6.e scrub ok 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.015950+0000 osd.7 (osd.7) 482 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:02:45.839 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:45 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.018375+0000 osd.7 (osd.7) 483 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:02:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.682914+0000 osd.1 (osd.1) 462 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:02:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.685302+0000 osd.1 (osd.1) 463 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:02:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.716557+0000 osd.0 (osd.0) 459 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:02:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:43.718733+0000 osd.0 (osd.0) 460 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:02:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:44.754378+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.e scrub starts 2024-09-13T23:02:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:44.756800+0000 osd.4 (osd.4) 291 : cluster [DBG] 6.e scrub ok 2024-09-13T23:02:45.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.015950+0000 osd.7 (osd.7) 482 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:02:45.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:45 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.018375+0000 osd.7 (osd.7) 483 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:02:46.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:02:46.474 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T23:02:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[24608]: cluster 2024-09-13T23:02:44.685825+0000 osd.0 (osd.0) 461 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:02:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[24608]: cluster 2024-09-13T23:02:44.688316+0000 osd.0 (osd.0) 462 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:02:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.040781+0000 mgr.y (mgr.14613) 930 : cluster [DBG] pgmap v1129: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:02:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.480754+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.c scrub starts 2024-09-13T23:02:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.483080+0000 osd.3 (osd.3) 446 : cluster [DBG] 6.c scrub ok 2024-09-13T23:02:46.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:46 smithi192 bash[24608]: audit 2024-09-13T23:02:46.197579+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? v1:172.21.15.178:0/3861890085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[22840]: cluster 2024-09-13T23:02:44.685825+0000 osd.0 (osd.0) 461 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[22840]: cluster 2024-09-13T23:02:44.688316+0000 osd.0 (osd.0) 462 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.040781+0000 mgr.y (mgr.14613) 930 : cluster [DBG] pgmap v1129: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.480754+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.c scrub starts 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.483080+0000 osd.3 (osd.3) 446 : cluster [DBG] 6.c scrub ok 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[22840]: audit 2024-09-13T23:02:46.197579+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? v1:172.21.15.178:0/3861890085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[29889]: cluster 2024-09-13T23:02:44.685825+0000 osd.0 (osd.0) 461 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:02:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[29889]: cluster 2024-09-13T23:02:44.688316+0000 osd.0 (osd.0) 462 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:02:46.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.040781+0000 mgr.y (mgr.14613) 930 : cluster [DBG] pgmap v1129: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:02:46.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.480754+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.c scrub starts 2024-09-13T23:02:46.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.483080+0000 osd.3 (osd.3) 446 : cluster [DBG] 6.c scrub ok 2024-09-13T23:02:46.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:46 smithi178 bash[29889]: audit 2024-09-13T23:02:46.197579+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? v1:172.21.15.178:0/3861890085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:47.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-13T23:02:47.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.674497+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:45.676825+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: audit 2024-09-13T23:02:46.128072+0000 mgr.y (mgr.14613) 931 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:46.441582+0000 osd.3 (osd.3) 447 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:46.443941+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: audit 2024-09-13T23:02:46.474771+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.178:0/3861890085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:46.482509+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:47.007807+0000 osd.7 (osd.7) 484 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:02:47.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[29889]: cluster 2024-09-13T23:02:47.011214+0000 osd.7 (osd.7) 485 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.674497+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:45.676825+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: audit 2024-09-13T23:02:46.128072+0000 mgr.y (mgr.14613) 931 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:46.441582+0000 osd.3 (osd.3) 447 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:46.443941+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: audit 2024-09-13T23:02:46.474771+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.178:0/3861890085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:46.482509+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:47.007807+0000 osd.7 (osd.7) 484 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:02:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:47 smithi178 bash[22840]: cluster 2024-09-13T23:02:47.011214+0000 osd.7 (osd.7) 485 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:02:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.674497+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.1d deep-scrub starts 2024-09-13T23:02:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:45.676825+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.1d deep-scrub ok 2024-09-13T23:02:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: audit 2024-09-13T23:02:46.128072+0000 mgr.y (mgr.14613) 931 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:46.441582+0000 osd.3 (osd.3) 447 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:02:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:46.443941+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:02:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: audit 2024-09-13T23:02:46.474771+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.178:0/3861890085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T23:02:47.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:46.482509+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2024-09-13T23:02:47.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:47.007807+0000 osd.7 (osd.7) 484 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:02:47.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:47 smithi192 bash[24608]: cluster 2024-09-13T23:02:47.011214+0000 osd.7 (osd.7) 485 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:02:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:48 smithi192 bash[24608]: cluster 2024-09-13T23:02:47.041615+0000 mgr.y (mgr.14613) 932 : cluster [DBG] pgmap v1131: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:48 smithi192 bash[24608]: cluster 2024-09-13T23:02:47.229625+0000 osd.1 (osd.1) 466 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:48 smithi192 bash[24608]: cluster 2024-09-13T23:02:47.229731+0000 osd.1 (osd.1) 467 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:48 smithi192 bash[24608]: cluster 2024-09-13T23:02:47.500814+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2024-09-13T23:02:48.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[22840]: cluster 2024-09-13T23:02:47.041615+0000 mgr.y (mgr.14613) 932 : cluster [DBG] pgmap v1131: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[22840]: cluster 2024-09-13T23:02:47.229625+0000 osd.1 (osd.1) 466 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[22840]: cluster 2024-09-13T23:02:47.229731+0000 osd.1 (osd.1) 467 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[22840]: cluster 2024-09-13T23:02:47.500814+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[29889]: cluster 2024-09-13T23:02:47.041615+0000 mgr.y (mgr.14613) 932 : cluster [DBG] pgmap v1131: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[29889]: cluster 2024-09-13T23:02:47.229625+0000 osd.1 (osd.1) 466 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[29889]: cluster 2024-09-13T23:02:47.229731+0000 osd.1 (osd.1) 467 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:48 smithi178 bash[29889]: cluster 2024-09-13T23:02:47.500814+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2024-09-13T23:02:49.814 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:49 smithi178 bash[29889]: cluster 2024-09-13T23:02:48.723123+0000 osd.5 (osd.5) 523 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:02:49.815 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:49 smithi178 bash[29889]: cluster 2024-09-13T23:02:48.725983+0000 osd.5 (osd.5) 524 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:02:49.815 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:49 smithi178 bash[22840]: cluster 2024-09-13T23:02:48.723123+0000 osd.5 (osd.5) 523 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:02:49.815 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:49 smithi178 bash[22840]: cluster 2024-09-13T23:02:48.725983+0000 osd.5 (osd.5) 524 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:02:49.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:49 smithi192 bash[24608]: cluster 2024-09-13T23:02:48.723123+0000 osd.5 (osd.5) 523 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:02:49.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:49 smithi192 bash[24608]: cluster 2024-09-13T23:02:48.725983+0000 osd.5 (osd.5) 524 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:02:50.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:02:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:02:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:02:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:50 smithi192 bash[24608]: cluster 2024-09-13T23:02:49.042613+0000 mgr.y (mgr.14613) 933 : cluster [DBG] pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:50 smithi192 bash[24608]: audit 2024-09-13T23:02:49.787309+0000 mon.b (mon.1) 305 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:50.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:50 smithi178 bash[29889]: cluster 2024-09-13T23:02:49.042613+0000 mgr.y (mgr.14613) 933 : cluster [DBG] pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:50.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:50 smithi178 bash[29889]: audit 2024-09-13T23:02:49.787309+0000 mon.b (mon.1) 305 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:50 smithi178 bash[22840]: cluster 2024-09-13T23:02:49.042613+0000 mgr.y (mgr.14613) 933 : cluster [DBG] pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:50 smithi178 bash[22840]: audit 2024-09-13T23:02:49.787309+0000 mon.b (mon.1) 305 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:02:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:51 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.192672+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:02:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:51 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.194804+0000 osd.6 (osd.6) 355 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:02:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:51 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.684569+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:02:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:51 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.687275+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:02:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:51 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.999442+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:02:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:51 smithi192 bash[24608]: cluster 2024-09-13T23:02:51.001927+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:02:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.192672+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.194804+0000 osd.6 (osd.6) 355 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.684569+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.687275+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.999442+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[29889]: cluster 2024-09-13T23:02:51.001927+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.192672+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.194804+0000 osd.6 (osd.6) 355 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.684569+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.687275+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:02:51.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.999442+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:02:51.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:51 smithi178 bash[22840]: cluster 2024-09-13T23:02:51.001927+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:02:52.159 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:02:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:52 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.799188+0000 osd.0 (osd.0) 463 : cluster [DBG] 5.8 scrub starts 2024-09-13T23:02:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:52 smithi192 bash[24608]: cluster 2024-09-13T23:02:50.801360+0000 osd.0 (osd.0) 464 : cluster [DBG] 5.8 scrub ok 2024-09-13T23:02:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:52 smithi192 bash[24608]: cluster 2024-09-13T23:02:51.043888+0000 mgr.y (mgr.14613) 934 : cluster [DBG] pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:52 smithi192 bash[24608]: cluster 2024-09-13T23:02:51.960873+0000 osd.7 (osd.7) 488 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:02:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:52 smithi192 bash[24608]: cluster 2024-09-13T23:02:51.963199+0000 osd.7 (osd.7) 489 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:02:52.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:52 smithi192 bash[24608]: audit 2024-09-13T23:02:51.970812+0000 mon.b (mon.1) 306 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:02:52.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.799188+0000 osd.0 (osd.0) 463 : cluster [DBG] 5.8 scrub starts 2024-09-13T23:02:52.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[29889]: cluster 2024-09-13T23:02:50.801360+0000 osd.0 (osd.0) 464 : cluster [DBG] 5.8 scrub ok 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[29889]: cluster 2024-09-13T23:02:51.043888+0000 mgr.y (mgr.14613) 934 : cluster [DBG] pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[29889]: cluster 2024-09-13T23:02:51.960873+0000 osd.7 (osd.7) 488 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[29889]: cluster 2024-09-13T23:02:51.963199+0000 osd.7 (osd.7) 489 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[29889]: audit 2024-09-13T23:02:51.970812+0000 mon.b (mon.1) 306 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.799188+0000 osd.0 (osd.0) 463 : cluster [DBG] 5.8 scrub starts 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[22840]: cluster 2024-09-13T23:02:50.801360+0000 osd.0 (osd.0) 464 : cluster [DBG] 5.8 scrub ok 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[22840]: cluster 2024-09-13T23:02:51.043888+0000 mgr.y (mgr.14613) 934 : cluster [DBG] pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[22840]: cluster 2024-09-13T23:02:51.960873+0000 osd.7 (osd.7) 488 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[22840]: cluster 2024-09-13T23:02:51.963199+0000 osd.7 (osd.7) 489 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:02:52.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:52 smithi178 bash[22840]: audit 2024-09-13T23:02:51.970812+0000 mon.b (mon.1) 306 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:02:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: cluster 2024-09-13T23:02:51.895890+0000 osd.1 (osd.1) 468 : cluster [DBG] 5.12 deep-scrub starts 2024-09-13T23:02:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: cluster 2024-09-13T23:02:51.898268+0000 osd.1 (osd.1) 469 : cluster [DBG] 5.12 deep-scrub ok 2024-09-13T23:02:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: cluster 2024-09-13T23:02:52.689239+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:02:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: cluster 2024-09-13T23:02:52.693346+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:02:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: audit 2024-09-13T23:02:52.728384+0000 mon.b (mon.1) 307 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:02:53.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: audit 2024-09-13T23:02:52.730454+0000 mon.b (mon.1) 308 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:02:53.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:53 smithi192 bash[24608]: audit 2024-09-13T23:02:52.740247+0000 mon.a (mon.0) 1884 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:02:53.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: cluster 2024-09-13T23:02:51.895890+0000 osd.1 (osd.1) 468 : cluster [DBG] 5.12 deep-scrub starts 2024-09-13T23:02:53.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: cluster 2024-09-13T23:02:51.898268+0000 osd.1 (osd.1) 469 : cluster [DBG] 5.12 deep-scrub ok 2024-09-13T23:02:53.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: cluster 2024-09-13T23:02:52.689239+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:02:53.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: cluster 2024-09-13T23:02:52.693346+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:02:53.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: audit 2024-09-13T23:02:52.728384+0000 mon.b (mon.1) 307 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: audit 2024-09-13T23:02:52.730454+0000 mon.b (mon.1) 308 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[22840]: audit 2024-09-13T23:02:52.740247+0000 mon.a (mon.0) 1884 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: cluster 2024-09-13T23:02:51.895890+0000 osd.1 (osd.1) 468 : cluster [DBG] 5.12 deep-scrub starts 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: cluster 2024-09-13T23:02:51.898268+0000 osd.1 (osd.1) 469 : cluster [DBG] 5.12 deep-scrub ok 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: cluster 2024-09-13T23:02:52.689239+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: cluster 2024-09-13T23:02:52.693346+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: audit 2024-09-13T23:02:52.728384+0000 mon.b (mon.1) 307 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: audit 2024-09-13T23:02:52.730454+0000 mon.b (mon.1) 308 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:02:53.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:53 smithi178 bash[29889]: audit 2024-09-13T23:02:52.740247+0000 mon.a (mon.0) 1884 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:02:54.227 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T23:02:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:52.910107+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:02:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:52.912409+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:02:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.044755+0000 mgr.y (mgr.14613) 935 : cluster [DBG] pgmap v1135: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.235453+0000 osd.3 (osd.3) 449 : cluster [DBG] 2.19 scrub starts 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.238432+0000 osd.3 (osd.3) 450 : cluster [DBG] 2.19 scrub ok 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.418260+0000 osd.4 (osd.4) 296 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.418402+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.669572+0000 osd.5 (osd.5) 525 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.672245+0000 osd.5 (osd.5) 526 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.951458+0000 osd.7 (osd.7) 490 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:53.953692+0000 osd.7 (osd.7) 491 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: audit 2024-09-13T23:02:54.072149+0000 mon.c (mon.2) 90 : audit [INF] from='client.? v1:172.21.15.178:0/594933384' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: audit 2024-09-13T23:02:54.072665+0000 mon.a (mon.0) 1885 : audit [INF] from='client.26015 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: audit 2024-09-13T23:02:54.220777+0000 mon.a (mon.0) 1886 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T23:02:54.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:54 smithi192 bash[24608]: cluster 2024-09-13T23:02:54.226969+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:52.910107+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:52.912409+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.044755+0000 mgr.y (mgr.14613) 935 : cluster [DBG] pgmap v1135: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.235453+0000 osd.3 (osd.3) 449 : cluster [DBG] 2.19 scrub starts 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.238432+0000 osd.3 (osd.3) 450 : cluster [DBG] 2.19 scrub ok 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.418260+0000 osd.4 (osd.4) 296 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.418402+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.669572+0000 osd.5 (osd.5) 525 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:02:54.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.672245+0000 osd.5 (osd.5) 526 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:02:54.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.951458+0000 osd.7 (osd.7) 490 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:02:54.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:53.953692+0000 osd.7 (osd.7) 491 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: audit 2024-09-13T23:02:54.072149+0000 mon.c (mon.2) 90 : audit [INF] from='client.? v1:172.21.15.178:0/594933384' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: audit 2024-09-13T23:02:54.072665+0000 mon.a (mon.0) 1885 : audit [INF] from='client.26015 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: audit 2024-09-13T23:02:54.220777+0000 mon.a (mon.0) 1886 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[22840]: cluster 2024-09-13T23:02:54.226969+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:52.910107+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:52.912409+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.044755+0000 mgr.y (mgr.14613) 935 : cluster [DBG] pgmap v1135: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.235453+0000 osd.3 (osd.3) 449 : cluster [DBG] 2.19 scrub starts 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.238432+0000 osd.3 (osd.3) 450 : cluster [DBG] 2.19 scrub ok 2024-09-13T23:02:54.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.418260+0000 osd.4 (osd.4) 296 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.418402+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.669572+0000 osd.5 (osd.5) 525 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.672245+0000 osd.5 (osd.5) 526 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.951458+0000 osd.7 (osd.7) 490 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:53.953692+0000 osd.7 (osd.7) 491 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: audit 2024-09-13T23:02:54.072149+0000 mon.c (mon.2) 90 : audit [INF] from='client.? v1:172.21.15.178:0/594933384' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: audit 2024-09-13T23:02:54.072665+0000 mon.a (mon.0) 1885 : audit [INF] from='client.26015 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: audit 2024-09-13T23:02:54.220777+0000 mon.a (mon.0) 1886 : audit [INF] from='client.26015 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T23:02:54.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:54 smithi178 bash[29889]: cluster 2024-09-13T23:02:54.226969+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2024-09-13T23:02:55.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T23:02:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:55 smithi192 bash[24608]: cluster 2024-09-13T23:02:54.256709+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:02:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:55 smithi192 bash[24608]: cluster 2024-09-13T23:02:54.259737+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:02:55.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:55 smithi178 bash[22840]: cluster 2024-09-13T23:02:54.256709+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:02:55.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:55 smithi178 bash[22840]: cluster 2024-09-13T23:02:54.259737+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:02:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:55 smithi178 bash[29889]: cluster 2024-09-13T23:02:54.256709+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:02:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:55 smithi178 bash[29889]: cluster 2024-09-13T23:02:54.259737+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:02:56.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:02:56 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:02:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:56 smithi192 bash[24608]: cluster 2024-09-13T23:02:55.046226+0000 mgr.y (mgr.14613) 936 : cluster [DBG] pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:56 smithi192 bash[24608]: cluster 2024-09-13T23:02:55.659125+0000 osd.5 (osd.5) 527 : cluster [DBG] 3.c scrub starts 2024-09-13T23:02:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:56 smithi192 bash[24608]: cluster 2024-09-13T23:02:55.663013+0000 osd.5 (osd.5) 528 : cluster [DBG] 3.c scrub ok 2024-09-13T23:02:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:56 smithi192 bash[24608]: cluster 2024-09-13T23:02:55.739945+0000 osd.4 (osd.4) 298 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:02:56.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:56 smithi192 bash[24608]: cluster 2024-09-13T23:02:55.742185+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:02:56.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[29889]: cluster 2024-09-13T23:02:55.046226+0000 mgr.y (mgr.14613) 936 : cluster [DBG] pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[29889]: cluster 2024-09-13T23:02:55.659125+0000 osd.5 (osd.5) 527 : cluster [DBG] 3.c scrub starts 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[29889]: cluster 2024-09-13T23:02:55.663013+0000 osd.5 (osd.5) 528 : cluster [DBG] 3.c scrub ok 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[29889]: cluster 2024-09-13T23:02:55.739945+0000 osd.4 (osd.4) 298 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[29889]: cluster 2024-09-13T23:02:55.742185+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[22840]: cluster 2024-09-13T23:02:55.046226+0000 mgr.y (mgr.14613) 936 : cluster [DBG] pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[22840]: cluster 2024-09-13T23:02:55.659125+0000 osd.5 (osd.5) 527 : cluster [DBG] 3.c scrub starts 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[22840]: cluster 2024-09-13T23:02:55.663013+0000 osd.5 (osd.5) 528 : cluster [DBG] 3.c scrub ok 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[22840]: cluster 2024-09-13T23:02:55.739945+0000 osd.4 (osd.4) 298 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:02:56.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:56 smithi178 bash[22840]: cluster 2024-09-13T23:02:55.742185+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:02:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:57 smithi192 bash[24608]: audit 2024-09-13T23:02:56.139488+0000 mgr.y (mgr.14613) 937 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:57 smithi192 bash[24608]: cluster 2024-09-13T23:02:56.710881+0000 osd.4 (osd.4) 300 : cluster [DBG] 5.a scrub starts 2024-09-13T23:02:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:57 smithi192 bash[24608]: cluster 2024-09-13T23:02:56.713511+0000 osd.4 (osd.4) 301 : cluster [DBG] 5.a scrub ok 2024-09-13T23:02:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:57 smithi192 bash[24608]: cluster 2024-09-13T23:02:56.990146+0000 osd.7 (osd.7) 492 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:02:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:57 smithi192 bash[24608]: cluster 2024-09-13T23:02:56.993219+0000 osd.7 (osd.7) 493 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:02:57.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[22840]: audit 2024-09-13T23:02:56.139488+0000 mgr.y (mgr.14613) 937 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:57.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[22840]: cluster 2024-09-13T23:02:56.710881+0000 osd.4 (osd.4) 300 : cluster [DBG] 5.a scrub starts 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[22840]: cluster 2024-09-13T23:02:56.713511+0000 osd.4 (osd.4) 301 : cluster [DBG] 5.a scrub ok 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[22840]: cluster 2024-09-13T23:02:56.990146+0000 osd.7 (osd.7) 492 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[22840]: cluster 2024-09-13T23:02:56.993219+0000 osd.7 (osd.7) 493 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[29889]: audit 2024-09-13T23:02:56.139488+0000 mgr.y (mgr.14613) 937 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[29889]: cluster 2024-09-13T23:02:56.710881+0000 osd.4 (osd.4) 300 : cluster [DBG] 5.a scrub starts 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[29889]: cluster 2024-09-13T23:02:56.713511+0000 osd.4 (osd.4) 301 : cluster [DBG] 5.a scrub ok 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[29889]: cluster 2024-09-13T23:02:56.990146+0000 osd.7 (osd.7) 492 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:02:57.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:57 smithi178 bash[29889]: cluster 2024-09-13T23:02:56.993219+0000 osd.7 (osd.7) 493 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:02:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:58 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.047233+0000 mgr.y (mgr.14613) 938 : cluster [DBG] pgmap v1138: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:02:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:58 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.244446+0000 osd.3 (osd.3) 453 : cluster [DBG] 6.1f deep-scrub starts 2024-09-13T23:02:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:58 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.250139+0000 osd.3 (osd.3) 454 : cluster [DBG] 6.1f deep-scrub ok 2024-09-13T23:02:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:58 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.969450+0000 osd.7 (osd.7) 494 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:02:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:58 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.971846+0000 osd.7 (osd.7) 495 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:02:58.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.047233+0000 mgr.y (mgr.14613) 938 : cluster [DBG] pgmap v1138: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:02:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.244446+0000 osd.3 (osd.3) 453 : cluster [DBG] 6.1f deep-scrub starts 2024-09-13T23:02:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.250139+0000 osd.3 (osd.3) 454 : cluster [DBG] 6.1f deep-scrub ok 2024-09-13T23:02:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.969450+0000 osd.7 (osd.7) 494 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:02:58.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.971846+0000 osd.7 (osd.7) 495 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:02:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.047233+0000 mgr.y (mgr.14613) 938 : cluster [DBG] pgmap v1138: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:02:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.244446+0000 osd.3 (osd.3) 453 : cluster [DBG] 6.1f deep-scrub starts 2024-09-13T23:02:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.250139+0000 osd.3 (osd.3) 454 : cluster [DBG] 6.1f deep-scrub ok 2024-09-13T23:02:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.969450+0000 osd.7 (osd.7) 494 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:02:58.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:58 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.971846+0000 osd.7 (osd.7) 495 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:02:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.704230+0000 osd.2 (osd.2) 337 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:02:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[22840]: cluster 2024-09-13T23:02:57.706504+0000 osd.2 (osd.2) 338 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:02:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[22840]: cluster 2024-09-13T23:02:58.969396+0000 osd.7 (osd.7) 496 : cluster [DBG] 5.f deep-scrub starts 2024-09-13T23:02:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[22840]: cluster 2024-09-13T23:02:58.971678+0000 osd.7 (osd.7) 497 : cluster [DBG] 5.f deep-scrub ok 2024-09-13T23:02:59.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.704230+0000 osd.2 (osd.2) 337 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:02:59.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[29889]: cluster 2024-09-13T23:02:57.706504+0000 osd.2 (osd.2) 338 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:02:59.812 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[29889]: cluster 2024-09-13T23:02:58.969396+0000 osd.7 (osd.7) 496 : cluster [DBG] 5.f deep-scrub starts 2024-09-13T23:02:59.812 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[29889]: cluster 2024-09-13T23:02:58.971678+0000 osd.7 (osd.7) 497 : cluster [DBG] 5.f deep-scrub ok 2024-09-13T23:02:59.889 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:02:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:59 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.704230+0000 osd.2 (osd.2) 337 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:02:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:59 smithi192 bash[24608]: cluster 2024-09-13T23:02:57.706504+0000 osd.2 (osd.2) 338 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:02:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:59 smithi192 bash[24608]: cluster 2024-09-13T23:02:58.969396+0000 osd.7 (osd.7) 496 : cluster [DBG] 5.f deep-scrub starts 2024-09-13T23:02:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:02:59 smithi192 bash[24608]: cluster 2024-09-13T23:02:58.971678+0000 osd.7 (osd.7) 497 : cluster [DBG] 5.f deep-scrub ok 2024-09-13T23:03:00.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:02:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:02:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:03:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:00 smithi192 bash[24608]: cluster 2024-09-13T23:02:58.693302+0000 osd.2 (osd.2) 339 : cluster [DBG] 2.c scrub starts 2024-09-13T23:03:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:00 smithi192 bash[24608]: cluster 2024-09-13T23:02:58.695860+0000 osd.2 (osd.2) 340 : cluster [DBG] 2.c scrub ok 2024-09-13T23:03:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:00 smithi192 bash[24608]: cluster 2024-09-13T23:02:59.048251+0000 mgr.y (mgr.14613) 939 : cluster [DBG] pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:00 smithi192 bash[24608]: cluster 2024-09-13T23:02:59.702511+0000 osd.4 (osd.4) 302 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:03:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:00 smithi192 bash[24608]: cluster 2024-09-13T23:02:59.706538+0000 osd.4 (osd.4) 303 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:03:00.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[29889]: cluster 2024-09-13T23:02:58.693302+0000 osd.2 (osd.2) 339 : cluster [DBG] 2.c scrub starts 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[29889]: cluster 2024-09-13T23:02:58.695860+0000 osd.2 (osd.2) 340 : cluster [DBG] 2.c scrub ok 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[29889]: cluster 2024-09-13T23:02:59.048251+0000 mgr.y (mgr.14613) 939 : cluster [DBG] pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[29889]: cluster 2024-09-13T23:02:59.702511+0000 osd.4 (osd.4) 302 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[29889]: cluster 2024-09-13T23:02:59.706538+0000 osd.4 (osd.4) 303 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[22840]: cluster 2024-09-13T23:02:58.693302+0000 osd.2 (osd.2) 339 : cluster [DBG] 2.c scrub starts 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[22840]: cluster 2024-09-13T23:02:58.695860+0000 osd.2 (osd.2) 340 : cluster [DBG] 2.c scrub ok 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[22840]: cluster 2024-09-13T23:02:59.048251+0000 mgr.y (mgr.14613) 939 : cluster [DBG] pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[22840]: cluster 2024-09-13T23:02:59.702511+0000 osd.4 (osd.4) 302 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:03:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:00 smithi178 bash[22840]: cluster 2024-09-13T23:02:59.706538+0000 osd.4 (osd.4) 303 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:03:01.567 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T23:03:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:59.723197+0000 osd.1 (osd.1) 472 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T23:03:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:01 smithi192 bash[24608]: cluster 2024-09-13T23:02:59.725440+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T23:03:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:01 smithi192 bash[24608]: audit 2024-09-13T23:03:01.463918+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.178:0/1434327778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:01 smithi192 bash[24608]: audit 2024-09-13T23:03:01.464461+0000 mon.a (mon.0) 1888 : audit [INF] from='client.26021 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:59.723197+0000 osd.1 (osd.1) 472 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[22840]: cluster 2024-09-13T23:02:59.725440+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[22840]: audit 2024-09-13T23:03:01.463918+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.178:0/1434327778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[22840]: audit 2024-09-13T23:03:01.464461+0000 mon.a (mon.0) 1888 : audit [INF] from='client.26021 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:59.723197+0000 osd.1 (osd.1) 472 : cluster [DBG] 6.4 deep-scrub starts 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[29889]: cluster 2024-09-13T23:02:59.725440+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.4 deep-scrub ok 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[29889]: audit 2024-09-13T23:03:01.463918+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.178:0/1434327778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:01.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:01 smithi178 bash[29889]: audit 2024-09-13T23:03:01.464461+0000 mon.a (mon.0) 1888 : audit [INF] from='client.26021 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:02.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T23:03:02.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: cluster 2024-09-13T23:03:00.913889+0000 osd.0 (osd.0) 465 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: cluster 2024-09-13T23:03:00.918332+0000 osd.0 (osd.0) 466 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: cluster 2024-09-13T23:03:01.049561+0000 mgr.y (mgr.14613) 940 : cluster [DBG] pgmap v1140: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: audit 2024-09-13T23:03:01.560153+0000 mon.a (mon.0) 1889 : audit [INF] from='client.26021 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: cluster 2024-09-13T23:03:01.568719+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: cluster 2024-09-13T23:03:02.014068+0000 osd.7 (osd.7) 498 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[29889]: cluster 2024-09-13T23:03:02.017733+0000 osd.7 (osd.7) 499 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: cluster 2024-09-13T23:03:00.913889+0000 osd.0 (osd.0) 465 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: cluster 2024-09-13T23:03:00.918332+0000 osd.0 (osd.0) 466 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: cluster 2024-09-13T23:03:01.049561+0000 mgr.y (mgr.14613) 940 : cluster [DBG] pgmap v1140: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:02.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: audit 2024-09-13T23:03:01.560153+0000 mon.a (mon.0) 1889 : audit [INF] from='client.26021 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:03:02.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: cluster 2024-09-13T23:03:01.568719+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2024-09-13T23:03:02.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: cluster 2024-09-13T23:03:02.014068+0000 osd.7 (osd.7) 498 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:03:02.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:02 smithi178 bash[22840]: cluster 2024-09-13T23:03:02.017733+0000 osd.7 (osd.7) 499 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: cluster 2024-09-13T23:03:00.913889+0000 osd.0 (osd.0) 465 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: cluster 2024-09-13T23:03:00.918332+0000 osd.0 (osd.0) 466 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: cluster 2024-09-13T23:03:01.049561+0000 mgr.y (mgr.14613) 940 : cluster [DBG] pgmap v1140: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: audit 2024-09-13T23:03:01.560153+0000 mon.a (mon.0) 1889 : audit [INF] from='client.26021 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: cluster 2024-09-13T23:03:01.568719+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: cluster 2024-09-13T23:03:02.014068+0000 osd.7 (osd.7) 498 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:03:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:02 smithi192 bash[24608]: cluster 2024-09-13T23:03:02.017733+0000 osd.7 (osd.7) 499 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:03:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:03 smithi192 bash[24608]: cluster 2024-09-13T23:03:01.884612+0000 osd.0 (osd.0) 467 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:03:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:03 smithi192 bash[24608]: cluster 2024-09-13T23:03:01.886835+0000 osd.0 (osd.0) 468 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:03:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:03 smithi192 bash[24608]: cluster 2024-09-13T23:03:02.282262+0000 osd.3 (osd.3) 455 : cluster [DBG] 3.b scrub starts 2024-09-13T23:03:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:03 smithi192 bash[24608]: cluster 2024-09-13T23:03:02.286233+0000 osd.3 (osd.3) 456 : cluster [DBG] 3.b scrub ok 2024-09-13T23:03:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[29889]: cluster 2024-09-13T23:03:01.884612+0000 osd.0 (osd.0) 467 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:03:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[29889]: cluster 2024-09-13T23:03:01.886835+0000 osd.0 (osd.0) 468 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:03:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[29889]: cluster 2024-09-13T23:03:02.282262+0000 osd.3 (osd.3) 455 : cluster [DBG] 3.b scrub starts 2024-09-13T23:03:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[29889]: cluster 2024-09-13T23:03:02.286233+0000 osd.3 (osd.3) 456 : cluster [DBG] 3.b scrub ok 2024-09-13T23:03:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[22840]: cluster 2024-09-13T23:03:01.884612+0000 osd.0 (osd.0) 467 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:03:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[22840]: cluster 2024-09-13T23:03:01.886835+0000 osd.0 (osd.0) 468 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:03:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[22840]: cluster 2024-09-13T23:03:02.282262+0000 osd.3 (osd.3) 455 : cluster [DBG] 3.b scrub starts 2024-09-13T23:03:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:03 smithi178 bash[22840]: cluster 2024-09-13T23:03:02.286233+0000 osd.3 (osd.3) 456 : cluster [DBG] 3.b scrub ok 2024-09-13T23:03:04.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:04 smithi192 bash[24608]: cluster 2024-09-13T23:03:03.050449+0000 mgr.y (mgr.14613) 941 : cluster [DBG] pgmap v1142: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:03:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:04 smithi178 bash[29889]: cluster 2024-09-13T23:03:03.050449+0000 mgr.y (mgr.14613) 941 : cluster [DBG] pgmap v1142: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:03:04.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:04 smithi178 bash[22840]: cluster 2024-09-13T23:03:03.050449+0000 mgr.y (mgr.14613) 941 : cluster [DBG] pgmap v1142: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:03:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:05 smithi192 bash[24608]: cluster 2024-09-13T23:03:04.721404+0000 osd.4 (osd.4) 304 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:03:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:05 smithi192 bash[24608]: cluster 2024-09-13T23:03:04.723952+0000 osd.4 (osd.4) 305 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:03:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:05 smithi192 bash[24608]: audit 2024-09-13T23:03:04.816614+0000 mon.b (mon.1) 309 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:05 smithi192 bash[24608]: cluster 2024-09-13T23:03:04.911943+0000 osd.7 (osd.7) 500 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:03:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:05 smithi192 bash[24608]: cluster 2024-09-13T23:03:04.914182+0000 osd.7 (osd.7) 501 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[22840]: cluster 2024-09-13T23:03:04.721404+0000 osd.4 (osd.4) 304 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[22840]: cluster 2024-09-13T23:03:04.723952+0000 osd.4 (osd.4) 305 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[22840]: audit 2024-09-13T23:03:04.816614+0000 mon.b (mon.1) 309 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[22840]: cluster 2024-09-13T23:03:04.911943+0000 osd.7 (osd.7) 500 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[22840]: cluster 2024-09-13T23:03:04.914182+0000 osd.7 (osd.7) 501 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[29889]: cluster 2024-09-13T23:03:04.721404+0000 osd.4 (osd.4) 304 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[29889]: cluster 2024-09-13T23:03:04.723952+0000 osd.4 (osd.4) 305 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[29889]: audit 2024-09-13T23:03:04.816614+0000 mon.b (mon.1) 309 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[29889]: cluster 2024-09-13T23:03:04.911943+0000 osd.7 (osd.7) 500 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:03:05.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:05 smithi178 bash[29889]: cluster 2024-09-13T23:03:04.914182+0000 osd.7 (osd.7) 501 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:03:06.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:03:06 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:03:06.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:06 smithi192 bash[24608]: cluster 2024-09-13T23:03:05.052005+0000 mgr.y (mgr.14613) 942 : cluster [DBG] pgmap v1143: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:07.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:06 smithi192 bash[24608]: cluster 2024-09-13T23:03:05.182234+0000 osd.3 (osd.3) 457 : cluster [DBG] 5.17 deep-scrub starts 2024-09-13T23:03:07.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:06 smithi192 bash[24608]: cluster 2024-09-13T23:03:05.183869+0000 osd.3 (osd.3) 458 : cluster [DBG] 5.17 deep-scrub ok 2024-09-13T23:03:07.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:06 smithi192 bash[24608]: cluster 2024-09-13T23:03:05.945911+0000 osd.7 (osd.7) 502 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:03:07.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:06 smithi192 bash[24608]: cluster 2024-09-13T23:03:05.948453+0000 osd.7 (osd.7) 503 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:03:07.066 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[22840]: cluster 2024-09-13T23:03:05.052005+0000 mgr.y (mgr.14613) 942 : cluster [DBG] pgmap v1143: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:07.066 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[22840]: cluster 2024-09-13T23:03:05.182234+0000 osd.3 (osd.3) 457 : cluster [DBG] 5.17 deep-scrub starts 2024-09-13T23:03:07.066 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[22840]: cluster 2024-09-13T23:03:05.183869+0000 osd.3 (osd.3) 458 : cluster [DBG] 5.17 deep-scrub ok 2024-09-13T23:03:07.066 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[22840]: cluster 2024-09-13T23:03:05.945911+0000 osd.7 (osd.7) 502 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:03:07.066 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[22840]: cluster 2024-09-13T23:03:05.948453+0000 osd.7 (osd.7) 503 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:03:07.067 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[29889]: cluster 2024-09-13T23:03:05.052005+0000 mgr.y (mgr.14613) 942 : cluster [DBG] pgmap v1143: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:07.067 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[29889]: cluster 2024-09-13T23:03:05.182234+0000 osd.3 (osd.3) 457 : cluster [DBG] 5.17 deep-scrub starts 2024-09-13T23:03:07.067 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[29889]: cluster 2024-09-13T23:03:05.183869+0000 osd.3 (osd.3) 458 : cluster [DBG] 5.17 deep-scrub ok 2024-09-13T23:03:07.067 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[29889]: cluster 2024-09-13T23:03:05.945911+0000 osd.7 (osd.7) 502 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:03:07.067 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:06 smithi178 bash[29889]: cluster 2024-09-13T23:03:05.948453+0000 osd.7 (osd.7) 503 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:03:07.159 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:07 smithi192 bash[24608]: audit 2024-09-13T23:03:06.150880+0000 mgr.y (mgr.14613) 943 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:07 smithi192 bash[24608]: cluster 2024-09-13T23:03:06.696046+0000 osd.5 (osd.5) 529 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:03:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:07 smithi192 bash[24608]: cluster 2024-09-13T23:03:06.698616+0000 osd.5 (osd.5) 530 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:03:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:07 smithi192 bash[24608]: cluster 2024-09-13T23:03:06.802515+0000 osd.4 (osd.4) 306 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:03:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:07 smithi192 bash[24608]: cluster 2024-09-13T23:03:06.804977+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:03:07.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[22840]: audit 2024-09-13T23:03:06.150880+0000 mgr.y (mgr.14613) 943 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:07.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[22840]: cluster 2024-09-13T23:03:06.696046+0000 osd.5 (osd.5) 529 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[22840]: cluster 2024-09-13T23:03:06.698616+0000 osd.5 (osd.5) 530 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[22840]: cluster 2024-09-13T23:03:06.802515+0000 osd.4 (osd.4) 306 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[22840]: cluster 2024-09-13T23:03:06.804977+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[29889]: audit 2024-09-13T23:03:06.150880+0000 mgr.y (mgr.14613) 943 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[29889]: cluster 2024-09-13T23:03:06.696046+0000 osd.5 (osd.5) 529 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[29889]: cluster 2024-09-13T23:03:06.698616+0000 osd.5 (osd.5) 530 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[29889]: cluster 2024-09-13T23:03:06.802515+0000 osd.4 (osd.4) 306 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:03:07.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:07 smithi178 bash[29889]: cluster 2024-09-13T23:03:06.804977+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.052908+0000 mgr.y (mgr.14613) 944 : cluster [DBG] pgmap v1144: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.258523+0000 osd.3 (osd.3) 459 : cluster [DBG] 2.e scrub starts 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.260182+0000 osd.3 (osd.3) 460 : cluster [DBG] 2.e scrub ok 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.741588+0000 osd.5 (osd.5) 531 : cluster [DBG] 6.a scrub starts 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.744240+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.a scrub ok 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.894883+0000 osd.7 (osd.7) 504 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:03:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:08 smithi192 bash[24608]: cluster 2024-09-13T23:03:07.897596+0000 osd.7 (osd.7) 505 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:03:08.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.052908+0000 mgr.y (mgr.14613) 944 : cluster [DBG] pgmap v1144: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:08.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.258523+0000 osd.3 (osd.3) 459 : cluster [DBG] 2.e scrub starts 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.260182+0000 osd.3 (osd.3) 460 : cluster [DBG] 2.e scrub ok 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.741588+0000 osd.5 (osd.5) 531 : cluster [DBG] 6.a scrub starts 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.744240+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.a scrub ok 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.894883+0000 osd.7 (osd.7) 504 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[22840]: cluster 2024-09-13T23:03:07.897596+0000 osd.7 (osd.7) 505 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.052908+0000 mgr.y (mgr.14613) 944 : cluster [DBG] pgmap v1144: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.258523+0000 osd.3 (osd.3) 459 : cluster [DBG] 2.e scrub starts 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.260182+0000 osd.3 (osd.3) 460 : cluster [DBG] 2.e scrub ok 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.741588+0000 osd.5 (osd.5) 531 : cluster [DBG] 6.a scrub starts 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.744240+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.a scrub ok 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.894883+0000 osd.7 (osd.7) 504 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:03:08.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:08 smithi178 bash[29889]: cluster 2024-09-13T23:03:07.897596+0000 osd.7 (osd.7) 505 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:03:09.612 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T23:03:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:09 smithi192 bash[24608]: cluster 2024-09-13T23:03:08.303175+0000 osd.3 (osd.3) 461 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:03:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:09 smithi192 bash[24608]: cluster 2024-09-13T23:03:08.305657+0000 osd.3 (osd.3) 462 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:03:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:09 smithi192 bash[24608]: audit 2024-09-13T23:03:09.018071+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.178:0/3590871403' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:09 smithi192 bash[24608]: audit 2024-09-13T23:03:09.018637+0000 mon.a (mon.0) 1891 : audit [INF] from='client.26027 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:09.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:03:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[29889]: cluster 2024-09-13T23:03:08.303175+0000 osd.3 (osd.3) 461 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[29889]: cluster 2024-09-13T23:03:08.305657+0000 osd.3 (osd.3) 462 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[29889]: audit 2024-09-13T23:03:09.018071+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.178:0/3590871403' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[29889]: audit 2024-09-13T23:03:09.018637+0000 mon.a (mon.0) 1891 : audit [INF] from='client.26027 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[22840]: cluster 2024-09-13T23:03:08.303175+0000 osd.3 (osd.3) 461 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[22840]: cluster 2024-09-13T23:03:08.305657+0000 osd.3 (osd.3) 462 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[22840]: audit 2024-09-13T23:03:09.018071+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.178:0/3590871403' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:09.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:09 smithi178 bash[22840]: audit 2024-09-13T23:03:09.018637+0000 mon.a (mon.0) 1891 : audit [INF] from='client.26027 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-13T23:03:10.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T23:03:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:09.053822+0000 mgr.y (mgr.14613) 945 : cluster [DBG] pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:09.331424+0000 osd.3 (osd.3) 463 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:03:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:09.333808+0000 osd.3 (osd.3) 464 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:03:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: audit 2024-09-13T23:03:09.605786+0000 mon.a (mon.0) 1892 : audit [INF] from='client.26027 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:03:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:09.613205+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2024-09-13T23:03:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:09.887083+0000 osd.4 (osd.4) 308 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T23:03:10.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:09.889606+0000 osd.4 (osd.4) 309 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T23:03:10.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:10.028087+0000 osd.6 (osd.6) 356 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:10.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:10 smithi192 bash[24608]: cluster 2024-09-13T23:03:10.028176+0000 osd.6 (osd.6) 357 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:09.053822+0000 mgr.y (mgr.14613) 945 : cluster [DBG] pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:09.331424+0000 osd.3 (osd.3) 463 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:09.333808+0000 osd.3 (osd.3) 464 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: audit 2024-09-13T23:03:09.605786+0000 mon.a (mon.0) 1892 : audit [INF] from='client.26027 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:09.613205+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:09.887083+0000 osd.4 (osd.4) 308 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:09.889606+0000 osd.4 (osd.4) 309 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:10.028087+0000 osd.6 (osd.6) 356 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[22840]: cluster 2024-09-13T23:03:10.028176+0000 osd.6 (osd.6) 357 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:09.053822+0000 mgr.y (mgr.14613) 945 : cluster [DBG] pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:09.331424+0000 osd.3 (osd.3) 463 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:09.333808+0000 osd.3 (osd.3) 464 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: audit 2024-09-13T23:03:09.605786+0000 mon.a (mon.0) 1892 : audit [INF] from='client.26027 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:09.613205+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:09.887083+0000 osd.4 (osd.4) 308 : cluster [DBG] 6.1a deep-scrub starts 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:09.889606+0000 osd.4 (osd.4) 309 : cluster [DBG] 6.1a deep-scrub ok 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:10.028087+0000 osd.6 (osd.6) 356 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:10.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:10 smithi178 bash[29889]: cluster 2024-09-13T23:03:10.028176+0000 osd.6 (osd.6) 357 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:11 smithi192 bash[24608]: cluster 2024-09-13T23:03:10.609050+0000 osd.1 (osd.1) 474 : cluster [DBG] 5.c scrub starts 2024-09-13T23:03:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:11 smithi192 bash[24608]: cluster 2024-09-13T23:03:10.611917+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.c scrub ok 2024-09-13T23:03:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:11 smithi192 bash[24608]: cluster 2024-09-13T23:03:10.614797+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2024-09-13T23:03:11.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:11 smithi178 bash[29889]: cluster 2024-09-13T23:03:10.609050+0000 osd.1 (osd.1) 474 : cluster [DBG] 5.c scrub starts 2024-09-13T23:03:11.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:11 smithi178 bash[29889]: cluster 2024-09-13T23:03:10.611917+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.c scrub ok 2024-09-13T23:03:11.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:11 smithi178 bash[29889]: cluster 2024-09-13T23:03:10.614797+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2024-09-13T23:03:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:11 smithi178 bash[22840]: cluster 2024-09-13T23:03:10.609050+0000 osd.1 (osd.1) 474 : cluster [DBG] 5.c scrub starts 2024-09-13T23:03:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:11 smithi178 bash[22840]: cluster 2024-09-13T23:03:10.611917+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.c scrub ok 2024-09-13T23:03:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:11 smithi178 bash[22840]: cluster 2024-09-13T23:03:10.614797+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2024-09-13T23:03:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:12 smithi192 bash[24608]: cluster 2024-09-13T23:03:11.054559+0000 mgr.y (mgr.14613) 946 : cluster [DBG] pgmap v1148: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:12 smithi192 bash[24608]: cluster 2024-09-13T23:03:11.422580+0000 osd.3 (osd.3) 465 : cluster [DBG] 6.f scrub starts 2024-09-13T23:03:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:12 smithi192 bash[24608]: cluster 2024-09-13T23:03:11.425800+0000 osd.3 (osd.3) 466 : cluster [DBG] 6.f scrub ok 2024-09-13T23:03:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:12 smithi178 bash[29889]: cluster 2024-09-13T23:03:11.054559+0000 mgr.y (mgr.14613) 946 : cluster [DBG] pgmap v1148: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:12 smithi178 bash[29889]: cluster 2024-09-13T23:03:11.422580+0000 osd.3 (osd.3) 465 : cluster [DBG] 6.f scrub starts 2024-09-13T23:03:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:12 smithi178 bash[29889]: cluster 2024-09-13T23:03:11.425800+0000 osd.3 (osd.3) 466 : cluster [DBG] 6.f scrub ok 2024-09-13T23:03:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:12 smithi178 bash[22840]: cluster 2024-09-13T23:03:11.054559+0000 mgr.y (mgr.14613) 946 : cluster [DBG] pgmap v1148: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:12 smithi178 bash[22840]: cluster 2024-09-13T23:03:11.422580+0000 osd.3 (osd.3) 465 : cluster [DBG] 6.f scrub starts 2024-09-13T23:03:12.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:12 smithi178 bash[22840]: cluster 2024-09-13T23:03:11.425800+0000 osd.3 (osd.3) 466 : cluster [DBG] 6.f scrub ok 2024-09-13T23:03:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:14 smithi192 bash[24608]: cluster 2024-09-13T23:03:12.905740+0000 osd.0 (osd.0) 469 : cluster [DBG] 6.0 scrub starts 2024-09-13T23:03:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:14 smithi192 bash[24608]: cluster 2024-09-13T23:03:12.908225+0000 osd.0 (osd.0) 470 : cluster [DBG] 6.0 scrub ok 2024-09-13T23:03:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:14 smithi192 bash[24608]: cluster 2024-09-13T23:03:13.055464+0000 mgr.y (mgr.14613) 947 : cluster [DBG] pgmap v1149: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:14.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:14 smithi178 bash[29889]: cluster 2024-09-13T23:03:12.905740+0000 osd.0 (osd.0) 469 : cluster [DBG] 6.0 scrub starts 2024-09-13T23:03:14.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:14 smithi178 bash[29889]: cluster 2024-09-13T23:03:12.908225+0000 osd.0 (osd.0) 470 : cluster [DBG] 6.0 scrub ok 2024-09-13T23:03:14.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:14 smithi178 bash[29889]: cluster 2024-09-13T23:03:13.055464+0000 mgr.y (mgr.14613) 947 : cluster [DBG] pgmap v1149: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:14.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:14 smithi178 bash[22840]: cluster 2024-09-13T23:03:12.905740+0000 osd.0 (osd.0) 469 : cluster [DBG] 6.0 scrub starts 2024-09-13T23:03:14.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:14 smithi178 bash[22840]: cluster 2024-09-13T23:03:12.908225+0000 osd.0 (osd.0) 470 : cluster [DBG] 6.0 scrub ok 2024-09-13T23:03:14.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:14 smithi178 bash[22840]: cluster 2024-09-13T23:03:13.055464+0000 mgr.y (mgr.14613) 947 : cluster [DBG] pgmap v1149: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:15.307 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:15 smithi178 bash[29889]: cluster 2024-09-13T23:03:15.182014+0000 osd.6 (osd.6) 358 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T23:03:15.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:15 smithi178 bash[29889]: cluster 2024-09-13T23:03:15.188014+0000 osd.6 (osd.6) 359 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T23:03:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:15 smithi178 bash[22840]: cluster 2024-09-13T23:03:15.182014+0000 osd.6 (osd.6) 358 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T23:03:15.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:15 smithi178 bash[22840]: cluster 2024-09-13T23:03:15.188014+0000 osd.6 (osd.6) 359 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T23:03:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:15 smithi192 bash[24608]: cluster 2024-09-13T23:03:15.182014+0000 osd.6 (osd.6) 358 : cluster [DBG] 3.a deep-scrub starts 2024-09-13T23:03:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:15 smithi192 bash[24608]: cluster 2024-09-13T23:03:15.188014+0000 osd.6 (osd.6) 359 : cluster [DBG] 3.a deep-scrub ok 2024-09-13T23:03:16.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:03:16 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:03:16.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:16 smithi192 bash[24608]: cluster 2024-09-13T23:03:14.925399+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.1 scrub starts 2024-09-13T23:03:16.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:16 smithi192 bash[24608]: cluster 2024-09-13T23:03:14.929838+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.1 scrub ok 2024-09-13T23:03:16.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:16 smithi192 bash[24608]: cluster 2024-09-13T23:03:15.056997+0000 mgr.y (mgr.14613) 948 : cluster [DBG] pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:16.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:16 smithi192 bash[24608]: cluster 2024-09-13T23:03:15.646348+0000 osd.1 (osd.1) 476 : cluster [DBG] 4.2 scrub starts 2024-09-13T23:03:16.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[29889]: cluster 2024-09-13T23:03:14.925399+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.1 scrub starts 2024-09-13T23:03:16.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[29889]: cluster 2024-09-13T23:03:14.929838+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.1 scrub ok 2024-09-13T23:03:16.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[29889]: cluster 2024-09-13T23:03:15.056997+0000 mgr.y (mgr.14613) 948 : cluster [DBG] pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:16.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[29889]: cluster 2024-09-13T23:03:15.646348+0000 osd.1 (osd.1) 476 : cluster [DBG] 4.2 scrub starts 2024-09-13T23:03:16.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[22840]: cluster 2024-09-13T23:03:14.925399+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.1 scrub starts 2024-09-13T23:03:16.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[22840]: cluster 2024-09-13T23:03:14.929838+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.1 scrub ok 2024-09-13T23:03:16.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[22840]: cluster 2024-09-13T23:03:15.056997+0000 mgr.y (mgr.14613) 948 : cluster [DBG] pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:16.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:16 smithi178 bash[22840]: cluster 2024-09-13T23:03:15.646348+0000 osd.1 (osd.1) 476 : cluster [DBG] 4.2 scrub starts 2024-09-13T23:03:17.645 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T23:03:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:17 smithi192 bash[24608]: cluster 2024-09-13T23:03:15.649918+0000 osd.1 (osd.1) 477 : cluster [DBG] 4.2 scrub ok 2024-09-13T23:03:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:17 smithi192 bash[24608]: audit 2024-09-13T23:03:16.162271+0000 mgr.y (mgr.14613) 949 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:17 smithi192 bash[24608]: cluster 2024-09-13T23:03:16.433442+0000 osd.3 (osd.3) 467 : cluster [DBG] 2.1e deep-scrub starts 2024-09-13T23:03:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:17 smithi192 bash[24608]: cluster 2024-09-13T23:03:16.435802+0000 osd.3 (osd.3) 468 : cluster [DBG] 2.1e deep-scrub ok 2024-09-13T23:03:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:17 smithi192 bash[24608]: audit 2024-09-13T23:03:16.971069+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? v1:172.21.15.178:0/2141434420' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:03:17.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[29889]: cluster 2024-09-13T23:03:15.649918+0000 osd.1 (osd.1) 477 : cluster [DBG] 4.2 scrub ok 2024-09-13T23:03:17.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[29889]: audit 2024-09-13T23:03:16.162271+0000 mgr.y (mgr.14613) 949 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[29889]: cluster 2024-09-13T23:03:16.433442+0000 osd.3 (osd.3) 467 : cluster [DBG] 2.1e deep-scrub starts 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[29889]: cluster 2024-09-13T23:03:16.435802+0000 osd.3 (osd.3) 468 : cluster [DBG] 2.1e deep-scrub ok 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[29889]: audit 2024-09-13T23:03:16.971069+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? v1:172.21.15.178:0/2141434420' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[22840]: cluster 2024-09-13T23:03:15.649918+0000 osd.1 (osd.1) 477 : cluster [DBG] 4.2 scrub ok 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[22840]: audit 2024-09-13T23:03:16.162271+0000 mgr.y (mgr.14613) 949 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[22840]: cluster 2024-09-13T23:03:16.433442+0000 osd.3 (osd.3) 467 : cluster [DBG] 2.1e deep-scrub starts 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[22840]: cluster 2024-09-13T23:03:16.435802+0000 osd.3 (osd.3) 468 : cluster [DBG] 2.1e deep-scrub ok 2024-09-13T23:03:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:17 smithi178 bash[22840]: audit 2024-09-13T23:03:16.971069+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? v1:172.21.15.178:0/2141434420' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:03:18.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T23:03:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: cluster 2024-09-13T23:03:17.057834+0000 mgr.y (mgr.14613) 950 : cluster [DBG] pgmap v1151: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: cluster 2024-09-13T23:03:17.633661+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:03:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: cluster 2024-09-13T23:03:17.640143+0000 osd.1 (osd.1) 479 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:03:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: audit 2024-09-13T23:03:17.645961+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? v1:172.21.15.178:0/2141434420' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:03:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: cluster 2024-09-13T23:03:17.652070+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2024-09-13T23:03:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: cluster 2024-09-13T23:03:17.778620+0000 osd.4 (osd.4) 310 : cluster [DBG] 2.f scrub starts 2024-09-13T23:03:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:18 smithi192 bash[24608]: cluster 2024-09-13T23:03:17.782556+0000 osd.4 (osd.4) 311 : cluster [DBG] 2.f scrub ok 2024-09-13T23:03:18.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: cluster 2024-09-13T23:03:17.057834+0000 mgr.y (mgr.14613) 950 : cluster [DBG] pgmap v1151: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:18.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: cluster 2024-09-13T23:03:17.633661+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: cluster 2024-09-13T23:03:17.640143+0000 osd.1 (osd.1) 479 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: audit 2024-09-13T23:03:17.645961+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? v1:172.21.15.178:0/2141434420' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: cluster 2024-09-13T23:03:17.652070+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: cluster 2024-09-13T23:03:17.778620+0000 osd.4 (osd.4) 310 : cluster [DBG] 2.f scrub starts 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[29889]: cluster 2024-09-13T23:03:17.782556+0000 osd.4 (osd.4) 311 : cluster [DBG] 2.f scrub ok 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: cluster 2024-09-13T23:03:17.057834+0000 mgr.y (mgr.14613) 950 : cluster [DBG] pgmap v1151: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: cluster 2024-09-13T23:03:17.633661+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: cluster 2024-09-13T23:03:17.640143+0000 osd.1 (osd.1) 479 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: audit 2024-09-13T23:03:17.645961+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? v1:172.21.15.178:0/2141434420' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: cluster 2024-09-13T23:03:17.652070+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2024-09-13T23:03:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: cluster 2024-09-13T23:03:17.778620+0000 osd.4 (osd.4) 310 : cluster [DBG] 2.f scrub starts 2024-09-13T23:03:18.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:18 smithi178 bash[22840]: cluster 2024-09-13T23:03:17.782556+0000 osd.4 (osd.4) 311 : cluster [DBG] 2.f scrub ok 2024-09-13T23:03:19.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:03:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[22840]: cluster 2024-09-13T23:03:18.449143+0000 osd.3 (osd.3) 469 : cluster [DBG] 6.6 deep-scrub starts 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[22840]: cluster 2024-09-13T23:03:18.451336+0000 osd.3 (osd.3) 470 : cluster [DBG] 6.6 deep-scrub ok 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[22840]: cluster 2024-09-13T23:03:18.763044+0000 osd.4 (osd.4) 312 : cluster [DBG] 6.1e scrub starts 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[22840]: cluster 2024-09-13T23:03:18.765530+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.1e scrub ok 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[22840]: cluster 2024-09-13T23:03:19.217272+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[22840]: cluster 2024-09-13T23:03:19.219694+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[29889]: cluster 2024-09-13T23:03:18.449143+0000 osd.3 (osd.3) 469 : cluster [DBG] 6.6 deep-scrub starts 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[29889]: cluster 2024-09-13T23:03:18.451336+0000 osd.3 (osd.3) 470 : cluster [DBG] 6.6 deep-scrub ok 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[29889]: cluster 2024-09-13T23:03:18.763044+0000 osd.4 (osd.4) 312 : cluster [DBG] 6.1e scrub starts 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[29889]: cluster 2024-09-13T23:03:18.765530+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.1e scrub ok 2024-09-13T23:03:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[29889]: cluster 2024-09-13T23:03:19.217272+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:03:19.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:19 smithi178 bash[29889]: cluster 2024-09-13T23:03:19.219694+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:03:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:19 smithi192 bash[24608]: cluster 2024-09-13T23:03:18.449143+0000 osd.3 (osd.3) 469 : cluster [DBG] 6.6 deep-scrub starts 2024-09-13T23:03:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:19 smithi192 bash[24608]: cluster 2024-09-13T23:03:18.451336+0000 osd.3 (osd.3) 470 : cluster [DBG] 6.6 deep-scrub ok 2024-09-13T23:03:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:19 smithi192 bash[24608]: cluster 2024-09-13T23:03:18.763044+0000 osd.4 (osd.4) 312 : cluster [DBG] 6.1e scrub starts 2024-09-13T23:03:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:19 smithi192 bash[24608]: cluster 2024-09-13T23:03:18.765530+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.1e scrub ok 2024-09-13T23:03:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:19 smithi192 bash[24608]: cluster 2024-09-13T23:03:19.217272+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:03:20.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:19 smithi192 bash[24608]: cluster 2024-09-13T23:03:19.219694+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:03:20.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[29889]: cluster 2024-09-13T23:03:19.058795+0000 mgr.y (mgr.14613) 951 : cluster [DBG] pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[29889]: cluster 2024-09-13T23:03:19.798631+0000 osd.7 (osd.7) 506 : cluster [DBG] 3.f scrub starts 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[29889]: cluster 2024-09-13T23:03:19.803154+0000 osd.7 (osd.7) 507 : cluster [DBG] 3.f scrub ok 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[29889]: audit 2024-09-13T23:03:19.844905+0000 mon.b (mon.1) 310 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[22840]: cluster 2024-09-13T23:03:19.058795+0000 mgr.y (mgr.14613) 951 : cluster [DBG] pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[22840]: cluster 2024-09-13T23:03:19.798631+0000 osd.7 (osd.7) 506 : cluster [DBG] 3.f scrub starts 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[22840]: cluster 2024-09-13T23:03:19.803154+0000 osd.7 (osd.7) 507 : cluster [DBG] 3.f scrub ok 2024-09-13T23:03:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:20 smithi178 bash[22840]: audit 2024-09-13T23:03:19.844905+0000 mon.b (mon.1) 310 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:21.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:20 smithi192 bash[24608]: cluster 2024-09-13T23:03:19.058795+0000 mgr.y (mgr.14613) 951 : cluster [DBG] pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-09-13T23:03:21.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:20 smithi192 bash[24608]: cluster 2024-09-13T23:03:19.798631+0000 osd.7 (osd.7) 506 : cluster [DBG] 3.f scrub starts 2024-09-13T23:03:21.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:20 smithi192 bash[24608]: cluster 2024-09-13T23:03:19.803154+0000 osd.7 (osd.7) 507 : cluster [DBG] 3.f scrub ok 2024-09-13T23:03:21.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:20 smithi192 bash[24608]: audit 2024-09-13T23:03:19.844905+0000 mon.b (mon.1) 310 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:21.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[29889]: cluster 2024-09-13T23:03:20.814371+0000 osd.7 (osd.7) 508 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:03:21.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[29889]: cluster 2024-09-13T23:03:20.816804+0000 osd.7 (osd.7) 509 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[29889]: cluster 2024-09-13T23:03:20.875273+0000 osd.0 (osd.0) 473 : cluster [DBG] 3.17 scrub starts 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[29889]: cluster 2024-09-13T23:03:20.878624+0000 osd.0 (osd.0) 474 : cluster [DBG] 3.17 scrub ok 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[29889]: cluster 2024-09-13T23:03:21.462753+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.8 deep-scrub starts 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[29889]: cluster 2024-09-13T23:03:21.476470+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.8 deep-scrub ok 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[22840]: cluster 2024-09-13T23:03:20.814371+0000 osd.7 (osd.7) 508 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[22840]: cluster 2024-09-13T23:03:20.816804+0000 osd.7 (osd.7) 509 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[22840]: cluster 2024-09-13T23:03:20.875273+0000 osd.0 (osd.0) 473 : cluster [DBG] 3.17 scrub starts 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[22840]: cluster 2024-09-13T23:03:20.878624+0000 osd.0 (osd.0) 474 : cluster [DBG] 3.17 scrub ok 2024-09-13T23:03:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[22840]: cluster 2024-09-13T23:03:21.462753+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.8 deep-scrub starts 2024-09-13T23:03:21.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:21 smithi178 bash[22840]: cluster 2024-09-13T23:03:21.476470+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.8 deep-scrub ok 2024-09-13T23:03:22.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:21 smithi192 bash[24608]: cluster 2024-09-13T23:03:20.814371+0000 osd.7 (osd.7) 508 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:03:22.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:21 smithi192 bash[24608]: cluster 2024-09-13T23:03:20.816804+0000 osd.7 (osd.7) 509 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:03:22.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:21 smithi192 bash[24608]: cluster 2024-09-13T23:03:20.875273+0000 osd.0 (osd.0) 473 : cluster [DBG] 3.17 scrub starts 2024-09-13T23:03:22.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:21 smithi192 bash[24608]: cluster 2024-09-13T23:03:20.878624+0000 osd.0 (osd.0) 474 : cluster [DBG] 3.17 scrub ok 2024-09-13T23:03:22.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:21 smithi192 bash[24608]: cluster 2024-09-13T23:03:21.462753+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.8 deep-scrub starts 2024-09-13T23:03:22.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:21 smithi192 bash[24608]: cluster 2024-09-13T23:03:21.476470+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.8 deep-scrub ok 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:21.060218+0000 mgr.y (mgr.14613) 952 : cluster [DBG] pgmap v1154: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:21.875232+0000 osd.0 (osd.0) 475 : cluster [DBG] 3.1b deep-scrub starts 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:21.879051+0000 osd.0 (osd.0) 476 : cluster [DBG] 3.1b deep-scrub ok 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:22.128206+0000 osd.6 (osd.6) 362 : cluster [DBG] 2.14 scrub starts 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:22.130413+0000 osd.6 (osd.6) 363 : cluster [DBG] 2.14 scrub ok 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:22.595458+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:23.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:22 smithi192 bash[24608]: cluster 2024-09-13T23:03:22.595547+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:21.060218+0000 mgr.y (mgr.14613) 952 : cluster [DBG] pgmap v1154: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:21.875232+0000 osd.0 (osd.0) 475 : cluster [DBG] 3.1b deep-scrub starts 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:21.879051+0000 osd.0 (osd.0) 476 : cluster [DBG] 3.1b deep-scrub ok 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:22.128206+0000 osd.6 (osd.6) 362 : cluster [DBG] 2.14 scrub starts 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:22.130413+0000 osd.6 (osd.6) 363 : cluster [DBG] 2.14 scrub ok 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:22.595458+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[22840]: cluster 2024-09-13T23:03:22.595547+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:23.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:21.060218+0000 mgr.y (mgr.14613) 952 : cluster [DBG] pgmap v1154: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:23.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:21.875232+0000 osd.0 (osd.0) 475 : cluster [DBG] 3.1b deep-scrub starts 2024-09-13T23:03:23.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:21.879051+0000 osd.0 (osd.0) 476 : cluster [DBG] 3.1b deep-scrub ok 2024-09-13T23:03:23.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:22.128206+0000 osd.6 (osd.6) 362 : cluster [DBG] 2.14 scrub starts 2024-09-13T23:03:23.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:22.130413+0000 osd.6 (osd.6) 363 : cluster [DBG] 2.14 scrub ok 2024-09-13T23:03:23.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:22.595458+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:23.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:22 smithi178 bash[29889]: cluster 2024-09-13T23:03:22.595547+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:23.289 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:24.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:23 smithi192 bash[24608]: cluster 2024-09-13T23:03:22.684214+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2024-09-13T23:03:24.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:23 smithi192 bash[24608]: cluster 2024-09-13T23:03:23.168253+0000 osd.6 (osd.6) 364 : cluster [DBG] 5.2 scrub starts 2024-09-13T23:03:24.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:23 smithi192 bash[24608]: cluster 2024-09-13T23:03:23.170722+0000 osd.6 (osd.6) 365 : cluster [DBG] 5.2 scrub ok 2024-09-13T23:03:24.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:23 smithi178 bash[29889]: cluster 2024-09-13T23:03:22.684214+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2024-09-13T23:03:24.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:23 smithi178 bash[29889]: cluster 2024-09-13T23:03:23.168253+0000 osd.6 (osd.6) 364 : cluster [DBG] 5.2 scrub starts 2024-09-13T23:03:24.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:23 smithi178 bash[29889]: cluster 2024-09-13T23:03:23.170722+0000 osd.6 (osd.6) 365 : cluster [DBG] 5.2 scrub ok 2024-09-13T23:03:24.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:23 smithi178 bash[22840]: cluster 2024-09-13T23:03:22.684214+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2024-09-13T23:03:24.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:23 smithi178 bash[22840]: cluster 2024-09-13T23:03:23.168253+0000 osd.6 (osd.6) 364 : cluster [DBG] 5.2 scrub starts 2024-09-13T23:03:24.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:23 smithi178 bash[22840]: cluster 2024-09-13T23:03:23.170722+0000 osd.6 (osd.6) 365 : cluster [DBG] 5.2 scrub ok 2024-09-13T23:03:25.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:24 smithi192 bash[24608]: cluster 2024-09-13T23:03:23.060937+0000 mgr.y (mgr.14613) 953 : cluster [DBG] pgmap v1156: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:25.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:24 smithi192 bash[24608]: cluster 2024-09-13T23:03:23.701065+0000 osd.5 (osd.5) 533 : cluster [DBG] 1.a scrub starts 2024-09-13T23:03:25.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:24 smithi192 bash[24608]: cluster 2024-09-13T23:03:23.703734+0000 osd.5 (osd.5) 534 : cluster [DBG] 1.a scrub ok 2024-09-13T23:03:25.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:24 smithi192 bash[24608]: cluster 2024-09-13T23:03:24.504601+0000 osd.3 (osd.3) 475 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:03:25.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:24 smithi192 bash[24608]: cluster 2024-09-13T23:03:24.508516+0000 osd.3 (osd.3) 476 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:03:25.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[29889]: cluster 2024-09-13T23:03:23.060937+0000 mgr.y (mgr.14613) 953 : cluster [DBG] pgmap v1156: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[29889]: cluster 2024-09-13T23:03:23.701065+0000 osd.5 (osd.5) 533 : cluster [DBG] 1.a scrub starts 2024-09-13T23:03:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[29889]: cluster 2024-09-13T23:03:23.703734+0000 osd.5 (osd.5) 534 : cluster [DBG] 1.a scrub ok 2024-09-13T23:03:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[29889]: cluster 2024-09-13T23:03:24.504601+0000 osd.3 (osd.3) 475 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:03:25.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[29889]: cluster 2024-09-13T23:03:24.508516+0000 osd.3 (osd.3) 476 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:03:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[22840]: cluster 2024-09-13T23:03:23.060937+0000 mgr.y (mgr.14613) 953 : cluster [DBG] pgmap v1156: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[22840]: cluster 2024-09-13T23:03:23.701065+0000 osd.5 (osd.5) 533 : cluster [DBG] 1.a scrub starts 2024-09-13T23:03:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[22840]: cluster 2024-09-13T23:03:23.703734+0000 osd.5 (osd.5) 534 : cluster [DBG] 1.a scrub ok 2024-09-13T23:03:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[22840]: cluster 2024-09-13T23:03:24.504601+0000 osd.3 (osd.3) 475 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:03:25.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:24 smithi178 bash[22840]: cluster 2024-09-13T23:03:24.508516+0000 osd.3 (osd.3) 476 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:03:25.706 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T23:03:26.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:25 smithi192 bash[24608]: audit 2024-09-13T23:03:25.202055+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.178:0/1100163724' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:03:26.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:25 smithi178 bash[22840]: audit 2024-09-13T23:03:25.202055+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.178:0/1100163724' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:03:26.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:25 smithi178 bash[29889]: audit 2024-09-13T23:03:25.202055+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.178:0/1100163724' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-13T23:03:26.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T23:03:26.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:03:26 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:03:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:26 smithi178 bash[22840]: cluster 2024-09-13T23:03:25.061929+0000 mgr.y (mgr.14613) 954 : cluster [DBG] pgmap v1157: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:26 smithi178 bash[22840]: audit 2024-09-13T23:03:25.707538+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? v1:172.21.15.178:0/1100163724' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:03:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:26 smithi178 bash[22840]: cluster 2024-09-13T23:03:25.714532+0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2024-09-13T23:03:26.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:26 smithi178 bash[29889]: cluster 2024-09-13T23:03:25.061929+0000 mgr.y (mgr.14613) 954 : cluster [DBG] pgmap v1157: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:26.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:26 smithi178 bash[29889]: audit 2024-09-13T23:03:25.707538+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? v1:172.21.15.178:0/1100163724' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:03:26.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:26 smithi178 bash[29889]: cluster 2024-09-13T23:03:25.714532+0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2024-09-13T23:03:27.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:26 smithi192 bash[24608]: cluster 2024-09-13T23:03:25.061929+0000 mgr.y (mgr.14613) 954 : cluster [DBG] pgmap v1157: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:27.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:26 smithi192 bash[24608]: audit 2024-09-13T23:03:25.707538+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? v1:172.21.15.178:0/1100163724' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T23:03:27.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:26 smithi192 bash[24608]: cluster 2024-09-13T23:03:25.714532+0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2024-09-13T23:03:28.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:27 smithi192 bash[24608]: audit 2024-09-13T23:03:26.170182+0000 mgr.y (mgr.14613) 955 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:28.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:27 smithi192 bash[24608]: cluster 2024-09-13T23:03:26.721616+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2024-09-13T23:03:28.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:27 smithi192 bash[24608]: cluster 2024-09-13T23:03:26.930983+0000 osd.0 (osd.0) 477 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:03:28.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:27 smithi192 bash[24608]: cluster 2024-09-13T23:03:26.936760+0000 osd.0 (osd.0) 478 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:03:28.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[29889]: audit 2024-09-13T23:03:26.170182+0000 mgr.y (mgr.14613) 955 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[29889]: cluster 2024-09-13T23:03:26.721616+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[29889]: cluster 2024-09-13T23:03:26.930983+0000 osd.0 (osd.0) 477 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[29889]: cluster 2024-09-13T23:03:26.936760+0000 osd.0 (osd.0) 478 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[22840]: audit 2024-09-13T23:03:26.170182+0000 mgr.y (mgr.14613) 955 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[22840]: cluster 2024-09-13T23:03:26.721616+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[22840]: cluster 2024-09-13T23:03:26.930983+0000 osd.0 (osd.0) 477 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:03:28.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:27 smithi178 bash[22840]: cluster 2024-09-13T23:03:26.936760+0000 osd.0 (osd.0) 478 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:03:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:28 smithi192 bash[24608]: cluster 2024-09-13T23:03:27.062639+0000 mgr.y (mgr.14613) 956 : cluster [DBG] pgmap v1160: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:03:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:28 smithi192 bash[24608]: cluster 2024-09-13T23:03:27.680468+0000 osd.5 (osd.5) 535 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:03:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:28 smithi192 bash[24608]: cluster 2024-09-13T23:03:27.682679+0000 osd.5 (osd.5) 536 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:03:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:28 smithi192 bash[24608]: cluster 2024-09-13T23:03:27.902409+0000 osd.7 (osd.7) 510 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:03:29.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:28 smithi192 bash[24608]: cluster 2024-09-13T23:03:27.904714+0000 osd.7 (osd.7) 511 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:03:29.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[29889]: cluster 2024-09-13T23:03:27.062639+0000 mgr.y (mgr.14613) 956 : cluster [DBG] pgmap v1160: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[29889]: cluster 2024-09-13T23:03:27.680468+0000 osd.5 (osd.5) 535 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[29889]: cluster 2024-09-13T23:03:27.682679+0000 osd.5 (osd.5) 536 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[29889]: cluster 2024-09-13T23:03:27.902409+0000 osd.7 (osd.7) 510 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[29889]: cluster 2024-09-13T23:03:27.904714+0000 osd.7 (osd.7) 511 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[22840]: cluster 2024-09-13T23:03:27.062639+0000 mgr.y (mgr.14613) 956 : cluster [DBG] pgmap v1160: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[22840]: cluster 2024-09-13T23:03:27.680468+0000 osd.5 (osd.5) 535 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[22840]: cluster 2024-09-13T23:03:27.682679+0000 osd.5 (osd.5) 536 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[22840]: cluster 2024-09-13T23:03:27.902409+0000 osd.7 (osd.7) 510 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:03:29.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:28 smithi178 bash[22840]: cluster 2024-09-13T23:03:27.904714+0000 osd.7 (osd.7) 511 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:03:30.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:29 smithi192 bash[24608]: cluster 2024-09-13T23:03:29.232989+0000 osd.6 (osd.6) 366 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:03:30.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:29 smithi192 bash[24608]: cluster 2024-09-13T23:03:29.235327+0000 osd.6 (osd.6) 367 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:03:30.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:29 smithi192 bash[24608]: cluster 2024-09-13T23:03:29.539928+0000 osd.2 (osd.2) 341 : cluster [DBG] 5.d scrub starts 2024-09-13T23:03:30.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:29 smithi192 bash[24608]: cluster 2024-09-13T23:03:29.543392+0000 osd.2 (osd.2) 342 : cluster [DBG] 5.d scrub ok 2024-09-13T23:03:30.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:03:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:03:30.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[29889]: cluster 2024-09-13T23:03:29.232989+0000 osd.6 (osd.6) 366 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[29889]: cluster 2024-09-13T23:03:29.235327+0000 osd.6 (osd.6) 367 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[29889]: cluster 2024-09-13T23:03:29.539928+0000 osd.2 (osd.2) 341 : cluster [DBG] 5.d scrub starts 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[29889]: cluster 2024-09-13T23:03:29.543392+0000 osd.2 (osd.2) 342 : cluster [DBG] 5.d scrub ok 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[22840]: cluster 2024-09-13T23:03:29.232989+0000 osd.6 (osd.6) 366 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[22840]: cluster 2024-09-13T23:03:29.235327+0000 osd.6 (osd.6) 367 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[22840]: cluster 2024-09-13T23:03:29.539928+0000 osd.2 (osd.2) 341 : cluster [DBG] 5.d scrub starts 2024-09-13T23:03:30.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:29 smithi178 bash[22840]: cluster 2024-09-13T23:03:29.543392+0000 osd.2 (osd.2) 342 : cluster [DBG] 5.d scrub ok 2024-09-13T23:03:31.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:30 smithi192 bash[24608]: cluster 2024-09-13T23:03:29.063753+0000 mgr.y (mgr.14613) 957 : cluster [DBG] pgmap v1161: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 802 B/s rd, 0 op/s 2024-09-13T23:03:31.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:30 smithi178 bash[29889]: cluster 2024-09-13T23:03:29.063753+0000 mgr.y (mgr.14613) 957 : cluster [DBG] pgmap v1161: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 802 B/s rd, 0 op/s 2024-09-13T23:03:31.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:30 smithi178 bash[22840]: cluster 2024-09-13T23:03:29.063753+0000 mgr.y (mgr.14613) 957 : cluster [DBG] pgmap v1161: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 802 B/s rd, 0 op/s 2024-09-13T23:03:31.337 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:30.653995+0000 osd.4 (osd.4) 314 : cluster [DBG] 4.1 scrub starts 2024-09-13T23:03:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:30.657058+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.1 scrub ok 2024-09-13T23:03:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:30.848324+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:03:32.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:30.852428+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:03:32.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:31.272604+0000 osd.6 (osd.6) 368 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T23:03:32.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:31.274820+0000 osd.6 (osd.6) 369 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T23:03:32.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:31.350026+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.1d scrub starts 2024-09-13T23:03:32.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:31 smithi192 bash[24608]: cluster 2024-09-13T23:03:31.351945+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub ok 2024-09-13T23:03:32.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:30.653995+0000 osd.4 (osd.4) 314 : cluster [DBG] 4.1 scrub starts 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:30.657058+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.1 scrub ok 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:30.848324+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:30.852428+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:31.272604+0000 osd.6 (osd.6) 368 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:31.274820+0000 osd.6 (osd.6) 369 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:31.350026+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.1d scrub starts 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[29889]: cluster 2024-09-13T23:03:31.351945+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub ok 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:30.653995+0000 osd.4 (osd.4) 314 : cluster [DBG] 4.1 scrub starts 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:30.657058+0000 osd.4 (osd.4) 315 : cluster [DBG] 4.1 scrub ok 2024-09-13T23:03:32.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:30.848324+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:03:32.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:30.852428+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:03:32.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:31.272604+0000 osd.6 (osd.6) 368 : cluster [DBG] 5.11 deep-scrub starts 2024-09-13T23:03:32.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:31.274820+0000 osd.6 (osd.6) 369 : cluster [DBG] 5.11 deep-scrub ok 2024-09-13T23:03:32.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:31.350026+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.1d scrub starts 2024-09-13T23:03:32.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:31 smithi178 bash[22840]: cluster 2024-09-13T23:03:31.351945+0000 osd.1 (osd.1) 481 : cluster [DBG] 6.1d scrub ok 2024-09-13T23:03:33.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:32 smithi192 bash[24608]: cluster 2024-09-13T23:03:31.065078+0000 mgr.y (mgr.14613) 958 : cluster [DBG] pgmap v1162: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:33.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:32 smithi178 bash[29889]: cluster 2024-09-13T23:03:31.065078+0000 mgr.y (mgr.14613) 958 : cluster [DBG] pgmap v1162: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:33.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:32 smithi178 bash[22840]: cluster 2024-09-13T23:03:31.065078+0000 mgr.y (mgr.14613) 958 : cluster [DBG] pgmap v1162: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:34.256 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T23:03:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:34 smithi192 bash[24608]: cluster 2024-09-13T23:03:32.637654+0000 osd.5 (osd.5) 537 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:03:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:34 smithi192 bash[24608]: cluster 2024-09-13T23:03:32.640046+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:03:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:34 smithi192 bash[24608]: cluster 2024-09-13T23:03:33.065585+0000 mgr.y (mgr.14613) 959 : cluster [DBG] pgmap v1163: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:34 smithi192 bash[24608]: audit 2024-09-13T23:03:33.254061+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.178:0/4260932360' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:34 smithi192 bash[24608]: audit 2024-09-13T23:03:33.254674+0000 mon.a (mon.0) 1903 : audit [INF] from='client.26045 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[29889]: cluster 2024-09-13T23:03:32.637654+0000 osd.5 (osd.5) 537 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:03:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[29889]: cluster 2024-09-13T23:03:32.640046+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[29889]: cluster 2024-09-13T23:03:33.065585+0000 mgr.y (mgr.14613) 959 : cluster [DBG] pgmap v1163: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[29889]: audit 2024-09-13T23:03:33.254061+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.178:0/4260932360' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[29889]: audit 2024-09-13T23:03:33.254674+0000 mon.a (mon.0) 1903 : audit [INF] from='client.26045 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[22840]: cluster 2024-09-13T23:03:32.637654+0000 osd.5 (osd.5) 537 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[22840]: cluster 2024-09-13T23:03:32.640046+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[22840]: cluster 2024-09-13T23:03:33.065585+0000 mgr.y (mgr.14613) 959 : cluster [DBG] pgmap v1163: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[22840]: audit 2024-09-13T23:03:33.254061+0000 mon.c (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.178:0/4260932360' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:34 smithi178 bash[22840]: audit 2024-09-13T23:03:33.254674+0000 mon.a (mon.0) 1903 : audit [INF] from='client.26045 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:35.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-13T23:03:35.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[29889]: audit 2024-09-13T23:03:34.249985+0000 mon.a (mon.0) 1904 : audit [INF] from='client.26045 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[29889]: cluster 2024-09-13T23:03:34.256220+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[29889]: cluster 2024-09-13T23:03:34.325790+0000 osd.1 (osd.1) 482 : cluster [DBG] 2.a scrub starts 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[29889]: cluster 2024-09-13T23:03:34.327480+0000 osd.1 (osd.1) 483 : cluster [DBG] 2.a scrub ok 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[29889]: audit 2024-09-13T23:03:34.873284+0000 mon.b (mon.1) 311 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[22840]: audit 2024-09-13T23:03:34.249985+0000 mon.a (mon.0) 1904 : audit [INF] from='client.26045 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[22840]: cluster 2024-09-13T23:03:34.256220+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[22840]: cluster 2024-09-13T23:03:34.325790+0000 osd.1 (osd.1) 482 : cluster [DBG] 2.a scrub starts 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[22840]: cluster 2024-09-13T23:03:34.327480+0000 osd.1 (osd.1) 483 : cluster [DBG] 2.a scrub ok 2024-09-13T23:03:35.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:35 smithi178 bash[22840]: audit 2024-09-13T23:03:34.873284+0000 mon.b (mon.1) 311 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:35 smithi192 bash[24608]: audit 2024-09-13T23:03:34.249985+0000 mon.a (mon.0) 1904 : audit [INF] from='client.26045 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:03:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:35 smithi192 bash[24608]: cluster 2024-09-13T23:03:34.256220+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2024-09-13T23:03:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:35 smithi192 bash[24608]: cluster 2024-09-13T23:03:34.325790+0000 osd.1 (osd.1) 482 : cluster [DBG] 2.a scrub starts 2024-09-13T23:03:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:35 smithi192 bash[24608]: cluster 2024-09-13T23:03:34.327480+0000 osd.1 (osd.1) 483 : cluster [DBG] 2.a scrub ok 2024-09-13T23:03:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:35 smithi192 bash[24608]: audit 2024-09-13T23:03:34.873284+0000 mon.b (mon.1) 311 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:36.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:03:36 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:03:36.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:36 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.067203+0000 mgr.y (mgr.14613) 960 : cluster [DBG] pgmap v1165: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:36.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:36 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.441147+0000 osd.3 (osd.3) 477 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:03:36.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:36 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.443597+0000 osd.3 (osd.3) 478 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:03:36.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:36 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.687637+0000 osd.4 (osd.4) 316 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:03:36.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:36 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.690003+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:03:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.067203+0000 mgr.y (mgr.14613) 960 : cluster [DBG] pgmap v1165: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.441147+0000 osd.3 (osd.3) 477 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.443597+0000 osd.3 (osd.3) 478 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.687637+0000 osd.4 (osd.4) 316 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.690003+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.067203+0000 mgr.y (mgr.14613) 960 : cluster [DBG] pgmap v1165: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.441147+0000 osd.3 (osd.3) 477 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.443597+0000 osd.3 (osd.3) 478 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.687637+0000 osd.4 (osd.4) 316 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:03:36.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:36 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.690003+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:03:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.977347+0000 osd.0 (osd.0) 479 : cluster [DBG] 6.9 scrub starts 2024-09-13T23:03:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: cluster 2024-09-13T23:03:35.979580+0000 osd.0 (osd.0) 480 : cluster [DBG] 6.9 scrub ok 2024-09-13T23:03:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: audit 2024-09-13T23:03:36.181537+0000 mgr.y (mgr.14613) 961 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: cluster 2024-09-13T23:03:36.300732+0000 osd.6 (osd.6) 370 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:03:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: cluster 2024-09-13T23:03:36.303153+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:03:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: cluster 2024-09-13T23:03:36.445704+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.b scrub starts 2024-09-13T23:03:37.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:37 smithi192 bash[24608]: cluster 2024-09-13T23:03:36.448365+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.b scrub ok 2024-09-13T23:03:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.977347+0000 osd.0 (osd.0) 479 : cluster [DBG] 6.9 scrub starts 2024-09-13T23:03:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: cluster 2024-09-13T23:03:35.979580+0000 osd.0 (osd.0) 480 : cluster [DBG] 6.9 scrub ok 2024-09-13T23:03:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: audit 2024-09-13T23:03:36.181537+0000 mgr.y (mgr.14613) 961 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: cluster 2024-09-13T23:03:36.300732+0000 osd.6 (osd.6) 370 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: cluster 2024-09-13T23:03:36.303153+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: cluster 2024-09-13T23:03:36.445704+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.b scrub starts 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[29889]: cluster 2024-09-13T23:03:36.448365+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.b scrub ok 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.977347+0000 osd.0 (osd.0) 479 : cluster [DBG] 6.9 scrub starts 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: cluster 2024-09-13T23:03:35.979580+0000 osd.0 (osd.0) 480 : cluster [DBG] 6.9 scrub ok 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: audit 2024-09-13T23:03:36.181537+0000 mgr.y (mgr.14613) 961 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: cluster 2024-09-13T23:03:36.300732+0000 osd.6 (osd.6) 370 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: cluster 2024-09-13T23:03:36.303153+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: cluster 2024-09-13T23:03:36.445704+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.b scrub starts 2024-09-13T23:03:37.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:37 smithi178 bash[22840]: cluster 2024-09-13T23:03:36.448365+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.b scrub ok 2024-09-13T23:03:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.068092+0000 mgr.y (mgr.14613) 962 : cluster [DBG] pgmap v1166: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.300335+0000 osd.1 (osd.1) 484 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:03:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.302654+0000 osd.1 (osd.1) 485 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:03:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.349767+0000 osd.0 (osd.0) 481 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.349867+0000 osd.0 (osd.0) 482 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.700970+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:03:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:38 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.705099+0000 osd.4 (osd.4) 319 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:03:38.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.068092+0000 mgr.y (mgr.14613) 962 : cluster [DBG] pgmap v1166: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.300335+0000 osd.1 (osd.1) 484 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.302654+0000 osd.1 (osd.1) 485 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.349767+0000 osd.0 (osd.0) 481 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.349867+0000 osd.0 (osd.0) 482 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.700970+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.705099+0000 osd.4 (osd.4) 319 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:03:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.068092+0000 mgr.y (mgr.14613) 962 : cluster [DBG] pgmap v1166: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:03:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.300335+0000 osd.1 (osd.1) 484 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:03:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.302654+0000 osd.1 (osd.1) 485 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:03:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.349767+0000 osd.0 (osd.0) 481 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.349867+0000 osd.0 (osd.0) 482 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.700970+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:03:38.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:38 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.705099+0000 osd.4 (osd.4) 319 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:03:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:39 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.953520+0000 osd.0 (osd.0) 483 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:03:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:39 smithi192 bash[24608]: cluster 2024-09-13T23:03:37.955984+0000 osd.0 (osd.0) 484 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:03:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:39 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.284964+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2024-09-13T23:03:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:39 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.397812+0000 osd.3 (osd.3) 481 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:03:39.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:39 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.403609+0000 osd.3 (osd.3) 482 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:03:39.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.953520+0000 osd.0 (osd.0) 483 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[29889]: cluster 2024-09-13T23:03:37.955984+0000 osd.0 (osd.0) 484 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.284964+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.397812+0000 osd.3 (osd.3) 481 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.403609+0000 osd.3 (osd.3) 482 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.953520+0000 osd.0 (osd.0) 483 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[22840]: cluster 2024-09-13T23:03:37.955984+0000 osd.0 (osd.0) 484 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.284964+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.397812+0000 osd.3 (osd.3) 481 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:03:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.403609+0000 osd.3 (osd.3) 482 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:03:39.893 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:40.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:03:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:03:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:03:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.555323+0000 osd.2 (osd.2) 343 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T23:03:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.561834+0000 osd.2 (osd.2) 344 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T23:03:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.614028+0000 osd.5 (osd.5) 539 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:03:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.616311+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.716253+0000 osd.7 (osd.7) 514 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:38.720842+0000 osd.7 (osd.7) 515 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.069060+0000 mgr.y (mgr.14613) 963 : cluster [DBG] pgmap v1168: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.374877+0000 osd.1 (osd.1) 486 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.377392+0000 osd.1 (osd.1) 487 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.401815+0000 osd.3 (osd.3) 483 : cluster [DBG] 5.0 deep-scrub starts 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.406178+0000 osd.3 (osd.3) 484 : cluster [DBG] 5.0 deep-scrub ok 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.757978+0000 osd.7 (osd.7) 516 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:03:40.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:40 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.760330+0000 osd.7 (osd.7) 517 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.555323+0000 osd.2 (osd.2) 343 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.561834+0000 osd.2 (osd.2) 344 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.614028+0000 osd.5 (osd.5) 539 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.616311+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.716253+0000 osd.7 (osd.7) 514 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:38.720842+0000 osd.7 (osd.7) 515 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.069060+0000 mgr.y (mgr.14613) 963 : cluster [DBG] pgmap v1168: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.374877+0000 osd.1 (osd.1) 486 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.377392+0000 osd.1 (osd.1) 487 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.401815+0000 osd.3 (osd.3) 483 : cluster [DBG] 5.0 deep-scrub starts 2024-09-13T23:03:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.406178+0000 osd.3 (osd.3) 484 : cluster [DBG] 5.0 deep-scrub ok 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.757978+0000 osd.7 (osd.7) 516 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.760330+0000 osd.7 (osd.7) 517 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.555323+0000 osd.2 (osd.2) 343 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.561834+0000 osd.2 (osd.2) 344 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.614028+0000 osd.5 (osd.5) 539 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.616311+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.716253+0000 osd.7 (osd.7) 514 : cluster [DBG] 3.11 deep-scrub starts 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:38.720842+0000 osd.7 (osd.7) 515 : cluster [DBG] 3.11 deep-scrub ok 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.069060+0000 mgr.y (mgr.14613) 963 : cluster [DBG] pgmap v1168: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.374877+0000 osd.1 (osd.1) 486 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:03:40.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.377392+0000 osd.1 (osd.1) 487 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:03:40.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.401815+0000 osd.3 (osd.3) 483 : cluster [DBG] 5.0 deep-scrub starts 2024-09-13T23:03:40.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.406178+0000 osd.3 (osd.3) 484 : cluster [DBG] 5.0 deep-scrub ok 2024-09-13T23:03:40.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.757978+0000 osd.7 (osd.7) 516 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:03:40.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:40 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.760330+0000 osd.7 (osd.7) 517 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:03:41.559 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:41 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.634184+0000 osd.5 (osd.5) 541 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:03:41.559 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:41 smithi178 bash[22840]: cluster 2024-09-13T23:03:39.639720+0000 osd.5 (osd.5) 542 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:03:41.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:41 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.634184+0000 osd.5 (osd.5) 541 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:03:41.559 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:41 smithi178 bash[29889]: cluster 2024-09-13T23:03:39.639720+0000 osd.5 (osd.5) 542 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:03:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:41 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.634184+0000 osd.5 (osd.5) 541 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:03:41.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:41 smithi192 bash[24608]: cluster 2024-09-13T23:03:39.639720+0000 osd.5 (osd.5) 542 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:03:42.306 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T23:03:42.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:42 smithi192 bash[24608]: cluster 2024-09-13T23:03:40.641408+0000 osd.5 (osd.5) 543 : cluster [DBG] 2.2 scrub starts 2024-09-13T23:03:42.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:42 smithi192 bash[24608]: cluster 2024-09-13T23:03:40.643878+0000 osd.5 (osd.5) 544 : cluster [DBG] 2.2 scrub ok 2024-09-13T23:03:42.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:42 smithi192 bash[24608]: cluster 2024-09-13T23:03:41.070374+0000 mgr.y (mgr.14613) 964 : cluster [DBG] pgmap v1169: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:42.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:42 smithi192 bash[24608]: audit 2024-09-13T23:03:41.886112+0000 mon.c (mon.2) 94 : audit [INF] from='client.? v1:172.21.15.178:0/395566204' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:42.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:42 smithi192 bash[24608]: audit 2024-09-13T23:03:41.886511+0000 mon.a (mon.0) 1907 : audit [INF] from='client.26051 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[29889]: cluster 2024-09-13T23:03:40.641408+0000 osd.5 (osd.5) 543 : cluster [DBG] 2.2 scrub starts 2024-09-13T23:03:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[29889]: cluster 2024-09-13T23:03:40.643878+0000 osd.5 (osd.5) 544 : cluster [DBG] 2.2 scrub ok 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[29889]: cluster 2024-09-13T23:03:41.070374+0000 mgr.y (mgr.14613) 964 : cluster [DBG] pgmap v1169: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[29889]: audit 2024-09-13T23:03:41.886112+0000 mon.c (mon.2) 94 : audit [INF] from='client.? v1:172.21.15.178:0/395566204' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[29889]: audit 2024-09-13T23:03:41.886511+0000 mon.a (mon.0) 1907 : audit [INF] from='client.26051 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[22840]: cluster 2024-09-13T23:03:40.641408+0000 osd.5 (osd.5) 543 : cluster [DBG] 2.2 scrub starts 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[22840]: cluster 2024-09-13T23:03:40.643878+0000 osd.5 (osd.5) 544 : cluster [DBG] 2.2 scrub ok 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[22840]: cluster 2024-09-13T23:03:41.070374+0000 mgr.y (mgr.14613) 964 : cluster [DBG] pgmap v1169: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[22840]: audit 2024-09-13T23:03:41.886112+0000 mon.c (mon.2) 94 : audit [INF] from='client.? v1:172.21.15.178:0/395566204' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:42 smithi178 bash[22840]: audit 2024-09-13T23:03:41.886511+0000 mon.a (mon.0) 1907 : audit [INF] from='client.26051 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-13T23:03:43.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T23:03:43.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:43 smithi178 bash[22840]: audit 2024-09-13T23:03:42.300014+0000 mon.a (mon.0) 1908 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:03:43.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:43 smithi178 bash[22840]: cluster 2024-09-13T23:03:42.306543+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2024-09-13T23:03:43.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:43 smithi178 bash[29889]: audit 2024-09-13T23:03:42.300014+0000 mon.a (mon.0) 1908 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:03:43.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:43 smithi178 bash[29889]: cluster 2024-09-13T23:03:42.306543+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2024-09-13T23:03:43.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:43 smithi192 bash[24608]: audit 2024-09-13T23:03:42.300014+0000 mon.a (mon.0) 1908 : audit [INF] from='client.26051 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T23:03:43.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:43 smithi192 bash[24608]: cluster 2024-09-13T23:03:42.306543+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2024-09-13T23:03:44.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:44 smithi192 bash[24608]: cluster 2024-09-13T23:03:43.071186+0000 mgr.y (mgr.14613) 965 : cluster [DBG] pgmap v1171: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:44.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:44 smithi178 bash[29889]: cluster 2024-09-13T23:03:43.071186+0000 mgr.y (mgr.14613) 965 : cluster [DBG] pgmap v1171: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:44.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:44 smithi178 bash[22840]: cluster 2024-09-13T23:03:43.071186+0000 mgr.y (mgr.14613) 965 : cluster [DBG] pgmap v1171: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:03:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:43.429825+0000 osd.5 (osd.5) 545 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:43.429958+0000 osd.5 (osd.5) 546 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:43.512629+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:03:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:43.518268+0000 osd.5 (osd.5) 548 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:03:45.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:44.328640+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2024-09-13T23:03:45.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:44.777580+0000 osd.7 (osd.7) 518 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:03:45.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:45 smithi192 bash[24608]: cluster 2024-09-13T23:03:44.779993+0000 osd.7 (osd.7) 519 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:03:45.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:43.429825+0000 osd.5 (osd.5) 545 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:43.429958+0000 osd.5 (osd.5) 546 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:43.512629+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:43.518268+0000 osd.5 (osd.5) 548 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:44.328640+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:44.777580+0000 osd.7 (osd.7) 518 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[29889]: cluster 2024-09-13T23:03:44.779993+0000 osd.7 (osd.7) 519 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:43.429825+0000 osd.5 (osd.5) 545 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:43.429958+0000 osd.5 (osd.5) 546 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:43.512629+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:43.518268+0000 osd.5 (osd.5) 548 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:03:45.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:44.328640+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2024-09-13T23:03:45.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:44.777580+0000 osd.7 (osd.7) 518 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:03:45.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:45 smithi178 bash[22840]: cluster 2024-09-13T23:03:44.779993+0000 osd.7 (osd.7) 519 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:03:46.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:03:46 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:03:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:46 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.072188+0000 mgr.y (mgr.14613) 966 : cluster [DBG] pgmap v1173: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:03:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:46 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.342297+0000 osd.3 (osd.3) 485 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:03:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:46 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.347075+0000 osd.3 (osd.3) 486 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:03:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:46 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.757472+0000 osd.7 (osd.7) 520 : cluster [DBG] 3.e scrub starts 2024-09-13T23:03:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:46 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.761498+0000 osd.7 (osd.7) 521 : cluster [DBG] 3.e scrub ok 2024-09-13T23:03:46.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.072188+0000 mgr.y (mgr.14613) 966 : cluster [DBG] pgmap v1173: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.342297+0000 osd.3 (osd.3) 485 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.347075+0000 osd.3 (osd.3) 486 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.757472+0000 osd.7 (osd.7) 520 : cluster [DBG] 3.e scrub starts 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.761498+0000 osd.7 (osd.7) 521 : cluster [DBG] 3.e scrub ok 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.072188+0000 mgr.y (mgr.14613) 966 : cluster [DBG] pgmap v1173: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.342297+0000 osd.3 (osd.3) 485 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.347075+0000 osd.3 (osd.3) 486 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.757472+0000 osd.7 (osd.7) 520 : cluster [DBG] 3.e scrub starts 2024-09-13T23:03:46.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:46 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.761498+0000 osd.7 (osd.7) 521 : cluster [DBG] 3.e scrub ok 2024-09-13T23:03:47.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.587044+0000 osd.5 (osd.5) 549 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:03:47.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.589280+0000 osd.5 (osd.5) 550 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:03:47.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.887056+0000 osd.2 (osd.2) 345 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:45.887143+0000 osd.2 (osd.2) 346 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: audit 2024-09-13T23:03:46.190316+0000 mgr.y (mgr.14613) 967 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.349031+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.365098+0000 osd.1 (osd.1) 488 : cluster [DBG] 2.d scrub starts 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.367385+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.d scrub ok 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.450221+0000 osd.6 (osd.6) 372 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.453280+0000 osd.6 (osd.6) 373 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.529288+0000 osd.4 (osd.4) 320 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:03:47.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:47 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.533285+0000 osd.4 (osd.4) 321 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.587044+0000 osd.5 (osd.5) 549 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.589280+0000 osd.5 (osd.5) 550 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.887056+0000 osd.2 (osd.2) 345 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:45.887143+0000 osd.2 (osd.2) 346 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: audit 2024-09-13T23:03:46.190316+0000 mgr.y (mgr.14613) 967 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.349031+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.365098+0000 osd.1 (osd.1) 488 : cluster [DBG] 2.d scrub starts 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.367385+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.d scrub ok 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.450221+0000 osd.6 (osd.6) 372 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:03:47.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.453280+0000 osd.6 (osd.6) 373 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.529288+0000 osd.4 (osd.4) 320 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.533285+0000 osd.4 (osd.4) 321 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.587044+0000 osd.5 (osd.5) 549 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.589280+0000 osd.5 (osd.5) 550 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.887056+0000 osd.2 (osd.2) 345 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:45.887143+0000 osd.2 (osd.2) 346 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: audit 2024-09-13T23:03:46.190316+0000 mgr.y (mgr.14613) 967 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.349031+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.365098+0000 osd.1 (osd.1) 488 : cluster [DBG] 2.d scrub starts 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.367385+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.d scrub ok 2024-09-13T23:03:47.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.450221+0000 osd.6 (osd.6) 372 : cluster [DBG] 1.6 scrub starts 2024-09-13T23:03:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.453280+0000 osd.6 (osd.6) 373 : cluster [DBG] 1.6 scrub ok 2024-09-13T23:03:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.529288+0000 osd.4 (osd.4) 320 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:03:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:47 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.533285+0000 osd.4 (osd.4) 321 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:03:47.981 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:48 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.545585+0000 osd.5 (osd.5) 551 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T23:03:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:48 smithi192 bash[24608]: cluster 2024-09-13T23:03:46.549394+0000 osd.5 (osd.5) 552 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T23:03:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:48 smithi192 bash[24608]: cluster 2024-09-13T23:03:47.073026+0000 mgr.y (mgr.14613) 968 : cluster [DBG] pgmap v1175: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:03:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:48 smithi192 bash[24608]: cluster 2024-09-13T23:03:47.739640+0000 osd.7 (osd.7) 522 : cluster [DBG] 2.b scrub starts 2024-09-13T23:03:48.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:48 smithi192 bash[24608]: cluster 2024-09-13T23:03:47.742051+0000 osd.7 (osd.7) 523 : cluster [DBG] 2.b scrub ok 2024-09-13T23:03:48.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.545585+0000 osd.5 (osd.5) 551 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[29889]: cluster 2024-09-13T23:03:46.549394+0000 osd.5 (osd.5) 552 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[29889]: cluster 2024-09-13T23:03:47.073026+0000 mgr.y (mgr.14613) 968 : cluster [DBG] pgmap v1175: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[29889]: cluster 2024-09-13T23:03:47.739640+0000 osd.7 (osd.7) 522 : cluster [DBG] 2.b scrub starts 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[29889]: cluster 2024-09-13T23:03:47.742051+0000 osd.7 (osd.7) 523 : cluster [DBG] 2.b scrub ok 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.545585+0000 osd.5 (osd.5) 551 : cluster [DBG] 1.1 deep-scrub starts 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[22840]: cluster 2024-09-13T23:03:46.549394+0000 osd.5 (osd.5) 552 : cluster [DBG] 1.1 deep-scrub ok 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[22840]: cluster 2024-09-13T23:03:47.073026+0000 mgr.y (mgr.14613) 968 : cluster [DBG] pgmap v1175: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[22840]: cluster 2024-09-13T23:03:47.739640+0000 osd.7 (osd.7) 522 : cluster [DBG] 2.b scrub starts 2024-09-13T23:03:48.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:48 smithi178 bash[22840]: cluster 2024-09-13T23:03:47.742051+0000 osd.7 (osd.7) 523 : cluster [DBG] 2.b scrub ok 2024-09-13T23:03:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[29889]: cluster 2024-09-13T23:03:47.561833+0000 osd.5 (osd.5) 553 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:03:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[29889]: cluster 2024-09-13T23:03:47.565302+0000 osd.5 (osd.5) 554 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:03:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[29889]: cluster 2024-09-13T23:03:48.722689+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 deep-scrub starts 2024-09-13T23:03:49.623 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[29889]: cluster 2024-09-13T23:03:48.725044+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 deep-scrub ok 2024-09-13T23:03:49.624 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[22840]: cluster 2024-09-13T23:03:47.561833+0000 osd.5 (osd.5) 553 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:03:49.624 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[22840]: cluster 2024-09-13T23:03:47.565302+0000 osd.5 (osd.5) 554 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:03:49.624 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[22840]: cluster 2024-09-13T23:03:48.722689+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 deep-scrub starts 2024-09-13T23:03:49.624 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[22840]: cluster 2024-09-13T23:03:48.725044+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 deep-scrub ok 2024-09-13T23:03:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:49 smithi192 bash[24608]: cluster 2024-09-13T23:03:47.561833+0000 osd.5 (osd.5) 553 : cluster [DBG] 5.15 scrub starts 2024-09-13T23:03:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:49 smithi192 bash[24608]: cluster 2024-09-13T23:03:47.565302+0000 osd.5 (osd.5) 554 : cluster [DBG] 5.15 scrub ok 2024-09-13T23:03:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:49 smithi192 bash[24608]: cluster 2024-09-13T23:03:48.722689+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 deep-scrub starts 2024-09-13T23:03:49.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:49 smithi192 bash[24608]: cluster 2024-09-13T23:03:48.725044+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 deep-scrub ok 2024-09-13T23:03:50.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:03:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:03:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:03:50.373 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T23:03:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:50 smithi192 bash[24608]: cluster 2024-09-13T23:03:49.074018+0000 mgr.y (mgr.14613) 969 : cluster [DBG] pgmap v1176: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 756 B/s rd, 0 op/s 2024-09-13T23:03:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:50 smithi192 bash[24608]: cluster 2024-09-13T23:03:49.407117+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T23:03:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:50 smithi192 bash[24608]: cluster 2024-09-13T23:03:49.410472+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T23:03:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:50 smithi192 bash[24608]: audit 2024-09-13T23:03:49.917312+0000 mon.b (mon.1) 312 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:50.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:50 smithi192 bash[24608]: audit 2024-09-13T23:03:49.940413+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? v1:172.21.15.178:0/1887362384' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[29889]: cluster 2024-09-13T23:03:49.074018+0000 mgr.y (mgr.14613) 969 : cluster [DBG] pgmap v1176: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 756 B/s rd, 0 op/s 2024-09-13T23:03:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[29889]: cluster 2024-09-13T23:03:49.407117+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T23:03:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[29889]: cluster 2024-09-13T23:03:49.410472+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T23:03:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[29889]: audit 2024-09-13T23:03:49.917312+0000 mon.b (mon.1) 312 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:50.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[29889]: audit 2024-09-13T23:03:49.940413+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? v1:172.21.15.178:0/1887362384' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[22840]: cluster 2024-09-13T23:03:49.074018+0000 mgr.y (mgr.14613) 969 : cluster [DBG] pgmap v1176: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 756 B/s rd, 0 op/s 2024-09-13T23:03:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[22840]: cluster 2024-09-13T23:03:49.407117+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T23:03:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[22840]: cluster 2024-09-13T23:03:49.410472+0000 osd.1 (osd.1) 491 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T23:03:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[22840]: audit 2024-09-13T23:03:49.917312+0000 mon.b (mon.1) 312 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:03:50.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:50 smithi178 bash[22840]: audit 2024-09-13T23:03:49.940413+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? v1:172.21.15.178:0/1887362384' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:51.274 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-13T23:03:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:51 smithi192 bash[24608]: audit 2024-09-13T23:03:50.373724+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? v1:172.21.15.178:0/1887362384' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T23:03:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:51 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.380380+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2024-09-13T23:03:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:51 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.417587+0000 osd.7 (osd.7) 526 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:51 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.417696+0000 osd.7 (osd.7) 527 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:51 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.528166+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e deep-scrub starts 2024-09-13T23:03:51.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:51 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.531472+0000 osd.4 (osd.4) 323 : cluster [DBG] 5.e deep-scrub ok 2024-09-13T23:03:51.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[29889]: audit 2024-09-13T23:03:50.373724+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? v1:172.21.15.178:0/1887362384' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.380380+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.417587+0000 osd.7 (osd.7) 526 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.417696+0000 osd.7 (osd.7) 527 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.528166+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e deep-scrub starts 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.531472+0000 osd.4 (osd.4) 323 : cluster [DBG] 5.e deep-scrub ok 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[22840]: audit 2024-09-13T23:03:50.373724+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? v1:172.21.15.178:0/1887362384' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.380380+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.417587+0000 osd.7 (osd.7) 526 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.417696+0000 osd.7 (osd.7) 527 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:03:51.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.528166+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e deep-scrub starts 2024-09-13T23:03:51.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:51 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.531472+0000 osd.4 (osd.4) 323 : cluster [DBG] 5.e deep-scrub ok 2024-09-13T23:03:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:52 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.624189+0000 osd.5 (osd.5) 555 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T23:03:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:52 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.627036+0000 osd.5 (osd.5) 556 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T23:03:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:52 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.920519+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:03:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:52 smithi192 bash[24608]: cluster 2024-09-13T23:03:50.924673+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:03:52.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:52 smithi192 bash[24608]: cluster 2024-09-13T23:03:51.074707+0000 mgr.y (mgr.14613) 970 : cluster [DBG] pgmap v1178: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:03:52.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:52 smithi192 bash[24608]: cluster 2024-09-13T23:03:51.400974+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2024-09-13T23:03:52.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.624189+0000 osd.5 (osd.5) 555 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.627036+0000 osd.5 (osd.5) 556 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.920519+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[29889]: cluster 2024-09-13T23:03:50.924673+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[29889]: cluster 2024-09-13T23:03:51.074707+0000 mgr.y (mgr.14613) 970 : cluster [DBG] pgmap v1178: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[29889]: cluster 2024-09-13T23:03:51.400974+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.624189+0000 osd.5 (osd.5) 555 : cluster [DBG] 2.12 deep-scrub starts 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.627036+0000 osd.5 (osd.5) 556 : cluster [DBG] 2.12 deep-scrub ok 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.920519+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.1f scrub starts 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[22840]: cluster 2024-09-13T23:03:50.924673+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.1f scrub ok 2024-09-13T23:03:52.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[22840]: cluster 2024-09-13T23:03:51.074707+0000 mgr.y (mgr.14613) 970 : cluster [DBG] pgmap v1178: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T23:03:52.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:52 smithi178 bash[22840]: cluster 2024-09-13T23:03:51.400974+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2024-09-13T23:03:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:53 smithi192 bash[24608]: cluster 2024-09-13T23:03:52.520595+0000 osd.4 (osd.4) 324 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T23:03:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:53 smithi192 bash[24608]: cluster 2024-09-13T23:03:52.522860+0000 osd.4 (osd.4) 325 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T23:03:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:53 smithi192 bash[24608]: cluster 2024-09-13T23:03:52.527318+0000 osd.6 (osd.6) 374 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:03:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:53 smithi192 bash[24608]: cluster 2024-09-13T23:03:52.530348+0000 osd.6 (osd.6) 375 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:03:53.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:53 smithi192 bash[24608]: audit 2024-09-13T23:03:52.866128+0000 mon.b (mon.1) 313 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:03:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[22840]: cluster 2024-09-13T23:03:52.520595+0000 osd.4 (osd.4) 324 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T23:03:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[22840]: cluster 2024-09-13T23:03:52.522860+0000 osd.4 (osd.4) 325 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T23:03:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[22840]: cluster 2024-09-13T23:03:52.527318+0000 osd.6 (osd.6) 374 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:03:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[22840]: cluster 2024-09-13T23:03:52.530348+0000 osd.6 (osd.6) 375 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:03:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[22840]: audit 2024-09-13T23:03:52.866128+0000 mon.b (mon.1) 313 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:03:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[29889]: cluster 2024-09-13T23:03:52.520595+0000 osd.4 (osd.4) 324 : cluster [DBG] 5.18 deep-scrub starts 2024-09-13T23:03:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[29889]: cluster 2024-09-13T23:03:52.522860+0000 osd.4 (osd.4) 325 : cluster [DBG] 5.18 deep-scrub ok 2024-09-13T23:03:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[29889]: cluster 2024-09-13T23:03:52.527318+0000 osd.6 (osd.6) 374 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:03:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[29889]: cluster 2024-09-13T23:03:52.530348+0000 osd.6 (osd.6) 375 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:03:53.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:53 smithi178 bash[29889]: audit 2024-09-13T23:03:52.866128+0000 mon.b (mon.1) 313 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:03:54.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:54 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.075499+0000 mgr.y (mgr.14613) 971 : cluster [DBG] pgmap v1180: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-09-13T23:03:54.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:54 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.474102+0000 osd.4 (osd.4) 326 : cluster [DBG] 6.e scrub starts 2024-09-13T23:03:54.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:54 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.475181+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.e scrub ok 2024-09-13T23:03:54.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:54 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.075499+0000 mgr.y (mgr.14613) 971 : cluster [DBG] pgmap v1180: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-09-13T23:03:54.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:54 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.474102+0000 osd.4 (osd.4) 326 : cluster [DBG] 6.e scrub starts 2024-09-13T23:03:54.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:54 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.475181+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.e scrub ok 2024-09-13T23:03:54.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:54 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.075499+0000 mgr.y (mgr.14613) 971 : cluster [DBG] pgmap v1180: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-09-13T23:03:54.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:54 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.474102+0000 osd.4 (osd.4) 326 : cluster [DBG] 6.e scrub starts 2024-09-13T23:03:54.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:54 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.475181+0000 osd.4 (osd.4) 327 : cluster [DBG] 6.e scrub ok 2024-09-13T23:03:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.870727+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T23:03:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.872713+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T23:03:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.887107+0000 osd.0 (osd.0) 487 : cluster [DBG] 5.5 deep-scrub starts 2024-09-13T23:03:55.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:53.889206+0000 osd.0 (osd.0) 488 : cluster [DBG] 5.5 deep-scrub ok 2024-09-13T23:03:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:54.398470+0000 osd.3 (osd.3) 487 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:03:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:54.401167+0000 osd.3 (osd.3) 488 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:03:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:54.796337+0000 osd.7 (osd.7) 528 : cluster [DBG] 3.1d deep-scrub starts 2024-09-13T23:03:55.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[29889]: cluster 2024-09-13T23:03:54.807543+0000 osd.7 (osd.7) 529 : cluster [DBG] 3.1d deep-scrub ok 2024-09-13T23:03:55.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.870727+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T23:03:55.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.872713+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T23:03:55.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.887107+0000 osd.0 (osd.0) 487 : cluster [DBG] 5.5 deep-scrub starts 2024-09-13T23:03:55.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:53.889206+0000 osd.0 (osd.0) 488 : cluster [DBG] 5.5 deep-scrub ok 2024-09-13T23:03:55.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:54.398470+0000 osd.3 (osd.3) 487 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:03:55.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:54.401167+0000 osd.3 (osd.3) 488 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:03:55.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:54.796337+0000 osd.7 (osd.7) 528 : cluster [DBG] 3.1d deep-scrub starts 2024-09-13T23:03:55.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:55 smithi178 bash[22840]: cluster 2024-09-13T23:03:54.807543+0000 osd.7 (osd.7) 529 : cluster [DBG] 3.1d deep-scrub ok 2024-09-13T23:03:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.870727+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.10 deep-scrub starts 2024-09-13T23:03:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.872713+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 deep-scrub ok 2024-09-13T23:03:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.887107+0000 osd.0 (osd.0) 487 : cluster [DBG] 5.5 deep-scrub starts 2024-09-13T23:03:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:53.889206+0000 osd.0 (osd.0) 488 : cluster [DBG] 5.5 deep-scrub ok 2024-09-13T23:03:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:54.398470+0000 osd.3 (osd.3) 487 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:03:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:54.401167+0000 osd.3 (osd.3) 488 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:03:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:54.796337+0000 osd.7 (osd.7) 528 : cluster [DBG] 3.1d deep-scrub starts 2024-09-13T23:03:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:55 smithi192 bash[24608]: cluster 2024-09-13T23:03:54.807543+0000 osd.7 (osd.7) 529 : cluster [DBG] 3.1d deep-scrub ok 2024-09-13T23:03:56.070 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:03:56.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:03:56 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:03:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:56 smithi192 bash[24608]: cluster 2024-09-13T23:03:55.076970+0000 mgr.y (mgr.14613) 972 : cluster [DBG] pgmap v1181: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:56 smithi192 bash[24608]: cluster 2024-09-13T23:03:55.603830+0000 osd.5 (osd.5) 557 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:03:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:56 smithi192 bash[24608]: cluster 2024-09-13T23:03:55.608045+0000 osd.5 (osd.5) 558 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:03:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:56 smithi178 bash[29889]: cluster 2024-09-13T23:03:55.076970+0000 mgr.y (mgr.14613) 972 : cluster [DBG] pgmap v1181: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:56 smithi178 bash[29889]: cluster 2024-09-13T23:03:55.603830+0000 osd.5 (osd.5) 557 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:03:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:56 smithi178 bash[29889]: cluster 2024-09-13T23:03:55.608045+0000 osd.5 (osd.5) 558 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:03:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:56 smithi178 bash[22840]: cluster 2024-09-13T23:03:55.076970+0000 mgr.y (mgr.14613) 972 : cluster [DBG] pgmap v1181: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:56 smithi178 bash[22840]: cluster 2024-09-13T23:03:55.603830+0000 osd.5 (osd.5) 557 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:03:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:56 smithi178 bash[22840]: cluster 2024-09-13T23:03:55.608045+0000 osd.5 (osd.5) 558 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:03:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:57 smithi178 bash[29889]: audit 2024-09-13T23:03:56.201553+0000 mgr.y (mgr.14613) 973 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:57 smithi178 bash[29889]: cluster 2024-09-13T23:03:56.758353+0000 osd.7 (osd.7) 530 : cluster [DBG] 1.9 deep-scrub starts 2024-09-13T23:03:57.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:57 smithi178 bash[29889]: cluster 2024-09-13T23:03:56.761021+0000 osd.7 (osd.7) 531 : cluster [DBG] 1.9 deep-scrub ok 2024-09-13T23:03:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:57 smithi178 bash[22840]: audit 2024-09-13T23:03:56.201553+0000 mgr.y (mgr.14613) 973 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:57 smithi178 bash[22840]: cluster 2024-09-13T23:03:56.758353+0000 osd.7 (osd.7) 530 : cluster [DBG] 1.9 deep-scrub starts 2024-09-13T23:03:57.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:57 smithi178 bash[22840]: cluster 2024-09-13T23:03:56.761021+0000 osd.7 (osd.7) 531 : cluster [DBG] 1.9 deep-scrub ok 2024-09-13T23:03:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:57 smithi192 bash[24608]: audit 2024-09-13T23:03:56.201553+0000 mgr.y (mgr.14613) 973 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:03:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:57 smithi192 bash[24608]: cluster 2024-09-13T23:03:56.758353+0000 osd.7 (osd.7) 530 : cluster [DBG] 1.9 deep-scrub starts 2024-09-13T23:03:57.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:57 smithi192 bash[24608]: cluster 2024-09-13T23:03:56.761021+0000 osd.7 (osd.7) 531 : cluster [DBG] 1.9 deep-scrub ok 2024-09-13T23:03:58.432 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T23:03:58.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: cluster 2024-09-13T23:03:57.077808+0000 mgr.y (mgr.14613) 974 : cluster [DBG] pgmap v1182: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: cluster 2024-09-13T23:03:57.314483+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: cluster 2024-09-13T23:03:57.316883+0000 osd.1 (osd.1) 493 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: cluster 2024-09-13T23:03:57.573938+0000 osd.5 (osd.5) 559 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: cluster 2024-09-13T23:03:57.577550+0000 osd.5 (osd.5) 560 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: audit 2024-09-13T23:03:57.926540+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.178:0/4249634017' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[22840]: audit 2024-09-13T23:03:57.927192+0000 mon.a (mon.0) 1916 : audit [INF] from='client.26063 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: cluster 2024-09-13T23:03:57.077808+0000 mgr.y (mgr.14613) 974 : cluster [DBG] pgmap v1182: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:58.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: cluster 2024-09-13T23:03:57.314483+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:03:58.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: cluster 2024-09-13T23:03:57.316883+0000 osd.1 (osd.1) 493 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:03:58.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: cluster 2024-09-13T23:03:57.573938+0000 osd.5 (osd.5) 559 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:03:58.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: cluster 2024-09-13T23:03:57.577550+0000 osd.5 (osd.5) 560 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:03:58.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: audit 2024-09-13T23:03:57.926540+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.178:0/4249634017' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:58.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:58 smithi178 bash[29889]: audit 2024-09-13T23:03:57.927192+0000 mon.a (mon.0) 1916 : audit [INF] from='client.26063 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: cluster 2024-09-13T23:03:57.077808+0000 mgr.y (mgr.14613) 974 : cluster [DBG] pgmap v1182: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:03:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: cluster 2024-09-13T23:03:57.314483+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:03:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: cluster 2024-09-13T23:03:57.316883+0000 osd.1 (osd.1) 493 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:03:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: cluster 2024-09-13T23:03:57.573938+0000 osd.5 (osd.5) 559 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:03:58.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: cluster 2024-09-13T23:03:57.577550+0000 osd.5 (osd.5) 560 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:03:58.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: audit 2024-09-13T23:03:57.926540+0000 mon.c (mon.2) 95 : audit [INF] from='client.? v1:172.21.15.178:0/4249634017' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:58.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:58 smithi192 bash[24608]: audit 2024-09-13T23:03:57.927192+0000 mon.a (mon.0) 1916 : audit [INF] from='client.26063 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-13T23:03:59.810 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[22840]: audit 2024-09-13T23:03:58.425189+0000 mon.a (mon.0) 1917 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.432408+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.529468+0000 osd.5 (osd.5) 561 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.531892+0000 osd.5 (osd.5) 562 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[29889]: audit 2024-09-13T23:03:58.425189+0000 mon.a (mon.0) 1917 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.432408+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.529468+0000 osd.5 (osd.5) 561 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:03:59.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.531892+0000 osd.5 (osd.5) 562 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:03:59.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T23:03:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:59 smithi192 bash[24608]: audit 2024-09-13T23:03:58.425189+0000 mon.a (mon.0) 1917 : audit [INF] from='client.26063 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T23:03:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:59 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.432408+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2024-09-13T23:03:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:59 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.529468+0000 osd.5 (osd.5) 561 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:03:59.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:03:59 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.531892+0000 osd.5 (osd.5) 562 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:04:00.075 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:03:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:03:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:04:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.563501+0000 osd.6 (osd.6) 376 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:04:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.566219+0000 osd.6 (osd.6) 377 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:04:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.939288+0000 osd.0 (osd.0) 489 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:04:00.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:58.941647+0000 osd.0 (osd.0) 490 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:04:00.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:59.078745+0000 mgr.y (mgr.14613) 975 : cluster [DBG] pgmap v1184: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:00.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:59.472931+0000 osd.3 (osd.3) 489 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:04:00.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:59.475962+0000 osd.3 (osd.3) 490 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:04:00.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:59.540821+0000 osd.5 (osd.5) 563 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:04:00.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:00 smithi192 bash[24608]: cluster 2024-09-13T23:03:59.543833+0000 osd.5 (osd.5) 564 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:04:00.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.563501+0000 osd.6 (osd.6) 376 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.566219+0000 osd.6 (osd.6) 377 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.939288+0000 osd.0 (osd.0) 489 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:58.941647+0000 osd.0 (osd.0) 490 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:59.078745+0000 mgr.y (mgr.14613) 975 : cluster [DBG] pgmap v1184: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:59.472931+0000 osd.3 (osd.3) 489 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:59.475962+0000 osd.3 (osd.3) 490 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:59.540821+0000 osd.5 (osd.5) 563 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[29889]: cluster 2024-09-13T23:03:59.543833+0000 osd.5 (osd.5) 564 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.563501+0000 osd.6 (osd.6) 376 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:04:00.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.566219+0000 osd.6 (osd.6) 377 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.939288+0000 osd.0 (osd.0) 489 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:58.941647+0000 osd.0 (osd.0) 490 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:59.078745+0000 mgr.y (mgr.14613) 975 : cluster [DBG] pgmap v1184: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:59.472931+0000 osd.3 (osd.3) 489 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:59.475962+0000 osd.3 (osd.3) 490 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:59.540821+0000 osd.5 (osd.5) 563 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:04:00.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:00 smithi178 bash[22840]: cluster 2024-09-13T23:03:59.543833+0000 osd.5 (osd.5) 564 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:04:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:01 smithi178 bash[29889]: cluster 2024-09-13T23:04:00.758470+0000 osd.7 (osd.7) 532 : cluster [DBG] 3.d scrub starts 2024-09-13T23:04:01.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:01 smithi178 bash[29889]: cluster 2024-09-13T23:04:00.763481+0000 osd.7 (osd.7) 533 : cluster [DBG] 3.d scrub ok 2024-09-13T23:04:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:01 smithi178 bash[22840]: cluster 2024-09-13T23:04:00.758470+0000 osd.7 (osd.7) 532 : cluster [DBG] 3.d scrub starts 2024-09-13T23:04:01.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:01 smithi178 bash[22840]: cluster 2024-09-13T23:04:00.763481+0000 osd.7 (osd.7) 533 : cluster [DBG] 3.d scrub ok 2024-09-13T23:04:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:01 smithi192 bash[24608]: cluster 2024-09-13T23:04:00.758470+0000 osd.7 (osd.7) 532 : cluster [DBG] 3.d scrub starts 2024-09-13T23:04:01.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:01 smithi192 bash[24608]: cluster 2024-09-13T23:04:00.763481+0000 osd.7 (osd.7) 533 : cluster [DBG] 3.d scrub ok 2024-09-13T23:04:02.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:02 smithi192 bash[24608]: cluster 2024-09-13T23:04:01.080159+0000 mgr.y (mgr.14613) 976 : cluster [DBG] pgmap v1185: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:02.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:02 smithi178 bash[22840]: cluster 2024-09-13T23:04:01.080159+0000 mgr.y (mgr.14613) 976 : cluster [DBG] pgmap v1185: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:02.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:02 smithi178 bash[29889]: cluster 2024-09-13T23:04:01.080159+0000 mgr.y (mgr.14613) 976 : cluster [DBG] pgmap v1185: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:01.581660+0000 osd.6 (osd.6) 378 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:04:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:01.584609+0000 osd.6 (osd.6) 379 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:04:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:01.970792+0000 osd.2 (osd.2) 349 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:04:03.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:01.973151+0000 osd.2 (osd.2) 350 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:04:03.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:02.523305+0000 osd.3 (osd.3) 491 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:04:03.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:02.525879+0000 osd.3 (osd.3) 492 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:04:03.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:02.540300+0000 osd.5 (osd.5) 565 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:04:03.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:03 smithi192 bash[24608]: cluster 2024-09-13T23:04:02.542455+0000 osd.5 (osd.5) 566 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:04:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:01.581660+0000 osd.6 (osd.6) 378 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:04:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:01.584609+0000 osd.6 (osd.6) 379 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:04:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:01.970792+0000 osd.2 (osd.2) 349 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:04:03.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:01.973151+0000 osd.2 (osd.2) 350 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:02.523305+0000 osd.3 (osd.3) 491 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:02.525879+0000 osd.3 (osd.3) 492 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:02.540300+0000 osd.5 (osd.5) 565 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[29889]: cluster 2024-09-13T23:04:02.542455+0000 osd.5 (osd.5) 566 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:01.581660+0000 osd.6 (osd.6) 378 : cluster [DBG] 1.7 scrub starts 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:01.584609+0000 osd.6 (osd.6) 379 : cluster [DBG] 1.7 scrub ok 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:01.970792+0000 osd.2 (osd.2) 349 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:01.973151+0000 osd.2 (osd.2) 350 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:02.523305+0000 osd.3 (osd.3) 491 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:02.525879+0000 osd.3 (osd.3) 492 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:04:03.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:02.540300+0000 osd.5 (osd.5) 565 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:04:03.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:03 smithi178 bash[22840]: cluster 2024-09-13T23:04:02.542455+0000 osd.5 (osd.5) 566 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:04:04.655 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:04.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:04 smithi192 bash[24608]: cluster 2024-09-13T23:04:02.918304+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:04:04.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:04 smithi192 bash[24608]: cluster 2024-09-13T23:04:02.920719+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:04:04.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:04 smithi192 bash[24608]: cluster 2024-09-13T23:04:03.081039+0000 mgr.y (mgr.14613) 977 : cluster [DBG] pgmap v1186: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:04 smithi178 bash[29889]: cluster 2024-09-13T23:04:02.918304+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:04:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:04 smithi178 bash[29889]: cluster 2024-09-13T23:04:02.920719+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:04:04.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:04 smithi178 bash[29889]: cluster 2024-09-13T23:04:03.081039+0000 mgr.y (mgr.14613) 977 : cluster [DBG] pgmap v1186: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:04 smithi178 bash[22840]: cluster 2024-09-13T23:04:02.918304+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:04:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:04 smithi178 bash[22840]: cluster 2024-09-13T23:04:02.920719+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:04:04.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:04 smithi178 bash[22840]: cluster 2024-09-13T23:04:03.081039+0000 mgr.y (mgr.14613) 977 : cluster [DBG] pgmap v1186: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: cluster 2024-09-13T23:04:03.563027+0000 osd.6 (osd.6) 380 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:04:05.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: cluster 2024-09-13T23:04:03.566147+0000 osd.6 (osd.6) 381 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: cluster 2024-09-13T23:04:04.789650+0000 osd.7 (osd.7) 534 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: cluster 2024-09-13T23:04:04.792211+0000 osd.7 (osd.7) 535 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: audit 2024-09-13T23:04:04.943254+0000 mon.b (mon.1) 314 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: audit 2024-09-13T23:04:05.162034+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: audit 2024-09-13T23:04:05.174434+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: audit 2024-09-13T23:04:05.177917+0000 mon.b (mon.1) 315 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: audit 2024-09-13T23:04:05.179858+0000 mon.b (mon.1) 316 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:04:05.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:05 smithi192 bash[24608]: audit 2024-09-13T23:04:05.190760+0000 mon.a (mon.0) 1921 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: cluster 2024-09-13T23:04:03.563027+0000 osd.6 (osd.6) 380 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: cluster 2024-09-13T23:04:03.566147+0000 osd.6 (osd.6) 381 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: cluster 2024-09-13T23:04:04.789650+0000 osd.7 (osd.7) 534 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: cluster 2024-09-13T23:04:04.792211+0000 osd.7 (osd.7) 535 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: audit 2024-09-13T23:04:04.943254+0000 mon.b (mon.1) 314 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: audit 2024-09-13T23:04:05.162034+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: audit 2024-09-13T23:04:05.174434+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: audit 2024-09-13T23:04:05.177917+0000 mon.b (mon.1) 315 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:04:05.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: audit 2024-09-13T23:04:05.179858+0000 mon.b (mon.1) 316 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[22840]: audit 2024-09-13T23:04:05.190760+0000 mon.a (mon.0) 1921 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: cluster 2024-09-13T23:04:03.563027+0000 osd.6 (osd.6) 380 : cluster [DBG] 3.10 scrub starts 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: cluster 2024-09-13T23:04:03.566147+0000 osd.6 (osd.6) 381 : cluster [DBG] 3.10 scrub ok 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: cluster 2024-09-13T23:04:04.789650+0000 osd.7 (osd.7) 534 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: cluster 2024-09-13T23:04:04.792211+0000 osd.7 (osd.7) 535 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: audit 2024-09-13T23:04:04.943254+0000 mon.b (mon.1) 314 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: audit 2024-09-13T23:04:05.162034+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: audit 2024-09-13T23:04:05.174434+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: audit 2024-09-13T23:04:05.177917+0000 mon.b (mon.1) 315 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: audit 2024-09-13T23:04:05.179858+0000 mon.b (mon.1) 316 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:04:05.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:05 smithi178 bash[29889]: audit 2024-09-13T23:04:05.190760+0000 mon.a (mon.0) 1921 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:04:06.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:04:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[29889]: cluster 2024-09-13T23:04:05.082534+0000 mgr.y (mgr.14613) 978 : cluster [DBG] pgmap v1187: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[29889]: cluster 2024-09-13T23:04:05.281761+0000 osd.4 (osd.4) 328 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[29889]: cluster 2024-09-13T23:04:05.281879+0000 osd.4 (osd.4) 329 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:06.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[29889]: cluster 2024-09-13T23:04:05.327761+0000 osd.1 (osd.1) 494 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[29889]: cluster 2024-09-13T23:04:05.330996+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.195517+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[22840]: cluster 2024-09-13T23:04:05.082534+0000 mgr.y (mgr.14613) 978 : cluster [DBG] pgmap v1187: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[22840]: cluster 2024-09-13T23:04:05.281761+0000 osd.4 (osd.4) 328 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[22840]: cluster 2024-09-13T23:04:05.281879+0000 osd.4 (osd.4) 329 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[22840]: cluster 2024-09-13T23:04:05.327761+0000 osd.1 (osd.1) 494 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[22840]: cluster 2024-09-13T23:04:05.330996+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:04:06.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:06 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.195517+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2024-09-13T23:04:06.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[24608]: cluster 2024-09-13T23:04:05.082534+0000 mgr.y (mgr.14613) 978 : cluster [DBG] pgmap v1187: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:06.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[24608]: cluster 2024-09-13T23:04:05.281761+0000 osd.4 (osd.4) 328 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:06.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[24608]: cluster 2024-09-13T23:04:05.281879+0000 osd.4 (osd.4) 329 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:06.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[24608]: cluster 2024-09-13T23:04:05.327761+0000 osd.1 (osd.1) 494 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:04:06.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[24608]: cluster 2024-09-13T23:04:05.330996+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:04:06.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:06 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.195517+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2024-09-13T23:04:07.487 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T23:04:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: audit 2024-09-13T23:04:06.210282+0000 mgr.y (mgr.14613) 979 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.359498+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:04:07.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.361958+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:04:07.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.545820+0000 osd.5 (osd.5) 567 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:04:07.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.547808+0000 osd.5 (osd.5) 568 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:04:07.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: audit 2024-09-13T23:04:06.701351+0000 mon.c (mon.2) 96 : audit [INF] from='client.? v1:172.21.15.178:0/3817669982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:07.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: audit 2024-09-13T23:04:06.701889+0000 mon.a (mon.0) 1923 : audit [INF] from='client.26069 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:07.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.858373+0000 osd.7 (osd.7) 536 : cluster [DBG] 6.3 deep-scrub starts 2024-09-13T23:04:07.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:07 smithi192 bash[24608]: cluster 2024-09-13T23:04:06.861048+0000 osd.7 (osd.7) 537 : cluster [DBG] 6.3 deep-scrub ok 2024-09-13T23:04:07.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: audit 2024-09-13T23:04:06.210282+0000 mgr.y (mgr.14613) 979 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:07.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.359498+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:04:07.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.361958+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:04:07.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.545820+0000 osd.5 (osd.5) 567 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.547808+0000 osd.5 (osd.5) 568 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: audit 2024-09-13T23:04:06.701351+0000 mon.c (mon.2) 96 : audit [INF] from='client.? v1:172.21.15.178:0/3817669982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: audit 2024-09-13T23:04:06.701889+0000 mon.a (mon.0) 1923 : audit [INF] from='client.26069 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.858373+0000 osd.7 (osd.7) 536 : cluster [DBG] 6.3 deep-scrub starts 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[22840]: cluster 2024-09-13T23:04:06.861048+0000 osd.7 (osd.7) 537 : cluster [DBG] 6.3 deep-scrub ok 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: audit 2024-09-13T23:04:06.210282+0000 mgr.y (mgr.14613) 979 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.359498+0000 osd.1 (osd.1) 496 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.361958+0000 osd.1 (osd.1) 497 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.545820+0000 osd.5 (osd.5) 567 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.547808+0000 osd.5 (osd.5) 568 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: audit 2024-09-13T23:04:06.701351+0000 mon.c (mon.2) 96 : audit [INF] from='client.? v1:172.21.15.178:0/3817669982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:07.933 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: audit 2024-09-13T23:04:06.701889+0000 mon.a (mon.0) 1923 : audit [INF] from='client.26069 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:07.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.858373+0000 osd.7 (osd.7) 536 : cluster [DBG] 6.3 deep-scrub starts 2024-09-13T23:04:07.934 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:07 smithi178 bash[29889]: cluster 2024-09-13T23:04:06.861048+0000 osd.7 (osd.7) 537 : cluster [DBG] 6.3 deep-scrub ok 2024-09-13T23:04:08.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T23:04:08.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.083449+0000 mgr.y (mgr.14613) 980 : cluster [DBG] pgmap v1189: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.473196+0000 osd.3 (osd.3) 493 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.475554+0000 osd.3 (osd.3) 494 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: audit 2024-09-13T23:04:07.481522+0000 mon.a (mon.0) 1924 : audit [INF] from='client.26069 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.488154+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.531135+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.533434+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.083449+0000 mgr.y (mgr.14613) 980 : cluster [DBG] pgmap v1189: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.473196+0000 osd.3 (osd.3) 493 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.475554+0000 osd.3 (osd.3) 494 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:04:08.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: audit 2024-09-13T23:04:07.481522+0000 mon.a (mon.0) 1924 : audit [INF] from='client.26069 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:04:08.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.488154+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2024-09-13T23:04:08.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.531135+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:04:08.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:08 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.533434+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:04:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.083449+0000 mgr.y (mgr.14613) 980 : cluster [DBG] pgmap v1189: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.473196+0000 osd.3 (osd.3) 493 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:04:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.475554+0000 osd.3 (osd.3) 494 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:04:08.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: audit 2024-09-13T23:04:07.481522+0000 mon.a (mon.0) 1924 : audit [INF] from='client.26069 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:04:08.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.488154+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2024-09-13T23:04:08.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.531135+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:04:08.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:08 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.533434+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:04:09.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.902781+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.b scrub starts 2024-09-13T23:04:09.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[29889]: cluster 2024-09-13T23:04:07.905349+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.b scrub ok 2024-09-13T23:04:09.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[29889]: cluster 2024-09-13T23:04:08.828840+0000 osd.7 (osd.7) 538 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:04:09.810 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[29889]: cluster 2024-09-13T23:04:08.831991+0000 osd.7 (osd.7) 539 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:04:09.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.902781+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.b scrub starts 2024-09-13T23:04:09.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[22840]: cluster 2024-09-13T23:04:07.905349+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.b scrub ok 2024-09-13T23:04:09.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[22840]: cluster 2024-09-13T23:04:08.828840+0000 osd.7 (osd.7) 538 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:04:09.811 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[22840]: cluster 2024-09-13T23:04:08.831991+0000 osd.7 (osd.7) 539 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:04:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:09 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.902781+0000 osd.2 (osd.2) 351 : cluster [DBG] 5.b scrub starts 2024-09-13T23:04:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:09 smithi192 bash[24608]: cluster 2024-09-13T23:04:07.905349+0000 osd.2 (osd.2) 352 : cluster [DBG] 5.b scrub ok 2024-09-13T23:04:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:09 smithi192 bash[24608]: cluster 2024-09-13T23:04:08.828840+0000 osd.7 (osd.7) 538 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:04:09.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:09 smithi192 bash[24608]: cluster 2024-09-13T23:04:08.831991+0000 osd.7 (osd.7) 539 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:04:10.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:04:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:04:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:04:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:10 smithi192 bash[24608]: cluster 2024-09-13T23:04:09.084405+0000 mgr.y (mgr.14613) 981 : cluster [DBG] pgmap v1191: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:10 smithi192 bash[24608]: cluster 2024-09-13T23:04:09.315384+0000 osd.1 (osd.1) 498 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:04:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:10 smithi192 bash[24608]: cluster 2024-09-13T23:04:09.318033+0000 osd.1 (osd.1) 499 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:04:10.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:10 smithi192 bash[24608]: cluster 2024-09-13T23:04:09.532463+0000 osd.5 (osd.5) 569 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:04:10.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:10 smithi192 bash[24608]: cluster 2024-09-13T23:04:09.535042+0000 osd.5 (osd.5) 570 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:04:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[29889]: cluster 2024-09-13T23:04:09.084405+0000 mgr.y (mgr.14613) 981 : cluster [DBG] pgmap v1191: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[29889]: cluster 2024-09-13T23:04:09.315384+0000 osd.1 (osd.1) 498 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:04:10.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[29889]: cluster 2024-09-13T23:04:09.318033+0000 osd.1 (osd.1) 499 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[29889]: cluster 2024-09-13T23:04:09.532463+0000 osd.5 (osd.5) 569 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[29889]: cluster 2024-09-13T23:04:09.535042+0000 osd.5 (osd.5) 570 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[22840]: cluster 2024-09-13T23:04:09.084405+0000 mgr.y (mgr.14613) 981 : cluster [DBG] pgmap v1191: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[22840]: cluster 2024-09-13T23:04:09.315384+0000 osd.1 (osd.1) 498 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[22840]: cluster 2024-09-13T23:04:09.318033+0000 osd.1 (osd.1) 499 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[22840]: cluster 2024-09-13T23:04:09.532463+0000 osd.5 (osd.5) 569 : cluster [DBG] 5.7 scrub starts 2024-09-13T23:04:10.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:10 smithi178 bash[22840]: cluster 2024-09-13T23:04:09.535042+0000 osd.5 (osd.5) 570 : cluster [DBG] 5.7 scrub ok 2024-09-13T23:04:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:11 smithi192 bash[24608]: cluster 2024-09-13T23:04:10.755464+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:04:11.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:11 smithi192 bash[24608]: cluster 2024-09-13T23:04:10.758499+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:04:11.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:11 smithi178 bash[29889]: cluster 2024-09-13T23:04:10.755464+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:04:11.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:11 smithi178 bash[29889]: cluster 2024-09-13T23:04:10.758499+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:04:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:11 smithi178 bash[22840]: cluster 2024-09-13T23:04:10.755464+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.1c scrub starts 2024-09-13T23:04:11.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:11 smithi178 bash[22840]: cluster 2024-09-13T23:04:10.758499+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.1c scrub ok 2024-09-13T23:04:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:12 smithi192 bash[24608]: cluster 2024-09-13T23:04:10.844132+0000 osd.0 (osd.0) 493 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:04:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:12 smithi192 bash[24608]: cluster 2024-09-13T23:04:10.848083+0000 osd.0 (osd.0) 494 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:04:12.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:12 smithi192 bash[24608]: cluster 2024-09-13T23:04:11.085677+0000 mgr.y (mgr.14613) 982 : cluster [DBG] pgmap v1192: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:12.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:12 smithi178 bash[22840]: cluster 2024-09-13T23:04:10.844132+0000 osd.0 (osd.0) 493 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:04:12.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:12 smithi178 bash[22840]: cluster 2024-09-13T23:04:10.848083+0000 osd.0 (osd.0) 494 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:04:12.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:12 smithi178 bash[22840]: cluster 2024-09-13T23:04:11.085677+0000 mgr.y (mgr.14613) 982 : cluster [DBG] pgmap v1192: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:12.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:12 smithi178 bash[29889]: cluster 2024-09-13T23:04:10.844132+0000 osd.0 (osd.0) 493 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:04:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:12 smithi178 bash[29889]: cluster 2024-09-13T23:04:10.848083+0000 osd.0 (osd.0) 494 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:04:12.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:12 smithi178 bash[29889]: cluster 2024-09-13T23:04:11.085677+0000 mgr.y (mgr.14613) 982 : cluster [DBG] pgmap v1192: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:13.135 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[22840]: cluster 2024-09-13T23:04:11.956166+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[22840]: cluster 2024-09-13T23:04:11.959219+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[22840]: cluster 2024-09-13T23:04:12.337976+0000 osd.1 (osd.1) 500 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[22840]: cluster 2024-09-13T23:04:12.340427+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[22840]: cluster 2024-09-13T23:04:12.611051+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.c scrub starts 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[22840]: cluster 2024-09-13T23:04:12.613998+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.c scrub ok 2024-09-13T23:04:13.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[29889]: cluster 2024-09-13T23:04:11.956166+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:04:13.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[29889]: cluster 2024-09-13T23:04:11.959219+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:04:13.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[29889]: cluster 2024-09-13T23:04:12.337976+0000 osd.1 (osd.1) 500 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:04:13.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[29889]: cluster 2024-09-13T23:04:12.340427+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:04:13.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[29889]: cluster 2024-09-13T23:04:12.611051+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.c scrub starts 2024-09-13T23:04:13.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:13 smithi178 bash[29889]: cluster 2024-09-13T23:04:12.613998+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.c scrub ok 2024-09-13T23:04:13.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:13 smithi192 bash[24608]: cluster 2024-09-13T23:04:11.956166+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:04:13.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:13 smithi192 bash[24608]: cluster 2024-09-13T23:04:11.959219+0000 osd.2 (osd.2) 354 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:04:13.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:13 smithi192 bash[24608]: cluster 2024-09-13T23:04:12.337976+0000 osd.1 (osd.1) 500 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:04:13.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:13 smithi192 bash[24608]: cluster 2024-09-13T23:04:12.340427+0000 osd.1 (osd.1) 501 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:04:13.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:13 smithi192 bash[24608]: cluster 2024-09-13T23:04:12.611051+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.c scrub starts 2024-09-13T23:04:13.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:13 smithi192 bash[24608]: cluster 2024-09-13T23:04:12.613998+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.c scrub ok 2024-09-13T23:04:14.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:14 smithi192 bash[24608]: cluster 2024-09-13T23:04:13.086521+0000 mgr.y (mgr.14613) 983 : cluster [DBG] pgmap v1193: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:14.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:14 smithi178 bash[29889]: cluster 2024-09-13T23:04:13.086521+0000 mgr.y (mgr.14613) 983 : cluster [DBG] pgmap v1193: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:14.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:14 smithi178 bash[22840]: cluster 2024-09-13T23:04:13.086521+0000 mgr.y (mgr.14613) 983 : cluster [DBG] pgmap v1193: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:15.539 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T23:04:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:15 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.424960+0000 osd.5 (osd.5) 571 : cluster [DBG] 3.c deep-scrub starts 2024-09-13T23:04:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:15 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.434649+0000 osd.5 (osd.5) 572 : cluster [DBG] 3.c deep-scrub ok 2024-09-13T23:04:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:15 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.725341+0000 osd.7 (osd.7) 542 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:04:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:15 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.727475+0000 osd.7 (osd.7) 543 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:04:15.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:15 smithi192 bash[24608]: audit 2024-09-13T23:04:15.064910+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? v1:172.21.15.178:0/3138357166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:15.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.424960+0000 osd.5 (osd.5) 571 : cluster [DBG] 3.c deep-scrub starts 2024-09-13T23:04:15.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.434649+0000 osd.5 (osd.5) 572 : cluster [DBG] 3.c deep-scrub ok 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.725341+0000 osd.7 (osd.7) 542 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.727475+0000 osd.7 (osd.7) 543 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[22840]: audit 2024-09-13T23:04:15.064910+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? v1:172.21.15.178:0/3138357166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.424960+0000 osd.5 (osd.5) 571 : cluster [DBG] 3.c deep-scrub starts 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.434649+0000 osd.5 (osd.5) 572 : cluster [DBG] 3.c deep-scrub ok 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.725341+0000 osd.7 (osd.7) 542 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.727475+0000 osd.7 (osd.7) 543 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:04:15.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:15 smithi178 bash[29889]: audit 2024-09-13T23:04:15.064910+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? v1:172.21.15.178:0/3138357166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-13T23:04:16.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T23:04:16.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:04:16.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.877087+0000 osd.0 (osd.0) 495 : cluster [DBG] 2.13 deep-scrub starts 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.878566+0000 osd.0 (osd.0) 496 : cluster [DBG] 2.13 deep-scrub ok 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.937288+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:14.939076+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.087998+0000 mgr.y (mgr.14613) 984 : cluster [DBG] pgmap v1194: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.366970+0000 osd.1 (osd.1) 502 : cluster [DBG] 2.9 deep-scrub starts 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.369383+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.9 deep-scrub ok 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: audit 2024-09-13T23:04:15.539807+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? v1:172.21.15.178:0/3138357166' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.544427+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.552241+0000 osd.4 (osd.4) 332 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:04:16.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.556950+0000 osd.4 (osd.4) 333 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.701074+0000 osd.7 (osd.7) 544 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[29889]: cluster 2024-09-13T23:04:15.703323+0000 osd.7 (osd.7) 545 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.877087+0000 osd.0 (osd.0) 495 : cluster [DBG] 2.13 deep-scrub starts 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.878566+0000 osd.0 (osd.0) 496 : cluster [DBG] 2.13 deep-scrub ok 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.937288+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:14.939076+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.087998+0000 mgr.y (mgr.14613) 984 : cluster [DBG] pgmap v1194: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.366970+0000 osd.1 (osd.1) 502 : cluster [DBG] 2.9 deep-scrub starts 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.369383+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.9 deep-scrub ok 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: audit 2024-09-13T23:04:15.539807+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? v1:172.21.15.178:0/3138357166' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:04:16.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.544427+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2024-09-13T23:04:16.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.552241+0000 osd.4 (osd.4) 332 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:04:16.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.556950+0000 osd.4 (osd.4) 333 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:04:16.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.701074+0000 osd.7 (osd.7) 544 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:04:16.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:16 smithi178 bash[22840]: cluster 2024-09-13T23:04:15.703323+0000 osd.7 (osd.7) 545 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:04:16.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.877087+0000 osd.0 (osd.0) 495 : cluster [DBG] 2.13 deep-scrub starts 2024-09-13T23:04:16.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.878566+0000 osd.0 (osd.0) 496 : cluster [DBG] 2.13 deep-scrub ok 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.937288+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.8 deep-scrub starts 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:14.939076+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.8 deep-scrub ok 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.087998+0000 mgr.y (mgr.14613) 984 : cluster [DBG] pgmap v1194: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.366970+0000 osd.1 (osd.1) 502 : cluster [DBG] 2.9 deep-scrub starts 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.369383+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.9 deep-scrub ok 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: audit 2024-09-13T23:04:15.539807+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? v1:172.21.15.178:0/3138357166' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.544427+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.552241+0000 osd.4 (osd.4) 332 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.556950+0000 osd.4 (osd.4) 333 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.701074+0000 osd.7 (osd.7) 544 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:04:16.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:16 smithi192 bash[24608]: cluster 2024-09-13T23:04:15.703323+0000 osd.7 (osd.7) 545 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:04:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:17 smithi192 bash[24608]: audit 2024-09-13T23:04:16.221555+0000 mgr.y (mgr.14613) 985 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:17 smithi192 bash[24608]: cluster 2024-09-13T23:04:16.500774+0000 osd.5 (osd.5) 573 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:04:17.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:17 smithi192 bash[24608]: cluster 2024-09-13T23:04:16.503144+0000 osd.5 (osd.5) 574 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:04:17.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:17 smithi178 bash[29889]: audit 2024-09-13T23:04:16.221555+0000 mgr.y (mgr.14613) 985 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:17.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:17 smithi178 bash[29889]: cluster 2024-09-13T23:04:16.500774+0000 osd.5 (osd.5) 573 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:04:17.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:17 smithi178 bash[29889]: cluster 2024-09-13T23:04:16.503144+0000 osd.5 (osd.5) 574 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:04:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:17 smithi178 bash[22840]: audit 2024-09-13T23:04:16.221555+0000 mgr.y (mgr.14613) 985 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:17 smithi178 bash[22840]: cluster 2024-09-13T23:04:16.500774+0000 osd.5 (osd.5) 573 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:04:17.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:17 smithi178 bash[22840]: cluster 2024-09-13T23:04:16.503144+0000 osd.5 (osd.5) 574 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:04:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:16.853766+0000 osd.0 (osd.0) 497 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:04:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:16.858383+0000 osd.0 (osd.0) 498 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:04:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:17.088878+0000 mgr.y (mgr.14613) 986 : cluster [DBG] pgmap v1196: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:18.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:17.157399+0000 osd.1 (osd.1) 504 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:17.157504+0000 osd.1 (osd.1) 505 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:17.551181+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2024-09-13T23:04:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:17.580113+0000 osd.4 (osd.4) 334 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:04:18.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:18 smithi192 bash[24608]: cluster 2024-09-13T23:04:17.582660+0000 osd.4 (osd.4) 335 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:04:18.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:16.853766+0000 osd.0 (osd.0) 497 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:04:18.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:16.858383+0000 osd.0 (osd.0) 498 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:17.088878+0000 mgr.y (mgr.14613) 986 : cluster [DBG] pgmap v1196: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:17.157399+0000 osd.1 (osd.1) 504 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:17.157504+0000 osd.1 (osd.1) 505 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:17.551181+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:17.580113+0000 osd.4 (osd.4) 334 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[22840]: cluster 2024-09-13T23:04:17.582660+0000 osd.4 (osd.4) 335 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:16.853766+0000 osd.0 (osd.0) 497 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:16.858383+0000 osd.0 (osd.0) 498 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:17.088878+0000 mgr.y (mgr.14613) 986 : cluster [DBG] pgmap v1196: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:17.157399+0000 osd.1 (osd.1) 504 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:18.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:17.157504+0000 osd.1 (osd.1) 505 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:18.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:17.551181+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2024-09-13T23:04:18.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:17.580113+0000 osd.4 (osd.4) 334 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:04:18.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:18 smithi178 bash[29889]: cluster 2024-09-13T23:04:17.582660+0000 osd.4 (osd.4) 335 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:04:19.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:19 smithi192 bash[24608]: cluster 2024-09-13T23:04:18.536386+0000 osd.4 (osd.4) 336 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:04:19.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:19 smithi192 bash[24608]: cluster 2024-09-13T23:04:18.541281+0000 osd.4 (osd.4) 337 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:04:19.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:19 smithi192 bash[24608]: cluster 2024-09-13T23:04:18.613966+0000 osd.3 (osd.3) 497 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:04:19.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:19 smithi192 bash[24608]: cluster 2024-09-13T23:04:18.617777+0000 osd.3 (osd.3) 498 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:04:19.930 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:04:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:04:19.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[29889]: cluster 2024-09-13T23:04:18.536386+0000 osd.4 (osd.4) 336 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[29889]: cluster 2024-09-13T23:04:18.541281+0000 osd.4 (osd.4) 337 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[29889]: cluster 2024-09-13T23:04:18.613966+0000 osd.3 (osd.3) 497 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[29889]: cluster 2024-09-13T23:04:18.617777+0000 osd.3 (osd.3) 498 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[22840]: cluster 2024-09-13T23:04:18.536386+0000 osd.4 (osd.4) 336 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[22840]: cluster 2024-09-13T23:04:18.541281+0000 osd.4 (osd.4) 337 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[22840]: cluster 2024-09-13T23:04:18.613966+0000 osd.3 (osd.3) 497 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:04:19.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:19 smithi178 bash[22840]: cluster 2024-09-13T23:04:18.617777+0000 osd.3 (osd.3) 498 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:04:20.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:20 smithi192 bash[24608]: cluster 2024-09-13T23:04:18.920402+0000 osd.0 (osd.0) 499 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:04:20.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:20 smithi192 bash[24608]: cluster 2024-09-13T23:04:18.922570+0000 osd.0 (osd.0) 500 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:04:20.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:20 smithi192 bash[24608]: cluster 2024-09-13T23:04:19.089837+0000 mgr.y (mgr.14613) 987 : cluster [DBG] pgmap v1198: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:20.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:20 smithi192 bash[24608]: audit 2024-09-13T23:04:19.970635+0000 mon.b (mon.1) 317 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:20.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[29889]: cluster 2024-09-13T23:04:18.920402+0000 osd.0 (osd.0) 499 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:04:20.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[29889]: cluster 2024-09-13T23:04:18.922570+0000 osd.0 (osd.0) 500 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:04:20.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[29889]: cluster 2024-09-13T23:04:19.089837+0000 mgr.y (mgr.14613) 987 : cluster [DBG] pgmap v1198: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:20.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[29889]: audit 2024-09-13T23:04:19.970635+0000 mon.b (mon.1) 317 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[22840]: cluster 2024-09-13T23:04:18.920402+0000 osd.0 (osd.0) 499 : cluster [DBG] 6.16 scrub starts 2024-09-13T23:04:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[22840]: cluster 2024-09-13T23:04:18.922570+0000 osd.0 (osd.0) 500 : cluster [DBG] 6.16 scrub ok 2024-09-13T23:04:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[22840]: cluster 2024-09-13T23:04:19.089837+0000 mgr.y (mgr.14613) 987 : cluster [DBG] pgmap v1198: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:20.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:20 smithi178 bash[22840]: audit 2024-09-13T23:04:19.970635+0000 mon.b (mon.1) 317 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:21.208 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:21.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:21 smithi192 bash[24608]: cluster 2024-09-13T23:04:20.890726+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-13T23:04:21.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:21 smithi192 bash[24608]: cluster 2024-09-13T23:04:20.893080+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-13T23:04:21.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:21 smithi178 bash[29889]: cluster 2024-09-13T23:04:20.890726+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-13T23:04:21.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:21 smithi178 bash[29889]: cluster 2024-09-13T23:04:20.893080+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-13T23:04:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:21 smithi178 bash[22840]: cluster 2024-09-13T23:04:20.890726+0000 osd.2 (osd.2) 357 : cluster [DBG] 5.a scrub starts 2024-09-13T23:04:21.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:21 smithi178 bash[22840]: cluster 2024-09-13T23:04:20.893080+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.a scrub ok 2024-09-13T23:04:22.830 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:22 smithi178 bash[29889]: cluster 2024-09-13T23:04:21.091162+0000 mgr.y (mgr.14613) 988 : cluster [DBG] pgmap v1199: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:22.831 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:22 smithi178 bash[29889]: cluster 2024-09-13T23:04:21.649115+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:04:22.831 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:22 smithi178 bash[29889]: cluster 2024-09-13T23:04:21.651674+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:04:22.831 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:22 smithi178 bash[22840]: cluster 2024-09-13T23:04:21.091162+0000 mgr.y (mgr.14613) 988 : cluster [DBG] pgmap v1199: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:22.831 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:22 smithi178 bash[22840]: cluster 2024-09-13T23:04:21.649115+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:04:22.831 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:22 smithi178 bash[22840]: cluster 2024-09-13T23:04:21.651674+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:04:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:22 smithi192 bash[24608]: cluster 2024-09-13T23:04:21.091162+0000 mgr.y (mgr.14613) 988 : cluster [DBG] pgmap v1199: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:22 smithi192 bash[24608]: cluster 2024-09-13T23:04:21.649115+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:04:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:22 smithi192 bash[24608]: cluster 2024-09-13T23:04:21.651674+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:04:23.579 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T23:04:23.854 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:23 smithi178 bash[29889]: cluster 2024-09-13T23:04:22.664918+0000 osd.3 (osd.3) 501 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:04:23.854 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:23 smithi178 bash[29889]: audit 2024-09-13T23:04:23.131972+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? v1:172.21.15.178:0/2174322456' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:23.854 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:23 smithi178 bash[22840]: cluster 2024-09-13T23:04:22.664918+0000 osd.3 (osd.3) 501 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:04:23.855 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:23 smithi178 bash[22840]: audit 2024-09-13T23:04:23.131972+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? v1:172.21.15.178:0/2174322456' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:23.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:23 smithi192 bash[24608]: cluster 2024-09-13T23:04:22.664918+0000 osd.3 (osd.3) 501 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:04:23.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:23 smithi192 bash[24608]: audit 2024-09-13T23:04:23.131972+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? v1:172.21.15.178:0/2174322456' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:24.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T23:04:24.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:24 smithi192 bash[24608]: cluster 2024-09-13T23:04:22.669164+0000 osd.3 (osd.3) 502 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:04:24.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:24 smithi192 bash[24608]: cluster 2024-09-13T23:04:22.938772+0000 osd.0 (osd.0) 501 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:04:24.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:24 smithi192 bash[24608]: cluster 2024-09-13T23:04:22.940292+0000 osd.0 (osd.0) 502 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:04:24.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:24 smithi192 bash[24608]: cluster 2024-09-13T23:04:23.092005+0000 mgr.y (mgr.14613) 989 : cluster [DBG] pgmap v1200: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:24.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:24 smithi192 bash[24608]: audit 2024-09-13T23:04:23.579953+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? v1:172.21.15.178:0/2174322456' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:04:24.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:24 smithi192 bash[24608]: cluster 2024-09-13T23:04:23.587597+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2024-09-13T23:04:24.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[22840]: cluster 2024-09-13T23:04:22.669164+0000 osd.3 (osd.3) 502 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[22840]: cluster 2024-09-13T23:04:22.938772+0000 osd.0 (osd.0) 501 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[22840]: cluster 2024-09-13T23:04:22.940292+0000 osd.0 (osd.0) 502 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[22840]: cluster 2024-09-13T23:04:23.092005+0000 mgr.y (mgr.14613) 989 : cluster [DBG] pgmap v1200: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[22840]: audit 2024-09-13T23:04:23.579953+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? v1:172.21.15.178:0/2174322456' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[22840]: cluster 2024-09-13T23:04:23.587597+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[29889]: cluster 2024-09-13T23:04:22.669164+0000 osd.3 (osd.3) 502 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[29889]: cluster 2024-09-13T23:04:22.938772+0000 osd.0 (osd.0) 501 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:04:24.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[29889]: cluster 2024-09-13T23:04:22.940292+0000 osd.0 (osd.0) 502 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:04:24.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[29889]: cluster 2024-09-13T23:04:23.092005+0000 mgr.y (mgr.14613) 989 : cluster [DBG] pgmap v1200: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:24.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[29889]: audit 2024-09-13T23:04:23.579953+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? v1:172.21.15.178:0/2174322456' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:04:24.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:24 smithi178 bash[29889]: cluster 2024-09-13T23:04:23.587597+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2024-09-13T23:04:26.598 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:04:26.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:24.672705+0000 osd.3 (osd.3) 503 : cluster [DBG] 5.13 deep-scrub starts 2024-09-13T23:04:26.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:24.673683+0000 osd.3 (osd.3) 504 : cluster [DBG] 5.13 deep-scrub ok 2024-09-13T23:04:26.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.093407+0000 mgr.y (mgr.14613) 990 : cluster [DBG] pgmap v1202: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:26.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.305429+0000 osd.1 (osd.1) 506 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T23:04:26.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.311642+0000 osd.1 (osd.1) 507 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T23:04:26.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.596597+0000 osd.7 (osd.7) 546 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:04:26.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.599057+0000 osd.7 (osd.7) 547 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:04:26.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.633130+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.b scrub starts 2024-09-13T23:04:26.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:26 smithi192 bash[24608]: cluster 2024-09-13T23:04:25.636956+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.b scrub ok 2024-09-13T23:04:26.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:24.672705+0000 osd.3 (osd.3) 503 : cluster [DBG] 5.13 deep-scrub starts 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:24.673683+0000 osd.3 (osd.3) 504 : cluster [DBG] 5.13 deep-scrub ok 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.093407+0000 mgr.y (mgr.14613) 990 : cluster [DBG] pgmap v1202: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.305429+0000 osd.1 (osd.1) 506 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.311642+0000 osd.1 (osd.1) 507 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.596597+0000 osd.7 (osd.7) 546 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.599057+0000 osd.7 (osd.7) 547 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.633130+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.b scrub starts 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[22840]: cluster 2024-09-13T23:04:25.636956+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.b scrub ok 2024-09-13T23:04:26.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:24.672705+0000 osd.3 (osd.3) 503 : cluster [DBG] 5.13 deep-scrub starts 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:24.673683+0000 osd.3 (osd.3) 504 : cluster [DBG] 5.13 deep-scrub ok 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.093407+0000 mgr.y (mgr.14613) 990 : cluster [DBG] pgmap v1202: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.305429+0000 osd.1 (osd.1) 506 : cluster [DBG] 3.4 deep-scrub starts 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.311642+0000 osd.1 (osd.1) 507 : cluster [DBG] 3.4 deep-scrub ok 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.596597+0000 osd.7 (osd.7) 546 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.599057+0000 osd.7 (osd.7) 547 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.633130+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.b scrub starts 2024-09-13T23:04:26.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:26 smithi178 bash[29889]: cluster 2024-09-13T23:04:25.636956+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.b scrub ok 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: audit 2024-09-13T23:04:26.232895+0000 mgr.y (mgr.14613) 991 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: cluster 2024-09-13T23:04:26.330986+0000 osd.1 (osd.1) 508 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: cluster 2024-09-13T23:04:26.333354+0000 osd.1 (osd.1) 509 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: cluster 2024-09-13T23:04:26.658063+0000 osd.3 (osd.3) 507 : cluster [DBG] 2.19 deep-scrub starts 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: cluster 2024-09-13T23:04:26.663501+0000 osd.3 (osd.3) 508 : cluster [DBG] 2.19 deep-scrub ok 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: cluster 2024-09-13T23:04:26.935527+0000 osd.6 (osd.6) 382 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:27.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:27 smithi192 bash[24608]: cluster 2024-09-13T23:04:26.935631+0000 osd.6 (osd.6) 383 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: audit 2024-09-13T23:04:26.232895+0000 mgr.y (mgr.14613) 991 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: cluster 2024-09-13T23:04:26.330986+0000 osd.1 (osd.1) 508 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: cluster 2024-09-13T23:04:26.333354+0000 osd.1 (osd.1) 509 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: cluster 2024-09-13T23:04:26.658063+0000 osd.3 (osd.3) 507 : cluster [DBG] 2.19 deep-scrub starts 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: cluster 2024-09-13T23:04:26.663501+0000 osd.3 (osd.3) 508 : cluster [DBG] 2.19 deep-scrub ok 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: cluster 2024-09-13T23:04:26.935527+0000 osd.6 (osd.6) 382 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[22840]: cluster 2024-09-13T23:04:26.935631+0000 osd.6 (osd.6) 383 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: audit 2024-09-13T23:04:26.232895+0000 mgr.y (mgr.14613) 991 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: cluster 2024-09-13T23:04:26.330986+0000 osd.1 (osd.1) 508 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:04:27.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: cluster 2024-09-13T23:04:26.333354+0000 osd.1 (osd.1) 509 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:04:27.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: cluster 2024-09-13T23:04:26.658063+0000 osd.3 (osd.3) 507 : cluster [DBG] 2.19 deep-scrub starts 2024-09-13T23:04:27.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: cluster 2024-09-13T23:04:26.663501+0000 osd.3 (osd.3) 508 : cluster [DBG] 2.19 deep-scrub ok 2024-09-13T23:04:27.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: cluster 2024-09-13T23:04:26.935527+0000 osd.6 (osd.6) 382 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:27.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:27 smithi178 bash[29889]: cluster 2024-09-13T23:04:26.935631+0000 osd.6 (osd.6) 383 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:28 smithi192 bash[24608]: cluster 2024-09-13T23:04:27.094283+0000 mgr.y (mgr.14613) 992 : cluster [DBG] pgmap v1203: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:28 smithi192 bash[24608]: cluster 2024-09-13T23:04:27.615811+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2024-09-13T23:04:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:28 smithi192 bash[24608]: cluster 2024-09-13T23:04:27.730129+0000 osd.4 (osd.4) 338 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:04:28.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:28 smithi192 bash[24608]: cluster 2024-09-13T23:04:27.732730+0000 osd.4 (osd.4) 339 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:04:28.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[22840]: cluster 2024-09-13T23:04:27.094283+0000 mgr.y (mgr.14613) 992 : cluster [DBG] pgmap v1203: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[22840]: cluster 2024-09-13T23:04:27.615811+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[22840]: cluster 2024-09-13T23:04:27.730129+0000 osd.4 (osd.4) 338 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[22840]: cluster 2024-09-13T23:04:27.732730+0000 osd.4 (osd.4) 339 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[29889]: cluster 2024-09-13T23:04:27.094283+0000 mgr.y (mgr.14613) 992 : cluster [DBG] pgmap v1203: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[29889]: cluster 2024-09-13T23:04:27.615811+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[29889]: cluster 2024-09-13T23:04:27.730129+0000 osd.4 (osd.4) 338 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:04:28.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:28 smithi178 bash[29889]: cluster 2024-09-13T23:04:27.732730+0000 osd.4 (osd.4) 339 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:04:29.252 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[22840]: cluster 2024-09-13T23:04:28.596700+0000 osd.7 (osd.7) 548 : cluster [DBG] 5.f scrub starts 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[22840]: cluster 2024-09-13T23:04:28.599001+0000 osd.7 (osd.7) 549 : cluster [DBG] 5.f scrub ok 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[22840]: cluster 2024-09-13T23:04:28.720125+0000 osd.4 (osd.4) 340 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[22840]: cluster 2024-09-13T23:04:28.722594+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[22840]: cluster 2024-09-13T23:04:28.942650+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[22840]: cluster 2024-09-13T23:04:28.945275+0000 osd.2 (osd.2) 360 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[29889]: cluster 2024-09-13T23:04:28.596700+0000 osd.7 (osd.7) 548 : cluster [DBG] 5.f scrub starts 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[29889]: cluster 2024-09-13T23:04:28.599001+0000 osd.7 (osd.7) 549 : cluster [DBG] 5.f scrub ok 2024-09-13T23:04:29.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[29889]: cluster 2024-09-13T23:04:28.720125+0000 osd.4 (osd.4) 340 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:04:29.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[29889]: cluster 2024-09-13T23:04:28.722594+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:04:29.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[29889]: cluster 2024-09-13T23:04:28.942650+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:04:29.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[29889]: cluster 2024-09-13T23:04:28.945275+0000 osd.2 (osd.2) 360 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:04:29.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:29 smithi192 bash[24608]: cluster 2024-09-13T23:04:28.596700+0000 osd.7 (osd.7) 548 : cluster [DBG] 5.f scrub starts 2024-09-13T23:04:29.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:29 smithi192 bash[24608]: cluster 2024-09-13T23:04:28.599001+0000 osd.7 (osd.7) 549 : cluster [DBG] 5.f scrub ok 2024-09-13T23:04:29.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:29 smithi192 bash[24608]: cluster 2024-09-13T23:04:28.720125+0000 osd.4 (osd.4) 340 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:04:29.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:29 smithi192 bash[24608]: cluster 2024-09-13T23:04:28.722594+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:04:29.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:29 smithi192 bash[24608]: cluster 2024-09-13T23:04:28.942650+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:04:29.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:29 smithi192 bash[24608]: cluster 2024-09-13T23:04:28.945275+0000 osd.2 (osd.2) 360 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:04:30.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:04:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:04:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:04:30.620 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.095250+0000 mgr.y (mgr.14613) 993 : cluster [DBG] pgmap v1205: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.636549+0000 osd.7 (osd.7) 550 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.639012+0000 osd.7 (osd.7) 551 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.694673+0000 osd.4 (osd.4) 342 : cluster [DBG] 6.1e deep-scrub starts 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.696986+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.1e deep-scrub ok 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.095250+0000 mgr.y (mgr.14613) 993 : cluster [DBG] pgmap v1205: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.636549+0000 osd.7 (osd.7) 550 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.639012+0000 osd.7 (osd.7) 551 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.694673+0000 osd.4 (osd.4) 342 : cluster [DBG] 6.1e deep-scrub starts 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.696986+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.1e deep-scrub ok 2024-09-13T23:04:30.621 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.706737+0000 osd.6 (osd.6) 384 : cluster [DBG] 3.a scrub starts 2024-09-13T23:04:30.622 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[22840]: cluster 2024-09-13T23:04:29.712065+0000 osd.6 (osd.6) 385 : cluster [DBG] 3.a scrub ok 2024-09-13T23:04:30.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.706737+0000 osd.6 (osd.6) 384 : cluster [DBG] 3.a scrub starts 2024-09-13T23:04:30.887 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:30 smithi178 bash[29889]: cluster 2024-09-13T23:04:29.712065+0000 osd.6 (osd.6) 385 : cluster [DBG] 3.a scrub ok 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.095250+0000 mgr.y (mgr.14613) 993 : cluster [DBG] pgmap v1205: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.636549+0000 osd.7 (osd.7) 550 : cluster [DBG] 2.1d scrub starts 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.639012+0000 osd.7 (osd.7) 551 : cluster [DBG] 2.1d scrub ok 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.694673+0000 osd.4 (osd.4) 342 : cluster [DBG] 6.1e deep-scrub starts 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.696986+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.1e deep-scrub ok 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.706737+0000 osd.6 (osd.6) 384 : cluster [DBG] 3.a scrub starts 2024-09-13T23:04:30.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:30 smithi192 bash[24608]: cluster 2024-09-13T23:04:29.712065+0000 osd.6 (osd.6) 385 : cluster [DBG] 3.a scrub ok 2024-09-13T23:04:31.628 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T23:04:31.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:31 smithi192 bash[24608]: cluster 2024-09-13T23:04:30.643327+0000 osd.7 (osd.7) 552 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:04:31.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:31 smithi192 bash[24608]: cluster 2024-09-13T23:04:30.645563+0000 osd.7 (osd.7) 553 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:04:31.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:31 smithi192 bash[24608]: cluster 2024-09-13T23:04:30.656910+0000 osd.3 (osd.3) 509 : cluster [DBG] 5.17 scrub starts 2024-09-13T23:04:31.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:31 smithi192 bash[24608]: cluster 2024-09-13T23:04:30.658746+0000 osd.3 (osd.3) 510 : cluster [DBG] 5.17 scrub ok 2024-09-13T23:04:31.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:31 smithi192 bash[24608]: audit 2024-09-13T23:04:31.194656+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.178:0/2290351656' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:31.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:31 smithi192 bash[24608]: audit 2024-09-13T23:04:31.195044+0000 mon.a (mon.0) 1934 : audit [INF] from='client.26087 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:31.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[29889]: cluster 2024-09-13T23:04:30.643327+0000 osd.7 (osd.7) 552 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:04:31.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[29889]: cluster 2024-09-13T23:04:30.645563+0000 osd.7 (osd.7) 553 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:04:31.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[29889]: cluster 2024-09-13T23:04:30.656910+0000 osd.3 (osd.3) 509 : cluster [DBG] 5.17 scrub starts 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[29889]: cluster 2024-09-13T23:04:30.658746+0000 osd.3 (osd.3) 510 : cluster [DBG] 5.17 scrub ok 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[29889]: audit 2024-09-13T23:04:31.194656+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.178:0/2290351656' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[29889]: audit 2024-09-13T23:04:31.195044+0000 mon.a (mon.0) 1934 : audit [INF] from='client.26087 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[22840]: cluster 2024-09-13T23:04:30.643327+0000 osd.7 (osd.7) 552 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[22840]: cluster 2024-09-13T23:04:30.645563+0000 osd.7 (osd.7) 553 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[22840]: cluster 2024-09-13T23:04:30.656910+0000 osd.3 (osd.3) 509 : cluster [DBG] 5.17 scrub starts 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[22840]: cluster 2024-09-13T23:04:30.658746+0000 osd.3 (osd.3) 510 : cluster [DBG] 5.17 scrub ok 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[22840]: audit 2024-09-13T23:04:31.194656+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.178:0/2290351656' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:31.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:31 smithi178 bash[22840]: audit 2024-09-13T23:04:31.195044+0000 mon.a (mon.0) 1934 : audit [INF] from='client.26087 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-13T23:04:32.494 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T23:04:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.096480+0000 mgr.y (mgr.14613) 994 : cluster [DBG] pgmap v1206: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.343534+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.c scrub starts 2024-09-13T23:04:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.345739+0000 osd.1 (osd.1) 511 : cluster [DBG] 5.c scrub ok 2024-09-13T23:04:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: audit 2024-09-13T23:04:31.622092+0000 mon.a (mon.0) 1935 : audit [INF] from='client.26087 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:04:32.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.628770+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2024-09-13T23:04:32.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.714356+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:04:32.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.716636+0000 osd.4 (osd.4) 345 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:04:32.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.932543+0000 osd.2 (osd.2) 361 : cluster [DBG] 2.c scrub starts 2024-09-13T23:04:32.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:32 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.936897+0000 osd.2 (osd.2) 362 : cluster [DBG] 2.c scrub ok 2024-09-13T23:04:32.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.096480+0000 mgr.y (mgr.14613) 994 : cluster [DBG] pgmap v1206: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:32.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.343534+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.c scrub starts 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.345739+0000 osd.1 (osd.1) 511 : cluster [DBG] 5.c scrub ok 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: audit 2024-09-13T23:04:31.622092+0000 mon.a (mon.0) 1935 : audit [INF] from='client.26087 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.628770+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.714356+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.716636+0000 osd.4 (osd.4) 345 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.932543+0000 osd.2 (osd.2) 361 : cluster [DBG] 2.c scrub starts 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.936897+0000 osd.2 (osd.2) 362 : cluster [DBG] 2.c scrub ok 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.096480+0000 mgr.y (mgr.14613) 994 : cluster [DBG] pgmap v1206: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.343534+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.c scrub starts 2024-09-13T23:04:32.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.345739+0000 osd.1 (osd.1) 511 : cluster [DBG] 5.c scrub ok 2024-09-13T23:04:32.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: audit 2024-09-13T23:04:31.622092+0000 mon.a (mon.0) 1935 : audit [INF] from='client.26087 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T23:04:32.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.628770+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2024-09-13T23:04:32.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.714356+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:04:32.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.716636+0000 osd.4 (osd.4) 345 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:04:32.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.932543+0000 osd.2 (osd.2) 361 : cluster [DBG] 2.c scrub starts 2024-09-13T23:04:32.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:32 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.936897+0000 osd.2 (osd.2) 362 : cluster [DBG] 2.c scrub ok 2024-09-13T23:04:33.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:33 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.959297+0000 osd.0 (osd.0) 503 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:04:33.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:33 smithi192 bash[24608]: cluster 2024-09-13T23:04:31.962731+0000 osd.0 (osd.0) 504 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:04:33.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:33 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.959297+0000 osd.0 (osd.0) 503 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:04:33.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:33 smithi178 bash[29889]: cluster 2024-09-13T23:04:31.962731+0000 osd.0 (osd.0) 504 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:04:33.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:33 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.959297+0000 osd.0 (osd.0) 503 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:04:33.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:33 smithi178 bash[22840]: cluster 2024-09-13T23:04:31.962731+0000 osd.0 (osd.0) 504 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:04:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:34 smithi192 bash[24608]: cluster 2024-09-13T23:04:33.097303+0000 mgr.y (mgr.14613) 995 : cluster [DBG] pgmap v1208: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:34 smithi192 bash[24608]: cluster 2024-09-13T23:04:33.672586+0000 osd.6 (osd.6) 386 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:04:34.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:34 smithi192 bash[24608]: cluster 2024-09-13T23:04:33.674893+0000 osd.6 (osd.6) 387 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:04:34.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:34 smithi178 bash[29889]: cluster 2024-09-13T23:04:33.097303+0000 mgr.y (mgr.14613) 995 : cluster [DBG] pgmap v1208: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:34.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:34 smithi178 bash[29889]: cluster 2024-09-13T23:04:33.672586+0000 osd.6 (osd.6) 386 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:04:34.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:34 smithi178 bash[29889]: cluster 2024-09-13T23:04:33.674893+0000 osd.6 (osd.6) 387 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:04:34.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:34 smithi178 bash[22840]: cluster 2024-09-13T23:04:33.097303+0000 mgr.y (mgr.14613) 995 : cluster [DBG] pgmap v1208: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:34.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:34 smithi178 bash[22840]: cluster 2024-09-13T23:04:33.672586+0000 osd.6 (osd.6) 386 : cluster [DBG] 2.7 scrub starts 2024-09-13T23:04:34.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:34 smithi178 bash[22840]: cluster 2024-09-13T23:04:33.674893+0000 osd.6 (osd.6) 387 : cluster [DBG] 2.7 scrub ok 2024-09-13T23:04:35.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:35 smithi192 bash[24608]: audit 2024-09-13T23:04:34.998752+0000 mon.b (mon.1) 318 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:35.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:35 smithi178 bash[29889]: audit 2024-09-13T23:04:34.998752+0000 mon.b (mon.1) 318 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:35.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:35 smithi178 bash[22840]: audit 2024-09-13T23:04:34.998752+0000 mon.b (mon.1) 318 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:36.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:04:36 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:04:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:36 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.098837+0000 mgr.y (mgr.14613) 996 : cluster [DBG] pgmap v1209: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:36 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.496033+0000 osd.5 (osd.5) 575 : cluster [DBG] 6.a scrub starts 2024-09-13T23:04:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:36 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.498307+0000 osd.5 (osd.5) 576 : cluster [DBG] 6.a scrub ok 2024-09-13T23:04:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:36 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.563546+0000 osd.7 (osd.7) 554 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:04:36.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:36 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.565957+0000 osd.7 (osd.7) 555 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:04:36.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.098837+0000 mgr.y (mgr.14613) 996 : cluster [DBG] pgmap v1209: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.496033+0000 osd.5 (osd.5) 575 : cluster [DBG] 6.a scrub starts 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.498307+0000 osd.5 (osd.5) 576 : cluster [DBG] 6.a scrub ok 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.563546+0000 osd.7 (osd.7) 554 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.565957+0000 osd.7 (osd.7) 555 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.098837+0000 mgr.y (mgr.14613) 996 : cluster [DBG] pgmap v1209: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.496033+0000 osd.5 (osd.5) 575 : cluster [DBG] 6.a scrub starts 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.498307+0000 osd.5 (osd.5) 576 : cluster [DBG] 6.a scrub ok 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.563546+0000 osd.7 (osd.7) 554 : cluster [DBG] 6.12 scrub starts 2024-09-13T23:04:36.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:36 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.565957+0000 osd.7 (osd.7) 555 : cluster [DBG] 6.12 scrub ok 2024-09-13T23:04:37.308 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:37.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.686261+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.688622+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.924880+0000 osd.0 (osd.0) 505 : cluster [DBG] 3.17 scrub starts 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:35.929501+0000 osd.0 (osd.0) 506 : cluster [DBG] 3.17 scrub ok 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: audit 2024-09-13T23:04:36.242427+0000 mgr.y (mgr.14613) 997 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:36.517425+0000 osd.5 (osd.5) 577 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:36.519839+0000 osd.5 (osd.5) 578 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:36.550019+0000 osd.7 (osd.7) 556 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[29889]: cluster 2024-09-13T23:04:36.552623+0000 osd.7 (osd.7) 557 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.686261+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.688622+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:04:37.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.924880+0000 osd.0 (osd.0) 505 : cluster [DBG] 3.17 scrub starts 2024-09-13T23:04:37.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:35.929501+0000 osd.0 (osd.0) 506 : cluster [DBG] 3.17 scrub ok 2024-09-13T23:04:37.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: audit 2024-09-13T23:04:36.242427+0000 mgr.y (mgr.14613) 997 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:37.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:36.517425+0000 osd.5 (osd.5) 577 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:04:37.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:36.519839+0000 osd.5 (osd.5) 578 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:04:37.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:36.550019+0000 osd.7 (osd.7) 556 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:04:37.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:37 smithi178 bash[22840]: cluster 2024-09-13T23:04:36.552623+0000 osd.7 (osd.7) 557 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:04:38.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.686261+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:04:38.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.688622+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:04:38.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.924880+0000 osd.0 (osd.0) 505 : cluster [DBG] 3.17 scrub starts 2024-09-13T23:04:38.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:35.929501+0000 osd.0 (osd.0) 506 : cluster [DBG] 3.17 scrub ok 2024-09-13T23:04:38.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: audit 2024-09-13T23:04:36.242427+0000 mgr.y (mgr.14613) 997 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:38.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:36.517425+0000 osd.5 (osd.5) 577 : cluster [DBG] 6.7 scrub starts 2024-09-13T23:04:38.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:36.519839+0000 osd.5 (osd.5) 578 : cluster [DBG] 6.7 scrub ok 2024-09-13T23:04:38.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:36.550019+0000 osd.7 (osd.7) 556 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:04:38.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:37 smithi192 bash[24608]: cluster 2024-09-13T23:04:36.552623+0000 osd.7 (osd.7) 557 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:04:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[22840]: cluster 2024-09-13T23:04:37.099744+0000 mgr.y (mgr.14613) 998 : cluster [DBG] pgmap v1210: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[22840]: cluster 2024-09-13T23:04:37.580748+0000 osd.7 (osd.7) 558 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:04:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[22840]: cluster 2024-09-13T23:04:37.584417+0000 osd.7 (osd.7) 559 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:04:38.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[22840]: cluster 2024-09-13T23:04:37.911612+0000 osd.0 (osd.0) 507 : cluster [DBG] 6.0 scrub starts 2024-09-13T23:04:38.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[22840]: cluster 2024-09-13T23:04:37.914142+0000 osd.0 (osd.0) 508 : cluster [DBG] 6.0 scrub ok 2024-09-13T23:04:38.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[29889]: cluster 2024-09-13T23:04:37.099744+0000 mgr.y (mgr.14613) 998 : cluster [DBG] pgmap v1210: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:38.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[29889]: cluster 2024-09-13T23:04:37.580748+0000 osd.7 (osd.7) 558 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:04:38.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[29889]: cluster 2024-09-13T23:04:37.584417+0000 osd.7 (osd.7) 559 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:04:38.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[29889]: cluster 2024-09-13T23:04:37.911612+0000 osd.0 (osd.0) 507 : cluster [DBG] 6.0 scrub starts 2024-09-13T23:04:38.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:38 smithi178 bash[29889]: cluster 2024-09-13T23:04:37.914142+0000 osd.0 (osd.0) 508 : cluster [DBG] 6.0 scrub ok 2024-09-13T23:04:39.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:38 smithi192 bash[24608]: cluster 2024-09-13T23:04:37.099744+0000 mgr.y (mgr.14613) 998 : cluster [DBG] pgmap v1210: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:39.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:38 smithi192 bash[24608]: cluster 2024-09-13T23:04:37.580748+0000 osd.7 (osd.7) 558 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:04:39.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:38 smithi192 bash[24608]: cluster 2024-09-13T23:04:37.584417+0000 osd.7 (osd.7) 559 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:04:39.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:38 smithi192 bash[24608]: cluster 2024-09-13T23:04:37.911612+0000 osd.0 (osd.0) 507 : cluster [DBG] 6.0 scrub starts 2024-09-13T23:04:39.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:38 smithi192 bash[24608]: cluster 2024-09-13T23:04:37.914142+0000 osd.0 (osd.0) 508 : cluster [DBG] 6.0 scrub ok 2024-09-13T23:04:39.675 INFO:teuthology.orchestra.run.smithi178.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T23:04:39.918 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:04:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:04:39.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[22840]: cluster 2024-09-13T23:04:38.943602+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1 scrub starts 2024-09-13T23:04:39.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[22840]: cluster 2024-09-13T23:04:38.946122+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1 scrub ok 2024-09-13T23:04:39.918 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[22840]: audit 2024-09-13T23:04:39.216038+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.178:0/194288704' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:04:39.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[29889]: cluster 2024-09-13T23:04:38.943602+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1 scrub starts 2024-09-13T23:04:39.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[29889]: cluster 2024-09-13T23:04:38.946122+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1 scrub ok 2024-09-13T23:04:39.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:39 smithi178 bash[29889]: audit 2024-09-13T23:04:39.216038+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.178:0/194288704' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:04:40.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:39 smithi192 bash[24608]: cluster 2024-09-13T23:04:38.943602+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1 scrub starts 2024-09-13T23:04:40.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:39 smithi192 bash[24608]: cluster 2024-09-13T23:04:38.946122+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1 scrub ok 2024-09-13T23:04:40.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:39 smithi192 bash[24608]: audit 2024-09-13T23:04:39.216038+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? v1:172.21.15.178:0/194288704' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:04:40.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:38.672652+0000 osd.3 (osd.3) 513 : cluster [DBG] 6.6 scrub starts 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:38.675125+0000 osd.3 (osd.3) 514 : cluster [DBG] 6.6 scrub ok 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:39.100689+0000 mgr.y (mgr.14613) 999 : cluster [DBG] pgmap v1211: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:39.439591+0000 osd.5 (osd.5) 579 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:39.441857+0000 osd.5 (osd.5) 580 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: audit 2024-09-13T23:04:39.675959+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.178:0/194288704' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:39.682088+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:39.982392+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f scrub starts 2024-09-13T23:04:40.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[29889]: cluster 2024-09-13T23:04:39.984832+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f scrub ok 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:38.672652+0000 osd.3 (osd.3) 513 : cluster [DBG] 6.6 scrub starts 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:38.675125+0000 osd.3 (osd.3) 514 : cluster [DBG] 6.6 scrub ok 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:39.100689+0000 mgr.y (mgr.14613) 999 : cluster [DBG] pgmap v1211: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:39.439591+0000 osd.5 (osd.5) 579 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:39.441857+0000 osd.5 (osd.5) 580 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: audit 2024-09-13T23:04:39.675959+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.178:0/194288704' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:39.682088+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:39.982392+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f scrub starts 2024-09-13T23:04:40.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:40 smithi178 bash[22840]: cluster 2024-09-13T23:04:39.984832+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f scrub ok 2024-09-13T23:04:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:38.672652+0000 osd.3 (osd.3) 513 : cluster [DBG] 6.6 scrub starts 2024-09-13T23:04:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:38.675125+0000 osd.3 (osd.3) 514 : cluster [DBG] 6.6 scrub ok 2024-09-13T23:04:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:39.100689+0000 mgr.y (mgr.14613) 999 : cluster [DBG] pgmap v1211: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:39.439591+0000 osd.5 (osd.5) 579 : cluster [DBG] 2.16 scrub starts 2024-09-13T23:04:41.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:39.441857+0000 osd.5 (osd.5) 580 : cluster [DBG] 2.16 scrub ok 2024-09-13T23:04:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: audit 2024-09-13T23:04:39.675959+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.178:0/194288704' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:04:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:39.682088+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2024-09-13T23:04:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:39.982392+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f scrub starts 2024-09-13T23:04:41.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:40 smithi192 bash[24608]: cluster 2024-09-13T23:04:39.984832+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f scrub ok 2024-09-13T23:04:41.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[22840]: cluster 2024-09-13T23:04:40.599202+0000 osd.7 (osd.7) 560 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[22840]: cluster 2024-09-13T23:04:40.601271+0000 osd.7 (osd.7) 561 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[22840]: cluster 2024-09-13T23:04:41.494163+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[22840]: cluster 2024-09-13T23:04:41.496659+0000 osd.1 (osd.1) 513 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[29889]: cluster 2024-09-13T23:04:40.599202+0000 osd.7 (osd.7) 560 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[29889]: cluster 2024-09-13T23:04:40.601271+0000 osd.7 (osd.7) 561 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[29889]: cluster 2024-09-13T23:04:41.494163+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:04:41.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:41 smithi178 bash[29889]: cluster 2024-09-13T23:04:41.496659+0000 osd.1 (osd.1) 513 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:04:42.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:41 smithi192 bash[24608]: cluster 2024-09-13T23:04:40.599202+0000 osd.7 (osd.7) 560 : cluster [DBG] 2.0 scrub starts 2024-09-13T23:04:42.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:41 smithi192 bash[24608]: cluster 2024-09-13T23:04:40.601271+0000 osd.7 (osd.7) 561 : cluster [DBG] 2.0 scrub ok 2024-09-13T23:04:42.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:41 smithi192 bash[24608]: cluster 2024-09-13T23:04:41.494163+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1 scrub starts 2024-09-13T23:04:42.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:41 smithi192 bash[24608]: cluster 2024-09-13T23:04:41.496659+0000 osd.1 (osd.1) 513 : cluster [DBG] 6.1 scrub ok 2024-09-13T23:04:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:42 smithi192 bash[24608]: cluster 2024-09-13T23:04:41.101651+0000 mgr.y (mgr.14613) 1000 : cluster [DBG] pgmap v1213: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:42 smithi192 bash[24608]: cluster 2024-09-13T23:04:41.952089+0000 osd.2 (osd.2) 365 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T23:04:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:42 smithi192 bash[24608]: cluster 2024-09-13T23:04:41.954630+0000 osd.2 (osd.2) 366 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T23:04:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:42 smithi192 bash[24608]: cluster 2024-09-13T23:04:42.232537+0000 osd.3 (osd.3) 515 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:43.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:42 smithi192 bash[24608]: cluster 2024-09-13T23:04:42.232659+0000 osd.3 (osd.3) 516 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:43.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[22840]: cluster 2024-09-13T23:04:41.101651+0000 mgr.y (mgr.14613) 1000 : cluster [DBG] pgmap v1213: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[22840]: cluster 2024-09-13T23:04:41.952089+0000 osd.2 (osd.2) 365 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[22840]: cluster 2024-09-13T23:04:41.954630+0000 osd.2 (osd.2) 366 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[22840]: cluster 2024-09-13T23:04:42.232537+0000 osd.3 (osd.3) 515 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[22840]: cluster 2024-09-13T23:04:42.232659+0000 osd.3 (osd.3) 516 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[29889]: cluster 2024-09-13T23:04:41.101651+0000 mgr.y (mgr.14613) 1000 : cluster [DBG] pgmap v1213: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[29889]: cluster 2024-09-13T23:04:41.952089+0000 osd.2 (osd.2) 365 : cluster [DBG] 2.e deep-scrub starts 2024-09-13T23:04:43.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[29889]: cluster 2024-09-13T23:04:41.954630+0000 osd.2 (osd.2) 366 : cluster [DBG] 2.e deep-scrub ok 2024-09-13T23:04:43.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[29889]: cluster 2024-09-13T23:04:42.232537+0000 osd.3 (osd.3) 515 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:43.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:42 smithi178 bash[29889]: cluster 2024-09-13T23:04:42.232659+0000 osd.3 (osd.3) 516 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:44.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:42.687385+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2024-09-13T23:04:44.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:42.732082+0000 osd.4 (osd.4) 346 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T23:04:44.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:42.736044+0000 osd.4 (osd.4) 347 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T23:04:44.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.284478+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:44.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.284640+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:44.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.432731+0000 osd.1 (osd.1) 514 : cluster [DBG] 4.2 deep-scrub starts 2024-09-13T23:04:44.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:43 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.435844+0000 osd.1 (osd.1) 515 : cluster [DBG] 4.2 deep-scrub ok 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:42.687385+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:42.732082+0000 osd.4 (osd.4) 346 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:42.736044+0000 osd.4 (osd.4) 347 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.284478+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.284640+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.432731+0000 osd.1 (osd.1) 514 : cluster [DBG] 4.2 deep-scrub starts 2024-09-13T23:04:44.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.435844+0000 osd.1 (osd.1) 515 : cluster [DBG] 4.2 deep-scrub ok 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:42.687385+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:42.732082+0000 osd.4 (osd.4) 346 : cluster [DBG] 2.f deep-scrub starts 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:42.736044+0000 osd.4 (osd.4) 347 : cluster [DBG] 2.f deep-scrub ok 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.284478+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.284640+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.432731+0000 osd.1 (osd.1) 514 : cluster [DBG] 4.2 deep-scrub starts 2024-09-13T23:04:44.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:43 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.435844+0000 osd.1 (osd.1) 515 : cluster [DBG] 4.2 deep-scrub ok 2024-09-13T23:04:45.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:44 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.102322+0000 mgr.y (mgr.14613) 1001 : cluster [DBG] pgmap v1215: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:45.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:44 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.654856+0000 osd.6 (osd.6) 388 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:04:45.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:44 smithi192 bash[24608]: cluster 2024-09-13T23:04:43.657450+0000 osd.6 (osd.6) 389 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:04:45.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:44 smithi192 bash[24608]: cluster 2024-09-13T23:04:44.263891+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2024-09-13T23:04:45.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.102322+0000 mgr.y (mgr.14613) 1001 : cluster [DBG] pgmap v1215: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.654856+0000 osd.6 (osd.6) 388 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[29889]: cluster 2024-09-13T23:04:43.657450+0000 osd.6 (osd.6) 389 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[29889]: cluster 2024-09-13T23:04:44.263891+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.102322+0000 mgr.y (mgr.14613) 1001 : cluster [DBG] pgmap v1215: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.654856+0000 osd.6 (osd.6) 388 : cluster [DBG] 2.17 scrub starts 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[22840]: cluster 2024-09-13T23:04:43.657450+0000 osd.6 (osd.6) 389 : cluster [DBG] 2.17 scrub ok 2024-09-13T23:04:45.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:44 smithi178 bash[22840]: cluster 2024-09-13T23:04:44.263891+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2024-09-13T23:04:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:45.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:45 smithi178 bash[22840]: cluster 2024-09-13T23:04:44.953502+0000 osd.2 (osd.2) 367 : cluster [DBG] 5.d scrub starts 2024-09-13T23:04:45.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:45 smithi178 bash[22840]: cluster 2024-09-13T23:04:44.956591+0000 osd.2 (osd.2) 368 : cluster [DBG] 5.d scrub ok 2024-09-13T23:04:45.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:45 smithi178 bash[29889]: cluster 2024-09-13T23:04:44.953502+0000 osd.2 (osd.2) 367 : cluster [DBG] 5.d scrub starts 2024-09-13T23:04:45.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:45 smithi178 bash[29889]: cluster 2024-09-13T23:04:44.956591+0000 osd.2 (osd.2) 368 : cluster [DBG] 5.d scrub ok 2024-09-13T23:04:46.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:45 smithi192 bash[24608]: cluster 2024-09-13T23:04:44.953502+0000 osd.2 (osd.2) 367 : cluster [DBG] 5.d scrub starts 2024-09-13T23:04:46.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:45 smithi192 bash[24608]: cluster 2024-09-13T23:04:44.956591+0000 osd.2 (osd.2) 368 : cluster [DBG] 5.d scrub ok 2024-09-13T23:04:46.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:04:46 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:04:47.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:46 smithi192 bash[24608]: cluster 2024-09-13T23:04:45.103447+0000 mgr.y (mgr.14613) 1002 : cluster [DBG] pgmap v1217: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T23:04:47.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:46 smithi192 bash[24608]: cluster 2024-09-13T23:04:45.660135+0000 osd.6 (osd.6) 390 : cluster [DBG] 2.14 deep-scrub starts 2024-09-13T23:04:47.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:46 smithi192 bash[24608]: cluster 2024-09-13T23:04:45.662714+0000 osd.7 (osd.7) 562 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:04:47.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:46 smithi192 bash[24608]: cluster 2024-09-13T23:04:45.662792+0000 osd.6 (osd.6) 391 : cluster [DBG] 2.14 deep-scrub ok 2024-09-13T23:04:47.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:46 smithi192 bash[24608]: cluster 2024-09-13T23:04:45.667050+0000 osd.7 (osd.7) 563 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:04:47.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[22840]: cluster 2024-09-13T23:04:45.103447+0000 mgr.y (mgr.14613) 1002 : cluster [DBG] pgmap v1217: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[22840]: cluster 2024-09-13T23:04:45.660135+0000 osd.6 (osd.6) 390 : cluster [DBG] 2.14 deep-scrub starts 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[22840]: cluster 2024-09-13T23:04:45.662714+0000 osd.7 (osd.7) 562 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[22840]: cluster 2024-09-13T23:04:45.662792+0000 osd.6 (osd.6) 391 : cluster [DBG] 2.14 deep-scrub ok 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[22840]: cluster 2024-09-13T23:04:45.667050+0000 osd.7 (osd.7) 563 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[29889]: cluster 2024-09-13T23:04:45.103447+0000 mgr.y (mgr.14613) 1002 : cluster [DBG] pgmap v1217: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[29889]: cluster 2024-09-13T23:04:45.660135+0000 osd.6 (osd.6) 390 : cluster [DBG] 2.14 deep-scrub starts 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[29889]: cluster 2024-09-13T23:04:45.662714+0000 osd.7 (osd.7) 562 : cluster [DBG] 3.13 scrub starts 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[29889]: cluster 2024-09-13T23:04:45.662792+0000 osd.6 (osd.6) 391 : cluster [DBG] 2.14 deep-scrub ok 2024-09-13T23:04:47.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:46 smithi178 bash[29889]: cluster 2024-09-13T23:04:45.667050+0000 osd.7 (osd.7) 563 : cluster [DBG] 3.13 scrub ok 2024-09-13T23:04:47.714 INFO:teuthology.orchestra.run.smithi178.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T23:04:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:47 smithi192 bash[24608]: audit 2024-09-13T23:04:46.253839+0000 mgr.y (mgr.14613) 1003 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:47 smithi192 bash[24608]: cluster 2024-09-13T23:04:46.918980+0000 osd.0 (osd.0) 513 : cluster [DBG] 6.9 deep-scrub starts 2024-09-13T23:04:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:47 smithi192 bash[24608]: cluster 2024-09-13T23:04:46.921547+0000 osd.0 (osd.0) 514 : cluster [DBG] 6.9 deep-scrub ok 2024-09-13T23:04:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:47 smithi192 bash[24608]: audit 2024-09-13T23:04:47.256542+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.178:0/1447503045' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:04:48.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:47 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.416593+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.a scrub starts 2024-09-13T23:04:48.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:47 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.419040+0000 osd.1 (osd.1) 517 : cluster [DBG] 2.a scrub ok 2024-09-13T23:04:48.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[29889]: audit 2024-09-13T23:04:46.253839+0000 mgr.y (mgr.14613) 1003 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:48.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[29889]: cluster 2024-09-13T23:04:46.918980+0000 osd.0 (osd.0) 513 : cluster [DBG] 6.9 deep-scrub starts 2024-09-13T23:04:48.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[29889]: cluster 2024-09-13T23:04:46.921547+0000 osd.0 (osd.0) 514 : cluster [DBG] 6.9 deep-scrub ok 2024-09-13T23:04:48.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[29889]: audit 2024-09-13T23:04:47.256542+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.178:0/1447503045' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:04:48.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.416593+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.a scrub starts 2024-09-13T23:04:48.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.419040+0000 osd.1 (osd.1) 517 : cluster [DBG] 2.a scrub ok 2024-09-13T23:04:48.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[22840]: audit 2024-09-13T23:04:46.253839+0000 mgr.y (mgr.14613) 1003 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:48.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[22840]: cluster 2024-09-13T23:04:46.918980+0000 osd.0 (osd.0) 513 : cluster [DBG] 6.9 deep-scrub starts 2024-09-13T23:04:48.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[22840]: cluster 2024-09-13T23:04:46.921547+0000 osd.0 (osd.0) 514 : cluster [DBG] 6.9 deep-scrub ok 2024-09-13T23:04:48.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[22840]: audit 2024-09-13T23:04:47.256542+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.178:0/1447503045' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-13T23:04:48.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.416593+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.a scrub starts 2024-09-13T23:04:48.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:47 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.419040+0000 osd.1 (osd.1) 517 : cluster [DBG] 2.a scrub ok 2024-09-13T23:04:48.568 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T23:04:48.569 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T23:04:48.569 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T23:04:48.569 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T23:04:48.569 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T23:04:48.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T23:04:48.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.104053+0000 mgr.y (mgr.14613) 1004 : cluster [DBG] pgmap v1218: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.650848+0000 osd.7 (osd.7) 564 : cluster [DBG] 3.f scrub starts 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.655671+0000 osd.7 (osd.7) 565 : cluster [DBG] 3.f scrub ok 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.660416+0000 osd.6 (osd.6) 392 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.662774+0000 osd.6 (osd.6) 393 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: audit 2024-09-13T23:04:47.715835+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.178:0/1447503045' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[29889]: cluster 2024-09-13T23:04:47.722296+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.104053+0000 mgr.y (mgr.14613) 1004 : cluster [DBG] pgmap v1218: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.650848+0000 osd.7 (osd.7) 564 : cluster [DBG] 3.f scrub starts 2024-09-13T23:04:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.655671+0000 osd.7 (osd.7) 565 : cluster [DBG] 3.f scrub ok 2024-09-13T23:04:48.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.660416+0000 osd.6 (osd.6) 392 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:04:48.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.662774+0000 osd.6 (osd.6) 393 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:04:48.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: audit 2024-09-13T23:04:47.715835+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.178:0/1447503045' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:04:48.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:48 smithi178 bash[22840]: cluster 2024-09-13T23:04:47.722296+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2024-09-13T23:04:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.104053+0000 mgr.y (mgr.14613) 1004 : cluster [DBG] pgmap v1218: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-09-13T23:04:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.650848+0000 osd.7 (osd.7) 564 : cluster [DBG] 3.f scrub starts 2024-09-13T23:04:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.655671+0000 osd.7 (osd.7) 565 : cluster [DBG] 3.f scrub ok 2024-09-13T23:04:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.660416+0000 osd.6 (osd.6) 392 : cluster [DBG] 2.1a scrub starts 2024-09-13T23:04:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.662774+0000 osd.6 (osd.6) 393 : cluster [DBG] 2.1a scrub ok 2024-09-13T23:04:49.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: audit 2024-09-13T23:04:47.715835+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.178:0/1447503045' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T23:04:49.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:48 smithi192 bash[24608]: cluster 2024-09-13T23:04:47.722296+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2024-09-13T23:04:50.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:49 smithi192 bash[24608]: cluster 2024-09-13T23:04:48.695445+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:04:50.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:49 smithi192 bash[24608]: cluster 2024-09-13T23:04:48.697586+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:04:50.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:04:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:04:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:04:50.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:49 smithi178 bash[29889]: cluster 2024-09-13T23:04:48.695445+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:04:50.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:49 smithi178 bash[29889]: cluster 2024-09-13T23:04:48.697586+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:04:50.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:49 smithi178 bash[22840]: cluster 2024-09-13T23:04:48.695445+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.5 scrub starts 2024-09-13T23:04:50.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:49 smithi178 bash[22840]: cluster 2024-09-13T23:04:48.697586+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.5 scrub ok 2024-09-13T23:04:51.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.104999+0000 mgr.y (mgr.14613) 1005 : cluster [DBG] pgmap v1220: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-09-13T23:04:51.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.457053+0000 osd.5 (osd.5) 581 : cluster [DBG] 1.a scrub starts 2024-09-13T23:04:51.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.459435+0000 osd.5 (osd.5) 582 : cluster [DBG] 1.a scrub ok 2024-09-13T23:04:51.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.664069+0000 osd.7 (osd.7) 568 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:04:51.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.666433+0000 osd.7 (osd.7) 569 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:04:51.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.668495+0000 osd.4 (osd.4) 348 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:04:51.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: cluster 2024-09-13T23:04:49.672603+0000 osd.4 (osd.4) 349 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:04:51.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:50 smithi192 bash[24608]: audit 2024-09-13T23:04:50.026381+0000 mon.b (mon.1) 319 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:51.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.104999+0000 mgr.y (mgr.14613) 1005 : cluster [DBG] pgmap v1220: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.457053+0000 osd.5 (osd.5) 581 : cluster [DBG] 1.a scrub starts 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.459435+0000 osd.5 (osd.5) 582 : cluster [DBG] 1.a scrub ok 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.664069+0000 osd.7 (osd.7) 568 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.666433+0000 osd.7 (osd.7) 569 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.668495+0000 osd.4 (osd.4) 348 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: cluster 2024-09-13T23:04:49.672603+0000 osd.4 (osd.4) 349 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[29889]: audit 2024-09-13T23:04:50.026381+0000 mon.b (mon.1) 319 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.104999+0000 mgr.y (mgr.14613) 1005 : cluster [DBG] pgmap v1220: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.457053+0000 osd.5 (osd.5) 581 : cluster [DBG] 1.a scrub starts 2024-09-13T23:04:51.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.459435+0000 osd.5 (osd.5) 582 : cluster [DBG] 1.a scrub ok 2024-09-13T23:04:51.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.664069+0000 osd.7 (osd.7) 568 : cluster [DBG] 1.2 scrub starts 2024-09-13T23:04:51.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.666433+0000 osd.7 (osd.7) 569 : cluster [DBG] 1.2 scrub ok 2024-09-13T23:04:51.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.668495+0000 osd.4 (osd.4) 348 : cluster [DBG] 3.14 scrub starts 2024-09-13T23:04:51.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: cluster 2024-09-13T23:04:49.672603+0000 osd.4 (osd.4) 349 : cluster [DBG] 3.14 scrub ok 2024-09-13T23:04:51.183 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:50 smithi178 bash[22840]: audit 2024-09-13T23:04:50.026381+0000 mon.b (mon.1) 319 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:04:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:51.106269+0000 mgr.y (mgr.14613) 1006 : cluster [DBG] pgmap v1221: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:51.718650+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.8 scrub starts 2024-09-13T23:04:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:51.723056+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.8 scrub ok 2024-09-13T23:04:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:51.846756+0000 osd.2 (osd.2) 369 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T23:04:53.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:51.852829+0000 osd.2 (osd.2) 370 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T23:04:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:52.453875+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.1d deep-scrub starts 2024-09-13T23:04:53.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:52 smithi192 bash[24608]: cluster 2024-09-13T23:04:52.456398+0000 osd.1 (osd.1) 519 : cluster [DBG] 6.1d deep-scrub ok 2024-09-13T23:04:53.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:51.106269+0000 mgr.y (mgr.14613) 1006 : cluster [DBG] pgmap v1221: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:51.718650+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.8 scrub starts 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:51.723056+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.8 scrub ok 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:51.846756+0000 osd.2 (osd.2) 369 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:51.852829+0000 osd.2 (osd.2) 370 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:52.453875+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.1d deep-scrub starts 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[29889]: cluster 2024-09-13T23:04:52.456398+0000 osd.1 (osd.1) 519 : cluster [DBG] 6.1d deep-scrub ok 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:51.106269+0000 mgr.y (mgr.14613) 1006 : cluster [DBG] pgmap v1221: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:51.718650+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.8 scrub starts 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:51.723056+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.8 scrub ok 2024-09-13T23:04:53.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:51.846756+0000 osd.2 (osd.2) 369 : cluster [DBG] 3.19 deep-scrub starts 2024-09-13T23:04:53.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:51.852829+0000 osd.2 (osd.2) 370 : cluster [DBG] 3.19 deep-scrub ok 2024-09-13T23:04:53.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:52.453875+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.1d deep-scrub starts 2024-09-13T23:04:53.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:52 smithi178 bash[22840]: cluster 2024-09-13T23:04:52.456398+0000 osd.1 (osd.1) 519 : cluster [DBG] 6.1d deep-scrub ok 2024-09-13T23:04:53.382 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:04:53.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:53 smithi178 bash[29889]: cluster 2024-09-13T23:04:52.781809+0000 osd.0 (osd.0) 515 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:04:53.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:53 smithi178 bash[29889]: cluster 2024-09-13T23:04:52.784812+0000 osd.0 (osd.0) 516 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:04:53.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:53 smithi178 bash[22840]: cluster 2024-09-13T23:04:52.781809+0000 osd.0 (osd.0) 515 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:04:53.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:53 smithi178 bash[22840]: cluster 2024-09-13T23:04:52.784812+0000 osd.0 (osd.0) 516 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:04:54.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:53 smithi192 bash[24608]: cluster 2024-09-13T23:04:52.781809+0000 osd.0 (osd.0) 515 : cluster [DBG] 5.3 scrub starts 2024-09-13T23:04:54.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:53 smithi192 bash[24608]: cluster 2024-09-13T23:04:52.784812+0000 osd.0 (osd.0) 516 : cluster [DBG] 5.3 scrub ok 2024-09-13T23:04:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:54 smithi192 bash[24608]: cluster 2024-09-13T23:04:53.107148+0000 mgr.y (mgr.14613) 1007 : cluster [DBG] pgmap v1222: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:54 smithi192 bash[24608]: cluster 2024-09-13T23:04:53.641945+0000 osd.6 (osd.6) 394 : cluster [DBG] 5.2 deep-scrub starts 2024-09-13T23:04:55.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:54 smithi192 bash[24608]: cluster 2024-09-13T23:04:53.644280+0000 osd.6 (osd.6) 395 : cluster [DBG] 5.2 deep-scrub ok 2024-09-13T23:04:55.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:54 smithi178 bash[29889]: cluster 2024-09-13T23:04:53.107148+0000 mgr.y (mgr.14613) 1007 : cluster [DBG] pgmap v1222: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:55.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:54 smithi178 bash[29889]: cluster 2024-09-13T23:04:53.641945+0000 osd.6 (osd.6) 394 : cluster [DBG] 5.2 deep-scrub starts 2024-09-13T23:04:55.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:54 smithi178 bash[29889]: cluster 2024-09-13T23:04:53.644280+0000 osd.6 (osd.6) 395 : cluster [DBG] 5.2 deep-scrub ok 2024-09-13T23:04:55.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:54 smithi178 bash[22840]: cluster 2024-09-13T23:04:53.107148+0000 mgr.y (mgr.14613) 1007 : cluster [DBG] pgmap v1222: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T23:04:55.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:54 smithi178 bash[22840]: cluster 2024-09-13T23:04:53.641945+0000 osd.6 (osd.6) 394 : cluster [DBG] 5.2 deep-scrub starts 2024-09-13T23:04:55.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:54 smithi178 bash[22840]: cluster 2024-09-13T23:04:53.644280+0000 osd.6 (osd.6) 395 : cluster [DBG] 5.2 deep-scrub ok 2024-09-13T23:04:55.238 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T23:04:55.239 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":524,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","created":"2024-09-13T22:27:31.500176+0000","modified":"2024-09-13T23:04:47.713575+0000","last_up_change":"2024-09-13T22:37:07.372146+0000","last_in_change":"2024-09-13T23:02:23.319337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T22:32:40.780155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"494","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"187","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-13T22:37:39.227580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-13T22:37:41.229251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-09-13T22:37:41.624329+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T22:37:43.199974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T22:37:45.423695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"394f2664-bba1-4b15-ace8-78f70f970cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6801","nonce":1489719354}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6802","nonce":1489719354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6804","nonce":1489719354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6803","nonce":1489719354}]},"public_addr":"172.21.15.178:6801/1489719354","cluster_addr":"172.21.15.178:6802/1489719354","heartbeat_back_addr":"172.21.15.178:6804/1489719354","heartbeat_front_addr":"172.21.15.178:6803/1489719354","state":["exists","up"]},{"osd":1,"uuid":"8e65c08b-6431-4849-a6af-2da9cb1efdfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":492,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6805","nonce":4187084287}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6806","nonce":4187084287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6808","nonce":4187084287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6807","nonce":4187084287}]},"public_addr":"172.21.15.178:6805/4187084287","cluster_addr":"172.21.15.178:6806/4187084287","heartbeat_back_addr":"172.21.15.178:6808/4187084287","heartbeat_front_addr":"172.21.15.178:6807/4187084287","state":["exists","up"]},{"osd":2,"uuid":"40a79075-5102-4f6c-b41a-255102064624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6809","nonce":2303802730}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6810","nonce":2303802730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6812","nonce":2303802730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6811","nonce":2303802730}]},"public_addr":"172.21.15.178:6809/2303802730","cluster_addr":"172.21.15.178:6810/2303802730","heartbeat_back_addr":"172.21.15.178:6812/2303802730","heartbeat_front_addr":"172.21.15.178:6811/2303802730","state":["exists","up"]},{"osd":3,"uuid":"3962c796-6f97-4cc6-a186-4b6ef58d67ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6813","nonce":320675660}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6814","nonce":320675660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6816","nonce":320675660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6815","nonce":320675660}]},"public_addr":"172.21.15.178:6813/320675660","cluster_addr":"172.21.15.178:6814/320675660","heartbeat_back_addr":"172.21.15.178:6816/320675660","heartbeat_front_addr":"172.21.15.178:6815/320675660","state":["exists","up"]},{"osd":4,"uuid":"1b89db2f-9e81-40b1-89bd-8de1dbff133b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":492,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":3593670527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":3593670527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":3593670527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":3593670527}]},"public_addr":"172.21.15.192:6800/3593670527","cluster_addr":"172.21.15.192:6801/3593670527","heartbeat_back_addr":"172.21.15.192:6803/3593670527","heartbeat_front_addr":"172.21.15.192:6802/3593670527","state":["exists","up"]},{"osd":5,"uuid":"efa849a2-d789-4fdd-acc7-04f2c1bebfa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":3271425338}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":3271425338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":3271425338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":3271425338}]},"public_addr":"172.21.15.192:6804/3271425338","cluster_addr":"172.21.15.192:6805/3271425338","heartbeat_back_addr":"172.21.15.192:6807/3271425338","heartbeat_front_addr":"172.21.15.192:6806/3271425338","state":["exists","up"]},{"osd":6,"uuid":"54a26ae9-bf31-49cf-a924-901eed4a0c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":684205162}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":684205162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":684205162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":684205162}]},"public_addr":"172.21.15.192:6808/684205162","cluster_addr":"172.21.15.192:6809/684205162","heartbeat_back_addr":"172.21.15.192:6811/684205162","heartbeat_front_addr":"172.21.15.192:6810/684205162","state":["exists","up"]},{"osd":7,"uuid":"13e146db-61ac-4fe6-8f26-cbf8d1623a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":492,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":1303971738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":1303971738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":1303971738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":1303971738}]},"public_addr":"172.21.15.192:6812/1303971738","cluster_addr":"172.21.15.192:6813/1303971738","heartbeat_back_addr":"172.21.15.192:6815/1303971738","heartbeat_front_addr":"172.21.15.192:6814/1303971738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:43.284647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:17.157509+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:03:45.887148+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:42.232665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:05.281884+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:03:43.429964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:26.935635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:03:50.417700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1f","mappings":[{"from":7,"to":2}]},{"pgid":"3.0","mappings":[{"from":1,"to":5}]},{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.18","mappings":[{"from":0,"to":5}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":7}]},{"pgid":"5.f","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:0/268210503":"2024-09-14T22:27:58.436396+0000","172.21.15.178:6800/2039160385":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/669877409":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/129655740":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/3632788039":"2024-09-14T22:28:30.168687+0000","172.21.15.178:6800/388635774":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3265105147":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2959920382":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/2494442689":"2024-09-14T22:38:32.396576+0000","172.21.15.178:6800/2676020773":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2675812306":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/1834442338":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2835351236":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3836928220":"2024-09-14T22:38:32.396576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T23:04:56.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:55 smithi178 bash[29889]: cluster 2024-09-13T23:04:54.642536+0000 osd.4 (osd.4) 350 : cluster [DBG] 4.1 deep-scrub starts 2024-09-13T23:04:56.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:55 smithi178 bash[29889]: cluster 2024-09-13T23:04:54.645635+0000 osd.4 (osd.4) 351 : cluster [DBG] 4.1 deep-scrub ok 2024-09-13T23:04:56.060 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:55 smithi178 bash[29889]: audit 2024-09-13T23:04:55.237760+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.178:0/4053412604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T23:04:56.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:55 smithi178 bash[22840]: cluster 2024-09-13T23:04:54.642536+0000 osd.4 (osd.4) 350 : cluster [DBG] 4.1 deep-scrub starts 2024-09-13T23:04:56.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:55 smithi178 bash[22840]: cluster 2024-09-13T23:04:54.645635+0000 osd.4 (osd.4) 351 : cluster [DBG] 4.1 deep-scrub ok 2024-09-13T23:04:56.061 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:55 smithi178 bash[22840]: audit 2024-09-13T23:04:55.237760+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.178:0/4053412604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T23:04:56.062 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T23:04:56.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T23:04:56.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:55 smithi192 bash[24608]: cluster 2024-09-13T23:04:54.642536+0000 osd.4 (osd.4) 350 : cluster [DBG] 4.1 deep-scrub starts 2024-09-13T23:04:56.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:55 smithi192 bash[24608]: cluster 2024-09-13T23:04:54.645635+0000 osd.4 (osd.4) 351 : cluster [DBG] 4.1 deep-scrub ok 2024-09-13T23:04:56.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:55 smithi192 bash[24608]: audit 2024-09-13T23:04:55.237760+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.178:0/4053412604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T23:04:56.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:04:57.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:55.108311+0000 mgr.y (mgr.14613) 1008 : cluster [DBG] pgmap v1223: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:57.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:55.690983+0000 osd.7 (osd.7) 570 : cluster [DBG] 3.11 scrub starts 2024-09-13T23:04:57.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:55.695418+0000 osd.7 (osd.7) 571 : cluster [DBG] 3.11 scrub ok 2024-09-13T23:04:57.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:55.777580+0000 osd.3 (osd.3) 519 : cluster [DBG] 2.1e scrub starts 2024-09-13T23:04:57.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:55.779941+0000 osd.3 (osd.3) 520 : cluster [DBG] 2.1e scrub ok 2024-09-13T23:04:57.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:56.396508+0000 osd.5 (osd.5) 583 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:04:57.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:56 smithi192 bash[24608]: cluster 2024-09-13T23:04:56.399140+0000 osd.5 (osd.5) 584 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:04:57.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:55.108311+0000 mgr.y (mgr.14613) 1008 : cluster [DBG] pgmap v1223: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:55.690983+0000 osd.7 (osd.7) 570 : cluster [DBG] 3.11 scrub starts 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:55.695418+0000 osd.7 (osd.7) 571 : cluster [DBG] 3.11 scrub ok 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:55.777580+0000 osd.3 (osd.3) 519 : cluster [DBG] 2.1e scrub starts 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:55.779941+0000 osd.3 (osd.3) 520 : cluster [DBG] 2.1e scrub ok 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:56.396508+0000 osd.5 (osd.5) 583 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[29889]: cluster 2024-09-13T23:04:56.399140+0000 osd.5 (osd.5) 584 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:55.108311+0000 mgr.y (mgr.14613) 1008 : cluster [DBG] pgmap v1223: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:55.690983+0000 osd.7 (osd.7) 570 : cluster [DBG] 3.11 scrub starts 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:55.695418+0000 osd.7 (osd.7) 571 : cluster [DBG] 3.11 scrub ok 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:55.777580+0000 osd.3 (osd.3) 519 : cluster [DBG] 2.1e scrub starts 2024-09-13T23:04:57.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:55.779941+0000 osd.3 (osd.3) 520 : cluster [DBG] 2.1e scrub ok 2024-09-13T23:04:57.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:56.396508+0000 osd.5 (osd.5) 583 : cluster [DBG] 2.3 scrub starts 2024-09-13T23:04:57.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:56 smithi178 bash[22840]: cluster 2024-09-13T23:04:56.399140+0000 osd.5 (osd.5) 584 : cluster [DBG] 2.3 scrub ok 2024-09-13T23:04:58.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:57 smithi192 bash[24608]: audit 2024-09-13T23:04:56.264951+0000 mgr.y (mgr.14613) 1009 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:58.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:57 smithi178 bash[22840]: audit 2024-09-13T23:04:56.264951+0000 mgr.y (mgr.14613) 1009 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:58.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:57 smithi178 bash[29889]: audit 2024-09-13T23:04:56.264951+0000 mgr.y (mgr.14613) 1009 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:04:59.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:58 smithi192 bash[24608]: cluster 2024-09-13T23:04:57.109128+0000 mgr.y (mgr.14613) 1010 : cluster [DBG] pgmap v1224: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:59.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:58 smithi192 bash[24608]: cluster 2024-09-13T23:04:57.406976+0000 osd.5 (osd.5) 585 : cluster [DBG] 5.15 deep-scrub starts 2024-09-13T23:04:59.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:58 smithi192 bash[24608]: cluster 2024-09-13T23:04:57.410529+0000 osd.5 (osd.5) 586 : cluster [DBG] 5.15 deep-scrub ok 2024-09-13T23:04:59.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:58 smithi192 bash[24608]: cluster 2024-09-13T23:04:57.830946+0000 osd.3 (osd.3) 521 : cluster [DBG] 5.0 scrub starts 2024-09-13T23:04:59.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:58 smithi192 bash[24608]: cluster 2024-09-13T23:04:57.834196+0000 osd.3 (osd.3) 522 : cluster [DBG] 5.0 scrub ok 2024-09-13T23:04:59.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[22840]: cluster 2024-09-13T23:04:57.109128+0000 mgr.y (mgr.14613) 1010 : cluster [DBG] pgmap v1224: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:59.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[22840]: cluster 2024-09-13T23:04:57.406976+0000 osd.5 (osd.5) 585 : cluster [DBG] 5.15 deep-scrub starts 2024-09-13T23:04:59.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[22840]: cluster 2024-09-13T23:04:57.410529+0000 osd.5 (osd.5) 586 : cluster [DBG] 5.15 deep-scrub ok 2024-09-13T23:04:59.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[22840]: cluster 2024-09-13T23:04:57.830946+0000 osd.3 (osd.3) 521 : cluster [DBG] 5.0 scrub starts 2024-09-13T23:04:59.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[22840]: cluster 2024-09-13T23:04:57.834196+0000 osd.3 (osd.3) 522 : cluster [DBG] 5.0 scrub ok 2024-09-13T23:04:59.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[29889]: cluster 2024-09-13T23:04:57.109128+0000 mgr.y (mgr.14613) 1010 : cluster [DBG] pgmap v1224: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:04:59.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[29889]: cluster 2024-09-13T23:04:57.406976+0000 osd.5 (osd.5) 585 : cluster [DBG] 5.15 deep-scrub starts 2024-09-13T23:04:59.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[29889]: cluster 2024-09-13T23:04:57.410529+0000 osd.5 (osd.5) 586 : cluster [DBG] 5.15 deep-scrub ok 2024-09-13T23:04:59.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[29889]: cluster 2024-09-13T23:04:57.830946+0000 osd.3 (osd.3) 521 : cluster [DBG] 5.0 scrub starts 2024-09-13T23:04:59.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:58 smithi178 bash[29889]: cluster 2024-09-13T23:04:57.834196+0000 osd.3 (osd.3) 522 : cluster [DBG] 5.0 scrub ok 2024-09-13T23:05:00.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:59 smithi192 bash[24608]: cluster 2024-09-13T23:04:58.698482+0000 osd.6 (osd.6) 396 : cluster [DBG] 5.11 scrub starts 2024-09-13T23:05:00.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:59 smithi192 bash[24608]: cluster 2024-09-13T23:04:58.700823+0000 osd.6 (osd.6) 397 : cluster [DBG] 5.11 scrub ok 2024-09-13T23:05:00.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:59 smithi192 bash[24608]: cluster 2024-09-13T23:04:58.729667+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:05:00.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:59 smithi192 bash[24608]: cluster 2024-09-13T23:04:58.732499+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:05:00.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:59 smithi192 bash[24608]: cluster 2024-09-13T23:04:58.806498+0000 osd.3 (osd.3) 523 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:05:00.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:04:59 smithi192 bash[24608]: cluster 2024-09-13T23:04:58.809024+0000 osd.3 (osd.3) 524 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:05:00.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:04:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:05:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[22840]: cluster 2024-09-13T23:04:58.698482+0000 osd.6 (osd.6) 396 : cluster [DBG] 5.11 scrub starts 2024-09-13T23:05:00.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[22840]: cluster 2024-09-13T23:04:58.700823+0000 osd.6 (osd.6) 397 : cluster [DBG] 5.11 scrub ok 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[22840]: cluster 2024-09-13T23:04:58.729667+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[22840]: cluster 2024-09-13T23:04:58.732499+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[22840]: cluster 2024-09-13T23:04:58.806498+0000 osd.3 (osd.3) 523 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[22840]: cluster 2024-09-13T23:04:58.809024+0000 osd.3 (osd.3) 524 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[29889]: cluster 2024-09-13T23:04:58.698482+0000 osd.6 (osd.6) 396 : cluster [DBG] 5.11 scrub starts 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[29889]: cluster 2024-09-13T23:04:58.700823+0000 osd.6 (osd.6) 397 : cluster [DBG] 5.11 scrub ok 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[29889]: cluster 2024-09-13T23:04:58.729667+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[29889]: cluster 2024-09-13T23:04:58.732499+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[29889]: cluster 2024-09-13T23:04:58.806498+0000 osd.3 (osd.3) 523 : cluster [DBG] 6.11 scrub starts 2024-09-13T23:05:00.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:04:59 smithi178 bash[29889]: cluster 2024-09-13T23:04:58.809024+0000 osd.3 (osd.3) 524 : cluster [DBG] 6.11 scrub ok 2024-09-13T23:05:00.858 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:00 smithi192 bash[24608]: cluster 2024-09-13T23:04:59.110092+0000 mgr.y (mgr.14613) 1011 : cluster [DBG] pgmap v1225: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-09-13T23:05:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:00 smithi192 bash[24608]: cluster 2024-09-13T23:04:59.695270+0000 osd.6 (osd.6) 398 : cluster [DBG] 1.6 deep-scrub starts 2024-09-13T23:05:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:00 smithi192 bash[24608]: cluster 2024-09-13T23:04:59.697727+0000 osd.6 (osd.6) 399 : cluster [DBG] 1.6 deep-scrub ok 2024-09-13T23:05:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:00 smithi192 bash[24608]: cluster 2024-09-13T23:05:00.417516+0000 osd.1 (osd.1) 520 : cluster [DBG] 2.d scrub starts 2024-09-13T23:05:01.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:00 smithi192 bash[24608]: cluster 2024-09-13T23:05:00.419774+0000 osd.1 (osd.1) 521 : cluster [DBG] 2.d scrub ok 2024-09-13T23:05:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[22840]: cluster 2024-09-13T23:04:59.110092+0000 mgr.y (mgr.14613) 1011 : cluster [DBG] pgmap v1225: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-09-13T23:05:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[22840]: cluster 2024-09-13T23:04:59.695270+0000 osd.6 (osd.6) 398 : cluster [DBG] 1.6 deep-scrub starts 2024-09-13T23:05:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[22840]: cluster 2024-09-13T23:04:59.697727+0000 osd.6 (osd.6) 399 : cluster [DBG] 1.6 deep-scrub ok 2024-09-13T23:05:01.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[22840]: cluster 2024-09-13T23:05:00.417516+0000 osd.1 (osd.1) 520 : cluster [DBG] 2.d scrub starts 2024-09-13T23:05:01.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[22840]: cluster 2024-09-13T23:05:00.419774+0000 osd.1 (osd.1) 521 : cluster [DBG] 2.d scrub ok 2024-09-13T23:05:01.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[29889]: cluster 2024-09-13T23:04:59.110092+0000 mgr.y (mgr.14613) 1011 : cluster [DBG] pgmap v1225: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-09-13T23:05:01.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[29889]: cluster 2024-09-13T23:04:59.695270+0000 osd.6 (osd.6) 398 : cluster [DBG] 1.6 deep-scrub starts 2024-09-13T23:05:01.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[29889]: cluster 2024-09-13T23:04:59.697727+0000 osd.6 (osd.6) 399 : cluster [DBG] 1.6 deep-scrub ok 2024-09-13T23:05:01.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[29889]: cluster 2024-09-13T23:05:00.417516+0000 osd.1 (osd.1) 520 : cluster [DBG] 2.d scrub starts 2024-09-13T23:05:01.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:00 smithi178 bash[29889]: cluster 2024-09-13T23:05:00.419774+0000 osd.1 (osd.1) 521 : cluster [DBG] 2.d scrub ok 2024-09-13T23:05:02.676 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T23:05:02.676 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":524,"fsid":"03ae43e2-721f-11ef-bceb-c7b262605968","created":"2024-09-13T22:27:31.500176+0000","modified":"2024-09-13T23:04:47.713575+0000","last_up_change":"2024-09-13T22:37:07.372146+0000","last_in_change":"2024-09-13T23:02:23.319337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T22:32:40.780155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"494","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"187","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-13T22:37:39.227580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-13T22:37:41.229251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-09-13T22:37:41.624329+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T22:37:43.199974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T22:37:45.423695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"394f2664-bba1-4b15-ace8-78f70f970cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6801","nonce":1489719354}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6802","nonce":1489719354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6804","nonce":1489719354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6803","nonce":1489719354}]},"public_addr":"172.21.15.178:6801/1489719354","cluster_addr":"172.21.15.178:6802/1489719354","heartbeat_back_addr":"172.21.15.178:6804/1489719354","heartbeat_front_addr":"172.21.15.178:6803/1489719354","state":["exists","up"]},{"osd":1,"uuid":"8e65c08b-6431-4849-a6af-2da9cb1efdfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":492,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6805","nonce":4187084287}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6806","nonce":4187084287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6808","nonce":4187084287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6807","nonce":4187084287}]},"public_addr":"172.21.15.178:6805/4187084287","cluster_addr":"172.21.15.178:6806/4187084287","heartbeat_back_addr":"172.21.15.178:6808/4187084287","heartbeat_front_addr":"172.21.15.178:6807/4187084287","state":["exists","up"]},{"osd":2,"uuid":"40a79075-5102-4f6c-b41a-255102064624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6809","nonce":2303802730}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6810","nonce":2303802730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6812","nonce":2303802730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6811","nonce":2303802730}]},"public_addr":"172.21.15.178:6809/2303802730","cluster_addr":"172.21.15.178:6810/2303802730","heartbeat_back_addr":"172.21.15.178:6812/2303802730","heartbeat_front_addr":"172.21.15.178:6811/2303802730","state":["exists","up"]},{"osd":3,"uuid":"3962c796-6f97-4cc6-a186-4b6ef58d67ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6813","nonce":320675660}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6814","nonce":320675660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6816","nonce":320675660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.178:6815","nonce":320675660}]},"public_addr":"172.21.15.178:6813/320675660","cluster_addr":"172.21.15.178:6814/320675660","heartbeat_back_addr":"172.21.15.178:6816/320675660","heartbeat_front_addr":"172.21.15.178:6815/320675660","state":["exists","up"]},{"osd":4,"uuid":"1b89db2f-9e81-40b1-89bd-8de1dbff133b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":492,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":3593670527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":3593670527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":3593670527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":3593670527}]},"public_addr":"172.21.15.192:6800/3593670527","cluster_addr":"172.21.15.192:6801/3593670527","heartbeat_back_addr":"172.21.15.192:6803/3593670527","heartbeat_front_addr":"172.21.15.192:6802/3593670527","state":["exists","up"]},{"osd":5,"uuid":"efa849a2-d789-4fdd-acc7-04f2c1bebfa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":3271425338}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":3271425338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":3271425338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":3271425338}]},"public_addr":"172.21.15.192:6804/3271425338","cluster_addr":"172.21.15.192:6805/3271425338","heartbeat_back_addr":"172.21.15.192:6807/3271425338","heartbeat_front_addr":"172.21.15.192:6806/3271425338","state":["exists","up"]},{"osd":6,"uuid":"54a26ae9-bf31-49cf-a924-901eed4a0c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":489,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":684205162}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":684205162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":684205162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":684205162}]},"public_addr":"172.21.15.192:6808/684205162","cluster_addr":"172.21.15.192:6809/684205162","heartbeat_back_addr":"172.21.15.192:6811/684205162","heartbeat_front_addr":"172.21.15.192:6810/684205162","state":["exists","up"]},{"osd":7,"uuid":"13e146db-61ac-4fe6-8f26-cbf8d1623a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":492,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":1303971738}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":1303971738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":1303971738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":1303971738}]},"public_addr":"172.21.15.192:6812/1303971738","cluster_addr":"172.21.15.192:6813/1303971738","heartbeat_back_addr":"172.21.15.192:6815/1303971738","heartbeat_front_addr":"172.21.15.192:6814/1303971738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:43.284647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:17.157509+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:03:45.887148+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:42.232665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:05.281884+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:03:43.429964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:04:26.935635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T23:03:50.417700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1f","mappings":[{"from":7,"to":2}]},{"pgid":"3.0","mappings":[{"from":1,"to":5}]},{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.18","mappings":[{"from":0,"to":5}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":7}]},{"pgid":"5.f","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:0/268210503":"2024-09-14T22:27:58.436396+0000","172.21.15.178:6800/2039160385":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/669877409":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/129655740":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/3632788039":"2024-09-14T22:28:30.168687+0000","172.21.15.178:6800/388635774":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3265105147":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2959920382":"2024-09-14T22:27:58.436396+0000","172.21.15.178:0/2494442689":"2024-09-14T22:38:32.396576+0000","172.21.15.178:6800/2676020773":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2675812306":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/1834442338":"2024-09-14T22:38:32.396576+0000","172.21.15.178:0/2835351236":"2024-09-14T22:28:30.168687+0000","172.21.15.178:0/3836928220":"2024-09-14T22:38:32.396576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T23:05:03.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:02 smithi192 bash[24608]: cluster 2024-09-13T23:05:01.111499+0000 mgr.y (mgr.14613) 1012 : cluster [DBG] pgmap v1226: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:03.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:02 smithi192 bash[24608]: audit 2024-09-13T23:05:02.673926+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.178:0/2371461756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T23:05:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:02 smithi178 bash[22840]: cluster 2024-09-13T23:05:01.111499+0000 mgr.y (mgr.14613) 1012 : cluster [DBG] pgmap v1226: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:03.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:02 smithi178 bash[22840]: audit 2024-09-13T23:05:02.673926+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.178:0/2371461756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T23:05:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:02 smithi178 bash[29889]: cluster 2024-09-13T23:05:01.111499+0000 mgr.y (mgr.14613) 1012 : cluster [DBG] pgmap v1226: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:03.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:02 smithi178 bash[29889]: audit 2024-09-13T23:05:02.673926+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.178:0/2371461756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T23:05:03.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T23:05:03.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T23:05:03.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T23:05:03.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T23:05:03.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T23:05:03.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T23:05:03.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T23:05:03.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T23:05:05.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:04 smithi192 bash[24608]: cluster 2024-09-13T23:05:03.112346+0000 mgr.y (mgr.14613) 1013 : cluster [DBG] pgmap v1227: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:05.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:04 smithi192 bash[24608]: cluster 2024-09-13T23:05:03.826190+0000 osd.3 (osd.3) 525 : cluster [DBG] 6.b scrub starts 2024-09-13T23:05:05.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:04 smithi192 bash[24608]: cluster 2024-09-13T23:05:03.828438+0000 osd.3 (osd.3) 526 : cluster [DBG] 6.b scrub ok 2024-09-13T23:05:05.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:04 smithi192 bash[24608]: cluster 2024-09-13T23:05:04.327504+0000 osd.5 (osd.5) 587 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:05:05.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:04 smithi192 bash[24608]: cluster 2024-09-13T23:05:04.329976+0000 osd.5 (osd.5) 588 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:05:05.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[29889]: cluster 2024-09-13T23:05:03.112346+0000 mgr.y (mgr.14613) 1013 : cluster [DBG] pgmap v1227: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:05.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[29889]: cluster 2024-09-13T23:05:03.826190+0000 osd.3 (osd.3) 525 : cluster [DBG] 6.b scrub starts 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[29889]: cluster 2024-09-13T23:05:03.828438+0000 osd.3 (osd.3) 526 : cluster [DBG] 6.b scrub ok 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[29889]: cluster 2024-09-13T23:05:04.327504+0000 osd.5 (osd.5) 587 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[29889]: cluster 2024-09-13T23:05:04.329976+0000 osd.5 (osd.5) 588 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[22840]: cluster 2024-09-13T23:05:03.112346+0000 mgr.y (mgr.14613) 1013 : cluster [DBG] pgmap v1227: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[22840]: cluster 2024-09-13T23:05:03.826190+0000 osd.3 (osd.3) 525 : cluster [DBG] 6.b scrub starts 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[22840]: cluster 2024-09-13T23:05:03.828438+0000 osd.3 (osd.3) 526 : cluster [DBG] 6.b scrub ok 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[22840]: cluster 2024-09-13T23:05:04.327504+0000 osd.5 (osd.5) 587 : cluster [DBG] 5.1b scrub starts 2024-09-13T23:05:05.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:04 smithi178 bash[22840]: cluster 2024-09-13T23:05:04.329976+0000 osd.5 (osd.5) 588 : cluster [DBG] 5.1b scrub ok 2024-09-13T23:05:06.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:05 smithi192 bash[24608]: cluster 2024-09-13T23:05:04.763554+0000 osd.7 (osd.7) 574 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:05:06.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:05 smithi192 bash[24608]: cluster 2024-09-13T23:05:04.765950+0000 osd.7 (osd.7) 575 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:05:06.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:05 smithi192 bash[24608]: cluster 2024-09-13T23:05:04.836720+0000 osd.3 (osd.3) 527 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:05:06.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:05 smithi192 bash[24608]: cluster 2024-09-13T23:05:04.840843+0000 osd.3 (osd.3) 528 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:05:06.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:05 smithi192 bash[24608]: audit 2024-09-13T23:05:05.054189+0000 mon.b (mon.1) 320 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:06.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:05 smithi192 bash[24608]: audit 2024-09-13T23:05:05.347684+0000 mon.b (mon.1) 321 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:05:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[29889]: cluster 2024-09-13T23:05:04.763554+0000 osd.7 (osd.7) 574 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:05:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[29889]: cluster 2024-09-13T23:05:04.765950+0000 osd.7 (osd.7) 575 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:05:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[29889]: cluster 2024-09-13T23:05:04.836720+0000 osd.3 (osd.3) 527 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:05:06.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[29889]: cluster 2024-09-13T23:05:04.840843+0000 osd.3 (osd.3) 528 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[29889]: audit 2024-09-13T23:05:05.054189+0000 mon.b (mon.1) 320 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[29889]: audit 2024-09-13T23:05:05.347684+0000 mon.b (mon.1) 321 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[22840]: cluster 2024-09-13T23:05:04.763554+0000 osd.7 (osd.7) 574 : cluster [DBG] 5.10 scrub starts 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[22840]: cluster 2024-09-13T23:05:04.765950+0000 osd.7 (osd.7) 575 : cluster [DBG] 5.10 scrub ok 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[22840]: cluster 2024-09-13T23:05:04.836720+0000 osd.3 (osd.3) 527 : cluster [DBG] 3.1e scrub starts 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[22840]: cluster 2024-09-13T23:05:04.840843+0000 osd.3 (osd.3) 528 : cluster [DBG] 3.1e scrub ok 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[22840]: audit 2024-09-13T23:05:05.054189+0000 mon.b (mon.1) 320 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:06.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:05 smithi178 bash[22840]: audit 2024-09-13T23:05:05.347684+0000 mon.b (mon.1) 321 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:05:06.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:05:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.113832+0000 mgr.y (mgr.14613) 1014 : cluster [DBG] pgmap v1228: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:07.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.715211+0000 osd.6 (osd.6) 400 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.717939+0000 osd.6 (osd.6) 401 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.806389+0000 osd.3 (osd.3) 529 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.810483+0000 osd.3 (osd.3) 530 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.867125+0000 osd.0 (osd.0) 517 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: cluster 2024-09-13T23:05:05.868449+0000 osd.0 (osd.0) 518 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: audit 2024-09-13T23:05:06.088569+0000 mon.b (mon.1) 322 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: audit 2024-09-13T23:05:06.090621+0000 mon.b (mon.1) 323 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:05:07.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:06 smithi192 bash[24608]: audit 2024-09-13T23:05:06.100704+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.113832+0000 mgr.y (mgr.14613) 1014 : cluster [DBG] pgmap v1228: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.715211+0000 osd.6 (osd.6) 400 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.717939+0000 osd.6 (osd.6) 401 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.806389+0000 osd.3 (osd.3) 529 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.810483+0000 osd.3 (osd.3) 530 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.867125+0000 osd.0 (osd.0) 517 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: cluster 2024-09-13T23:05:05.868449+0000 osd.0 (osd.0) 518 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: audit 2024-09-13T23:05:06.088569+0000 mon.b (mon.1) 322 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: audit 2024-09-13T23:05:06.090621+0000 mon.b (mon.1) 323 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:05:07.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[22840]: audit 2024-09-13T23:05:06.100704+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.113832+0000 mgr.y (mgr.14613) 1014 : cluster [DBG] pgmap v1228: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.715211+0000 osd.6 (osd.6) 400 : cluster [DBG] 5.1f scrub starts 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.717939+0000 osd.6 (osd.6) 401 : cluster [DBG] 5.1f scrub ok 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.806389+0000 osd.3 (osd.3) 529 : cluster [DBG] 3.2 scrub starts 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.810483+0000 osd.3 (osd.3) 530 : cluster [DBG] 3.2 scrub ok 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.867125+0000 osd.0 (osd.0) 517 : cluster [DBG] 5.5 scrub starts 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: cluster 2024-09-13T23:05:05.868449+0000 osd.0 (osd.0) 518 : cluster [DBG] 5.5 scrub ok 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: audit 2024-09-13T23:05:06.088569+0000 mon.b (mon.1) 322 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: audit 2024-09-13T23:05:06.090621+0000 mon.b (mon.1) 323 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:05:07.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:06 smithi178 bash[29889]: audit 2024-09-13T23:05:06.100704+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:05:08.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:07 smithi192 bash[24608]: audit 2024-09-13T23:05:06.266479+0000 mgr.y (mgr.14613) 1015 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:08.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:07 smithi192 bash[24608]: cluster 2024-09-13T23:05:06.807319+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:05:08.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:07 smithi192 bash[24608]: cluster 2024-09-13T23:05:06.809925+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:05:08.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:07 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.575315+0000 osd.1 (osd.1) 522 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:05:08.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:07 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.578053+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:05:08.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[29889]: audit 2024-09-13T23:05:06.266479+0000 mgr.y (mgr.14613) 1015 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[29889]: cluster 2024-09-13T23:05:06.807319+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[29889]: cluster 2024-09-13T23:05:06.809925+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.575315+0000 osd.1 (osd.1) 522 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.578053+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[22840]: audit 2024-09-13T23:05:06.266479+0000 mgr.y (mgr.14613) 1015 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[22840]: cluster 2024-09-13T23:05:06.807319+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.15 scrub starts 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[22840]: cluster 2024-09-13T23:05:06.809925+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.15 scrub ok 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.575315+0000 osd.1 (osd.1) 522 : cluster [DBG] 1.3 scrub starts 2024-09-13T23:05:08.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:07 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.578053+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub ok 2024-09-13T23:05:08.509 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.509 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.509 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.511 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.511 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.512 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.514 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:08.515 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:09.062 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:08 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.114716+0000 mgr.y (mgr.14613) 1016 : cluster [DBG] pgmap v1229: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:09.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:08 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.809969+0000 osd.5 (osd.5) 589 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:09.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:08 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.810074+0000 osd.5 (osd.5) 590 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:09.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:08 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.893255+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:05:09.063 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:08 smithi178 bash[22840]: cluster 2024-09-13T23:05:07.895777+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:05:09.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:08 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.114716+0000 mgr.y (mgr.14613) 1016 : cluster [DBG] pgmap v1229: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:09.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:08 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.809969+0000 osd.5 (osd.5) 589 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:09.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:08 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.810074+0000 osd.5 (osd.5) 590 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:09.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:08 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.893255+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:05:09.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:08 smithi192 bash[24608]: cluster 2024-09-13T23:05:07.895777+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:05:09.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.114716+0000 mgr.y (mgr.14613) 1016 : cluster [DBG] pgmap v1229: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:09.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.809969+0000 osd.5 (osd.5) 589 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:09.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.810074+0000 osd.5 (osd.5) 590 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.893255+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.10 scrub starts 2024-09-13T23:05:09.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:07.895777+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.10 scrub ok 2024-09-13T23:05:10.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:08.820839+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2024-09-13T23:05:10.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:08.857426+0000 osd.0 (osd.0) 521 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:05:10.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:08.859737+0000 osd.0 (osd.0) 522 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:05:10.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:08.866537+0000 osd.7 (osd.7) 578 : cluster [DBG] 3.e deep-scrub starts 2024-09-13T23:05:10.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:08.877943+0000 osd.7 (osd.7) 579 : cluster [DBG] 3.e deep-scrub ok 2024-09-13T23:05:10.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:09.010257+0000 osd.2 (osd.2) 371 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:10.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:09 smithi192 bash[24608]: cluster 2024-09-13T23:05:09.010359+0000 osd.2 (osd.2) 372 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:10.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:05:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:05:10.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:08.820839+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:08.857426+0000 osd.0 (osd.0) 521 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:08.859737+0000 osd.0 (osd.0) 522 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:08.866537+0000 osd.7 (osd.7) 578 : cluster [DBG] 3.e deep-scrub starts 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:08.877943+0000 osd.7 (osd.7) 579 : cluster [DBG] 3.e deep-scrub ok 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:09.010257+0000 osd.2 (osd.2) 371 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[29889]: cluster 2024-09-13T23:05:09.010359+0000 osd.2 (osd.2) 372 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:08.820839+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:08.857426+0000 osd.0 (osd.0) 521 : cluster [DBG] 5.1e scrub starts 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:08.859737+0000 osd.0 (osd.0) 522 : cluster [DBG] 5.1e scrub ok 2024-09-13T23:05:10.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:08.866537+0000 osd.7 (osd.7) 578 : cluster [DBG] 3.e deep-scrub starts 2024-09-13T23:05:10.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:08.877943+0000 osd.7 (osd.7) 579 : cluster [DBG] 3.e deep-scrub ok 2024-09-13T23:05:10.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:09.010257+0000 osd.2 (osd.2) 371 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:10.182 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:09 smithi178 bash[22840]: cluster 2024-09-13T23:05:09.010359+0000 osd.2 (osd.2) 372 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:11.144 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:10 smithi178 bash[29889]: cluster 2024-09-13T23:05:09.115398+0000 mgr.y (mgr.14613) 1017 : cluster [DBG] pgmap v1231: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:11.144 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:10 smithi178 bash[29889]: cluster 2024-09-13T23:05:09.839955+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2024-09-13T23:05:11.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:10 smithi178 bash[22840]: cluster 2024-09-13T23:05:09.115398+0000 mgr.y (mgr.14613) 1017 : cluster [DBG] pgmap v1231: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:11.145 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:10 smithi178 bash[22840]: cluster 2024-09-13T23:05:09.839955+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2024-09-13T23:05:11.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:10 smithi192 bash[24608]: cluster 2024-09-13T23:05:09.115398+0000 mgr.y (mgr.14613) 1017 : cluster [DBG] pgmap v1231: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:11.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:10 smithi192 bash[24608]: cluster 2024-09-13T23:05:09.839955+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2024-09-13T23:05:12.117 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:11 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.307502+0000 osd.5 (osd.5) 591 : cluster [DBG] 1.1 scrub starts 2024-09-13T23:05:12.118 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:11 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.309792+0000 osd.5 (osd.5) 592 : cluster [DBG] 1.1 scrub ok 2024-09-13T23:05:12.118 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:11 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.307502+0000 osd.5 (osd.5) 591 : cluster [DBG] 1.1 scrub starts 2024-09-13T23:05:12.118 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:11 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.309792+0000 osd.5 (osd.5) 592 : cluster [DBG] 1.1 scrub ok 2024-09-13T23:05:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:11 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.307502+0000 osd.5 (osd.5) 591 : cluster [DBG] 1.1 scrub starts 2024-09-13T23:05:12.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:11 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.309792+0000 osd.5 (osd.5) 592 : cluster [DBG] 1.1 scrub ok 2024-09-13T23:05:13.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.116272+0000 mgr.y (mgr.14613) 1018 : cluster [DBG] pgmap v1233: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:13.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.593426+0000 osd.4 (osd.4) 352 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:05:13.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.595644+0000 osd.4 (osd.4) 353 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:05:13.158 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.794841+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-09-13T23:05:13.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.797675+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-09-13T23:05:13.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.827500+0000 osd.7 (osd.7) 580 : cluster [DBG] 1.9 scrub starts 2024-09-13T23:05:13.159 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:12 smithi192 bash[24608]: cluster 2024-09-13T23:05:11.829808+0000 osd.7 (osd.7) 581 : cluster [DBG] 1.9 scrub ok 2024-09-13T23:05:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.116272+0000 mgr.y (mgr.14613) 1018 : cluster [DBG] pgmap v1233: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:13.180 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.593426+0000 osd.4 (osd.4) 352 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.595644+0000 osd.4 (osd.4) 353 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.794841+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.797675+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.827500+0000 osd.7 (osd.7) 580 : cluster [DBG] 1.9 scrub starts 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[22840]: cluster 2024-09-13T23:05:11.829808+0000 osd.7 (osd.7) 581 : cluster [DBG] 1.9 scrub ok 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.116272+0000 mgr.y (mgr.14613) 1018 : cluster [DBG] pgmap v1233: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.593426+0000 osd.4 (osd.4) 352 : cluster [DBG] 5.1 scrub starts 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.595644+0000 osd.4 (osd.4) 353 : cluster [DBG] 5.1 scrub ok 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.794841+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.797675+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-09-13T23:05:13.181 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.827500+0000 osd.7 (osd.7) 580 : cluster [DBG] 1.9 scrub starts 2024-09-13T23:05:13.182 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:12 smithi178 bash[29889]: cluster 2024-09-13T23:05:11.829808+0000 osd.7 (osd.7) 581 : cluster [DBG] 1.9 scrub ok 2024-09-13T23:05:14.004 INFO:teuthology.orchestra.run.smithi178.stdout:193273528681 2024-09-13T23:05:14.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T23:05:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[29889]: cluster 2024-09-13T23:05:12.752033+0000 osd.3 (osd.3) 531 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:05:14.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[29889]: cluster 2024-09-13T23:05:12.756256+0000 osd.3 (osd.3) 532 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:05:14.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[29889]: cluster 2024-09-13T23:05:13.117480+0000 mgr.y (mgr.14613) 1019 : cluster [DBG] pgmap v1234: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:14.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[29889]: cluster 2024-09-13T23:05:13.282554+0000 osd.5 (osd.5) 593 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:05:14.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[29889]: cluster 2024-09-13T23:05:13.287874+0000 osd.5 (osd.5) 594 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:05:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[22840]: cluster 2024-09-13T23:05:12.752033+0000 osd.3 (osd.3) 531 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:05:14.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[22840]: cluster 2024-09-13T23:05:12.756256+0000 osd.3 (osd.3) 532 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:05:14.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[22840]: cluster 2024-09-13T23:05:13.117480+0000 mgr.y (mgr.14613) 1019 : cluster [DBG] pgmap v1234: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:14.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[22840]: cluster 2024-09-13T23:05:13.282554+0000 osd.5 (osd.5) 593 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:05:14.432 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:14 smithi178 bash[22840]: cluster 2024-09-13T23:05:13.287874+0000 osd.5 (osd.5) 594 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:05:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:14 smithi192 bash[24608]: cluster 2024-09-13T23:05:12.752033+0000 osd.3 (osd.3) 531 : cluster [DBG] 3.7 scrub starts 2024-09-13T23:05:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:14 smithi192 bash[24608]: cluster 2024-09-13T23:05:12.756256+0000 osd.3 (osd.3) 532 : cluster [DBG] 3.7 scrub ok 2024-09-13T23:05:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:14 smithi192 bash[24608]: cluster 2024-09-13T23:05:13.117480+0000 mgr.y (mgr.14613) 1019 : cluster [DBG] pgmap v1234: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:14 smithi192 bash[24608]: cluster 2024-09-13T23:05:13.282554+0000 osd.5 (osd.5) 593 : cluster [DBG] 3.5 scrub starts 2024-09-13T23:05:14.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:14 smithi192 bash[24608]: cluster 2024-09-13T23:05:13.287874+0000 osd.5 (osd.5) 594 : cluster [DBG] 3.5 scrub ok 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:13.834270+0000 osd.7 (osd.7) 582 : cluster [DBG] 2.5 scrub starts 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:13.836571+0000 osd.7 (osd.7) 583 : cluster [DBG] 2.5 scrub ok 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:14.279465+0000 osd.5 (osd.5) 595 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:14.283779+0000 osd.5 (osd.5) 596 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:14.479020+0000 osd.1 (osd.1) 524 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:14.482009+0000 osd.1 (osd.1) 525 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:14.564398+0000 osd.4 (osd.4) 354 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:05:15.484 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[22840]: cluster 2024-09-13T23:05:14.566766+0000 osd.4 (osd.4) 355 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:13.834270+0000 osd.7 (osd.7) 582 : cluster [DBG] 2.5 scrub starts 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:13.836571+0000 osd.7 (osd.7) 583 : cluster [DBG] 2.5 scrub ok 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:14.279465+0000 osd.5 (osd.5) 595 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:14.283779+0000 osd.5 (osd.5) 596 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:14.479020+0000 osd.1 (osd.1) 524 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:14.482009+0000 osd.1 (osd.1) 525 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:14.564398+0000 osd.4 (osd.4) 354 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:05:15.485 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:15 smithi178 bash[29889]: cluster 2024-09-13T23:05:14.566766+0000 osd.4 (osd.4) 355 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:05:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:13.834270+0000 osd.7 (osd.7) 582 : cluster [DBG] 2.5 scrub starts 2024-09-13T23:05:15.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:13.836571+0000 osd.7 (osd.7) 583 : cluster [DBG] 2.5 scrub ok 2024-09-13T23:05:15.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:14.279465+0000 osd.5 (osd.5) 595 : cluster [DBG] 3.1c scrub starts 2024-09-13T23:05:15.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:14.283779+0000 osd.5 (osd.5) 596 : cluster [DBG] 3.1c scrub ok 2024-09-13T23:05:15.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:14.479020+0000 osd.1 (osd.1) 524 : cluster [DBG] 2.4 deep-scrub starts 2024-09-13T23:05:15.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:14.482009+0000 osd.1 (osd.1) 525 : cluster [DBG] 2.4 deep-scrub ok 2024-09-13T23:05:15.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:14.564398+0000 osd.4 (osd.4) 354 : cluster [DBG] 5.18 scrub starts 2024-09-13T23:05:15.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:15 smithi192 bash[24608]: cluster 2024-09-13T23:05:14.566766+0000 osd.4 (osd.4) 355 : cluster [DBG] 5.18 scrub ok 2024-09-13T23:05:16.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:16 smithi178 bash[22840]: cluster 2024-09-13T23:05:15.118944+0000 mgr.y (mgr.14613) 1020 : cluster [DBG] pgmap v1235: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:16.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:16 smithi178 bash[29889]: cluster 2024-09-13T23:05:15.118944+0000 mgr.y (mgr.14613) 1020 : cluster [DBG] pgmap v1235: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:16.460 INFO:teuthology.orchestra.run.smithi178.stdout:55834575251 2024-09-13T23:05:16.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T23:05:16.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:05:16 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:05:16.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:16 smithi192 bash[24608]: cluster 2024-09-13T23:05:15.118944+0000 mgr.y (mgr.14613) 1020 : cluster [DBG] pgmap v1235: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:16.662 INFO:teuthology.orchestra.run.smithi178.stdout:287762809171 2024-09-13T23:05:16.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T23:05:16.688 INFO:teuthology.orchestra.run.smithi178.stdout:34359738782 2024-09-13T23:05:16.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T23:05:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:17 smithi192 bash[24608]: cluster 2024-09-13T23:05:15.802300+0000 osd.3 (osd.3) 533 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:05:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:17 smithi192 bash[24608]: cluster 2024-09-13T23:05:15.805291+0000 osd.3 (osd.3) 534 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:05:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:17 smithi192 bash[24608]: audit 2024-09-13T23:05:16.277838+0000 mgr.y (mgr.14613) 1021 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:17.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:17 smithi192 bash[24608]: cluster 2024-09-13T23:05:16.497572+0000 osd.1 (osd.1) 526 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:05:17.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:17 smithi192 bash[24608]: cluster 2024-09-13T23:05:16.499283+0000 osd.1 (osd.1) 527 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:05:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[22840]: cluster 2024-09-13T23:05:15.802300+0000 osd.3 (osd.3) 533 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:05:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[22840]: cluster 2024-09-13T23:05:15.805291+0000 osd.3 (osd.3) 534 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:05:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[22840]: audit 2024-09-13T23:05:16.277838+0000 mgr.y (mgr.14613) 1021 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:17.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[22840]: cluster 2024-09-13T23:05:16.497572+0000 osd.1 (osd.1) 526 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:05:17.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[22840]: cluster 2024-09-13T23:05:16.499283+0000 osd.1 (osd.1) 527 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:05:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[29889]: cluster 2024-09-13T23:05:15.802300+0000 osd.3 (osd.3) 533 : cluster [DBG] 4.0 scrub starts 2024-09-13T23:05:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[29889]: cluster 2024-09-13T23:05:15.805291+0000 osd.3 (osd.3) 534 : cluster [DBG] 4.0 scrub ok 2024-09-13T23:05:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[29889]: audit 2024-09-13T23:05:16.277838+0000 mgr.y (mgr.14613) 1021 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[29889]: cluster 2024-09-13T23:05:16.497572+0000 osd.1 (osd.1) 526 : cluster [DBG] 5.19 scrub starts 2024-09-13T23:05:17.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:17 smithi178 bash[29889]: cluster 2024-09-13T23:05:16.499283+0000 osd.1 (osd.1) 527 : cluster [DBG] 5.19 scrub ok 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[29889]: cluster 2024-09-13T23:05:16.787096+0000 osd.7 (osd.7) 584 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[29889]: cluster 2024-09-13T23:05:16.790116+0000 osd.7 (osd.7) 585 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[29889]: cluster 2024-09-13T23:05:17.120002+0000 mgr.y (mgr.14613) 1022 : cluster [DBG] pgmap v1236: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[29889]: cluster 2024-09-13T23:05:17.672270+0000 osd.4 (osd.4) 356 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[29889]: cluster 2024-09-13T23:05:17.676451+0000 osd.4 (osd.4) 357 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[22840]: cluster 2024-09-13T23:05:16.787096+0000 osd.7 (osd.7) 584 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[22840]: cluster 2024-09-13T23:05:16.790116+0000 osd.7 (osd.7) 585 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:05:18.523 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[22840]: cluster 2024-09-13T23:05:17.120002+0000 mgr.y (mgr.14613) 1022 : cluster [DBG] pgmap v1236: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T23:05:18.524 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[22840]: cluster 2024-09-13T23:05:17.672270+0000 osd.4 (osd.4) 356 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:05:18.524 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:18 smithi178 bash[22840]: cluster 2024-09-13T23:05:17.676451+0000 osd.4 (osd.4) 357 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:05:18.524 INFO:teuthology.orchestra.run.smithi178.stdout:146028888436 2024-09-13T23:05:18.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T23:05:18.587 INFO:teuthology.orchestra.run.smithi178.stdout:244813136222 2024-09-13T23:05:18.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T23:05:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:18 smithi192 bash[24608]: cluster 2024-09-13T23:05:16.787096+0000 osd.7 (osd.7) 584 : cluster [DBG] 5.9 scrub starts 2024-09-13T23:05:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:18 smithi192 bash[24608]: cluster 2024-09-13T23:05:16.790116+0000 osd.7 (osd.7) 585 : cluster [DBG] 5.9 scrub ok 2024-09-13T23:05:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:18 smithi192 bash[24608]: cluster 2024-09-13T23:05:17.120002+0000 mgr.y (mgr.14613) 1022 : cluster [DBG] pgmap v1236: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T23:05:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:18 smithi192 bash[24608]: cluster 2024-09-13T23:05:17.672270+0000 osd.4 (osd.4) 356 : cluster [DBG] 3.1a scrub starts 2024-09-13T23:05:18.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:18 smithi192 bash[24608]: cluster 2024-09-13T23:05:17.676451+0000 osd.4 (osd.4) 357 : cluster [DBG] 3.1a scrub ok 2024-09-13T23:05:18.724 INFO:teuthology.orchestra.run.smithi178.stdout:120259084670 2024-09-13T23:05:18.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T23:05:18.740 INFO:teuthology.orchestra.run.smithi178.stdout:81604379018 2024-09-13T23:05:18.740 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T23:05:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:19 smithi192 bash[24608]: cluster 2024-09-13T23:05:17.774359+0000 osd.7 (osd.7) 586 : cluster [DBG] 2.b scrub starts 2024-09-13T23:05:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:19 smithi192 bash[24608]: cluster 2024-09-13T23:05:17.776697+0000 osd.7 (osd.7) 587 : cluster [DBG] 2.b scrub ok 2024-09-13T23:05:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:19 smithi192 bash[24608]: cluster 2024-09-13T23:05:18.309508+0000 osd.5 (osd.5) 597 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T23:05:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:19 smithi192 bash[24608]: cluster 2024-09-13T23:05:18.311910+0000 osd.5 (osd.5) 598 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T23:05:19.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:19 smithi192 bash[24608]: cluster 2024-09-13T23:05:18.697152+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.e scrub starts 2024-09-13T23:05:19.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:19 smithi192 bash[24608]: cluster 2024-09-13T23:05:18.699796+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.e scrub ok 2024-09-13T23:05:19.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[29889]: cluster 2024-09-13T23:05:17.774359+0000 osd.7 (osd.7) 586 : cluster [DBG] 2.b scrub starts 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[29889]: cluster 2024-09-13T23:05:17.776697+0000 osd.7 (osd.7) 587 : cluster [DBG] 2.b scrub ok 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[29889]: cluster 2024-09-13T23:05:18.309508+0000 osd.5 (osd.5) 597 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[29889]: cluster 2024-09-13T23:05:18.311910+0000 osd.5 (osd.5) 598 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[29889]: cluster 2024-09-13T23:05:18.697152+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.e scrub starts 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[29889]: cluster 2024-09-13T23:05:18.699796+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.e scrub ok 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[22840]: cluster 2024-09-13T23:05:17.774359+0000 osd.7 (osd.7) 586 : cluster [DBG] 2.b scrub starts 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[22840]: cluster 2024-09-13T23:05:17.776697+0000 osd.7 (osd.7) 587 : cluster [DBG] 2.b scrub ok 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[22840]: cluster 2024-09-13T23:05:18.309508+0000 osd.5 (osd.5) 597 : cluster [DBG] 2.2 deep-scrub starts 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[22840]: cluster 2024-09-13T23:05:18.311910+0000 osd.5 (osd.5) 598 : cluster [DBG] 2.2 deep-scrub ok 2024-09-13T23:05:19.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[22840]: cluster 2024-09-13T23:05:18.697152+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.e scrub starts 2024-09-13T23:05:19.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[22840]: cluster 2024-09-13T23:05:18.699796+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.e scrub ok 2024-09-13T23:05:20.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:19 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:05:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:05:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:20 smithi192 bash[24608]: cluster 2024-09-13T23:05:18.605551+0000 osd.7 (osd.7) 588 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:20 smithi192 bash[24608]: cluster 2024-09-13T23:05:18.605665+0000 osd.7 (osd.7) 589 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:20 smithi192 bash[24608]: cluster 2024-09-13T23:05:19.121009+0000 mgr.y (mgr.14613) 1023 : cluster [DBG] pgmap v1237: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:20 smithi192 bash[24608]: cluster 2024-09-13T23:05:19.298924+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2024-09-13T23:05:20.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:20 smithi192 bash[24608]: audit 2024-09-13T23:05:20.082856+0000 mon.b (mon.1) 324 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[29889]: cluster 2024-09-13T23:05:18.605551+0000 osd.7 (osd.7) 588 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:20.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[29889]: cluster 2024-09-13T23:05:18.605665+0000 osd.7 (osd.7) 589 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[29889]: cluster 2024-09-13T23:05:19.121009+0000 mgr.y (mgr.14613) 1023 : cluster [DBG] pgmap v1237: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[29889]: cluster 2024-09-13T23:05:19.298924+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[29889]: audit 2024-09-13T23:05:20.082856+0000 mon.b (mon.1) 324 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[22840]: cluster 2024-09-13T23:05:18.605551+0000 osd.7 (osd.7) 588 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[22840]: cluster 2024-09-13T23:05:18.605665+0000 osd.7 (osd.7) 589 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[22840]: cluster 2024-09-13T23:05:19.121009+0000 mgr.y (mgr.14613) 1023 : cluster [DBG] pgmap v1237: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[22840]: cluster 2024-09-13T23:05:19.298924+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2024-09-13T23:05:20.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:20 smithi178 bash[22840]: audit 2024-09-13T23:05:20.082856+0000 mon.b (mon.1) 324 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:20.826 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:20.829 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:20.829 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:20.830 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:21.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:21 smithi178 bash[22840]: cluster 2024-09-13T23:05:19.908294+0000 osd.3 (osd.3) 535 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:05:21.615 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:21 smithi178 bash[22840]: cluster 2024-09-13T23:05:19.911105+0000 osd.3 (osd.3) 536 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:05:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:21 smithi192 bash[24608]: cluster 2024-09-13T23:05:19.908294+0000 osd.3 (osd.3) 535 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:05:21.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:21 smithi192 bash[24608]: cluster 2024-09-13T23:05:19.911105+0000 osd.3 (osd.3) 536 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:05:21.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:21 smithi178 bash[29889]: cluster 2024-09-13T23:05:19.908294+0000 osd.3 (osd.3) 535 : cluster [DBG] 6.13 scrub starts 2024-09-13T23:05:21.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:21 smithi178 bash[29889]: cluster 2024-09-13T23:05:19.911105+0000 osd.3 (osd.3) 536 : cluster [DBG] 6.13 scrub ok 2024-09-13T23:05:22.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:22 smithi192 bash[24608]: cluster 2024-09-13T23:05:21.122067+0000 mgr.y (mgr.14613) 1024 : cluster [DBG] pgmap v1239: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:22.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:22 smithi178 bash[29889]: cluster 2024-09-13T23:05:21.122067+0000 mgr.y (mgr.14613) 1024 : cluster [DBG] pgmap v1239: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:22.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:22 smithi178 bash[22840]: cluster 2024-09-13T23:05:21.122067+0000 mgr.y (mgr.14613) 1024 : cluster [DBG] pgmap v1239: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:22.876 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:22.876 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:22.877 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:22.877 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:23 smithi192 bash[24608]: cluster 2024-09-13T23:05:22.703608+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:05:23.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:23 smithi192 bash[24608]: cluster 2024-09-13T23:05:22.705576+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:05:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:23 smithi178 bash[29889]: cluster 2024-09-13T23:05:22.703608+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:05:23.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:23 smithi178 bash[29889]: cluster 2024-09-13T23:05:22.705576+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:05:23.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:23 smithi178 bash[22840]: cluster 2024-09-13T23:05:22.703608+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.2 scrub starts 2024-09-13T23:05:23.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:23 smithi178 bash[22840]: cluster 2024-09-13T23:05:22.705576+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.2 scrub ok 2024-09-13T23:05:24.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:24 smithi192 bash[24608]: cluster 2024-09-13T23:05:23.122942+0000 mgr.y (mgr.14613) 1025 : cluster [DBG] pgmap v1240: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:24.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:24 smithi178 bash[29889]: cluster 2024-09-13T23:05:23.122942+0000 mgr.y (mgr.14613) 1025 : cluster [DBG] pgmap v1240: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:24.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:24 smithi178 bash[22840]: cluster 2024-09-13T23:05:23.122942+0000 mgr.y (mgr.14613) 1025 : cluster [DBG] pgmap v1240: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:24.722 INFO:teuthology.orchestra.run.smithi178.stdout:55834575253 2024-09-13T23:05:25.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:25 smithi192 bash[24608]: audit 2024-09-13T23:05:24.723149+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? v1:172.21.15.178:0/2202315482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T23:05:25.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:25 smithi178 bash[29889]: audit 2024-09-13T23:05:24.723149+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? v1:172.21.15.178:0/2202315482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T23:05:25.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:25 smithi178 bash[22840]: audit 2024-09-13T23:05:24.723149+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? v1:172.21.15.178:0/2202315482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T23:05:26.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:24.910981+0000 osd.3 (osd.3) 537 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:05:26.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:24.913947+0000 osd.3 (osd.3) 538 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:05:26.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:24.955804+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:05:26.430 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:24.958068+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:25.124390+0000 mgr.y (mgr.14613) 1026 : cluster [DBG] pgmap v1241: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:25.563456+0000 osd.1 (osd.1) 528 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:25.565902+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:25.864111+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.1d deep-scrub starts 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[22840]: cluster 2024-09-13T23:05:25.869273+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.1d deep-scrub ok 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:24.910981+0000 osd.3 (osd.3) 537 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:24.913947+0000 osd.3 (osd.3) 538 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:24.955804+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:24.958068+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:05:26.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:25.124390+0000 mgr.y (mgr.14613) 1026 : cluster [DBG] pgmap v1241: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:25.563456+0000 osd.1 (osd.1) 528 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:05:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:25.565902+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:05:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:25.864111+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.1d deep-scrub starts 2024-09-13T23:05:26.432 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:26 smithi178 bash[29889]: cluster 2024-09-13T23:05:25.869273+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.1d deep-scrub ok 2024-09-13T23:05:26.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575251 got 55834575253 for osd.1 2024-09-13T23:05:26.477 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:26.505 INFO:teuthology.orchestra.run.smithi178.stdout:34359738784 2024-09-13T23:05:26.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:05:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:24.910981+0000 osd.3 (osd.3) 537 : cluster [DBG] 1.4 scrub starts 2024-09-13T23:05:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:24.913947+0000 osd.3 (osd.3) 538 : cluster [DBG] 1.4 scrub ok 2024-09-13T23:05:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:24.955804+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.10 scrub starts 2024-09-13T23:05:26.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:24.958068+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.10 scrub ok 2024-09-13T23:05:26.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:25.124390+0000 mgr.y (mgr.14613) 1026 : cluster [DBG] pgmap v1241: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:26.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:25.563456+0000 osd.1 (osd.1) 528 : cluster [DBG] 2.15 scrub starts 2024-09-13T23:05:26.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:25.565902+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 scrub ok 2024-09-13T23:05:26.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:25.864111+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.1d deep-scrub starts 2024-09-13T23:05:26.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:26 smithi192 bash[24608]: cluster 2024-09-13T23:05:25.869273+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.1d deep-scrub ok 2024-09-13T23:05:27.320 INFO:teuthology.orchestra.run.smithi178.stdout:193273528684 2024-09-13T23:05:27.321 INFO:teuthology.orchestra.run.smithi178.stdout:287762809174 2024-09-13T23:05:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:27 smithi192 bash[24608]: audit 2024-09-13T23:05:26.288946+0000 mgr.y (mgr.14613) 1027 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:27 smithi192 bash[24608]: audit 2024-09-13T23:05:26.507117+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.178:0/2765146618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T23:05:27.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:27 smithi192 bash[24608]: cluster 2024-09-13T23:05:26.902302+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.d scrub starts 2024-09-13T23:05:27.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:27 smithi192 bash[24608]: cluster 2024-09-13T23:05:26.907163+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.d scrub ok 2024-09-13T23:05:27.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:27 smithi192 bash[24608]: audit 2024-09-13T23:05:27.322096+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.178:0/1481914639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T23:05:27.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:27 smithi192 bash[24608]: audit 2024-09-13T23:05:27.322102+0000 mon.a (mon.0) 1950 : audit [DBG] from='client.? v1:172.21.15.178:0/3896902482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T23:05:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[29889]: audit 2024-09-13T23:05:26.288946+0000 mgr.y (mgr.14613) 1027 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:27.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[29889]: audit 2024-09-13T23:05:26.507117+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.178:0/2765146618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[29889]: cluster 2024-09-13T23:05:26.902302+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.d scrub starts 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[29889]: cluster 2024-09-13T23:05:26.907163+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.d scrub ok 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[29889]: audit 2024-09-13T23:05:27.322096+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.178:0/1481914639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[29889]: audit 2024-09-13T23:05:27.322102+0000 mon.a (mon.0) 1950 : audit [DBG] from='client.? v1:172.21.15.178:0/3896902482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[22840]: audit 2024-09-13T23:05:26.288946+0000 mgr.y (mgr.14613) 1027 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[22840]: audit 2024-09-13T23:05:26.507117+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.178:0/2765146618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[22840]: cluster 2024-09-13T23:05:26.902302+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.d scrub starts 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[22840]: cluster 2024-09-13T23:05:26.907163+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.d scrub ok 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[22840]: audit 2024-09-13T23:05:27.322096+0000 mon.c (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.178:0/1481914639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T23:05:27.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:27 smithi178 bash[22840]: audit 2024-09-13T23:05:27.322102+0000 mon.a (mon.0) 1950 : audit [DBG] from='client.? v1:172.21.15.178:0/3896902482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T23:05:28.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:28 smithi192 bash[24608]: cluster 2024-09-13T23:05:27.124945+0000 mgr.y (mgr.14613) 1028 : cluster [DBG] pgmap v1242: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-13T23:05:28.667 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:28 smithi178 bash[22840]: cluster 2024-09-13T23:05:27.124945+0000 mgr.y (mgr.14613) 1028 : cluster [DBG] pgmap v1242: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-13T23:05:28.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:28 smithi178 bash[29889]: cluster 2024-09-13T23:05:27.124945+0000 mgr.y (mgr.14613) 1028 : cluster [DBG] pgmap v1242: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-13T23:05:29.044 INFO:teuthology.orchestra.run.smithi178.stdout:120259084672 2024-09-13T23:05:29.082 INFO:teuthology.orchestra.run.smithi178.stdout:146028888438 2024-09-13T23:05:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[29889]: cluster 2024-09-13T23:05:27.684348+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:05:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[29889]: cluster 2024-09-13T23:05:27.686825+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:05:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[29889]: cluster 2024-09-13T23:05:28.198382+0000 osd.5 (osd.5) 599 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:05:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[29889]: cluster 2024-09-13T23:05:28.201586+0000 osd.5 (osd.5) 600 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:05:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[29889]: audit 2024-09-13T23:05:29.046000+0000 mon.a (mon.0) 1951 : audit [DBG] from='client.? v1:172.21.15.178:0/1115631308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[29889]: audit 2024-09-13T23:05:29.084145+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.178:0/1224702023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[22840]: cluster 2024-09-13T23:05:27.684348+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[22840]: cluster 2024-09-13T23:05:27.686825+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[22840]: cluster 2024-09-13T23:05:28.198382+0000 osd.5 (osd.5) 599 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[22840]: cluster 2024-09-13T23:05:28.201586+0000 osd.5 (osd.5) 600 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[22840]: audit 2024-09-13T23:05:29.046000+0000 mon.a (mon.0) 1951 : audit [DBG] from='client.? v1:172.21.15.178:0/1115631308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T23:05:29.596 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[22840]: audit 2024-09-13T23:05:29.084145+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.178:0/1224702023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T23:05:29.597 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738782 got 34359738784 for osd.0 2024-09-13T23:05:29.597 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:29 smithi192 bash[24608]: cluster 2024-09-13T23:05:27.684348+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.18 scrub starts 2024-09-13T23:05:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:29 smithi192 bash[24608]: cluster 2024-09-13T23:05:27.686825+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.18 scrub ok 2024-09-13T23:05:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:29 smithi192 bash[24608]: cluster 2024-09-13T23:05:28.198382+0000 osd.5 (osd.5) 599 : cluster [DBG] 1.8 scrub starts 2024-09-13T23:05:29.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:29 smithi192 bash[24608]: cluster 2024-09-13T23:05:28.201586+0000 osd.5 (osd.5) 600 : cluster [DBG] 1.8 scrub ok 2024-09-13T23:05:29.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:29 smithi192 bash[24608]: audit 2024-09-13T23:05:29.046000+0000 mon.a (mon.0) 1951 : audit [DBG] from='client.? v1:172.21.15.178:0/1115631308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T23:05:29.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:29 smithi192 bash[24608]: audit 2024-09-13T23:05:29.084145+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.178:0/1224702023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T23:05:29.963 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:29 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:05:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:05:30.033 INFO:teuthology.orchestra.run.smithi178.stdout:81604379020 2024-09-13T23:05:30.042 INFO:teuthology.orchestra.run.smithi178.stdout:244813136225 2024-09-13T23:05:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: cluster 2024-09-13T23:05:28.925300+0000 osd.3 (osd.3) 539 : cluster [DBG] 3.b scrub starts 2024-09-13T23:05:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: cluster 2024-09-13T23:05:28.927563+0000 osd.3 (osd.3) 540 : cluster [DBG] 3.b scrub ok 2024-09-13T23:05:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.125475+0000 mgr.y (mgr.14613) 1029 : cluster [DBG] pgmap v1243: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-13T23:05:30.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.539396+0000 osd.1 (osd.1) 530 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:05:30.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.541837+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:05:30.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.668567+0000 osd.0 (osd.0) 527 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:05:30.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: audit 2024-09-13T23:05:30.034285+0000 mon.a (mon.0) 1952 : audit [DBG] from='client.? v1:172.21.15.178:0/4129663679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T23:05:30.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:30 smithi192 bash[24608]: audit 2024-09-13T23:05:30.043839+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.178:0/1477964489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: cluster 2024-09-13T23:05:28.925300+0000 osd.3 (osd.3) 539 : cluster [DBG] 3.b scrub starts 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: cluster 2024-09-13T23:05:28.927563+0000 osd.3 (osd.3) 540 : cluster [DBG] 3.b scrub ok 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.125475+0000 mgr.y (mgr.14613) 1029 : cluster [DBG] pgmap v1243: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.539396+0000 osd.1 (osd.1) 530 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.541837+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.668567+0000 osd.0 (osd.0) 527 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: audit 2024-09-13T23:05:30.034285+0000 mon.a (mon.0) 1952 : audit [DBG] from='client.? v1:172.21.15.178:0/4129663679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T23:05:30.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[29889]: audit 2024-09-13T23:05:30.043839+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.178:0/1477964489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: cluster 2024-09-13T23:05:28.925300+0000 osd.3 (osd.3) 539 : cluster [DBG] 3.b scrub starts 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: cluster 2024-09-13T23:05:28.927563+0000 osd.3 (osd.3) 540 : cluster [DBG] 3.b scrub ok 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.125475+0000 mgr.y (mgr.14613) 1029 : cluster [DBG] pgmap v1243: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.539396+0000 osd.1 (osd.1) 530 : cluster [DBG] 5.12 scrub starts 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.541837+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.12 scrub ok 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.668567+0000 osd.0 (osd.0) 527 : cluster [DBG] 3.12 scrub starts 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: audit 2024-09-13T23:05:30.034285+0000 mon.a (mon.0) 1952 : audit [DBG] from='client.? v1:172.21.15.178:0/4129663679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T23:05:30.683 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:30 smithi178 bash[22840]: audit 2024-09-13T23:05:30.043839+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.178:0/1477964489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T23:05:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:31 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.672391+0000 osd.0 (osd.0) 528 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:05:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:31 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.972023+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:05:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:31 smithi192 bash[24608]: cluster 2024-09-13T23:05:29.974943+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:05:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:31 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.121604+0000 osd.6 (osd.6) 402 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:31.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:31 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.121735+0000 osd.6 (osd.6) 403 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:31.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:31 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.350666+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2024-09-13T23:05:31.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.672391+0000 osd.0 (osd.0) 528 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.972023+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[22840]: cluster 2024-09-13T23:05:29.974943+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.121604+0000 osd.6 (osd.6) 402 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.121735+0000 osd.6 (osd.6) 403 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.350666+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.672391+0000 osd.0 (osd.0) 528 : cluster [DBG] 3.12 scrub ok 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.972023+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.14 scrub starts 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[29889]: cluster 2024-09-13T23:05:29.974943+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.14 scrub ok 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.121604+0000 osd.6 (osd.6) 402 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.121735+0000 osd.6 (osd.6) 403 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:31.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:31 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.350666+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2024-09-13T23:05:32.065 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762809171 got 287762809174 for osd.7 2024-09-13T23:05:32.066 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:32.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888436 got 146028888438 for osd.4 2024-09-13T23:05:32.149 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:32.150 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528681 got 193273528684 for osd.5 2024-09-13T23:05:32.150 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:32.245 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084670 got 120259084672 for osd.3 2024-09-13T23:05:32.246 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:32 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.933820+0000 osd.1 (osd.1) 532 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:32 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.933925+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:32 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.986937+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.a scrub starts 2024-09-13T23:05:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:32 smithi192 bash[24608]: cluster 2024-09-13T23:05:30.990391+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.a scrub ok 2024-09-13T23:05:32.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:32 smithi192 bash[24608]: cluster 2024-09-13T23:05:31.126150+0000 mgr.y (mgr.14613) 1030 : cluster [DBG] pgmap v1245: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:32.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:32 smithi192 bash[24608]: cluster 2024-09-13T23:05:31.374507+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2024-09-13T23:05:32.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.933820+0000 osd.1 (osd.1) 532 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:32.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.933925+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:32.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.986937+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.a scrub starts 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[22840]: cluster 2024-09-13T23:05:30.990391+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.a scrub ok 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[22840]: cluster 2024-09-13T23:05:31.126150+0000 mgr.y (mgr.14613) 1030 : cluster [DBG] pgmap v1245: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[22840]: cluster 2024-09-13T23:05:31.374507+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.933820+0000 osd.1 (osd.1) 532 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.933925+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.986937+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.a scrub starts 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[29889]: cluster 2024-09-13T23:05:30.990391+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.a scrub ok 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[29889]: cluster 2024-09-13T23:05:31.126150+0000 mgr.y (mgr.14613) 1030 : cluster [DBG] pgmap v1245: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:32.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:32 smithi178 bash[29889]: cluster 2024-09-13T23:05:31.374507+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2024-09-13T23:05:33.423 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813136222 got 244813136225 for osd.6 2024-09-13T23:05:33.423 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:33.433 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604379018 got 81604379020 for osd.2 2024-09-13T23:05:33.433 DEBUG:teuthology.parallel:result is None 2024-09-13T23:05:33.433 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T23:05:33.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T23:05:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:33 smithi192 bash[24608]: cluster 2024-09-13T23:05:32.208549+0000 osd.5 (osd.5) 601 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:05:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:33 smithi192 bash[24608]: cluster 2024-09-13T23:05:32.213055+0000 osd.5 (osd.5) 602 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:05:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:33 smithi192 bash[24608]: cluster 2024-09-13T23:05:32.613187+0000 osd.0 (osd.0) 529 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:05:33.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:33 smithi192 bash[24608]: cluster 2024-09-13T23:05:32.617265+0000 osd.0 (osd.0) 530 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[29889]: cluster 2024-09-13T23:05:32.208549+0000 osd.5 (osd.5) 601 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[29889]: cluster 2024-09-13T23:05:32.213055+0000 osd.5 (osd.5) 602 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[29889]: cluster 2024-09-13T23:05:32.613187+0000 osd.0 (osd.0) 529 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[29889]: cluster 2024-09-13T23:05:32.617265+0000 osd.0 (osd.0) 530 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[22840]: cluster 2024-09-13T23:05:32.208549+0000 osd.5 (osd.5) 601 : cluster [DBG] 3.0 scrub starts 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[22840]: cluster 2024-09-13T23:05:32.213055+0000 osd.5 (osd.5) 602 : cluster [DBG] 3.0 scrub ok 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[22840]: cluster 2024-09-13T23:05:32.613187+0000 osd.0 (osd.0) 529 : cluster [DBG] 3.6 scrub starts 2024-09-13T23:05:33.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:33 smithi178 bash[22840]: cluster 2024-09-13T23:05:32.617265+0000 osd.0 (osd.0) 530 : cluster [DBG] 3.6 scrub ok 2024-09-13T23:05:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:32.925982+0000 osd.3 (osd.3) 543 : cluster [DBG] 6.c scrub starts 2024-09-13T23:05:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:32.928472+0000 osd.3 (osd.3) 544 : cluster [DBG] 6.c scrub ok 2024-09-13T23:05:34.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.126969+0000 mgr.y (mgr.14613) 1031 : cluster [DBG] pgmap v1247: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:34.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.348423+0000 osd.4 (osd.4) 362 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:34.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.348535+0000 osd.4 (osd.4) 363 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:34.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.551124+0000 osd.4 (osd.4) 364 : cluster [DBG] 6.e scrub starts 2024-09-13T23:05:34.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.553206+0000 osd.4 (osd.4) 365 : cluster [DBG] 6.e scrub ok 2024-09-13T23:05:34.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:34 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.280006+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2024-09-13T23:05:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:32.925982+0000 osd.3 (osd.3) 543 : cluster [DBG] 6.c scrub starts 2024-09-13T23:05:34.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:32.928472+0000 osd.3 (osd.3) 544 : cluster [DBG] 6.c scrub ok 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.126969+0000 mgr.y (mgr.14613) 1031 : cluster [DBG] pgmap v1247: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.348423+0000 osd.4 (osd.4) 362 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.348535+0000 osd.4 (osd.4) 363 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.551124+0000 osd.4 (osd.4) 364 : cluster [DBG] 6.e scrub starts 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.553206+0000 osd.4 (osd.4) 365 : cluster [DBG] 6.e scrub ok 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.280006+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:32.925982+0000 osd.3 (osd.3) 543 : cluster [DBG] 6.c scrub starts 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:32.928472+0000 osd.3 (osd.3) 544 : cluster [DBG] 6.c scrub ok 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.126969+0000 mgr.y (mgr.14613) 1031 : cluster [DBG] pgmap v1247: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:34.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.348423+0000 osd.4 (osd.4) 362 : cluster [DBG] purged_snaps scrub starts 2024-09-13T23:05:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.348535+0000 osd.4 (osd.4) 363 : cluster [DBG] purged_snaps scrub ok 2024-09-13T23:05:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.551124+0000 osd.4 (osd.4) 364 : cluster [DBG] 6.e scrub starts 2024-09-13T23:05:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.553206+0000 osd.4 (osd.4) 365 : cluster [DBG] 6.e scrub ok 2024-09-13T23:05:34.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:34 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.280006+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2024-09-13T23:05:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.562167+0000 osd.6 (osd.6) 404 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T23:05:35.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.564488+0000 osd.6 (osd.6) 405 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.976143+0000 osd.3 (osd.3) 545 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.978719+0000 osd.3 (osd.3) 546 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:33.998436+0000 osd.2 (osd.2) 377 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.001444+0000 osd.2 (osd.2) 378 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.396084+0000 osd.1 (osd.1) 534 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.398653+0000 osd.1 (osd.1) 535 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.525206+0000 osd.4 (osd.4) 366 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.527531+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:05:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.891678+0000 osd.7 (osd.7) 594 : cluster [DBG] 6.3 scrub starts 2024-09-13T23:05:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: cluster 2024-09-13T23:05:34.893971+0000 osd.7 (osd.7) 595 : cluster [DBG] 6.3 scrub ok 2024-09-13T23:05:35.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:35 smithi192 bash[24608]: audit 2024-09-13T23:05:35.111138+0000 mon.b (mon.1) 325 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:35.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.562167+0000 osd.6 (osd.6) 404 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.564488+0000 osd.6 (osd.6) 405 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.976143+0000 osd.3 (osd.3) 545 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.978719+0000 osd.3 (osd.3) 546 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:33.998436+0000 osd.2 (osd.2) 377 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.001444+0000 osd.2 (osd.2) 378 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.396084+0000 osd.1 (osd.1) 534 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.398653+0000 osd.1 (osd.1) 535 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.525206+0000 osd.4 (osd.4) 366 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.527531+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.891678+0000 osd.7 (osd.7) 594 : cluster [DBG] 6.3 scrub starts 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: cluster 2024-09-13T23:05:34.893971+0000 osd.7 (osd.7) 595 : cluster [DBG] 6.3 scrub ok 2024-09-13T23:05:35.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[29889]: audit 2024-09-13T23:05:35.111138+0000 mon.b (mon.1) 325 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.562167+0000 osd.6 (osd.6) 404 : cluster [DBG] 1.7 deep-scrub starts 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.564488+0000 osd.6 (osd.6) 405 : cluster [DBG] 1.7 deep-scrub ok 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.976143+0000 osd.3 (osd.3) 545 : cluster [DBG] 6.1b scrub starts 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.978719+0000 osd.3 (osd.3) 546 : cluster [DBG] 6.1b scrub ok 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:33.998436+0000 osd.2 (osd.2) 377 : cluster [DBG] 2.1 scrub starts 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.001444+0000 osd.2 (osd.2) 378 : cluster [DBG] 2.1 scrub ok 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.396084+0000 osd.1 (osd.1) 534 : cluster [DBG] 6.4 scrub starts 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.398653+0000 osd.1 (osd.1) 535 : cluster [DBG] 6.4 scrub ok 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.525206+0000 osd.4 (osd.4) 366 : cluster [DBG] 6.17 scrub starts 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.527531+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub ok 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.891678+0000 osd.7 (osd.7) 594 : cluster [DBG] 6.3 scrub starts 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: cluster 2024-09-13T23:05:34.893971+0000 osd.7 (osd.7) 595 : cluster [DBG] 6.3 scrub ok 2024-09-13T23:05:35.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:35 smithi178 bash[22840]: audit 2024-09-13T23:05:35.111138+0000 mon.b (mon.1) 325 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:36.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:05:36.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.022314+0000 osd.2 (osd.2) 379 : cluster [DBG] 5.b scrub starts 2024-09-13T23:05:36.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.024619+0000 osd.2 (osd.2) 380 : cluster [DBG] 5.b scrub ok 2024-09-13T23:05:36.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.128044+0000 mgr.y (mgr.14613) 1032 : cluster [DBG] pgmap v1249: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T23:05:36.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.422251+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:05:36.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.424630+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:05:36.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.504403+0000 osd.4 (osd.4) 368 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:05:36.660 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:36 smithi192 bash[24608]: cluster 2024-09-13T23:05:35.506916+0000 osd.4 (osd.4) 369 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:05:36.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.022314+0000 osd.2 (osd.2) 379 : cluster [DBG] 5.b scrub starts 2024-09-13T23:05:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.024619+0000 osd.2 (osd.2) 380 : cluster [DBG] 5.b scrub ok 2024-09-13T23:05:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.128044+0000 mgr.y (mgr.14613) 1032 : cluster [DBG] pgmap v1249: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T23:05:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.422251+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:05:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.424630+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:05:36.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.504403+0000 osd.4 (osd.4) 368 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:05:36.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[22840]: cluster 2024-09-13T23:05:35.506916+0000 osd.4 (osd.4) 369 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:05:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.022314+0000 osd.2 (osd.2) 379 : cluster [DBG] 5.b scrub starts 2024-09-13T23:05:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.024619+0000 osd.2 (osd.2) 380 : cluster [DBG] 5.b scrub ok 2024-09-13T23:05:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.128044+0000 mgr.y (mgr.14613) 1032 : cluster [DBG] pgmap v1249: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T23:05:36.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.422251+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.9 scrub starts 2024-09-13T23:05:36.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.424630+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.9 scrub ok 2024-09-13T23:05:36.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.504403+0000 osd.4 (osd.4) 368 : cluster [DBG] 1.5 scrub starts 2024-09-13T23:05:36.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:36 smithi178 bash[29889]: cluster 2024-09-13T23:05:35.506916+0000 osd.4 (osd.4) 369 : cluster [DBG] 1.5 scrub ok 2024-09-13T23:05:37.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:37 smithi192 bash[24608]: audit 2024-09-13T23:05:36.300254+0000 mgr.y (mgr.14613) 1033 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:37.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:37 smithi178 bash[22840]: audit 2024-09-13T23:05:36.300254+0000 mgr.y (mgr.14613) 1033 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:37.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:37 smithi178 bash[29889]: audit 2024-09-13T23:05:36.300254+0000 mgr.y (mgr.14613) 1033 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:38.279 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:38 smithi192 bash[24608]: cluster 2024-09-13T23:05:36.685652+0000 osd.0 (osd.0) 531 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:05:38.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:38 smithi192 bash[24608]: cluster 2024-09-13T23:05:36.687895+0000 osd.0 (osd.0) 532 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:05:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:38 smithi192 bash[24608]: cluster 2024-09-13T23:05:37.128874+0000 mgr.y (mgr.14613) 1034 : cluster [DBG] pgmap v1250: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-09-13T23:05:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:38 smithi192 bash[24608]: cluster 2024-09-13T23:05:37.535274+0000 osd.4 (osd.4) 370 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:05:38.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:38 smithi192 bash[24608]: cluster 2024-09-13T23:05:37.540002+0000 osd.4 (osd.4) 371 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:05:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[22840]: cluster 2024-09-13T23:05:36.685652+0000 osd.0 (osd.0) 531 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:05:38.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[22840]: cluster 2024-09-13T23:05:36.687895+0000 osd.0 (osd.0) 532 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[22840]: cluster 2024-09-13T23:05:37.128874+0000 mgr.y (mgr.14613) 1034 : cluster [DBG] pgmap v1250: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[22840]: cluster 2024-09-13T23:05:37.535274+0000 osd.4 (osd.4) 370 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[22840]: cluster 2024-09-13T23:05:37.540002+0000 osd.4 (osd.4) 371 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[29889]: cluster 2024-09-13T23:05:36.685652+0000 osd.0 (osd.0) 531 : cluster [DBG] 2.1f scrub starts 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[29889]: cluster 2024-09-13T23:05:36.687895+0000 osd.0 (osd.0) 532 : cluster [DBG] 2.1f scrub ok 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[29889]: cluster 2024-09-13T23:05:37.128874+0000 mgr.y (mgr.14613) 1034 : cluster [DBG] pgmap v1250: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[29889]: cluster 2024-09-13T23:05:37.535274+0000 osd.4 (osd.4) 370 : cluster [DBG] 3.3 scrub starts 2024-09-13T23:05:38.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:38 smithi178 bash[29889]: cluster 2024-09-13T23:05:37.540002+0000 osd.4 (osd.4) 371 : cluster [DBG] 3.3 scrub ok 2024-09-13T23:05:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:39 smithi192 bash[24608]: cluster 2024-09-13T23:05:37.971713+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:05:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:39 smithi192 bash[24608]: cluster 2024-09-13T23:05:37.974943+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:05:39.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:39 smithi192 bash[24608]: cluster 2024-09-13T23:05:38.241149+0000 osd.5 (osd.5) 603 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:05:39.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:39 smithi192 bash[24608]: cluster 2024-09-13T23:05:38.243847+0000 osd.5 (osd.5) 604 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:05:39.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[29889]: cluster 2024-09-13T23:05:37.971713+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:05:39.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[29889]: cluster 2024-09-13T23:05:37.974943+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:05:39.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[29889]: cluster 2024-09-13T23:05:38.241149+0000 osd.5 (osd.5) 603 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:05:39.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[29889]: cluster 2024-09-13T23:05:38.243847+0000 osd.5 (osd.5) 604 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:05:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[22840]: cluster 2024-09-13T23:05:37.971713+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-09-13T23:05:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[22840]: cluster 2024-09-13T23:05:37.974943+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-09-13T23:05:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[22840]: cluster 2024-09-13T23:05:38.241149+0000 osd.5 (osd.5) 603 : cluster [DBG] 6.d deep-scrub starts 2024-09-13T23:05:39.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[22840]: cluster 2024-09-13T23:05:38.243847+0000 osd.5 (osd.5) 604 : cluster [DBG] 6.d deep-scrub ok 2024-09-13T23:05:40.136 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T23:05:40.137 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T23:05:40.150 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:39 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:05:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:05:40.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:38.706714+0000 osd.0 (osd.0) 533 : cluster [DBG] 6.0 deep-scrub starts 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:38.709213+0000 osd.0 (osd.0) 534 : cluster [DBG] 6.0 deep-scrub ok 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.007630+0000 osd.3 (osd.3) 549 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.010126+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.010193+0000 osd.3 (osd.3) 550 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.012719+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.129836+0000 mgr.y (mgr.14613) 1035 : cluster [DBG] pgmap v1251: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.495762+0000 osd.4 (osd.4) 372 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.497943+0000 osd.4 (osd.4) 373 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.957341+0000 osd.7 (osd.7) 596 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.959403+0000 osd.7 (osd.7) 597 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:05:40.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:38.706714+0000 osd.0 (osd.0) 533 : cluster [DBG] 6.0 deep-scrub starts 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:38.709213+0000 osd.0 (osd.0) 534 : cluster [DBG] 6.0 deep-scrub ok 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.007630+0000 osd.3 (osd.3) 549 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.010126+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.010193+0000 osd.3 (osd.3) 550 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.012719+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.129836+0000 mgr.y (mgr.14613) 1035 : cluster [DBG] pgmap v1251: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.495762+0000 osd.4 (osd.4) 372 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.497943+0000 osd.4 (osd.4) 373 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.957341+0000 osd.7 (osd.7) 596 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:05:40.682 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:40 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.959403+0000 osd.7 (osd.7) 597 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:05:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:38.706714+0000 osd.0 (osd.0) 533 : cluster [DBG] 6.0 deep-scrub starts 2024-09-13T23:05:40.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:38.709213+0000 osd.0 (osd.0) 534 : cluster [DBG] 6.0 deep-scrub ok 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.007630+0000 osd.3 (osd.3) 549 : cluster [DBG] 5.13 scrub starts 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.010126+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 scrub starts 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.010193+0000 osd.3 (osd.3) 550 : cluster [DBG] 5.13 scrub ok 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.012719+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 scrub ok 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.129836+0000 mgr.y (mgr.14613) 1035 : cluster [DBG] pgmap v1251: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.495762+0000 osd.4 (osd.4) 372 : cluster [DBG] 6.1a scrub starts 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.497943+0000 osd.4 (osd.4) 373 : cluster [DBG] 6.1a scrub ok 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.957341+0000 osd.7 (osd.7) 596 : cluster [DBG] 6.8 scrub starts 2024-09-13T23:05:40.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:40 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.959403+0000 osd.7 (osd.7) 597 : cluster [DBG] 6.8 scrub ok 2024-09-13T23:05:41.003 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":1251,"stamp":"2024-09-13T23:05:39.129168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3234,"num_read_kb":2986,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":11196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2669,"ondisk_log_size":2669,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5780384,"kb_used_data":8024,"kb_used_omap":12,"kb_used_meta":5771955,"kb_avail":744180832,"statfs":{"total":767960285184,"available":762041171968,"internally_reserved":0,"allocated":8216576,"data_stored":4891070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":5910482480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003401"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":1007,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:33.978862+0000","last_change":"2024-09-13T23:05:33.978937+0000","last_active":"2024-09-13T23:05:33.978862+0000","last_peered":"2024-09-13T23:05:33.978862+0000","last_clean":"2024-09-13T23:05:33.978862+0000","last_became_active":"2024-09-13T22:58:42.763279+0000","last_became_peered":"2024-09-13T22:58:42.763279+0000","last_unstale":"2024-09-13T23:05:33.978862+0000","last_undegraded":"2024-09-13T23:05:33.978862+0000","last_fullsized":"2024-09-13T23:05:33.978862+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:33.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T23:05:33.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:43.296664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1070,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278404+0000","last_change":"2024-09-13T23:04:22.940443+0000","last_active":"2024-09-13T23:05:34.278404+0000","last_peered":"2024-09-13T23:05:34.278404+0000","last_clean":"2024-09-13T23:05:34.278404+0000","last_became_active":"2024-09-13T23:02:24.401461+0000","last_became_peered":"2024-09-13T23:02:24.401461+0000","last_unstale":"2024-09-13T23:05:34.278404+0000","last_undegraded":"2024-09-13T23:05:34.278404+0000","last_fullsized":"2024-09-13T23:05:34.278404+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:22.940298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:55.756398+0000","last_clean_scrub_stamp":"2024-09-13T23:04:22.940298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":1099,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392751+0000","last_change":"2024-09-13T23:05:04.841182+0000","last_active":"2024-09-13T23:05:31.392751+0000","last_peered":"2024-09-13T23:05:31.392751+0000","last_clean":"2024-09-13T23:05:31.392751+0000","last_became_active":"2024-09-13T22:58:42.747843+0000","last_became_peered":"2024-09-13T22:58:42.747843+0000","last_unstale":"2024-09-13T23:05:31.392751+0000","last_undegraded":"2024-09-13T23:05:31.392751+0000","last_fullsized":"2024-09-13T23:05:31.392751+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-09-13T23:05:04.840856+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T23:05:04.840856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:34.033479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277629+0000","last_change":"2024-09-13T23:05:14.567019+0000","last_active":"2024-09-13T23:05:34.277629+0000","last_peered":"2024-09-13T23:05:34.277629+0000","last_clean":"2024-09-13T23:05:34.277629+0000","last_became_active":"2024-09-13T23:02:24.389370+0000","last_became_peered":"2024-09-13T23:02:24.389370+0000","last_unstale":"2024-09-13T23:05:34.277629+0000","last_undegraded":"2024-09-13T23:05:34.277629+0000","last_fullsized":"2024-09-13T23:05:34.277629+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:14.566776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:52.522870+0000","last_clean_scrub_stamp":"2024-09-13T23:05:14.566776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:38.340930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1009,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392808+0000","last_change":"2024-09-13T23:04:55.780152+0000","last_active":"2024-09-13T23:05:31.392808+0000","last_peered":"2024-09-13T23:05:31.392808+0000","last_clean":"2024-09-13T23:05:31.392808+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T23:05:31.392808+0000","last_undegraded":"2024-09-13T23:05:31.392808+0000","last_fullsized":"2024-09-13T23:05:31.392808+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:55.779951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:16.435813+0000","last_clean_scrub_stamp":"2024-09-13T23:04:55.779951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.625494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":1080,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277928+0000","last_change":"2024-09-13T23:05:11.797789+0000","last_active":"2024-09-13T23:05:34.277928+0000","last_peered":"2024-09-13T23:05:34.277928+0000","last_clean":"2024-09-13T23:05:34.277928+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T23:05:34.277928+0000","last_undegraded":"2024-09-13T23:05:34.277928+0000","last_fullsized":"2024-09-13T23:05:34.277928+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:05:11.797680+0000","last_deep_scrub":"397'32","last_deep_scrub_stamp":"2024-09-13T23:05:11.797680+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.797680+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:27.259403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1016,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.389880+0000","last_change":"2024-09-13T23:05:16.499406+0000","last_active":"2024-09-13T23:05:31.389880+0000","last_peered":"2024-09-13T23:05:31.389880+0000","last_clean":"2024-09-13T23:05:31.389880+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T23:05:31.389880+0000","last_undegraded":"2024-09-13T23:05:31.389880+0000","last_fullsized":"2024-09-13T23:05:31.389880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:16.499290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T23:05:16.499290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:25.613237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278991+0000","last_change":"2024-09-13T23:04:31.716839+0000","last_active":"2024-09-13T23:05:34.278991+0000","last_peered":"2024-09-13T23:05:34.278991+0000","last_clean":"2024-09-13T23:05:34.278991+0000","last_became_active":"2024-09-13T23:02:24.378760+0000","last_became_peered":"2024-09-13T23:02:24.378760+0000","last_unstale":"2024-09-13T23:05:34.278991+0000","last_undegraded":"2024-09-13T23:05:34.278991+0000","last_fullsized":"2024-09-13T23:05:34.278991+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:31.716648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:09.889616+0000","last_clean_scrub_stamp":"2024-09-13T23:04:31.716648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:35.419049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1019,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.397701+0000","last_change":"2024-09-13T23:04:29.639210+0000","last_active":"2024-09-13T23:05:31.397701+0000","last_peered":"2024-09-13T23:05:31.397701+0000","last_clean":"2024-09-13T23:05:31.397701+0000","last_became_active":"2024-09-13T22:58:42.770668+0000","last_became_peered":"2024-09-13T22:58:42.770668+0000","last_unstale":"2024-09-13T23:05:31.397701+0000","last_undegraded":"2024-09-13T23:05:31.397701+0000","last_fullsized":"2024-09-13T23:05:31.397701+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:29.639022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:29.639022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.503427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":1230,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524064+0000","last_change":"2024-09-13T23:05:14.284015+0000","last_active":"2024-09-13T23:05:34.524064+0000","last_peered":"2024-09-13T23:05:34.524064+0000","last_clean":"2024-09-13T23:05:34.524064+0000","last_became_active":"2024-09-13T23:02:24.384594+0000","last_became_peered":"2024-09-13T23:02:24.384594+0000","last_unstale":"2024-09-13T23:05:34.524064+0000","last_undegraded":"2024-09-13T23:05:34.524064+0000","last_fullsized":"2024-09-13T23:05:34.524064+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:05:14.283792+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T23:05:14.283792+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:30.484409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1015,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396125+0000","last_change":"2024-09-13T23:04:30.645760+0000","last_active":"2024-09-13T23:05:31.396125+0000","last_peered":"2024-09-13T23:05:31.396125+0000","last_clean":"2024-09-13T23:05:31.396125+0000","last_became_active":"2024-09-13T23:02:24.396141+0000","last_became_peered":"2024-09-13T23:02:24.396141+0000","last_unstale":"2024-09-13T23:05:31.396125+0000","last_undegraded":"2024-09-13T23:05:31.396125+0000","last_fullsized":"2024-09-13T23:05:31.396125+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:30.645572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T23:04:30.645572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.135104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1014,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523348+0000","last_change":"2024-09-13T23:04:16.503342+0000","last_active":"2024-09-13T23:05:34.523348+0000","last_peered":"2024-09-13T23:05:34.523348+0000","last_clean":"2024-09-13T23:05:34.523348+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T23:05:34.523348+0000","last_undegraded":"2024-09-13T23:05:34.523348+0000","last_fullsized":"2024-09-13T23:05:34.523348+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:16.503153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T23:04:16.503153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1007,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.398088+0000","last_change":"2024-09-13T23:04:15.703588+0000","last_active":"2024-09-13T23:05:31.398088+0000","last_peered":"2024-09-13T23:05:31.398088+0000","last_clean":"2024-09-13T23:05:31.398088+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T23:05:31.398088+0000","last_undegraded":"2024-09-13T23:05:31.398088+0000","last_fullsized":"2024-09-13T23:05:31.398088+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:15.703332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:15.703332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"512'29","reported_seq":610,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.395628+0000","last_change":"2024-09-13T23:05:25.869524+0000","last_active":"2024-09-13T23:05:31.395628+0000","last_peered":"2024-09-13T23:05:31.395628+0000","last_clean":"2024-09-13T23:05:31.395628+0000","last_became_active":"2024-09-13T23:02:24.395749+0000","last_became_peered":"2024-09-13T23:02:24.395749+0000","last_unstale":"2024-09-13T23:05:31.395628+0000","last_undegraded":"2024-09-13T23:05:31.395628+0000","last_fullsized":"2024-09-13T23:05:31.395628+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'29","last_scrub_stamp":"2024-09-13T23:05:25.869284+0000","last_deep_scrub":"512'29","last_deep_scrub_stamp":"2024-09-13T23:05:25.869284+0000","last_clean_scrub_stamp":"2024-09-13T23:05:25.869284+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.554368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1018,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523624+0000","last_change":"2024-09-13T23:05:04.330173+0000","last_active":"2024-09-13T23:05:34.523624+0000","last_peered":"2024-09-13T23:05:34.523624+0000","last_clean":"2024-09-13T23:05:34.523624+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T23:05:34.523624+0000","last_undegraded":"2024-09-13T23:05:34.523624+0000","last_fullsized":"2024-09-13T23:05:34.523624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:04.329986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T23:05:04.329986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:26.234421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1022,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277180+0000","last_change":"2024-09-13T23:05:27.687051+0000","last_active":"2024-09-13T23:05:34.277180+0000","last_peered":"2024-09-13T23:05:34.277180+0000","last_clean":"2024-09-13T23:05:34.277180+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T23:05:34.277180+0000","last_undegraded":"2024-09-13T23:05:34.277180+0000","last_fullsized":"2024-09-13T23:05:34.277180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:27.686834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:27.686834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:40.877367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":1082,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.522939+0000","last_change":"2024-09-13T23:05:28.201795+0000","last_active":"2024-09-13T23:05:34.522939+0000","last_peered":"2024-09-13T23:05:34.522939+0000","last_clean":"2024-09-13T23:05:34.522939+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T23:05:34.522939+0000","last_undegraded":"2024-09-13T23:05:34.522939+0000","last_fullsized":"2024-09-13T23:05:34.522939+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:28.201596+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:05:28.201596+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:40.647798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":313,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.400181+0000","last_change":"2024-09-13T23:04:29.712288+0000","last_active":"2024-09-13T23:05:31.400181+0000","last_peered":"2024-09-13T23:05:31.400181+0000","last_clean":"2024-09-13T23:05:31.400181+0000","last_became_active":"2024-09-13T23:02:24.401801+0000","last_became_peered":"2024-09-13T23:02:24.401801+0000","last_unstale":"2024-09-13T23:05:31.400181+0000","last_undegraded":"2024-09-13T23:05:31.400181+0000","last_fullsized":"2024-09-13T23:05:31.400181+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'60","last_scrub_stamp":"2024-09-13T23:04:29.712075+0000","last_deep_scrub":"397'60","last_deep_scrub_stamp":"2024-09-13T23:03:15.188024+0000","last_clean_scrub_stamp":"2024-09-13T23:04:29.712075+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:35.947352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1032,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396323+0000","last_change":"2024-09-13T23:05:17.776973+0000","last_active":"2024-09-13T23:05:31.396323+0000","last_peered":"2024-09-13T23:05:31.396323+0000","last_clean":"2024-09-13T23:05:31.396323+0000","last_became_active":"2024-09-13T23:02:24.394946+0000","last_became_peered":"2024-09-13T23:02:24.394946+0000","last_unstale":"2024-09-13T23:05:31.396323+0000","last_undegraded":"2024-09-13T23:05:31.396323+0000","last_fullsized":"2024-09-13T23:05:31.396323+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:17.776706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T23:05:17.776706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:45.439143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1053,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391110+0000","last_change":"2024-09-13T23:04:31.345930+0000","last_active":"2024-09-13T23:05:31.391110+0000","last_peered":"2024-09-13T23:05:31.391110+0000","last_clean":"2024-09-13T23:05:31.391110+0000","last_became_active":"2024-09-13T23:02:24.395018+0000","last_became_peered":"2024-09-13T23:02:24.395018+0000","last_unstale":"2024-09-13T23:05:31.391110+0000","last_undegraded":"2024-09-13T23:05:31.391110+0000","last_fullsized":"2024-09-13T23:05:31.391110+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:31.345747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_clean_scrub_stamp":"2024-09-13T23:04:31.345747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:53.527854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":913,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.519218+0000","last_change":"2024-09-13T23:04:39.985031+0000","last_active":"2024-09-13T23:05:34.519218+0000","last_peered":"2024-09-13T23:05:34.519218+0000","last_clean":"2024-09-13T23:05:34.519218+0000","last_became_active":"2024-09-13T23:03:27.005831+0000","last_became_peered":"2024-09-13T23:03:27.005831+0000","last_unstale":"2024-09-13T23:05:34.519218+0000","last_undegraded":"2024-09-13T23:05:34.519218+0000","last_fullsized":"2024-09-13T23:05:34.519218+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:39.984844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:39.984844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.278333+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":1288,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.395688+0000","last_change":"2024-09-13T23:05:11.830051+0000","last_active":"2024-09-13T23:05:31.395688+0000","last_peered":"2024-09-13T23:05:31.395688+0000","last_clean":"2024-09-13T23:05:31.395688+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T23:05:31.395688+0000","last_undegraded":"2024-09-13T23:05:31.395688+0000","last_fullsized":"2024-09-13T23:05:31.395688+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:11.829818+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:03:56.761033+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.829818+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:23.388677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":1114,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392845+0000","last_change":"2024-09-13T23:05:28.927659+0000","last_active":"2024-09-13T23:05:31.392845+0000","last_peered":"2024-09-13T23:05:31.392845+0000","last_clean":"2024-09-13T23:05:31.392845+0000","last_became_active":"2024-09-13T23:02:24.393869+0000","last_became_peered":"2024-09-13T23:02:24.393869+0000","last_unstale":"2024-09-13T23:05:31.392845+0000","last_undegraded":"2024-09-13T23:05:31.392845+0000","last_fullsized":"2024-09-13T23:05:31.392845+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:05:28.927568+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T23:05:28.927568+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.824407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1025,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392310+0000","last_change":"2024-09-13T23:04:47.419240+0000","last_active":"2024-09-13T23:05:31.392310+0000","last_peered":"2024-09-13T23:05:31.392310+0000","last_clean":"2024-09-13T23:05:31.392310+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T23:05:31.392310+0000","last_undegraded":"2024-09-13T23:05:31.392310+0000","last_fullsized":"2024-09-13T23:05:31.392310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:47.419049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:47.419049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:51.993357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"505'13","reported_seq":675,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518763+0000","last_change":"2024-09-13T23:04:44.956800+0000","last_active":"2024-09-13T23:05:34.518763+0000","last_peered":"2024-09-13T23:05:34.518763+0000","last_clean":"2024-09-13T23:05:34.518763+0000","last_became_active":"2024-09-13T23:03:26.731853+0000","last_became_peered":"2024-09-13T23:03:26.731853+0000","last_unstale":"2024-09-13T23:05:34.518763+0000","last_undegraded":"2024-09-13T23:05:34.518763+0000","last_fullsized":"2024-09-13T23:05:34.518763+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'13","last_scrub_stamp":"2024-09-13T23:04:44.956604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:04:44.956604+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.966454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.279511+0000","last_change":"2024-09-13T23:05:33.553416+0000","last_active":"2024-09-13T23:05:34.279511+0000","last_peered":"2024-09-13T23:05:34.279511+0000","last_clean":"2024-09-13T23:05:34.279511+0000","last_became_active":"2024-09-13T23:02:24.373934+0000","last_became_peered":"2024-09-13T23:02:24.373934+0000","last_unstale":"2024-09-13T23:05:34.279511+0000","last_undegraded":"2024-09-13T23:05:34.279511+0000","last_fullsized":"2024-09-13T23:05:34.279511+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:33.553216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:33.553216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:41.839342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":1100,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523181+0000","last_change":"2024-09-13T23:04:49.459653+0000","last_active":"2024-09-13T23:05:34.523181+0000","last_peered":"2024-09-13T23:05:34.523181+0000","last_clean":"2024-09-13T23:05:34.523181+0000","last_became_active":"2024-09-13T22:58:45.828166+0000","last_became_peered":"2024-09-13T22:58:45.828166+0000","last_unstale":"2024-09-13T23:05:34.523181+0000","last_undegraded":"2024-09-13T23:05:34.523181+0000","last_fullsized":"2024-09-13T23:05:34.523181+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:49.459445+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:04:49.459445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.357664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"502'46","reported_seq":1072,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390310+0000","last_change":"2024-09-13T23:04:51.723277+0000","last_active":"2024-09-13T23:05:31.390310+0000","last_peered":"2024-09-13T23:05:31.390310+0000","last_clean":"2024-09-13T23:05:31.390310+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T23:05:31.390310+0000","last_undegraded":"2024-09-13T23:05:31.390310+0000","last_fullsized":"2024-09-13T23:05:31.390310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'46","last_scrub_stamp":"2024-09-13T23:04:51.723066+0000","last_deep_scrub":"502'46","last_deep_scrub_stamp":"2024-09-13T23:03:21.476482+0000","last_clean_scrub_stamp":"2024-09-13T23:04:51.723066+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.018580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1025,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391171+0000","last_change":"2024-09-13T23:04:15.369601+0000","last_active":"2024-09-13T23:05:31.391171+0000","last_peered":"2024-09-13T23:05:31.391171+0000","last_clean":"2024-09-13T23:05:31.391171+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T23:05:31.391171+0000","last_undegraded":"2024-09-13T23:05:31.391171+0000","last_fullsized":"2024-09-13T23:05:31.391171+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:15.369396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:15.369396+0000","last_clean_scrub_stamp":"2024-09-13T23:04:15.369396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"492'16","reported_seq":159,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.279775+0000","last_change":"2024-09-13T23:05:18.700022+0000","last_active":"2024-09-13T23:05:34.279775+0000","last_peered":"2024-09-13T23:05:34.279775+0000","last_clean":"2024-09-13T23:05:34.279775+0000","last_became_active":"2024-09-13T23:02:24.386882+0000","last_became_peered":"2024-09-13T23:02:24.386882+0000","last_unstale":"2024-09-13T23:05:34.279775+0000","last_undegraded":"2024-09-13T23:05:34.279775+0000","last_fullsized":"2024-09-13T23:05:34.279775+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'16","last_scrub_stamp":"2024-09-13T23:05:18.699810+0000","last_deep_scrub":"492'16","last_deep_scrub_stamp":"2024-09-13T23:03:50.531482+0000","last_clean_scrub_stamp":"2024-09-13T23:05:18.699810+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:36.058171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1015,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:38.244003+0000","last_change":"2024-09-13T23:05:38.244057+0000","last_active":"2024-09-13T23:05:38.244003+0000","last_peered":"2024-09-13T23:05:38.244003+0000","last_clean":"2024-09-13T23:05:38.244003+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T23:05:38.244003+0000","last_undegraded":"2024-09-13T23:05:38.244003+0000","last_fullsized":"2024-09-13T23:05:38.244003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:38.243858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:05:38.243858+0000","last_clean_scrub_stamp":"2024-09-13T23:05:38.243858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:49.557447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":120,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277876+0000","last_change":"2024-09-13T23:04:15.557204+0000","last_active":"2024-09-13T23:05:34.277876+0000","last_peered":"2024-09-13T23:05:34.277876+0000","last_clean":"2024-09-13T23:05:34.277876+0000","last_became_active":"2024-09-13T23:02:24.394573+0000","last_became_peered":"2024-09-13T23:02:24.394573+0000","last_unstale":"2024-09-13T23:05:34.277876+0000","last_undegraded":"2024-09-13T23:05:34.277876+0000","last_fullsized":"2024-09-13T23:05:34.277876+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T23:04:15.556961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:15.556961+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:37.522199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1008,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392778+0000","last_change":"2024-09-13T23:04:58.732694+0000","last_active":"2024-09-13T23:05:31.392778+0000","last_peered":"2024-09-13T23:05:31.392778+0000","last_clean":"2024-09-13T23:05:31.392778+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T23:05:31.392778+0000","last_undegraded":"2024-09-13T23:05:31.392778+0000","last_fullsized":"2024-09-13T23:05:31.392778+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:58.732508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:58.732508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:27.985410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":841,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396197+0000","last_change":"2024-09-13T23:04:28.599203+0000","last_active":"2024-09-13T23:05:31.396197+0000","last_peered":"2024-09-13T23:05:31.396197+0000","last_clean":"2024-09-13T23:05:31.396197+0000","last_became_active":"2024-09-13T23:02:24.393998+0000","last_became_peered":"2024-09-13T23:02:24.393998+0000","last_unstale":"2024-09-13T23:05:31.396197+0000","last_undegraded":"2024-09-13T23:05:31.396197+0000","last_fullsized":"2024-09-13T23:05:31.396197+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:28.599012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:58.971688+0000","last_clean_scrub_stamp":"2024-09-13T23:04:28.599012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:34.510142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1008,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:32.928649+0000","last_change":"2024-09-13T23:05:32.928758+0000","last_active":"2024-09-13T23:05:32.928649+0000","last_peered":"2024-09-13T23:05:32.928649+0000","last_clean":"2024-09-13T23:05:32.928649+0000","last_became_active":"2024-09-13T22:58:42.746385+0000","last_became_peered":"2024-09-13T22:58:42.746385+0000","last_unstale":"2024-09-13T23:05:32.928649+0000","last_undegraded":"2024-09-13T23:05:32.928649+0000","last_fullsized":"2024-09-13T23:05:32.928649+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:32.928485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_clean_scrub_stamp":"2024-09-13T23:05:32.928485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:56.175784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":1062,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392053+0000","last_change":"2024-09-13T23:05:24.914145+0000","last_active":"2024-09-13T23:05:31.392053+0000","last_peered":"2024-09-13T23:05:31.392053+0000","last_clean":"2024-09-13T23:05:31.392053+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T23:05:31.392053+0000","last_undegraded":"2024-09-13T23:05:31.392053+0000","last_fullsized":"2024-09-13T23:05:31.392053+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:24.913957+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T23:05:24.913957+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:39.029738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":1165,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278724+0000","last_change":"2024-09-13T23:05:32.617484+0000","last_active":"2024-09-13T23:05:34.278724+0000","last_peered":"2024-09-13T23:05:34.278724+0000","last_clean":"2024-09-13T23:05:34.278724+0000","last_became_active":"2024-09-13T23:02:24.401486+0000","last_became_peered":"2024-09-13T23:02:24.401486+0000","last_unstale":"2024-09-13T23:05:34.278724+0000","last_undegraded":"2024-09-13T23:05:34.278724+0000","last_fullsized":"2024-09-13T23:05:34.278724+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'36","last_scrub_stamp":"2024-09-13T23:05:32.617275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:32.617275+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:34.054021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393689+0000","last_change":"2024-09-13T23:04:33.675184+0000","last_active":"2024-09-13T23:05:31.393689+0000","last_peered":"2024-09-13T23:05:31.393689+0000","last_clean":"2024-09-13T23:05:31.393689+0000","last_became_active":"2024-09-13T22:58:42.747048+0000","last_became_peered":"2024-09-13T22:58:42.747048+0000","last_unstale":"2024-09-13T23:05:31.393689+0000","last_undegraded":"2024-09-13T23:05:31.393689+0000","last_fullsized":"2024-09-13T23:05:31.393689+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:33.674906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:33.674906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:53.493378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":112,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278562+0000","last_change":"2024-09-13T23:04:54.645977+0000","last_active":"2024-09-13T23:05:34.278562+0000","last_peered":"2024-09-13T23:05:34.278562+0000","last_clean":"2024-09-13T23:05:34.278562+0000","last_became_active":"2024-09-13T23:02:24.401765+0000","last_became_peered":"2024-09-13T23:02:24.401765+0000","last_unstale":"2024-09-13T23:05:34.278562+0000","last_undegraded":"2024-09-13T23:05:34.278562+0000","last_fullsized":"2024-09-13T23:05:34.278562+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T23:04:54.645649+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T23:04:54.645649+0000","last_clean_scrub_stamp":"2024-09-13T23:04:54.645649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:00.803577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"507'16","reported_seq":1389,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.389737+0000","last_change":"2024-09-13T23:04:57.834420+0000","last_active":"2024-09-13T23:05:31.389737+0000","last_peered":"2024-09-13T23:05:31.389737+0000","last_clean":"2024-09-13T23:05:31.389737+0000","last_became_active":"2024-09-13T23:02:24.376006+0000","last_became_peered":"2024-09-13T23:02:24.376006+0000","last_unstale":"2024-09-13T23:05:31.389737+0000","last_undegraded":"2024-09-13T23:05:31.389737+0000","last_fullsized":"2024-09-13T23:05:31.389737+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'16","last_scrub_stamp":"2024-09-13T23:04:57.834206+0000","last_deep_scrub":"507'16","last_deep_scrub_stamp":"2024-09-13T23:03:39.406184+0000","last_clean_scrub_stamp":"2024-09-13T23:04:57.834206+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:07.319089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1001,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.394482+0000","last_change":"2024-09-13T23:04:06.861262+0000","last_active":"2024-09-13T23:05:31.394482+0000","last_peered":"2024-09-13T23:05:31.394482+0000","last_clean":"2024-09-13T23:05:31.394482+0000","last_became_active":"2024-09-13T22:58:42.770701+0000","last_became_peered":"2024-09-13T22:58:42.770701+0000","last_unstale":"2024-09-13T23:05:31.394482+0000","last_undegraded":"2024-09-13T23:05:31.394482+0000","last_fullsized":"2024-09-13T23:05:31.394482+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:06.861058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:06.861058+0000","last_clean_scrub_stamp":"2024-09-13T23:04:06.861058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":901,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.507067+0000","last_change":"2024-09-13T23:05:35.507130+0000","last_active":"2024-09-13T23:05:35.507067+0000","last_peered":"2024-09-13T23:05:35.507067+0000","last_clean":"2024-09-13T23:05:35.507067+0000","last_became_active":"2024-09-13T23:02:27.393901+0000","last_became_peered":"2024-09-13T23:02:27.393901+0000","last_unstale":"2024-09-13T23:05:35.507067+0000","last_undegraded":"2024-09-13T23:05:35.507067+0000","last_fullsized":"2024-09-13T23:05:35.507067+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:35.506928+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:01:31.604112+0000","last_clean_scrub_stamp":"2024-09-13T23:05:35.506928+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.450384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":1048,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.394635+0000","last_change":"2024-09-13T23:05:12.756474+0000","last_active":"2024-09-13T23:05:31.394635+0000","last_peered":"2024-09-13T23:05:31.394635+0000","last_clean":"2024-09-13T23:05:31.394635+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T23:05:31.394635+0000","last_undegraded":"2024-09-13T23:05:31.394635+0000","last_fullsized":"2024-09-13T23:05:31.394635+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'26","last_scrub_stamp":"2024-09-13T23:05:12.756269+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T23:05:12.756269+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:29.774314+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":1087,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392698+0000","last_change":"2024-09-13T23:04:05.331264+0000","last_active":"2024-09-13T23:05:31.392698+0000","last_peered":"2024-09-13T23:05:31.392698+0000","last_clean":"2024-09-13T23:05:31.392698+0000","last_became_active":"2024-09-13T23:02:24.370112+0000","last_became_peered":"2024-09-13T23:02:24.370112+0000","last_unstale":"2024-09-13T23:05:31.392698+0000","last_undegraded":"2024-09-13T23:05:31.392698+0000","last_fullsized":"2024-09-13T23:05:31.392698+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T23:04:05.331012+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T23:04:05.331012+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":2661,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:33.172282+0000","last_change":"2024-09-13T23:05:15.805499+0000","last_active":"2024-09-13T23:05:33.172282+0000","last_peered":"2024-09-13T23:05:33.172282+0000","last_clean":"2024-09-13T23:05:33.172282+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T23:05:33.172282+0000","last_undegraded":"2024-09-13T23:05:33.172282+0000","last_fullsized":"2024-09-13T23:05:33.172282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T23:05:15.805301+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T23:05:15.805301+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:39.761119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1645,"num_read_kb":1640,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278321+0000","last_change":"2024-09-13T23:05:11.595837+0000","last_active":"2024-09-13T23:05:34.278321+0000","last_peered":"2024-09-13T23:05:34.278321+0000","last_clean":"2024-09-13T23:05:34.278321+0000","last_became_active":"2024-09-13T23:02:24.393819+0000","last_became_peered":"2024-09-13T23:02:24.393819+0000","last_unstale":"2024-09-13T23:05:34.278321+0000","last_undegraded":"2024-09-13T23:05:34.278321+0000","last_fullsized":"2024-09-13T23:05:34.278321+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:11.595653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.595653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:28.923365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278320+0000","last_change":"2024-09-13T23:05:22.705784+0000","last_active":"2024-09-13T23:05:34.278320+0000","last_peered":"2024-09-13T23:05:34.278320+0000","last_clean":"2024-09-13T23:05:34.278320+0000","last_became_active":"2024-09-13T23:02:24.399413+0000","last_became_peered":"2024-09-13T23:02:24.399413+0000","last_unstale":"2024-09-13T23:05:34.278320+0000","last_undegraded":"2024-09-13T23:05:34.278320+0000","last_fullsized":"2024-09-13T23:05:34.278320+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:22.705589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T23:05:22.705589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:26.685351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":716,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.402392+0000","last_change":"2024-09-13T23:04:59.697982+0000","last_active":"2024-09-13T23:05:31.402392+0000","last_peered":"2024-09-13T23:05:31.402392+0000","last_clean":"2024-09-13T23:05:31.402392+0000","last_became_active":"2024-09-13T22:58:45.829511+0000","last_became_peered":"2024-09-13T22:58:45.829511+0000","last_unstale":"2024-09-13T23:05:31.402392+0000","last_undegraded":"2024-09-13T23:05:31.402392+0000","last_fullsized":"2024-09-13T23:05:31.402392+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:59.697738+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:04:59.697738+0000","last_clean_scrub_stamp":"2024-09-13T23:04:59.697738+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:07.670107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1453,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391468+0000","last_change":"2024-09-13T23:04:25.311998+0000","last_active":"2024-09-13T23:05:31.391468+0000","last_peered":"2024-09-13T23:05:31.391468+0000","last_clean":"2024-09-13T23:05:31.391468+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T23:05:31.391468+0000","last_undegraded":"2024-09-13T23:05:31.391468+0000","last_fullsized":"2024-09-13T23:05:31.391468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'64","last_scrub_stamp":"2024-09-13T23:04:25.311656+0000","last_deep_scrub":"397'64","last_deep_scrub_stamp":"2024-09-13T23:04:25.311656+0000","last_clean_scrub_stamp":"2024-09-13T23:04:25.311656+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:40.821982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1033,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.398299+0000","last_change":"2024-09-13T23:05:13.836769+0000","last_active":"2024-09-13T23:05:31.398299+0000","last_peered":"2024-09-13T23:05:31.398299+0000","last_clean":"2024-09-13T23:05:31.398299+0000","last_became_active":"2024-09-13T23:02:24.396411+0000","last_became_peered":"2024-09-13T23:02:24.396411+0000","last_unstale":"2024-09-13T23:05:31.398299+0000","last_undegraded":"2024-09-13T23:05:31.398299+0000","last_fullsized":"2024-09-13T23:05:31.398299+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:13.836583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:48.725054+0000","last_clean_scrub_stamp":"2024-09-13T23:05:13.836583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:41.008871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":267,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.399826+0000","last_change":"2024-09-13T23:04:53.644576+0000","last_active":"2024-09-13T23:05:31.399826+0000","last_peered":"2024-09-13T23:05:31.399826+0000","last_clean":"2024-09-13T23:05:31.399826+0000","last_became_active":"2024-09-13T22:58:42.740862+0000","last_became_peered":"2024-09-13T22:58:42.740862+0000","last_unstale":"2024-09-13T23:05:31.399826+0000","last_undegraded":"2024-09-13T23:05:31.399826+0000","last_fullsized":"2024-09-13T23:05:31.399826+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:53.644292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:53.644292+0000","last_clean_scrub_stamp":"2024-09-13T23:04:53.644292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:00.888838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1026,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.389612+0000","last_change":"2024-09-13T23:04:41.496868+0000","last_active":"2024-09-13T23:05:31.389612+0000","last_peered":"2024-09-13T23:05:31.389612+0000","last_clean":"2024-09-13T23:05:31.389612+0000","last_became_active":"2024-09-13T22:58:42.742606+0000","last_became_peered":"2024-09-13T22:58:42.742606+0000","last_unstale":"2024-09-13T23:05:31.389612+0000","last_undegraded":"2024-09-13T23:05:31.389612+0000","last_fullsized":"2024-09-13T23:05:31.389612+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:41.496672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:41.496672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:02.102150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":719,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:33.564641+0000","last_change":"2024-09-13T23:05:33.564698+0000","last_active":"2024-09-13T23:05:33.564641+0000","last_peered":"2024-09-13T23:05:33.564641+0000","last_clean":"2024-09-13T23:05:33.564641+0000","last_became_active":"2024-09-13T22:58:45.832840+0000","last_became_peered":"2024-09-13T22:58:45.832840+0000","last_unstale":"2024-09-13T23:05:33.564641+0000","last_undegraded":"2024-09-13T23:05:33.564641+0000","last_fullsized":"2024-09-13T23:05:33.564641+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:33.564501+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:05:33.564501+0000","last_clean_scrub_stamp":"2024-09-13T23:05:33.564501+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:46.370294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1426,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524673+0000","last_change":"2024-09-13T23:05:13.288114+0000","last_active":"2024-09-13T23:05:34.524673+0000","last_peered":"2024-09-13T23:05:34.524673+0000","last_clean":"2024-09-13T23:05:34.524673+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T23:05:34.524673+0000","last_undegraded":"2024-09-13T23:05:34.524673+0000","last_fullsized":"2024-09-13T23:05:34.524673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'48","last_scrub_stamp":"2024-09-13T23:05:13.287884+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T23:05:13.287884+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:32.385851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":1043,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391315+0000","last_change":"2024-09-13T23:05:14.482221+0000","last_active":"2024-09-13T23:05:31.391315+0000","last_peered":"2024-09-13T23:05:31.391315+0000","last_clean":"2024-09-13T23:05:31.391315+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T23:05:31.391315+0000","last_undegraded":"2024-09-13T23:05:31.391315+0000","last_fullsized":"2024-09-13T23:05:31.391315+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T23:05:14.482020+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T23:05:14.482020+0000","last_clean_scrub_stamp":"2024-09-13T23:05:14.482020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:22.426259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":1072,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391886+0000","last_change":"2024-09-13T23:04:43.436113+0000","last_active":"2024-09-13T23:05:31.391886+0000","last_peered":"2024-09-13T23:05:31.391886+0000","last_clean":"2024-09-13T23:05:31.391886+0000","last_became_active":"2024-09-13T23:02:24.394526+0000","last_became_peered":"2024-09-13T23:02:24.394526+0000","last_unstale":"2024-09-13T23:05:31.391886+0000","last_undegraded":"2024-09-13T23:05:31.391886+0000","last_fullsized":"2024-09-13T23:05:31.391886+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T23:04:43.435857+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T23:04:43.435857+0000","last_clean_scrub_stamp":"2024-09-13T23:04:43.435857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.531892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"414'14","reported_seq":1395,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277808+0000","last_change":"2024-09-13T23:04:52.785033+0000","last_active":"2024-09-13T23:05:34.277808+0000","last_peered":"2024-09-13T23:05:34.277808+0000","last_clean":"2024-09-13T23:05:34.277808+0000","last_became_active":"2024-09-13T22:58:42.740603+0000","last_became_peered":"2024-09-13T22:58:42.740603+0000","last_unstale":"2024-09-13T23:05:34.277808+0000","last_undegraded":"2024-09-13T23:05:34.277808+0000","last_fullsized":"2024-09-13T23:05:34.277808+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-09-13T23:04:52.784826+0000","last_deep_scrub":"358'13","last_deep_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_clean_scrub_stamp":"2024-09-13T23:04:52.784826+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:00.899732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1016,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.276857+0000","last_change":"2024-09-13T23:04:37.914343+0000","last_active":"2024-09-13T23:05:34.276857+0000","last_peered":"2024-09-13T23:05:34.276857+0000","last_clean":"2024-09-13T23:05:34.276857+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T23:05:34.276857+0000","last_undegraded":"2024-09-13T23:05:34.276857+0000","last_fullsized":"2024-09-13T23:05:34.276857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:37.914155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:59.741159+0000","last_clean_scrub_stamp":"2024-09-13T23:04:37.914155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:38.122315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":1074,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.522982+0000","last_change":"2024-09-13T23:05:11.310119+0000","last_active":"2024-09-13T23:05:34.522982+0000","last_peered":"2024-09-13T23:05:34.522982+0000","last_clean":"2024-09-13T23:05:34.522982+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T23:05:34.522982+0000","last_undegraded":"2024-09-13T23:05:34.522982+0000","last_fullsized":"2024-09-13T23:05:34.522982+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:11.309805+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:03:46.549405+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.309805+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:34.948253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"468'47","reported_seq":134,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.540160+0000","last_change":"2024-09-13T23:05:37.540257+0000","last_active":"2024-09-13T23:05:37.540160+0000","last_peered":"2024-09-13T23:05:37.540160+0000","last_clean":"2024-09-13T23:05:37.540160+0000","last_became_active":"2024-09-13T23:02:24.392564+0000","last_became_peered":"2024-09-13T23:02:24.392564+0000","last_unstale":"2024-09-13T23:05:37.540160+0000","last_undegraded":"2024-09-13T23:05:37.540160+0000","last_fullsized":"2024-09-13T23:05:37.540160+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'47","last_scrub_stamp":"2024-09-13T23:05:37.540012+0000","last_deep_scrub":"468'47","last_deep_scrub_stamp":"2024-09-13T23:01:23.540241+0000","last_clean_scrub_stamp":"2024-09-13T23:05:37.540012+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:43.279521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1048,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524061+0000","last_change":"2024-09-13T23:05:18.312117+0000","last_active":"2024-09-13T23:05:34.524061+0000","last_peered":"2024-09-13T23:05:34.524061+0000","last_clean":"2024-09-13T23:05:34.524061+0000","last_became_active":"2024-09-13T22:58:42.741601+0000","last_became_peered":"2024-09-13T22:58:42.741601+0000","last_unstale":"2024-09-13T23:05:34.524061+0000","last_undegraded":"2024-09-13T23:05:34.524061+0000","last_fullsized":"2024-09-13T23:05:34.524061+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:18.311920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:05:18.311920+0000","last_clean_scrub_stamp":"2024-09-13T23:05:18.311920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:32.363297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1064,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278203+0000","last_change":"2024-09-13T23:05:05.868562+0000","last_active":"2024-09-13T23:05:34.278203+0000","last_peered":"2024-09-13T23:05:34.278203+0000","last_clean":"2024-09-13T23:05:34.278203+0000","last_became_active":"2024-09-13T23:02:24.404336+0000","last_became_peered":"2024-09-13T23:02:24.404336+0000","last_unstale":"2024-09-13T23:05:34.278203+0000","last_undegraded":"2024-09-13T23:05:34.278203+0000","last_fullsized":"2024-09-13T23:05:34.278203+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:05.868455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:53.889215+0000","last_clean_scrub_stamp":"2024-09-13T23:05:05.868455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:33.441126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1032,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392507+0000","last_change":"2024-09-13T23:04:38.675390+0000","last_active":"2024-09-13T23:05:31.392507+0000","last_peered":"2024-09-13T23:05:31.392507+0000","last_clean":"2024-09-13T23:05:31.392507+0000","last_became_active":"2024-09-13T23:02:24.375086+0000","last_became_peered":"2024-09-13T23:02:24.375086+0000","last_unstale":"2024-09-13T23:05:31.392507+0000","last_undegraded":"2024-09-13T23:05:31.392507+0000","last_fullsized":"2024-09-13T23:05:31.392507+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:38.675138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:18.451343+0000","last_clean_scrub_stamp":"2024-09-13T23:04:38.675138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:41.137984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1324,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396531+0000","last_change":"2024-09-13T23:04:49.666637+0000","last_active":"2024-09-13T23:05:31.396531+0000","last_peered":"2024-09-13T23:05:31.396531+0000","last_clean":"2024-09-13T23:05:31.396531+0000","last_became_active":"2024-09-13T23:02:27.391190+0000","last_became_peered":"2024-09-13T23:02:27.391190+0000","last_unstale":"2024-09-13T23:05:31.396531+0000","last_undegraded":"2024-09-13T23:05:31.396531+0000","last_fullsized":"2024-09-13T23:05:31.396531+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:49.666443+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:04:49.666443+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:52.085906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":383,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.522899+0000","last_change":"2024-09-13T23:05:32.213315+0000","last_active":"2024-09-13T23:05:34.522899+0000","last_peered":"2024-09-13T23:05:34.522899+0000","last_clean":"2024-09-13T23:05:34.522899+0000","last_became_active":"2024-09-13T22:58:42.727647+0000","last_became_peered":"2024-09-13T22:58:42.727647+0000","last_unstale":"2024-09-13T23:05:34.522899+0000","last_undegraded":"2024-09-13T23:05:34.522899+0000","last_fullsized":"2024-09-13T23:05:34.522899+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-09-13T23:05:32.213065+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T23:05:32.213065+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:52.221244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"470'2","reported_seq":1078,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.517402+0000","last_change":"2024-09-13T23:05:34.001657+0000","last_active":"2024-09-13T23:05:34.517402+0000","last_peered":"2024-09-13T23:05:34.517402+0000","last_clean":"2024-09-13T23:05:34.517402+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T23:05:34.517402+0000","last_undegraded":"2024-09-13T23:05:34.517402+0000","last_fullsized":"2024-09-13T23:05:34.517402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'2","last_scrub_stamp":"2024-09-13T23:05:34.001454+0000","last_deep_scrub":"470'2","last_deep_scrub_stamp":"2024-09-13T23:01:27.631212+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.001454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:39.769192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1052,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518994+0000","last_change":"2024-09-13T23:04:28.945472+0000","last_active":"2024-09-13T23:05:34.518994+0000","last_peered":"2024-09-13T23:05:34.518994+0000","last_clean":"2024-09-13T23:05:34.518994+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T23:05:34.518994+0000","last_undegraded":"2024-09-13T23:05:34.518994+0000","last_fullsized":"2024-09-13T23:05:34.518994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:28.945284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T23:04:28.945284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.567460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1009,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.395017+0000","last_change":"2024-09-13T23:04:48.697783+0000","last_active":"2024-09-13T23:05:31.395017+0000","last_peered":"2024-09-13T23:05:31.395017+0000","last_clean":"2024-09-13T23:05:31.395017+0000","last_became_active":"2024-09-13T22:58:42.764039+0000","last_became_peered":"2024-09-13T22:58:42.764039+0000","last_unstale":"2024-09-13T23:05:31.395017+0000","last_undegraded":"2024-09-13T23:05:31.395017+0000","last_fullsized":"2024-09-13T23:05:31.395017+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:48.697595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:48.697595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.917387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":1133,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390651+0000","last_change":"2024-09-13T23:05:07.578268+0000","last_active":"2024-09-13T23:05:31.390651+0000","last_peered":"2024-09-13T23:05:31.390651+0000","last_clean":"2024-09-13T23:05:31.390651+0000","last_became_active":"2024-09-13T23:02:27.391398+0000","last_became_peered":"2024-09-13T23:02:27.391398+0000","last_unstale":"2024-09-13T23:05:31.390651+0000","last_undegraded":"2024-09-13T23:05:31.390651+0000","last_fullsized":"2024-09-13T23:05:31.390651+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:07.578066+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T23:05:07.578066+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:29.562152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"485'49","reported_seq":1190,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277072+0000","last_change":"2024-09-13T23:04:38.946230+0000","last_active":"2024-09-13T23:05:34.277072+0000","last_peered":"2024-09-13T23:05:34.277072+0000","last_clean":"2024-09-13T23:05:34.277072+0000","last_became_active":"2024-09-13T23:02:24.403148+0000","last_became_peered":"2024-09-13T23:02:24.403148+0000","last_unstale":"2024-09-13T23:05:34.277072+0000","last_undegraded":"2024-09-13T23:05:34.277072+0000","last_fullsized":"2024-09-13T23:05:34.277072+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'49","last_scrub_stamp":"2024-09-13T23:04:38.946127+0000","last_deep_scrub":"485'49","last_deep_scrub_stamp":"2024-09-13T23:02:11.668680+0000","last_clean_scrub_stamp":"2024-09-13T23:04:38.946127+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:03.311497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":784,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391241+0000","last_change":"2024-09-13T23:04:40.601476+0000","last_active":"2024-09-13T23:05:31.391241+0000","last_peered":"2024-09-13T23:05:31.391241+0000","last_clean":"2024-09-13T23:05:31.391241+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T23:05:31.391241+0000","last_undegraded":"2024-09-13T23:05:31.391241+0000","last_fullsized":"2024-09-13T23:05:31.391241+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:40.601280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T23:04:40.601280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.214708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1019,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524913+0000","last_change":"2024-09-13T23:04:09.535246+0000","last_active":"2024-09-13T23:05:34.524913+0000","last_peered":"2024-09-13T23:05:34.524913+0000","last_clean":"2024-09-13T23:05:34.524913+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T23:05:34.524913+0000","last_undegraded":"2024-09-13T23:05:34.524913+0000","last_fullsized":"2024-09-13T23:05:34.524913+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:09.535055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:49.271179+0000","last_clean_scrub_stamp":"2024-09-13T23:04:09.535055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1022,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.398834+0000","last_change":"2024-09-13T23:05:34.398965+0000","last_active":"2024-09-13T23:05:34.398834+0000","last_peered":"2024-09-13T23:05:34.398834+0000","last_clean":"2024-09-13T23:05:34.398834+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T23:05:34.398834+0000","last_undegraded":"2024-09-13T23:05:34.398834+0000","last_fullsized":"2024-09-13T23:05:34.398834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:34.398662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:59.725449+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.398662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:46.970537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":1086,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391973+0000","last_change":"2024-09-13T23:05:05.810722+0000","last_active":"2024-09-13T23:05:31.391973+0000","last_peered":"2024-09-13T23:05:31.391973+0000","last_clean":"2024-09-13T23:05:31.391973+0000","last_became_active":"2024-09-13T22:58:42.746819+0000","last_became_peered":"2024-09-13T22:58:42.746819+0000","last_unstale":"2024-09-13T23:05:31.391973+0000","last_undegraded":"2024-09-13T23:05:31.391973+0000","last_fullsized":"2024-09-13T23:05:31.391973+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:05:05.810496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:05.810496+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:29.324903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1036,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523970+0000","last_change":"2024-09-13T23:04:56.399458+0000","last_active":"2024-09-13T23:05:34.523970+0000","last_peered":"2024-09-13T23:05:34.523970+0000","last_clean":"2024-09-13T23:05:34.523970+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T23:05:34.523970+0000","last_undegraded":"2024-09-13T23:05:34.523970+0000","last_fullsized":"2024-09-13T23:05:34.523970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:56.399161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:56.399161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:04.611430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1299,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393204+0000","last_change":"2024-09-13T23:04:36.552833+0000","last_active":"2024-09-13T23:05:31.393204+0000","last_peered":"2024-09-13T23:05:31.393204+0000","last_clean":"2024-09-13T23:05:31.393204+0000","last_became_active":"2024-09-13T22:58:45.829938+0000","last_became_peered":"2024-09-13T22:58:45.829938+0000","last_unstale":"2024-09-13T23:05:31.393204+0000","last_undegraded":"2024-09-13T23:05:31.393204+0000","last_fullsized":"2024-09-13T23:05:31.393204+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:36.552637+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:04:36.552637+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:38.051986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1001,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.398387+0000","last_change":"2024-09-13T23:04:25.599319+0000","last_active":"2024-09-13T23:05:31.398387+0000","last_peered":"2024-09-13T23:05:31.398387+0000","last_clean":"2024-09-13T23:05:31.398387+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T23:05:31.398387+0000","last_undegraded":"2024-09-13T23:05:31.398387+0000","last_fullsized":"2024-09-13T23:05:31.398387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:25.599066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T23:04:25.599066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:46.802603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1047,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524768+0000","last_change":"2024-09-13T23:04:36.520054+0000","last_active":"2024-09-13T23:05:34.524768+0000","last_peered":"2024-09-13T23:05:34.524768+0000","last_clean":"2024-09-13T23:05:34.524768+0000","last_became_active":"2024-09-13T23:02:24.384320+0000","last_became_peered":"2024-09-13T23:02:24.384320+0000","last_unstale":"2024-09-13T23:05:34.524768+0000","last_undegraded":"2024-09-13T23:05:34.524768+0000","last_fullsized":"2024-09-13T23:05:34.524768+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:36.519848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:36.519848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:05.163742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":1122,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.397899+0000","last_change":"2024-09-13T23:05:26.907389+0000","last_active":"2024-09-13T23:05:31.397899+0000","last_peered":"2024-09-13T23:05:31.397899+0000","last_clean":"2024-09-13T23:05:31.397899+0000","last_became_active":"2024-09-13T22:58:42.764099+0000","last_became_peered":"2024-09-13T22:58:42.764099+0000","last_unstale":"2024-09-13T23:05:31.397899+0000","last_undegraded":"2024-09-13T23:05:31.397899+0000","last_fullsized":"2024-09-13T23:05:31.397899+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-09-13T23:05:26.907173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:26.907173+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:40.532856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1060,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.519255+0000","last_change":"2024-09-13T23:04:31.937104+0000","last_active":"2024-09-13T23:05:34.519255+0000","last_peered":"2024-09-13T23:05:34.519255+0000","last_clean":"2024-09-13T23:05:34.519255+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T23:05:34.519255+0000","last_undegraded":"2024-09-13T23:05:34.519255+0000","last_fullsized":"2024-09-13T23:05:34.519255+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:31.936914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:31.936914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:34.659341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1044,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518370+0000","last_change":"2024-09-13T23:04:07.905673+0000","last_active":"2024-09-13T23:05:34.518370+0000","last_peered":"2024-09-13T23:05:34.518370+0000","last_clean":"2024-09-13T23:05:34.518370+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T23:05:34.518370+0000","last_undegraded":"2024-09-13T23:05:34.518370+0000","last_fullsized":"2024-09-13T23:05:34.518370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:07.905365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:04:07.905365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":988,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390763+0000","last_change":"2024-09-13T23:04:14.727683+0000","last_active":"2024-09-13T23:05:31.390763+0000","last_peered":"2024-09-13T23:05:31.390763+0000","last_clean":"2024-09-13T23:05:31.390763+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T23:05:31.390763+0000","last_undegraded":"2024-09-13T23:05:31.390763+0000","last_fullsized":"2024-09-13T23:05:31.390763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:14.727486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T23:04:14.727486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"515'35","reported_seq":1109,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524320+0000","last_change":"2024-09-13T23:04:14.434888+0000","last_active":"2024-09-13T23:05:34.524320+0000","last_peered":"2024-09-13T23:05:34.524320+0000","last_clean":"2024-09-13T23:05:34.524320+0000","last_became_active":"2024-09-13T22:58:42.731142+0000","last_became_peered":"2024-09-13T22:58:42.731142+0000","last_unstale":"2024-09-13T23:05:34.524320+0000","last_undegraded":"2024-09-13T23:05:34.524320+0000","last_fullsized":"2024-09-13T23:05:34.524320+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'35","last_scrub_stamp":"2024-09-13T23:04:14.434660+0000","last_deep_scrub":"515'35","last_deep_scrub_stamp":"2024-09-13T23:04:14.434660+0000","last_clean_scrub_stamp":"2024-09-13T23:04:14.434660+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1057,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393988+0000","last_change":"2024-09-13T23:05:00.419977+0000","last_active":"2024-09-13T23:05:31.393988+0000","last_peered":"2024-09-13T23:05:31.393988+0000","last_clean":"2024-09-13T23:05:31.393988+0000","last_became_active":"2024-09-13T23:02:24.393975+0000","last_became_peered":"2024-09-13T23:02:24.393975+0000","last_unstale":"2024-09-13T23:05:31.393988+0000","last_undegraded":"2024-09-13T23:05:31.393988+0000","last_fullsized":"2024-09-13T23:05:31.393988+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:00.419783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:33.671179+0000","last_clean_scrub_stamp":"2024-09-13T23:05:00.419783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:09.887151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":508,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518171+0000","last_change":"2024-09-13T23:05:30.990576+0000","last_active":"2024-09-13T23:05:34.518171+0000","last_peered":"2024-09-13T23:05:34.518171+0000","last_clean":"2024-09-13T23:05:34.518171+0000","last_became_active":"2024-09-13T23:03:27.008041+0000","last_became_peered":"2024-09-13T23:03:27.008041+0000","last_unstale":"2024-09-13T23:05:34.518171+0000","last_undegraded":"2024-09-13T23:05:34.518171+0000","last_fullsized":"2024-09-13T23:05:34.518171+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:30.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:30.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.913782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1023,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277041+0000","last_change":"2024-09-13T23:04:46.921759+0000","last_active":"2024-09-13T23:05:34.277041+0000","last_peered":"2024-09-13T23:05:34.277041+0000","last_clean":"2024-09-13T23:05:34.277041+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T23:05:34.277041+0000","last_undegraded":"2024-09-13T23:05:34.277041+0000","last_fullsized":"2024-09-13T23:05:34.277041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:46.921560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:46.921560+0000","last_clean_scrub_stamp":"2024-09-13T23:04:46.921560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.585062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":1160,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396839+0000","last_change":"2024-09-13T23:04:47.655901+0000","last_active":"2024-09-13T23:05:31.396839+0000","last_peered":"2024-09-13T23:05:31.396839+0000","last_clean":"2024-09-13T23:05:31.396839+0000","last_became_active":"2024-09-13T23:02:24.394074+0000","last_became_peered":"2024-09-13T23:02:24.394074+0000","last_unstale":"2024-09-13T23:05:31.396839+0000","last_undegraded":"2024-09-13T23:05:31.396839+0000","last_fullsized":"2024-09-13T23:05:31.396839+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:04:47.655681+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-09-13T23:01:36.239714+0000","last_clean_scrub_stamp":"2024-09-13T23:04:47.655681+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.281721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":494,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.519208+0000","last_change":"2024-09-13T23:04:41.954847+0000","last_active":"2024-09-13T23:05:34.519208+0000","last_peered":"2024-09-13T23:05:34.519208+0000","last_clean":"2024-09-13T23:05:34.519208+0000","last_became_active":"2024-09-13T23:03:27.005723+0000","last_became_peered":"2024-09-13T23:03:27.005723+0000","last_unstale":"2024-09-13T23:05:34.519208+0000","last_undegraded":"2024-09-13T23:05:34.519208+0000","last_fullsized":"2024-09-13T23:05:34.519208+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:41.954640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:41.954640+0000","last_clean_scrub_stamp":"2024-09-13T23:04:41.954640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.483498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"492'17","reported_seq":1416,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.394554+0000","last_change":"2024-09-13T23:05:16.790332+0000","last_active":"2024-09-13T23:05:31.394554+0000","last_peered":"2024-09-13T23:05:31.394554+0000","last_clean":"2024-09-13T23:05:31.394554+0000","last_became_active":"2024-09-13T23:02:24.394764+0000","last_became_peered":"2024-09-13T23:02:24.394764+0000","last_unstale":"2024-09-13T23:05:31.394554+0000","last_undegraded":"2024-09-13T23:05:31.394554+0000","last_fullsized":"2024-09-13T23:05:31.394554+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'17","last_scrub_stamp":"2024-09-13T23:05:16.790128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:16.790128+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:42.432413+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1027,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523375+0000","last_change":"2024-09-13T23:04:35.498511+0000","last_active":"2024-09-13T23:05:34.523375+0000","last_peered":"2024-09-13T23:05:34.523375+0000","last_clean":"2024-09-13T23:05:34.523375+0000","last_became_active":"2024-09-13T22:58:42.732057+0000","last_became_peered":"2024-09-13T22:58:42.732057+0000","last_unstale":"2024-09-13T23:05:34.523375+0000","last_undegraded":"2024-09-13T23:05:34.523375+0000","last_fullsized":"2024-09-13T23:05:34.523375+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:35.498321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.498321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:04.131799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"525'32","reported_seq":1132,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396766+0000","last_change":"2024-09-13T23:05:08.878192+0000","last_active":"2024-09-13T23:05:31.396766+0000","last_peered":"2024-09-13T23:05:31.396766+0000","last_clean":"2024-09-13T23:05:31.396766+0000","last_became_active":"2024-09-13T23:02:24.402569+0000","last_became_peered":"2024-09-13T23:02:24.402569+0000","last_unstale":"2024-09-13T23:05:31.396766+0000","last_undegraded":"2024-09-13T23:05:31.396766+0000","last_fullsized":"2024-09-13T23:05:31.396766+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'32","last_scrub_stamp":"2024-09-13T23:05:08.877954+0000","last_deep_scrub":"525'32","last_deep_scrub_stamp":"2024-09-13T23:05:08.877954+0000","last_clean_scrub_stamp":"2024-09-13T23:05:08.877954+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:30.064240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":114,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.280050+0000","last_change":"2024-09-13T23:04:42.736278+0000","last_active":"2024-09-13T23:05:34.280050+0000","last_peered":"2024-09-13T23:05:34.280050+0000","last_clean":"2024-09-13T23:05:34.280050+0000","last_became_active":"2024-09-13T23:02:24.393934+0000","last_became_peered":"2024-09-13T23:02:24.393934+0000","last_unstale":"2024-09-13T23:05:34.280050+0000","last_undegraded":"2024-09-13T23:05:34.280050+0000","last_fullsized":"2024-09-13T23:05:34.280050+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T23:04:42.736058+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T23:04:42.736058+0000","last_clean_scrub_stamp":"2024-09-13T23:04:42.736058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:44.223373+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":486,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.517810+0000","last_change":"2024-09-13T23:04:14.939239+0000","last_active":"2024-09-13T23:05:34.517810+0000","last_peered":"2024-09-13T23:05:34.517810+0000","last_clean":"2024-09-13T23:05:34.517810+0000","last_became_active":"2024-09-13T23:03:26.934191+0000","last_became_peered":"2024-09-13T23:03:26.934191+0000","last_unstale":"2024-09-13T23:05:34.517810+0000","last_undegraded":"2024-09-13T23:05:34.517810+0000","last_fullsized":"2024-09-13T23:05:34.517810+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:14.939085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:14.939085+0000","last_clean_scrub_stamp":"2024-09-13T23:04:14.939085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1004,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392202+0000","last_change":"2024-09-13T23:05:03.828589+0000","last_active":"2024-09-13T23:05:31.392202+0000","last_peered":"2024-09-13T23:05:31.392202+0000","last_clean":"2024-09-13T23:05:31.392202+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T23:05:31.392202+0000","last_undegraded":"2024-09-13T23:05:31.392202+0000","last_fullsized":"2024-09-13T23:05:31.392202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:03.828445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:12.611718+0000","last_clean_scrub_stamp":"2024-09-13T23:05:03.828445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.566595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":1159,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390989+0000","last_change":"2024-09-13T23:04:55.695652+0000","last_active":"2024-09-13T23:05:31.390989+0000","last_peered":"2024-09-13T23:05:31.390989+0000","last_clean":"2024-09-13T23:05:31.390989+0000","last_became_active":"2024-09-13T23:02:24.395338+0000","last_became_peered":"2024-09-13T23:02:24.395338+0000","last_unstale":"2024-09-13T23:05:31.390989+0000","last_undegraded":"2024-09-13T23:05:31.390989+0000","last_fullsized":"2024-09-13T23:05:31.390989+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:04:55.695431+0000","last_deep_scrub":"397'32","last_deep_scrub_stamp":"2024-09-13T23:03:38.720852+0000","last_clean_scrub_stamp":"2024-09-13T23:04:55.695431+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:19.318431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1059,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518435+0000","last_change":"2024-09-13T23:05:24.958255+0000","last_active":"2024-09-13T23:05:34.518435+0000","last_peered":"2024-09-13T23:05:34.518435+0000","last_clean":"2024-09-13T23:05:34.518435+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T23:05:34.518435+0000","last_undegraded":"2024-09-13T23:05:34.518435+0000","last_fullsized":"2024-09-13T23:05:34.518435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:24.958078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:53.872730+0000","last_clean_scrub_stamp":"2024-09-13T23:05:24.958078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:26.778270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":999,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.394502+0000","last_change":"2024-09-13T23:04:30.658861+0000","last_active":"2024-09-13T23:05:31.394502+0000","last_peered":"2024-09-13T23:05:31.394502+0000","last_clean":"2024-09-13T23:05:31.394502+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T23:05:31.394502+0000","last_undegraded":"2024-09-13T23:05:31.394502+0000","last_fullsized":"2024-09-13T23:05:31.394502+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:30.658752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:05.183873+0000","last_clean_scrub_stamp":"2024-09-13T23:04:30.658752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.535714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1071,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518136+0000","last_change":"2024-09-13T23:04:01.973357+0000","last_active":"2024-09-13T23:05:34.518136+0000","last_peered":"2024-09-13T23:05:34.518136+0000","last_clean":"2024-09-13T23:05:34.518136+0000","last_became_active":"2024-09-13T23:02:24.371637+0000","last_became_peered":"2024-09-13T23:02:24.371637+0000","last_unstale":"2024-09-13T23:05:34.518136+0000","last_undegraded":"2024-09-13T23:05:34.518136+0000","last_fullsized":"2024-09-13T23:05:34.518136+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:01.973166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T23:04:01.973166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":272,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.399403+0000","last_change":"2024-09-13T23:04:03.566365+0000","last_active":"2024-09-13T23:05:31.399403+0000","last_peered":"2024-09-13T23:05:31.399403+0000","last_clean":"2024-09-13T23:05:31.399403+0000","last_became_active":"2024-09-13T22:58:42.736178+0000","last_became_peered":"2024-09-13T22:58:42.736178+0000","last_unstale":"2024-09-13T23:05:31.399403+0000","last_undegraded":"2024-09-13T23:05:31.399403+0000","last_fullsized":"2024-09-13T23:05:31.399403+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T23:04:03.566160+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T23:04:03.566160+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:31.547456+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":272,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.399638+0000","last_change":"2024-09-13T23:03:58.566417+0000","last_active":"2024-09-13T23:05:31.399638+0000","last_peered":"2024-09-13T23:05:31.399638+0000","last_clean":"2024-09-13T23:05:31.399638+0000","last_became_active":"2024-09-13T23:02:24.403343+0000","last_became_peered":"2024-09-13T23:02:24.403343+0000","last_unstale":"2024-09-13T23:05:31.399638+0000","last_undegraded":"2024-09-13T23:05:31.399638+0000","last_fullsized":"2024-09-13T23:05:31.399638+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:03:58.566228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T23:03:58.566228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1018,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.525395+0000","last_change":"2024-09-13T23:04:02.542656+0000","last_active":"2024-09-13T23:05:34.525395+0000","last_peered":"2024-09-13T23:05:34.525395+0000","last_clean":"2024-09-13T23:05:34.525395+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T23:05:34.525395+0000","last_undegraded":"2024-09-13T23:05:34.525395+0000","last_fullsized":"2024-09-13T23:05:34.525395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:02.542465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:04:02.542465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1031,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.396180+0000","last_change":"2024-09-13T23:05:06.810218+0000","last_active":"2024-09-13T23:05:31.396180+0000","last_peered":"2024-09-13T23:05:31.396180+0000","last_clean":"2024-09-13T23:05:31.396180+0000","last_became_active":"2024-09-13T23:02:24.394992+0000","last_became_peered":"2024-09-13T23:02:24.394992+0000","last_unstale":"2024-09-13T23:05:31.396180+0000","last_undegraded":"2024-09-13T23:05:31.396180+0000","last_fullsized":"2024-09-13T23:05:31.396180+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:06.809938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:06.809938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:14.691502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":1154,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.395668+0000","last_change":"2024-09-13T23:04:45.667263+0000","last_active":"2024-09-13T23:05:31.395668+0000","last_peered":"2024-09-13T23:05:31.395668+0000","last_clean":"2024-09-13T23:05:31.395668+0000","last_became_active":"2024-09-13T23:02:24.395825+0000","last_became_peered":"2024-09-13T23:02:24.395825+0000","last_unstale":"2024-09-13T23:05:31.395668+0000","last_undegraded":"2024-09-13T23:05:31.395668+0000","last_fullsized":"2024-09-13T23:05:31.395668+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T23:04:45.667060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:45.667060+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:14.137058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1029,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523896+0000","last_change":"2024-09-13T23:03:50.627244+0000","last_active":"2024-09-13T23:05:34.523896+0000","last_peered":"2024-09-13T23:05:34.523896+0000","last_clean":"2024-09-13T23:05:34.523896+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T23:05:34.523896+0000","last_undegraded":"2024-09-13T23:05:34.523896+0000","last_fullsized":"2024-09-13T23:05:34.523896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:03:50.627046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:50.627046+0000","last_clean_scrub_stamp":"2024-09-13T23:03:50.627046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":1376,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524389+0000","last_change":"2024-09-13T23:04:57.410753+0000","last_active":"2024-09-13T23:05:34.524389+0000","last_peered":"2024-09-13T23:05:34.524389+0000","last_clean":"2024-09-13T23:05:34.524389+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T23:05:34.524389+0000","last_undegraded":"2024-09-13T23:05:34.524389+0000","last_fullsized":"2024-09-13T23:05:34.524389+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T23:04:57.410540+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T23:04:57.410540+0000","last_clean_scrub_stamp":"2024-09-13T23:04:57.410540+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:16.645440+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1014,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278097+0000","last_change":"2024-09-13T23:04:18.922767+0000","last_active":"2024-09-13T23:05:34.278097+0000","last_peered":"2024-09-13T23:05:34.278097+0000","last_clean":"2024-09-13T23:05:34.278097+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T23:05:34.278097+0000","last_undegraded":"2024-09-13T23:05:34.278097+0000","last_fullsized":"2024-09-13T23:05:34.278097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:18.922585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:18.922585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:43.268564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":1064,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277835+0000","last_change":"2024-09-13T23:05:29.672609+0000","last_active":"2024-09-13T23:05:34.277835+0000","last_peered":"2024-09-13T23:05:34.277835+0000","last_clean":"2024-09-13T23:05:34.277835+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T23:05:34.277835+0000","last_undegraded":"2024-09-13T23:05:34.277835+0000","last_fullsized":"2024-09-13T23:05:34.277835+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:05:29.672403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:29.672403+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:50.208059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1050,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278427+0000","last_change":"2024-09-13T23:04:14.878645+0000","last_active":"2024-09-13T23:05:34.278427+0000","last_peered":"2024-09-13T23:05:34.278427+0000","last_clean":"2024-09-13T23:05:34.278427+0000","last_became_active":"2024-09-13T23:02:24.401203+0000","last_became_peered":"2024-09-13T23:02:24.401203+0000","last_unstale":"2024-09-13T23:05:34.278427+0000","last_undegraded":"2024-09-13T23:05:34.278427+0000","last_fullsized":"2024-09-13T23:05:34.278427+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:14.878570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:14.878570+0000","last_clean_scrub_stamp":"2024-09-13T23:04:14.878570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:40.029415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":1352,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393105+0000","last_change":"2024-09-13T23:05:29.975146+0000","last_active":"2024-09-13T23:05:31.393105+0000","last_peered":"2024-09-13T23:05:31.393105+0000","last_clean":"2024-09-13T23:05:31.393105+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T23:05:31.393105+0000","last_undegraded":"2024-09-13T23:05:31.393105+0000","last_fullsized":"2024-09-13T23:05:31.393105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T23:05:29.974953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:29.974953+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:43.737369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.527677+0000","last_change":"2024-09-13T23:05:34.527731+0000","last_active":"2024-09-13T23:05:34.527677+0000","last_peered":"2024-09-13T23:05:34.527677+0000","last_clean":"2024-09-13T23:05:34.527677+0000","last_became_active":"2024-09-13T23:02:24.374176+0000","last_became_peered":"2024-09-13T23:02:24.374176+0000","last_unstale":"2024-09-13T23:05:34.527677+0000","last_undegraded":"2024-09-13T23:05:34.527677+0000","last_fullsized":"2024-09-13T23:05:34.527677+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:34.527540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.527540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:51.848295+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":1101,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.397137+0000","last_change":"2024-09-13T23:04:37.584638+0000","last_active":"2024-09-13T23:05:31.397137+0000","last_peered":"2024-09-13T23:05:31.397137+0000","last_clean":"2024-09-13T23:05:31.397137+0000","last_became_active":"2024-09-13T23:02:24.397236+0000","last_became_peered":"2024-09-13T23:02:24.397236+0000","last_unstale":"2024-09-13T23:05:31.397137+0000","last_undegraded":"2024-09-13T23:05:31.397137+0000","last_fullsized":"2024-09-13T23:05:31.397137+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:04:37.584429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:37.584429+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.354922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":266,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390942+0000","last_change":"2024-09-13T23:04:45.663056+0000","last_active":"2024-09-13T23:05:31.390942+0000","last_peered":"2024-09-13T23:05:31.390942+0000","last_clean":"2024-09-13T23:05:31.390942+0000","last_became_active":"2024-09-13T22:58:42.749062+0000","last_became_peered":"2024-09-13T22:58:42.749062+0000","last_unstale":"2024-09-13T23:05:31.390942+0000","last_undegraded":"2024-09-13T23:05:31.390942+0000","last_fullsized":"2024-09-13T23:05:31.390942+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:45.662801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:45.662801+0000","last_clean_scrub_stamp":"2024-09-13T23:04:45.662801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.902824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":999,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.392514+0000","last_change":"2024-09-13T23:04:24.673769+0000","last_active":"2024-09-13T23:05:31.392514+0000","last_peered":"2024-09-13T23:05:31.392514+0000","last_clean":"2024-09-13T23:05:31.392514+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T23:05:31.392514+0000","last_undegraded":"2024-09-13T23:05:31.392514+0000","last_fullsized":"2024-09-13T23:05:31.392514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:24.673687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:24.673687+0000","last_clean_scrub_stamp":"2024-09-13T23:04:24.673687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:32.726167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1014,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277448+0000","last_change":"2024-09-13T23:05:07.896069+0000","last_active":"2024-09-13T23:05:34.277448+0000","last_peered":"2024-09-13T23:05:34.277448+0000","last_clean":"2024-09-13T23:05:34.277448+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T23:05:34.277448+0000","last_undegraded":"2024-09-13T23:05:34.277448+0000","last_fullsized":"2024-09-13T23:05:34.277448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:07.895792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:07.895792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.567379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":128,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278671+0000","last_change":"2024-09-13T23:04:49.672838+0000","last_active":"2024-09-13T23:05:34.278671+0000","last_peered":"2024-09-13T23:05:34.278671+0000","last_clean":"2024-09-13T23:05:34.278671+0000","last_became_active":"2024-09-13T23:02:24.401214+0000","last_became_peered":"2024-09-13T23:02:24.401214+0000","last_unstale":"2024-09-13T23:05:34.278671+0000","last_undegraded":"2024-09-13T23:05:34.278671+0000","last_fullsized":"2024-09-13T23:05:34.278671+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:04:49.672613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:49.672613+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:05.325793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1032,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393871+0000","last_change":"2024-09-13T23:05:25.566110+0000","last_active":"2024-09-13T23:05:31.393871+0000","last_peered":"2024-09-13T23:05:31.393871+0000","last_clean":"2024-09-13T23:05:31.393871+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T23:05:31.393871+0000","last_undegraded":"2024-09-13T23:05:31.393871+0000","last_fullsized":"2024-09-13T23:05:31.393871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:25.565914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T23:05:25.565914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:50.066578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1024,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391409+0000","last_change":"2024-09-13T23:05:29.542039+0000","last_active":"2024-09-13T23:05:31.391409+0000","last_peered":"2024-09-13T23:05:31.391409+0000","last_clean":"2024-09-13T23:05:31.391409+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T23:05:31.391409+0000","last_undegraded":"2024-09-13T23:05:31.391409+0000","last_fullsized":"2024-09-13T23:05:31.391409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:29.541845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:51.898279+0000","last_clean_scrub_stamp":"2024-09-13T23:05:29.541845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:37.539412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":995,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390451+0000","last_change":"2024-09-13T23:04:58.809306+0000","last_active":"2024-09-13T23:05:31.390451+0000","last_peered":"2024-09-13T23:05:31.390451+0000","last_clean":"2024-09-13T23:05:31.390451+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T23:05:31.390451+0000","last_undegraded":"2024-09-13T23:05:31.390451+0000","last_fullsized":"2024-09-13T23:05:31.390451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:58.809033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T23:04:58.809033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:18.494703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"486'21","reported_seq":1060,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278156+0000","last_change":"2024-09-13T23:04:35.929849+0000","last_active":"2024-09-13T23:05:34.278156+0000","last_peered":"2024-09-13T23:05:34.278156+0000","last_clean":"2024-09-13T23:05:34.278156+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T23:05:34.278156+0000","last_undegraded":"2024-09-13T23:05:34.278156+0000","last_fullsized":"2024-09-13T23:05:34.278156+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'21","last_scrub_stamp":"2024-09-13T23:04:35.929519+0000","last_deep_scrub":"486'21","last_deep_scrub_stamp":"2024-09-13T23:02:16.665442+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.929519+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:03.393944+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1038,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.525652+0000","last_change":"2024-09-13T23:04:39.442052+0000","last_active":"2024-09-13T23:05:34.525652+0000","last_peered":"2024-09-13T23:05:34.525652+0000","last_clean":"2024-09-13T23:05:34.525652+0000","last_became_active":"2024-09-13T22:58:42.726569+0000","last_became_peered":"2024-09-13T22:58:42.726569+0000","last_unstale":"2024-09-13T23:05:34.525652+0000","last_undegraded":"2024-09-13T23:05:34.525652+0000","last_fullsized":"2024-09-13T23:05:34.525652+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:39.441867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T23:04:39.441867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:01.565561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":272,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.399187+0000","last_change":"2024-09-13T23:04:58.701028+0000","last_active":"2024-09-13T23:05:31.399187+0000","last_peered":"2024-09-13T23:05:31.399187+0000","last_clean":"2024-09-13T23:05:31.399187+0000","last_became_active":"2024-09-13T23:02:24.402278+0000","last_became_peered":"2024-09-13T23:02:24.402278+0000","last_unstale":"2024-09-13T23:05:31.399187+0000","last_undegraded":"2024-09-13T23:05:31.399187+0000","last_fullsized":"2024-09-13T23:05:31.399187+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:58.700833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:31.274829+0000","last_clean_scrub_stamp":"2024-09-13T23:04:58.700833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.749174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1019,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.395880+0000","last_change":"2024-09-13T23:04:35.566223+0000","last_active":"2024-09-13T23:05:31.395880+0000","last_peered":"2024-09-13T23:05:31.395880+0000","last_clean":"2024-09-13T23:05:31.395880+0000","last_became_active":"2024-09-13T23:02:24.397699+0000","last_became_peered":"2024-09-13T23:02:24.397699+0000","last_unstale":"2024-09-13T23:05:31.395880+0000","last_undegraded":"2024-09-13T23:05:31.395880+0000","last_fullsized":"2024-09-13T23:05:31.395880+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:35.565966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.565966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.870688+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":1056,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.525489+0000","last_change":"2024-09-13T23:03:57.577789+0000","last_active":"2024-09-13T23:05:34.525489+0000","last_peered":"2024-09-13T23:05:34.525489+0000","last_clean":"2024-09-13T23:05:34.525489+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T23:05:34.525489+0000","last_undegraded":"2024-09-13T23:05:34.525489+0000","last_fullsized":"2024-09-13T23:05:34.525489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:03:57.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:03:57.577560+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":266,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.397811+0000","last_change":"2024-09-13T23:04:43.657652+0000","last_active":"2024-09-13T23:05:31.397811+0000","last_peered":"2024-09-13T23:05:31.397811+0000","last_clean":"2024-09-13T23:05:31.397811+0000","last_became_active":"2024-09-13T22:58:42.742229+0000","last_became_peered":"2024-09-13T22:58:42.742229+0000","last_unstale":"2024-09-13T23:05:31.397811+0000","last_undegraded":"2024-09-13T23:05:31.397811+0000","last_fullsized":"2024-09-13T23:05:31.397811+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:43.657462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T23:04:43.657462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:43.708479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1041,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391573+0000","last_change":"2024-09-13T23:05:04.766151+0000","last_active":"2024-09-13T23:05:31.391573+0000","last_peered":"2024-09-13T23:05:31.391573+0000","last_clean":"2024-09-13T23:05:31.391573+0000","last_became_active":"2024-09-13T23:02:24.368599+0000","last_became_peered":"2024-09-13T23:02:24.368599+0000","last_unstale":"2024-09-13T23:05:31.391573+0000","last_undegraded":"2024-09-13T23:05:31.391573+0000","last_fullsized":"2024-09-13T23:05:31.391573+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:04.765960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:04.765960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:09.752576+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1015,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391873+0000","last_change":"2024-09-13T23:05:19.911310+0000","last_active":"2024-09-13T23:05:31.391873+0000","last_peered":"2024-09-13T23:05:31.391873+0000","last_clean":"2024-09-13T23:05:31.391873+0000","last_became_active":"2024-09-13T22:58:42.749118+0000","last_became_peered":"2024-09-13T22:58:42.749118+0000","last_unstale":"2024-09-13T23:05:31.391873+0000","last_undegraded":"2024-09-13T23:05:31.391873+0000","last_fullsized":"2024-09-13T23:05:31.391873+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:19.911118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T23:05:19.911118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:35.119533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":998,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.397240+0000","last_change":"2024-09-13T23:04:10.758721+0000","last_active":"2024-09-13T23:05:31.397240+0000","last_peered":"2024-09-13T23:05:31.397240+0000","last_clean":"2024-09-13T23:05:31.397240+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T23:05:31.397240+0000","last_undegraded":"2024-09-13T23:05:31.397240+0000","last_fullsized":"2024-09-13T23:05:31.397240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T23:04:10.758512+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T22:56:53.115999+0000","last_clean_scrub_stamp":"2024-09-13T23:04:10.758512+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":400,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.517668+0000","last_change":"2024-09-13T23:04:51.853177+0000","last_active":"2024-09-13T23:05:34.517668+0000","last_peered":"2024-09-13T23:05:34.517668+0000","last_clean":"2024-09-13T23:05:34.517668+0000","last_became_active":"2024-09-13T23:02:24.364155+0000","last_became_peered":"2024-09-13T23:02:24.364155+0000","last_unstale":"2024-09-13T23:05:34.517668+0000","last_undegraded":"2024-09-13T23:05:34.517668+0000","last_fullsized":"2024-09-13T23:05:34.517668+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:04:51.852841+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-09-13T23:04:51.852841+0000","last_clean_scrub_stamp":"2024-09-13T23:04:51.852841+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:20.534556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1062,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523225+0000","last_change":"2024-09-13T23:03:58.532100+0000","last_active":"2024-09-13T23:05:34.523225+0000","last_peered":"2024-09-13T23:05:34.523225+0000","last_clean":"2024-09-13T23:05:34.523225+0000","last_became_active":"2024-09-13T23:02:24.379222+0000","last_became_peered":"2024-09-13T23:02:24.379222+0000","last_unstale":"2024-09-13T23:05:34.523225+0000","last_undegraded":"2024-09-13T23:05:34.523225+0000","last_fullsized":"2024-09-13T23:05:34.523225+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:03:58.531905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T23:03:58.531905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"492'18","reported_seq":372,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.403124+0000","last_change":"2024-09-13T23:05:05.718076+0000","last_active":"2024-09-13T23:05:31.403124+0000","last_peered":"2024-09-13T23:05:31.403124+0000","last_clean":"2024-09-13T23:05:31.403124+0000","last_became_active":"2024-09-13T23:02:24.402311+0000","last_became_peered":"2024-09-13T23:02:24.402311+0000","last_unstale":"2024-09-13T23:05:31.403124+0000","last_undegraded":"2024-09-13T23:05:31.403124+0000","last_fullsized":"2024-09-13T23:05:31.403124+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'18","last_scrub_stamp":"2024-09-13T23:05:05.717946+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T23:05:05.717946+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:33.439629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1044,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390653+0000","last_change":"2024-09-13T23:04:52.456602+0000","last_active":"2024-09-13T23:05:31.390653+0000","last_peered":"2024-09-13T23:05:31.390653+0000","last_clean":"2024-09-13T23:05:31.390653+0000","last_became_active":"2024-09-13T23:02:24.396581+0000","last_became_peered":"2024-09-13T23:02:24.396581+0000","last_unstale":"2024-09-13T23:05:31.390653+0000","last_undegraded":"2024-09-13T23:05:31.390653+0000","last_fullsized":"2024-09-13T23:05:31.390653+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:52.456408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:52.456408+0000","last_clean_scrub_stamp":"2024-09-13T23:04:52.456408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:20.159285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":1053,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.390040+0000","last_change":"2024-09-13T23:04:22.669506+0000","last_active":"2024-09-13T23:05:31.390040+0000","last_peered":"2024-09-13T23:05:31.390040+0000","last_clean":"2024-09-13T23:05:31.390040+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T23:05:31.390040+0000","last_undegraded":"2024-09-13T23:05:31.390040+0000","last_fullsized":"2024-09-13T23:05:31.390040+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:04:22.669174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:22.669174+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.716790+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"518'2","reported_seq":1021,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.391261+0000","last_change":"2024-09-13T23:04:26.663817+0000","last_active":"2024-09-13T23:05:31.391261+0000","last_peered":"2024-09-13T23:05:31.391261+0000","last_clean":"2024-09-13T23:05:31.391261+0000","last_became_active":"2024-09-13T22:58:42.748914+0000","last_became_peered":"2024-09-13T22:58:42.748914+0000","last_unstale":"2024-09-13T23:05:31.391261+0000","last_undegraded":"2024-09-13T23:05:31.391261+0000","last_fullsized":"2024-09-13T23:05:31.391261+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'2","last_scrub_stamp":"2024-09-13T23:04:26.663512+0000","last_deep_scrub":"518'2","last_deep_scrub_stamp":"2024-09-13T23:04:26.663512+0000","last_clean_scrub_stamp":"2024-09-13T23:04:26.663512+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:50.222772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1028,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278348+0000","last_change":"2024-09-13T23:05:08.860000+0000","last_active":"2024-09-13T23:05:34.278348+0000","last_peered":"2024-09-13T23:05:34.278348+0000","last_clean":"2024-09-13T23:05:34.278348+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T23:05:34.278348+0000","last_undegraded":"2024-09-13T23:05:34.278348+0000","last_fullsized":"2024-09-13T23:05:34.278348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:08.859747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T23:05:08.859747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.405458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.279543+0000","last_change":"2024-09-13T23:04:29.697193+0000","last_active":"2024-09-13T23:05:34.279543+0000","last_peered":"2024-09-13T23:05:34.279543+0000","last_clean":"2024-09-13T23:05:34.279543+0000","last_became_active":"2024-09-13T23:02:24.389985+0000","last_became_peered":"2024-09-13T23:02:24.389985+0000","last_unstale":"2024-09-13T23:05:34.279543+0000","last_undegraded":"2024-09-13T23:05:34.279543+0000","last_fullsized":"2024-09-13T23:05:34.279543+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:29.696999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:29.696999+0000","last_clean_scrub_stamp":"2024-09-13T23:04:29.696999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.339215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":280,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.400763+0000","last_change":"2024-09-13T23:04:47.663076+0000","last_active":"2024-09-13T23:05:31.400763+0000","last_peered":"2024-09-13T23:05:31.400763+0000","last_clean":"2024-09-13T23:05:31.400763+0000","last_became_active":"2024-09-13T23:02:24.375223+0000","last_became_peered":"2024-09-13T23:02:24.375223+0000","last_unstale":"2024-09-13T23:05:31.400763+0000","last_undegraded":"2024-09-13T23:05:31.400763+0000","last_fullsized":"2024-09-13T23:05:31.400763+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:47.662786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:47.662786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:55.870847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":1096,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278192+0000","last_change":"2024-09-13T23:04:31.962978+0000","last_active":"2024-09-13T23:05:34.278192+0000","last_peered":"2024-09-13T23:05:34.278192+0000","last_clean":"2024-09-13T23:05:34.278192+0000","last_became_active":"2024-09-13T23:02:24.401750+0000","last_became_peered":"2024-09-13T23:02:24.401750+0000","last_unstale":"2024-09-13T23:05:34.278192+0000","last_undegraded":"2024-09-13T23:05:34.278192+0000","last_fullsized":"2024-09-13T23:05:34.278192+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'11","last_scrub_stamp":"2024-09-13T23:04:31.962741+0000","last_deep_scrub":"397'11","last_deep_scrub_stamp":"2024-09-13T23:03:21.879061+0000","last_clean_scrub_stamp":"2024-09-13T23:04:31.962741+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:47.575075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1040,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393722+0000","last_change":"2024-09-13T23:04:12.340625+0000","last_active":"2024-09-13T23:05:31.393722+0000","last_peered":"2024-09-13T23:05:31.393722+0000","last_clean":"2024-09-13T23:05:31.393722+0000","last_became_active":"2024-09-13T23:02:24.396469+0000","last_became_peered":"2024-09-13T23:02:24.396469+0000","last_unstale":"2024-09-13T23:05:31.393722+0000","last_undegraded":"2024-09-13T23:05:31.393722+0000","last_fullsized":"2024-09-13T23:05:31.393722+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:12.340435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:45.676834+0000","last_clean_scrub_stamp":"2024-09-13T23:04:12.340435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:31.449367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"498'2","reported_seq":1019,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.394416+0000","last_change":"2024-09-13T23:04:18.617979+0000","last_active":"2024-09-13T23:05:31.394416+0000","last_peered":"2024-09-13T23:05:31.394416+0000","last_clean":"2024-09-13T23:05:31.394416+0000","last_became_active":"2024-09-13T22:58:42.748249+0000","last_became_peered":"2024-09-13T22:58:42.748249+0000","last_unstale":"2024-09-13T23:05:31.394416+0000","last_undegraded":"2024-09-13T23:05:31.394416+0000","last_fullsized":"2024-09-13T23:05:31.394416+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'2","last_scrub_stamp":"2024-09-13T23:04:18.617787+0000","last_deep_scrub":"498'2","last_deep_scrub_stamp":"2024-09-13T23:02:57.250150+0000","last_clean_scrub_stamp":"2024-09-13T23:04:18.617787+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:32.491476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1019,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-13T23:05:31.393299+0000","last_change":"2024-09-13T23:04:35.688824+0000","last_active":"2024-09-13T23:05:31.393299+0000","last_peered":"2024-09-13T23:05:31.393299+0000","last_clean":"2024-09-13T23:05:31.393299+0000","last_became_active":"2024-09-13T22:58:42.769318+0000","last_became_peered":"2024-09-13T22:58:42.769318+0000","last_unstale":"2024-09-13T23:05:31.393299+0000","last_undegraded":"2024-09-13T23:05:31.393299+0000","last_fullsized":"2024-09-13T23:05:31.393299+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:35.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.626403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":126,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278746+0000","last_change":"2024-09-13T23:05:17.676676+0000","last_active":"2024-09-13T23:05:34.278746+0000","last_peered":"2024-09-13T23:05:34.278746+0000","last_clean":"2024-09-13T23:05:34.278746+0000","last_became_active":"2024-09-13T23:02:24.374040+0000","last_became_peered":"2024-09-13T23:02:24.374040+0000","last_unstale":"2024-09-13T23:05:34.278746+0000","last_undegraded":"2024-09-13T23:05:34.278746+0000","last_fullsized":"2024-09-13T23:05:34.278746+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:05:17.676461+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T23:05:17.676461+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:45.465858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278365+0000","last_change":"2024-09-13T23:04:28.722797+0000","last_active":"2024-09-13T23:05:34.278365+0000","last_peered":"2024-09-13T23:05:34.278365+0000","last_clean":"2024-09-13T23:05:34.278365+0000","last_became_active":"2024-09-13T23:02:24.400638+0000","last_became_peered":"2024-09-13T23:02:24.400638+0000","last_unstale":"2024-09-13T23:05:34.278365+0000","last_undegraded":"2024-09-13T23:05:34.278365+0000","last_fullsized":"2024-09-13T23:05:34.278365+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:28.722603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T23:04:28.722603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.172556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1645,"num_read_kb":1640,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1552,"num_read_kb":1309,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":522,"num_bytes_recovered":10450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809176,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726500,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93018652,"statfs":{"total":95995035648,"available":95251099648,"internally_reserved":0,"allocated":1196032,"data_stored":741801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136227,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762240,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982912,"statfs":{"total":95995035648,"available":95214501888,"internally_reserved":0,"allocated":831488,"data_stored":411579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528687,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668440,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93076712,"statfs":{"total":95995035648,"available":95310553088,"internally_reserved":0,"allocated":1323008,"data_stored":941978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888441,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473396,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271756,"statfs":{"total":95995035648,"available":95510278144,"internally_reserved":0,"allocated":757760,"data_stored":352030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084674,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893316,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":92851836,"statfs":{"total":95995035648,"available":95080280064,"internally_reserved":0,"allocated":1429504,"data_stored":1002770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":913308103},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379022,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618140,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93127012,"statfs":{"total":95995035648,"available":95362060288,"internally_reserved":0,"allocated":1118208,"data_stored":675623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575256,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812224,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92932928,"statfs":{"total":95995035648,"available":95163318272,"internally_reserved":0,"allocated":835584,"data_stored":412560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738787,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826128,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92919024,"statfs":{"total":95995035648,"available":95149080576,"internally_reserved":0,"allocated":724992,"data_stored":352729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T23:05:41.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T23:05:41.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.962310+0000 osd.2 (osd.2) 383 : cluster [DBG] 5.8 scrub starts 2024-09-13T23:05:41.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: cluster 2024-09-13T23:05:39.963517+0000 osd.2 (osd.2) 384 : cluster [DBG] 5.8 scrub ok 2024-09-13T23:05:41.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: audit 2024-09-13T23:05:40.105847+0000 mgr.y (mgr.14613) 1036 : audit [DBG] from='client.16311 v1:172.21.15.178:0/1948276642' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: cluster 2024-09-13T23:05:40.538871+0000 osd.4 (osd.4) 374 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: cluster 2024-09-13T23:05:40.543121+0000 osd.4 (osd.4) 375 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.198292+0000 osd.5 (osd.5) 605 : cluster [DBG] 5.7 deep-scrub starts 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.201187+0000 osd.5 (osd.5) 606 : cluster [DBG] 5.7 deep-scrub ok 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.962310+0000 osd.2 (osd.2) 383 : cluster [DBG] 5.8 scrub starts 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: cluster 2024-09-13T23:05:39.963517+0000 osd.2 (osd.2) 384 : cluster [DBG] 5.8 scrub ok 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: audit 2024-09-13T23:05:40.105847+0000 mgr.y (mgr.14613) 1036 : audit [DBG] from='client.16311 v1:172.21.15.178:0/1948276642' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: cluster 2024-09-13T23:05:40.538871+0000 osd.4 (osd.4) 374 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: cluster 2024-09-13T23:05:40.543121+0000 osd.4 (osd.4) 375 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.198292+0000 osd.5 (osd.5) 605 : cluster [DBG] 5.7 deep-scrub starts 2024-09-13T23:05:41.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:41 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.201187+0000 osd.5 (osd.5) 606 : cluster [DBG] 5.7 deep-scrub ok 2024-09-13T23:05:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.962310+0000 osd.2 (osd.2) 383 : cluster [DBG] 5.8 scrub starts 2024-09-13T23:05:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: cluster 2024-09-13T23:05:39.963517+0000 osd.2 (osd.2) 384 : cluster [DBG] 5.8 scrub ok 2024-09-13T23:05:41.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: audit 2024-09-13T23:05:40.105847+0000 mgr.y (mgr.14613) 1036 : audit [DBG] from='client.16311 v1:172.21.15.178:0/1948276642' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:05:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: cluster 2024-09-13T23:05:40.538871+0000 osd.4 (osd.4) 374 : cluster [DBG] 3.9 scrub starts 2024-09-13T23:05:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: cluster 2024-09-13T23:05:40.543121+0000 osd.4 (osd.4) 375 : cluster [DBG] 3.9 scrub ok 2024-09-13T23:05:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.198292+0000 osd.5 (osd.5) 605 : cluster [DBG] 5.7 deep-scrub starts 2024-09-13T23:05:41.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:41 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.201187+0000 osd.5 (osd.5) 606 : cluster [DBG] 5.7 deep-scrub ok 2024-09-13T23:05:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.130707+0000 mgr.y (mgr.14613) 1037 : cluster [DBG] pgmap v1252: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:05:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.894365+0000 osd.7 (osd.7) 598 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:05:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.896408+0000 osd.7 (osd.7) 599 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:05:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[29889]: cluster 2024-09-13T23:05:42.200477+0000 osd.5 (osd.5) 607 : cluster [DBG] 2.12 scrub starts 2024-09-13T23:05:42.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[29889]: cluster 2024-09-13T23:05:42.202655+0000 osd.5 (osd.5) 608 : cluster [DBG] 2.12 scrub ok 2024-09-13T23:05:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.130707+0000 mgr.y (mgr.14613) 1037 : cluster [DBG] pgmap v1252: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:05:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.894365+0000 osd.7 (osd.7) 598 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:05:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.896408+0000 osd.7 (osd.7) 599 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:05:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[22840]: cluster 2024-09-13T23:05:42.200477+0000 osd.5 (osd.5) 607 : cluster [DBG] 2.12 scrub starts 2024-09-13T23:05:42.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:42 smithi178 bash[22840]: cluster 2024-09-13T23:05:42.202655+0000 osd.5 (osd.5) 608 : cluster [DBG] 2.12 scrub ok 2024-09-13T23:05:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:42 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.130707+0000 mgr.y (mgr.14613) 1037 : cluster [DBG] pgmap v1252: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T23:05:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:42 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.894365+0000 osd.7 (osd.7) 598 : cluster [DBG] 2.1c scrub starts 2024-09-13T23:05:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:42 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.896408+0000 osd.7 (osd.7) 599 : cluster [DBG] 2.1c scrub ok 2024-09-13T23:05:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:42 smithi192 bash[24608]: cluster 2024-09-13T23:05:42.200477+0000 osd.5 (osd.5) 607 : cluster [DBG] 2.12 scrub starts 2024-09-13T23:05:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:42 smithi192 bash[24608]: cluster 2024-09-13T23:05:42.202655+0000 osd.5 (osd.5) 608 : cluster [DBG] 2.12 scrub ok 2024-09-13T23:05:43.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:43 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.714209+0000 osd.6 (osd.6) 406 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:05:43.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:43 smithi192 bash[24608]: cluster 2024-09-13T23:05:41.716672+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:05:43.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:43 smithi192 bash[24608]: cluster 2024-09-13T23:05:42.864678+0000 osd.7 (osd.7) 600 : cluster [DBG] 6.1c deep-scrub starts 2024-09-13T23:05:43.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:43 smithi192 bash[24608]: cluster 2024-09-13T23:05:42.868445+0000 osd.7 (osd.7) 601 : cluster [DBG] 6.1c deep-scrub ok 2024-09-13T23:05:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.714209+0000 osd.6 (osd.6) 406 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:05:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[29889]: cluster 2024-09-13T23:05:41.716672+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:05:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[29889]: cluster 2024-09-13T23:05:42.864678+0000 osd.7 (osd.7) 600 : cluster [DBG] 6.1c deep-scrub starts 2024-09-13T23:05:43.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[29889]: cluster 2024-09-13T23:05:42.868445+0000 osd.7 (osd.7) 601 : cluster [DBG] 6.1c deep-scrub ok 2024-09-13T23:05:43.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.714209+0000 osd.6 (osd.6) 406 : cluster [DBG] 2.11 scrub starts 2024-09-13T23:05:43.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[22840]: cluster 2024-09-13T23:05:41.716672+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.11 scrub ok 2024-09-13T23:05:43.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[22840]: cluster 2024-09-13T23:05:42.864678+0000 osd.7 (osd.7) 600 : cluster [DBG] 6.1c deep-scrub starts 2024-09-13T23:05:43.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:43 smithi178 bash[22840]: cluster 2024-09-13T23:05:42.868445+0000 osd.7 (osd.7) 601 : cluster [DBG] 6.1c deep-scrub ok 2024-09-13T23:05:44.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:42.915708+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.c scrub starts 2024-09-13T23:05:44.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:42.918016+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.c scrub ok 2024-09-13T23:05:44.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:43.131475+0000 mgr.y (mgr.14613) 1038 : cluster [DBG] pgmap v1253: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:44.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:43.831456+0000 osd.7 (osd.7) 602 : cluster [DBG] 5.f scrub starts 2024-09-13T23:05:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:43.833757+0000 osd.7 (osd.7) 603 : cluster [DBG] 5.f scrub ok 2024-09-13T23:05:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:44.258339+0000 osd.5 (osd.5) 609 : cluster [DBG] 3.c scrub starts 2024-09-13T23:05:44.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:44 smithi192 bash[24608]: cluster 2024-09-13T23:05:44.262295+0000 osd.5 (osd.5) 610 : cluster [DBG] 3.c scrub ok 2024-09-13T23:05:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:42.915708+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.c scrub starts 2024-09-13T23:05:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:42.918016+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.c scrub ok 2024-09-13T23:05:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:43.131475+0000 mgr.y (mgr.14613) 1038 : cluster [DBG] pgmap v1253: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:44.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:43.831456+0000 osd.7 (osd.7) 602 : cluster [DBG] 5.f scrub starts 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:43.833757+0000 osd.7 (osd.7) 603 : cluster [DBG] 5.f scrub ok 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:44.258339+0000 osd.5 (osd.5) 609 : cluster [DBG] 3.c scrub starts 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[29889]: cluster 2024-09-13T23:05:44.262295+0000 osd.5 (osd.5) 610 : cluster [DBG] 3.c scrub ok 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:42.915708+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.c scrub starts 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:42.918016+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.c scrub ok 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:43.131475+0000 mgr.y (mgr.14613) 1038 : cluster [DBG] pgmap v1253: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:43.831456+0000 osd.7 (osd.7) 602 : cluster [DBG] 5.f scrub starts 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:43.833757+0000 osd.7 (osd.7) 603 : cluster [DBG] 5.f scrub ok 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:44.258339+0000 osd.5 (osd.5) 609 : cluster [DBG] 3.c scrub starts 2024-09-13T23:05:44.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:44 smithi178 bash[22840]: cluster 2024-09-13T23:05:44.262295+0000 osd.5 (osd.5) 610 : cluster [DBG] 3.c scrub ok 2024-09-13T23:05:45.853 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:45 smithi192 bash[24608]: cluster 2024-09-13T23:05:44.823871+0000 osd.7 (osd.7) 604 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:05:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:45 smithi192 bash[24608]: cluster 2024-09-13T23:05:44.826716+0000 osd.7 (osd.7) 605 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:05:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:45 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.236273+0000 osd.5 (osd.5) 611 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:05:45.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:45 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.238668+0000 osd.5 (osd.5) 612 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:05:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[29889]: cluster 2024-09-13T23:05:44.823871+0000 osd.7 (osd.7) 604 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:05:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[29889]: cluster 2024-09-13T23:05:44.826716+0000 osd.7 (osd.7) 605 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:05:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.236273+0000 osd.5 (osd.5) 611 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:05:45.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.238668+0000 osd.5 (osd.5) 612 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:05:45.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[22840]: cluster 2024-09-13T23:05:44.823871+0000 osd.7 (osd.7) 604 : cluster [DBG] 1.0 scrub starts 2024-09-13T23:05:45.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[22840]: cluster 2024-09-13T23:05:44.826716+0000 osd.7 (osd.7) 605 : cluster [DBG] 1.0 scrub ok 2024-09-13T23:05:45.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.236273+0000 osd.5 (osd.5) 611 : cluster [DBG] 6.19 scrub starts 2024-09-13T23:05:45.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:45 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.238668+0000 osd.5 (osd.5) 612 : cluster [DBG] 6.19 scrub ok 2024-09-13T23:05:46.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:05:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:44.820546+0000 osd.6 (osd.6) 408 : cluster [DBG] 3.10 deep-scrub starts 2024-09-13T23:05:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:44.824577+0000 osd.6 (osd.6) 409 : cluster [DBG] 3.10 deep-scrub ok 2024-09-13T23:05:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.132901+0000 mgr.y (mgr.14613) 1039 : cluster [DBG] pgmap v1254: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T23:05:46.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.522862+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:05:46.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.526194+0000 osd.1 (osd.1) 539 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:05:46.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.825357+0000 osd.7 (osd.7) 606 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:05:46.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.827447+0000 osd.7 (osd.7) 607 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:05:46.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.217287+0000 osd.5 (osd.5) 613 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:05:46.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:46 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.219728+0000 osd.5 (osd.5) 614 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:05:46.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:44.820546+0000 osd.6 (osd.6) 408 : cluster [DBG] 3.10 deep-scrub starts 2024-09-13T23:05:46.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:44.824577+0000 osd.6 (osd.6) 409 : cluster [DBG] 3.10 deep-scrub ok 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.132901+0000 mgr.y (mgr.14613) 1039 : cluster [DBG] pgmap v1254: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.522862+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.526194+0000 osd.1 (osd.1) 539 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.825357+0000 osd.7 (osd.7) 606 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.827447+0000 osd.7 (osd.7) 607 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.217287+0000 osd.5 (osd.5) 613 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.219728+0000 osd.5 (osd.5) 614 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:44.820546+0000 osd.6 (osd.6) 408 : cluster [DBG] 3.10 deep-scrub starts 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:44.824577+0000 osd.6 (osd.6) 409 : cluster [DBG] 3.10 deep-scrub ok 2024-09-13T23:05:46.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.132901+0000 mgr.y (mgr.14613) 1039 : cluster [DBG] pgmap v1254: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T23:05:46.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.522862+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.6 scrub starts 2024-09-13T23:05:46.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.526194+0000 osd.1 (osd.1) 539 : cluster [DBG] 2.6 scrub ok 2024-09-13T23:05:46.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.825357+0000 osd.7 (osd.7) 606 : cluster [DBG] 5.1a scrub starts 2024-09-13T23:05:46.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.827447+0000 osd.7 (osd.7) 607 : cluster [DBG] 5.1a scrub ok 2024-09-13T23:05:46.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.217287+0000 osd.5 (osd.5) 613 : cluster [DBG] 2.18 scrub starts 2024-09-13T23:05:46.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:46 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.219728+0000 osd.5 (osd.5) 614 : cluster [DBG] 2.18 scrub ok 2024-09-13T23:05:47.681 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.817810+0000 osd.0 (osd.0) 535 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: cluster 2024-09-13T23:05:45.820181+0000 osd.0 (osd.0) 536 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.019693+0000 osd.3 (osd.3) 551 : cluster [DBG] 6.6 scrub starts 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.022384+0000 osd.3 (osd.3) 552 : cluster [DBG] 6.6 scrub ok 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: audit 2024-09-13T23:05:46.311652+0000 mgr.y (mgr.14613) 1040 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.856275+0000 osd.7 (osd.7) 608 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:05:47.682 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.858616+0000 osd.7 (osd.7) 609 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.817810+0000 osd.0 (osd.0) 535 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: cluster 2024-09-13T23:05:45.820181+0000 osd.0 (osd.0) 536 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.019693+0000 osd.3 (osd.3) 551 : cluster [DBG] 6.6 scrub starts 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.022384+0000 osd.3 (osd.3) 552 : cluster [DBG] 6.6 scrub ok 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: audit 2024-09-13T23:05:46.311652+0000 mgr.y (mgr.14613) 1040 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.856275+0000 osd.7 (osd.7) 608 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:05:47.683 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:47 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.858616+0000 osd.7 (osd.7) 609 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:05:47.686 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-09-13T23:05:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.817810+0000 osd.0 (osd.0) 535 : cluster [DBG] 2.13 scrub starts 2024-09-13T23:05:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: cluster 2024-09-13T23:05:45.820181+0000 osd.0 (osd.0) 536 : cluster [DBG] 2.13 scrub ok 2024-09-13T23:05:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.019693+0000 osd.3 (osd.3) 551 : cluster [DBG] 6.6 scrub starts 2024-09-13T23:05:47.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.022384+0000 osd.3 (osd.3) 552 : cluster [DBG] 6.6 scrub ok 2024-09-13T23:05:47.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: audit 2024-09-13T23:05:46.311652+0000 mgr.y (mgr.14613) 1040 : audit [DBG] from='client.24502 v1:172.21.15.192:0/3674496335' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T23:05:47.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.856275+0000 osd.7 (osd.7) 608 : cluster [DBG] 5.4 scrub starts 2024-09-13T23:05:47.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:47 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.858616+0000 osd.7 (osd.7) 609 : cluster [DBG] 5.4 scrub ok 2024-09-13T23:05:48.491 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":1255,"stamp":"2024-09-13T23:05:47.133185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3244,"num_read_kb":2996,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":11196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2669,"ondisk_log_size":2669,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5779580,"kb_used_data":7484,"kb_used_omap":12,"kb_used_meta":5771955,"kb_avail":744181636,"statfs":{"total":767960285184,"available":762041995264,"internally_reserved":0,"allocated":7663616,"data_stored":4588665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":5910482480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005964"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":1009,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963410+0000","last_change":"2024-09-13T23:05:33.978937+0000","last_active":"2024-09-13T23:05:34.963410+0000","last_peered":"2024-09-13T23:05:34.963410+0000","last_clean":"2024-09-13T23:05:34.963410+0000","last_became_active":"2024-09-13T22:58:42.763279+0000","last_became_peered":"2024-09-13T22:58:42.763279+0000","last_unstale":"2024-09-13T23:05:34.963410+0000","last_undegraded":"2024-09-13T23:05:34.963410+0000","last_fullsized":"2024-09-13T23:05:34.963410+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:33.978729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:02.514742+0000","last_clean_scrub_stamp":"2024-09-13T23:05:33.978729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:43.296664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1078,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:36.688038+0000","last_change":"2024-09-13T23:05:36.688090+0000","last_active":"2024-09-13T23:05:36.688038+0000","last_peered":"2024-09-13T23:05:36.688038+0000","last_clean":"2024-09-13T23:05:36.688038+0000","last_became_active":"2024-09-13T23:02:24.401461+0000","last_became_peered":"2024-09-13T23:02:24.401461+0000","last_unstale":"2024-09-13T23:05:36.688038+0000","last_undegraded":"2024-09-13T23:05:36.688038+0000","last_fullsized":"2024-09-13T23:05:36.688038+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:36.687905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:01:55.756398+0000","last_clean_scrub_stamp":"2024-09-13T23:05:36.687905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:58.460838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"397'35","reported_seq":1101,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.965168+0000","last_change":"2024-09-13T23:05:04.841182+0000","last_active":"2024-09-13T23:05:34.965168+0000","last_peered":"2024-09-13T23:05:34.965168+0000","last_clean":"2024-09-13T23:05:34.965168+0000","last_became_active":"2024-09-13T22:58:42.747843+0000","last_became_peered":"2024-09-13T22:58:42.747843+0000","last_unstale":"2024-09-13T23:05:34.965168+0000","last_undegraded":"2024-09-13T23:05:34.965168+0000","last_fullsized":"2024-09-13T23:05:34.965168+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-09-13T23:05:04.840856+0000","last_deep_scrub":"99'15","last_deep_scrub_stamp":"2024-09-13T22:41:49.765993+0000","last_clean_scrub_stamp":"2024-09-13T23:05:04.840856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:34.033479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277629+0000","last_change":"2024-09-13T23:05:14.567019+0000","last_active":"2024-09-13T23:05:34.277629+0000","last_peered":"2024-09-13T23:05:34.277629+0000","last_clean":"2024-09-13T23:05:34.277629+0000","last_became_active":"2024-09-13T23:02:24.389370+0000","last_became_peered":"2024-09-13T23:02:24.389370+0000","last_unstale":"2024-09-13T23:05:34.277629+0000","last_undegraded":"2024-09-13T23:05:34.277629+0000","last_fullsized":"2024-09-13T23:05:34.277629+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:14.566776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:52.522870+0000","last_clean_scrub_stamp":"2024-09-13T23:05:14.566776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:38.340930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1011,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.965246+0000","last_change":"2024-09-13T23:04:55.780152+0000","last_active":"2024-09-13T23:05:34.965246+0000","last_peered":"2024-09-13T23:05:34.965246+0000","last_clean":"2024-09-13T23:05:34.965246+0000","last_became_active":"2024-09-13T22:37:40.736926+0000","last_became_peered":"2024-09-13T22:37:40.736926+0000","last_unstale":"2024-09-13T23:05:34.965246+0000","last_undegraded":"2024-09-13T23:05:34.965246+0000","last_fullsized":"2024-09-13T23:05:34.965246+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:55.779951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:16.435813+0000","last_clean_scrub_stamp":"2024-09-13T23:04:55.779951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.625494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"397'32","reported_seq":1080,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277928+0000","last_change":"2024-09-13T23:05:11.797789+0000","last_active":"2024-09-13T23:05:34.277928+0000","last_peered":"2024-09-13T23:05:34.277928+0000","last_clean":"2024-09-13T23:05:34.277928+0000","last_became_active":"2024-09-13T22:37:43.301243+0000","last_became_peered":"2024-09-13T22:37:43.301243+0000","last_unstale":"2024-09-13T23:05:34.277928+0000","last_undegraded":"2024-09-13T23:05:34.277928+0000","last_fullsized":"2024-09-13T23:05:34.277928+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:05:11.797680+0000","last_deep_scrub":"397'32","last_deep_scrub_stamp":"2024-09-13T23:05:11.797680+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.797680+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:27.259403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1018,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.283700+0000","last_change":"2024-09-13T23:05:16.499406+0000","last_active":"2024-09-13T23:05:35.283700+0000","last_peered":"2024-09-13T23:05:35.283700+0000","last_clean":"2024-09-13T23:05:35.283700+0000","last_became_active":"2024-09-13T22:37:44.876275+0000","last_became_peered":"2024-09-13T22:37:44.876275+0000","last_unstale":"2024-09-13T23:05:35.283700+0000","last_undegraded":"2024-09-13T23:05:35.283700+0000","last_fullsized":"2024-09-13T23:05:35.283700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:16.499290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:42:59.307067+0000","last_clean_scrub_stamp":"2024-09-13T23:05:16.499290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:25.613237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:39.498099+0000","last_change":"2024-09-13T23:05:39.498153+0000","last_active":"2024-09-13T23:05:39.498099+0000","last_peered":"2024-09-13T23:05:39.498099+0000","last_clean":"2024-09-13T23:05:39.498099+0000","last_became_active":"2024-09-13T23:02:24.378760+0000","last_became_peered":"2024-09-13T23:02:24.378760+0000","last_unstale":"2024-09-13T23:05:39.498099+0000","last_undegraded":"2024-09-13T23:05:39.498099+0000","last_fullsized":"2024-09-13T23:05:39.498099+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:39.497952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:09.889616+0000","last_clean_scrub_stamp":"2024-09-13T23:05:39.497952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:42.607872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1021,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498726+0000","last_change":"2024-09-13T23:04:29.639210+0000","last_active":"2024-09-13T23:05:34.498726+0000","last_peered":"2024-09-13T23:05:34.498726+0000","last_clean":"2024-09-13T23:05:34.498726+0000","last_became_active":"2024-09-13T22:58:42.770668+0000","last_became_peered":"2024-09-13T22:58:42.770668+0000","last_unstale":"2024-09-13T23:05:34.498726+0000","last_undegraded":"2024-09-13T23:05:34.498726+0000","last_fullsized":"2024-09-13T23:05:34.498726+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:29.639022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:29.639022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.503427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"397'46","reported_seq":1230,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524064+0000","last_change":"2024-09-13T23:05:14.284015+0000","last_active":"2024-09-13T23:05:34.524064+0000","last_peered":"2024-09-13T23:05:34.524064+0000","last_clean":"2024-09-13T23:05:34.524064+0000","last_became_active":"2024-09-13T23:02:24.384594+0000","last_became_peered":"2024-09-13T23:02:24.384594+0000","last_unstale":"2024-09-13T23:05:34.524064+0000","last_undegraded":"2024-09-13T23:05:34.524064+0000","last_fullsized":"2024-09-13T23:05:34.524064+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:05:14.283792+0000","last_deep_scrub":"103'22","last_deep_scrub_stamp":"2024-09-13T22:39:26.318753+0000","last_clean_scrub_stamp":"2024-09-13T23:05:14.283792+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:30.484409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1017,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.496946+0000","last_change":"2024-09-13T23:04:30.645760+0000","last_active":"2024-09-13T23:05:34.496946+0000","last_peered":"2024-09-13T23:05:34.496946+0000","last_clean":"2024-09-13T23:05:34.496946+0000","last_became_active":"2024-09-13T23:02:24.396141+0000","last_became_peered":"2024-09-13T23:02:24.396141+0000","last_unstale":"2024-09-13T23:05:34.496946+0000","last_undegraded":"2024-09-13T23:05:34.496946+0000","last_fullsized":"2024-09-13T23:05:34.496946+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:30.645572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:39:48.735986+0000","last_clean_scrub_stamp":"2024-09-13T23:04:30.645572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.135104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1014,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523348+0000","last_change":"2024-09-13T23:04:16.503342+0000","last_active":"2024-09-13T23:05:34.523348+0000","last_peered":"2024-09-13T23:05:34.523348+0000","last_clean":"2024-09-13T23:05:34.523348+0000","last_became_active":"2024-09-13T22:37:46.912404+0000","last_became_peered":"2024-09-13T22:37:46.912404+0000","last_unstale":"2024-09-13T23:05:34.523348+0000","last_undegraded":"2024-09-13T23:05:34.523348+0000","last_fullsized":"2024-09-13T23:05:34.523348+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:16.503153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:45:01.888825+0000","last_clean_scrub_stamp":"2024-09-13T23:04:16.503153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1017,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:41.896552+0000","last_change":"2024-09-13T23:05:41.896606+0000","last_active":"2024-09-13T23:05:41.896552+0000","last_peered":"2024-09-13T23:05:41.896552+0000","last_clean":"2024-09-13T23:05:41.896552+0000","last_became_active":"2024-09-13T22:37:40.752894+0000","last_became_peered":"2024-09-13T22:37:40.752894+0000","last_unstale":"2024-09-13T23:05:41.896552+0000","last_undegraded":"2024-09-13T23:05:41.896552+0000","last_fullsized":"2024-09-13T23:05:41.896552+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:41.896421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:05:41.896421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:07:08.449775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"512'29","reported_seq":612,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.497746+0000","last_change":"2024-09-13T23:05:25.869524+0000","last_active":"2024-09-13T23:05:34.497746+0000","last_peered":"2024-09-13T23:05:34.497746+0000","last_clean":"2024-09-13T23:05:34.497746+0000","last_became_active":"2024-09-13T23:02:24.395749+0000","last_became_peered":"2024-09-13T23:02:24.395749+0000","last_unstale":"2024-09-13T23:05:34.497746+0000","last_undegraded":"2024-09-13T23:05:34.497746+0000","last_fullsized":"2024-09-13T23:05:34.497746+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'29","last_scrub_stamp":"2024-09-13T23:05:25.869284+0000","last_deep_scrub":"512'29","last_deep_scrub_stamp":"2024-09-13T23:05:25.869284+0000","last_clean_scrub_stamp":"2024-09-13T23:05:25.869284+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.554368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1018,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523624+0000","last_change":"2024-09-13T23:05:04.330173+0000","last_active":"2024-09-13T23:05:34.523624+0000","last_peered":"2024-09-13T23:05:34.523624+0000","last_clean":"2024-09-13T23:05:34.523624+0000","last_became_active":"2024-09-13T22:37:44.886680+0000","last_became_peered":"2024-09-13T22:37:44.886680+0000","last_unstale":"2024-09-13T23:05:34.523624+0000","last_undegraded":"2024-09-13T23:05:34.523624+0000","last_fullsized":"2024-09-13T23:05:34.523624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:04.329986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:49:42.177430+0000","last_clean_scrub_stamp":"2024-09-13T23:05:04.329986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:26.234421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1022,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277180+0000","last_change":"2024-09-13T23:05:27.687051+0000","last_active":"2024-09-13T23:05:34.277180+0000","last_peered":"2024-09-13T23:05:34.277180+0000","last_clean":"2024-09-13T23:05:34.277180+0000","last_became_active":"2024-09-13T22:37:46.911436+0000","last_became_peered":"2024-09-13T22:37:46.911436+0000","last_unstale":"2024-09-13T23:05:34.277180+0000","last_undegraded":"2024-09-13T23:05:34.277180+0000","last_fullsized":"2024-09-13T23:05:34.277180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:27.686834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:27.686834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:40.877367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"96'130","reported_seq":1082,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.522939+0000","last_change":"2024-09-13T23:05:28.201795+0000","last_active":"2024-09-13T23:05:34.522939+0000","last_peered":"2024-09-13T23:05:34.522939+0000","last_clean":"2024-09-13T23:05:34.522939+0000","last_became_active":"2024-09-13T22:45:58.875334+0000","last_became_peered":"2024-09-13T22:45:58.875334+0000","last_unstale":"2024-09-13T23:05:34.522939+0000","last_undegraded":"2024-09-13T23:05:34.522939+0000","last_fullsized":"2024-09-13T23:05:34.522939+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:28.201596+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:05:28.201596+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:40.647798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"397'60","reported_seq":315,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.507273+0000","last_change":"2024-09-13T23:04:29.712288+0000","last_active":"2024-09-13T23:05:34.507273+0000","last_peered":"2024-09-13T23:05:34.507273+0000","last_clean":"2024-09-13T23:05:34.507273+0000","last_became_active":"2024-09-13T23:02:24.401801+0000","last_became_peered":"2024-09-13T23:02:24.401801+0000","last_unstale":"2024-09-13T23:05:34.507273+0000","last_undegraded":"2024-09-13T23:05:34.507273+0000","last_fullsized":"2024-09-13T23:05:34.507273+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'60","last_scrub_stamp":"2024-09-13T23:04:29.712075+0000","last_deep_scrub":"397'60","last_deep_scrub_stamp":"2024-09-13T23:03:15.188024+0000","last_clean_scrub_stamp":"2024-09-13T23:04:29.712075+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:35.947352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1034,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498213+0000","last_change":"2024-09-13T23:05:17.776973+0000","last_active":"2024-09-13T23:05:34.498213+0000","last_peered":"2024-09-13T23:05:34.498213+0000","last_clean":"2024-09-13T23:05:34.498213+0000","last_became_active":"2024-09-13T23:02:24.394946+0000","last_became_peered":"2024-09-13T23:02:24.394946+0000","last_unstale":"2024-09-13T23:05:34.498213+0000","last_undegraded":"2024-09-13T23:05:34.498213+0000","last_fullsized":"2024-09-13T23:05:34.498213+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:17.776706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:46:09.789477+0000","last_clean_scrub_stamp":"2024-09-13T23:05:17.776706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:45.439143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1055,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284178+0000","last_change":"2024-09-13T23:04:31.345930+0000","last_active":"2024-09-13T23:05:35.284178+0000","last_peered":"2024-09-13T23:05:35.284178+0000","last_clean":"2024-09-13T23:05:35.284178+0000","last_became_active":"2024-09-13T23:02:24.395018+0000","last_became_peered":"2024-09-13T23:02:24.395018+0000","last_unstale":"2024-09-13T23:05:35.284178+0000","last_undegraded":"2024-09-13T23:05:35.284178+0000","last_fullsized":"2024-09-13T23:05:35.284178+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:31.345747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:10.123464+0000","last_clean_scrub_stamp":"2024-09-13T23:04:31.345747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:53.527854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":914,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.519284+0000","last_change":"2024-09-13T23:04:39.985031+0000","last_active":"2024-09-13T23:05:34.519284+0000","last_peered":"2024-09-13T23:05:34.519284+0000","last_clean":"2024-09-13T23:05:34.519284+0000","last_became_active":"2024-09-13T23:03:27.005831+0000","last_became_peered":"2024-09-13T23:03:27.005831+0000","last_unstale":"2024-09-13T23:05:34.519284+0000","last_undegraded":"2024-09-13T23:05:34.519284+0000","last_fullsized":"2024-09-13T23:05:34.519284+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:39.984844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:39.984844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.278333+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"96'130","reported_seq":1290,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.497516+0000","last_change":"2024-09-13T23:05:11.830051+0000","last_active":"2024-09-13T23:05:34.497516+0000","last_peered":"2024-09-13T23:05:34.497516+0000","last_clean":"2024-09-13T23:05:34.497516+0000","last_became_active":"2024-09-13T22:45:58.864917+0000","last_became_peered":"2024-09-13T22:45:58.864917+0000","last_unstale":"2024-09-13T23:05:34.497516+0000","last_undegraded":"2024-09-13T23:05:34.497516+0000","last_fullsized":"2024-09-13T23:05:34.497516+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:11.829818+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:03:56.761033+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.829818+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:23.388677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"397'25","reported_seq":1116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.964336+0000","last_change":"2024-09-13T23:05:28.927659+0000","last_active":"2024-09-13T23:05:34.964336+0000","last_peered":"2024-09-13T23:05:34.964336+0000","last_clean":"2024-09-13T23:05:34.964336+0000","last_became_active":"2024-09-13T23:02:24.393869+0000","last_became_peered":"2024-09-13T23:02:24.393869+0000","last_unstale":"2024-09-13T23:05:34.964336+0000","last_undegraded":"2024-09-13T23:05:34.964336+0000","last_fullsized":"2024-09-13T23:05:34.964336+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:05:28.927568+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-09-13T22:40:15.285976+0000","last_clean_scrub_stamp":"2024-09-13T23:05:28.927568+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.824407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1027,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284744+0000","last_change":"2024-09-13T23:04:47.419240+0000","last_active":"2024-09-13T23:05:35.284744+0000","last_peered":"2024-09-13T23:05:35.284744+0000","last_clean":"2024-09-13T23:05:35.284744+0000","last_became_active":"2024-09-13T22:37:40.741722+0000","last_became_peered":"2024-09-13T22:37:40.741722+0000","last_unstale":"2024-09-13T23:05:35.284744+0000","last_undegraded":"2024-09-13T23:05:35.284744+0000","last_fullsized":"2024-09-13T23:05:35.284744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:47.419049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:47.419049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:51.993357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"505'13","reported_seq":678,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:42.115896+0000","last_change":"2024-09-13T23:04:44.956800+0000","last_active":"2024-09-13T23:05:42.115896+0000","last_peered":"2024-09-13T23:05:42.115896+0000","last_clean":"2024-09-13T23:05:42.115896+0000","last_became_active":"2024-09-13T23:03:26.731853+0000","last_became_peered":"2024-09-13T23:03:26.731853+0000","last_unstale":"2024-09-13T23:05:42.115896+0000","last_undegraded":"2024-09-13T23:05:42.115896+0000","last_fullsized":"2024-09-13T23:05:42.115896+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'13","last_scrub_stamp":"2024-09-13T23:04:44.956604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:04:44.956604+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.966454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.279511+0000","last_change":"2024-09-13T23:05:33.553416+0000","last_active":"2024-09-13T23:05:34.279511+0000","last_peered":"2024-09-13T23:05:34.279511+0000","last_clean":"2024-09-13T23:05:34.279511+0000","last_became_active":"2024-09-13T23:02:24.373934+0000","last_became_peered":"2024-09-13T23:02:24.373934+0000","last_unstale":"2024-09-13T23:05:34.279511+0000","last_undegraded":"2024-09-13T23:05:34.279511+0000","last_fullsized":"2024-09-13T23:05:34.279511+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:33.553216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:33.553216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:41.839342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"96'130","reported_seq":1100,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523181+0000","last_change":"2024-09-13T23:04:49.459653+0000","last_active":"2024-09-13T23:05:34.523181+0000","last_peered":"2024-09-13T23:05:34.523181+0000","last_clean":"2024-09-13T23:05:34.523181+0000","last_became_active":"2024-09-13T22:58:45.828166+0000","last_became_peered":"2024-09-13T22:58:45.828166+0000","last_unstale":"2024-09-13T23:05:34.523181+0000","last_undegraded":"2024-09-13T23:05:34.523181+0000","last_fullsized":"2024-09-13T23:05:34.523181+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:49.459445+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:04:49.459445+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.357664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"502'46","reported_seq":1074,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963433+0000","last_change":"2024-09-13T23:04:51.723277+0000","last_active":"2024-09-13T23:05:34.963433+0000","last_peered":"2024-09-13T23:05:34.963433+0000","last_clean":"2024-09-13T23:05:34.963433+0000","last_became_active":"2024-09-13T22:37:42.761294+0000","last_became_peered":"2024-09-13T22:37:42.761294+0000","last_unstale":"2024-09-13T23:05:34.963433+0000","last_undegraded":"2024-09-13T23:05:34.963433+0000","last_fullsized":"2024-09-13T23:05:34.963433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'46","last_scrub_stamp":"2024-09-13T23:04:51.723066+0000","last_deep_scrub":"502'46","last_deep_scrub_stamp":"2024-09-13T23:03:21.476482+0000","last_clean_scrub_stamp":"2024-09-13T23:04:51.723066+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.018580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1035,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.424785+0000","last_change":"2024-09-13T23:05:35.424839+0000","last_active":"2024-09-13T23:05:35.424785+0000","last_peered":"2024-09-13T23:05:35.424785+0000","last_clean":"2024-09-13T23:05:35.424785+0000","last_became_active":"2024-09-13T22:37:40.741184+0000","last_became_peered":"2024-09-13T22:37:40.741184+0000","last_unstale":"2024-09-13T23:05:35.424785+0000","last_undegraded":"2024-09-13T23:05:35.424785+0000","last_fullsized":"2024-09-13T23:05:35.424785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:35.424644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:15.369396+0000","last_clean_scrub_stamp":"2024-09-13T23:05:35.424644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:41.286246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"492'16","reported_seq":160,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112211+0000","last_change":"2024-09-13T23:05:18.700022+0000","last_active":"2024-09-13T23:05:37.112211+0000","last_peered":"2024-09-13T23:05:37.112211+0000","last_clean":"2024-09-13T23:05:37.112211+0000","last_became_active":"2024-09-13T23:02:24.386882+0000","last_became_peered":"2024-09-13T23:02:24.386882+0000","last_unstale":"2024-09-13T23:05:37.112211+0000","last_undegraded":"2024-09-13T23:05:37.112211+0000","last_fullsized":"2024-09-13T23:05:37.112211+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'16","last_scrub_stamp":"2024-09-13T23:05:18.699810+0000","last_deep_scrub":"492'16","last_deep_scrub_stamp":"2024-09-13T23:03:50.531482+0000","last_clean_scrub_stamp":"2024-09-13T23:05:18.699810+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:36.058171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1015,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:38.244003+0000","last_change":"2024-09-13T23:05:38.244057+0000","last_active":"2024-09-13T23:05:38.244003+0000","last_peered":"2024-09-13T23:05:38.244003+0000","last_clean":"2024-09-13T23:05:38.244003+0000","last_became_active":"2024-09-13T22:37:46.914501+0000","last_became_peered":"2024-09-13T22:37:46.914501+0000","last_unstale":"2024-09-13T23:05:38.244003+0000","last_undegraded":"2024-09-13T23:05:38.244003+0000","last_fullsized":"2024-09-13T23:05:38.244003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:38.243858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:05:38.243858+0000","last_clean_scrub_stamp":"2024-09-13T23:05:38.243858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:49.557447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"397'24","reported_seq":128,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:40.543273+0000","last_change":"2024-09-13T23:05:40.543345+0000","last_active":"2024-09-13T23:05:40.543273+0000","last_peered":"2024-09-13T23:05:40.543273+0000","last_clean":"2024-09-13T23:05:40.543273+0000","last_became_active":"2024-09-13T23:02:24.394573+0000","last_became_peered":"2024-09-13T23:02:24.394573+0000","last_unstale":"2024-09-13T23:05:40.543273+0000","last_undegraded":"2024-09-13T23:05:40.543273+0000","last_fullsized":"2024-09-13T23:05:40.543273+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-09-13T23:05:40.543131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:40.543131+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:07:05.221412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1010,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.497394+0000","last_change":"2024-09-13T23:04:58.732694+0000","last_active":"2024-09-13T23:05:34.497394+0000","last_peered":"2024-09-13T23:05:34.497394+0000","last_clean":"2024-09-13T23:05:34.497394+0000","last_became_active":"2024-09-13T22:37:40.753343+0000","last_became_peered":"2024-09-13T22:37:40.753343+0000","last_unstale":"2024-09-13T23:05:34.497394+0000","last_undegraded":"2024-09-13T23:05:34.497394+0000","last_fullsized":"2024-09-13T23:05:34.497394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:58.732508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:58.732508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:27.985410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":851,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:43.833919+0000","last_change":"2024-09-13T23:05:43.833972+0000","last_active":"2024-09-13T23:05:43.833919+0000","last_peered":"2024-09-13T23:05:43.833919+0000","last_clean":"2024-09-13T23:05:43.833919+0000","last_became_active":"2024-09-13T23:02:24.393998+0000","last_became_peered":"2024-09-13T23:02:24.393998+0000","last_unstale":"2024-09-13T23:05:43.833919+0000","last_undegraded":"2024-09-13T23:05:43.833919+0000","last_fullsized":"2024-09-13T23:05:43.833919+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:43.833767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:58.971688+0000","last_clean_scrub_stamp":"2024-09-13T23:05:43.833767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:49.799736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1010,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.964269+0000","last_change":"2024-09-13T23:05:32.928758+0000","last_active":"2024-09-13T23:05:34.964269+0000","last_peered":"2024-09-13T23:05:34.964269+0000","last_clean":"2024-09-13T23:05:34.964269+0000","last_became_active":"2024-09-13T22:58:42.746385+0000","last_became_peered":"2024-09-13T22:58:42.746385+0000","last_unstale":"2024-09-13T23:05:34.964269+0000","last_undegraded":"2024-09-13T23:05:34.964269+0000","last_fullsized":"2024-09-13T23:05:34.964269+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:32.928485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:58:12.718269+0000","last_clean_scrub_stamp":"2024-09-13T23:05:32.928485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:56.175784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"96'130","reported_seq":1064,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.964173+0000","last_change":"2024-09-13T23:05:24.914145+0000","last_active":"2024-09-13T23:05:34.964173+0000","last_peered":"2024-09-13T23:05:34.964173+0000","last_clean":"2024-09-13T23:05:34.964173+0000","last_became_active":"2024-09-13T22:45:58.867512+0000","last_became_peered":"2024-09-13T22:45:58.867512+0000","last_unstale":"2024-09-13T23:05:34.964173+0000","last_undegraded":"2024-09-13T23:05:34.964173+0000","last_fullsized":"2024-09-13T23:05:34.964173+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:24.913957+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:17.983227+0000","last_clean_scrub_stamp":"2024-09-13T23:05:24.913957+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:39.029738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"397'36","reported_seq":1165,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278724+0000","last_change":"2024-09-13T23:05:32.617484+0000","last_active":"2024-09-13T23:05:34.278724+0000","last_peered":"2024-09-13T23:05:34.278724+0000","last_clean":"2024-09-13T23:05:34.278724+0000","last_became_active":"2024-09-13T23:02:24.401486+0000","last_became_peered":"2024-09-13T23:02:24.401486+0000","last_unstale":"2024-09-13T23:05:34.278724+0000","last_undegraded":"2024-09-13T23:05:34.278724+0000","last_fullsized":"2024-09-13T23:05:34.278724+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'36","last_scrub_stamp":"2024-09-13T23:05:32.617275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:32.617275+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:34.054021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":269,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.508174+0000","last_change":"2024-09-13T23:04:33.675184+0000","last_active":"2024-09-13T23:05:34.508174+0000","last_peered":"2024-09-13T23:05:34.508174+0000","last_clean":"2024-09-13T23:05:34.508174+0000","last_became_active":"2024-09-13T22:58:42.747048+0000","last_became_peered":"2024-09-13T22:58:42.747048+0000","last_unstale":"2024-09-13T23:05:34.508174+0000","last_undegraded":"2024-09-13T23:05:34.508174+0000","last_fullsized":"2024-09-13T23:05:34.508174+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:33.674906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:33.674906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:53.493378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"194'2","reported_seq":112,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278562+0000","last_change":"2024-09-13T23:04:54.645977+0000","last_active":"2024-09-13T23:05:34.278562+0000","last_peered":"2024-09-13T23:05:34.278562+0000","last_clean":"2024-09-13T23:05:34.278562+0000","last_became_active":"2024-09-13T23:02:24.401765+0000","last_became_peered":"2024-09-13T23:02:24.401765+0000","last_unstale":"2024-09-13T23:05:34.278562+0000","last_undegraded":"2024-09-13T23:05:34.278562+0000","last_fullsized":"2024-09-13T23:05:34.278562+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'2","last_scrub_stamp":"2024-09-13T23:04:54.645649+0000","last_deep_scrub":"194'2","last_deep_scrub_stamp":"2024-09-13T23:04:54.645649+0000","last_clean_scrub_stamp":"2024-09-13T23:04:54.645649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:00.803577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193188,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"507'16","reported_seq":1393,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112140+0000","last_change":"2024-09-13T23:04:57.834420+0000","last_active":"2024-09-13T23:05:37.112140+0000","last_peered":"2024-09-13T23:05:37.112140+0000","last_clean":"2024-09-13T23:05:37.112140+0000","last_became_active":"2024-09-13T23:02:24.376006+0000","last_became_peered":"2024-09-13T23:02:24.376006+0000","last_unstale":"2024-09-13T23:05:37.112140+0000","last_undegraded":"2024-09-13T23:05:37.112140+0000","last_fullsized":"2024-09-13T23:05:37.112140+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'16","last_scrub_stamp":"2024-09-13T23:04:57.834206+0000","last_deep_scrub":"507'16","last_deep_scrub_stamp":"2024-09-13T23:03:39.406184+0000","last_clean_scrub_stamp":"2024-09-13T23:04:57.834206+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:07.319089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1011,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.894115+0000","last_change":"2024-09-13T23:05:34.894168+0000","last_active":"2024-09-13T23:05:34.894115+0000","last_peered":"2024-09-13T23:05:34.894115+0000","last_clean":"2024-09-13T23:05:34.894115+0000","last_became_active":"2024-09-13T22:58:42.770701+0000","last_became_peered":"2024-09-13T22:58:42.770701+0000","last_unstale":"2024-09-13T23:05:34.894115+0000","last_undegraded":"2024-09-13T23:05:34.894115+0000","last_fullsized":"2024-09-13T23:05:34.894115+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:34.893984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:06.861058+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.893984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:38.279007+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"96'130","reported_seq":901,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.507067+0000","last_change":"2024-09-13T23:05:35.507130+0000","last_active":"2024-09-13T23:05:35.507067+0000","last_peered":"2024-09-13T23:05:35.507067+0000","last_clean":"2024-09-13T23:05:35.507067+0000","last_became_active":"2024-09-13T23:02:27.393901+0000","last_became_peered":"2024-09-13T23:02:27.393901+0000","last_unstale":"2024-09-13T23:05:35.507067+0000","last_undegraded":"2024-09-13T23:05:35.507067+0000","last_fullsized":"2024-09-13T23:05:35.507067+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:35.506928+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:01:31.604112+0000","last_clean_scrub_stamp":"2024-09-13T23:05:35.506928+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.450384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"397'26","reported_seq":1050,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.965665+0000","last_change":"2024-09-13T23:05:12.756474+0000","last_active":"2024-09-13T23:05:34.965665+0000","last_peered":"2024-09-13T23:05:34.965665+0000","last_clean":"2024-09-13T23:05:34.965665+0000","last_became_active":"2024-09-13T22:37:42.761778+0000","last_became_peered":"2024-09-13T22:37:42.761778+0000","last_unstale":"2024-09-13T23:05:34.965665+0000","last_undegraded":"2024-09-13T23:05:34.965665+0000","last_fullsized":"2024-09-13T23:05:34.965665+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'26","last_scrub_stamp":"2024-09-13T23:05:12.756269+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-13T22:39:16.975244+0000","last_clean_scrub_stamp":"2024-09-13T23:05:12.756269+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:29.774314+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"214'2","reported_seq":1089,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284985+0000","last_change":"2024-09-13T23:04:05.331264+0000","last_active":"2024-09-13T23:05:35.284985+0000","last_peered":"2024-09-13T23:05:35.284985+0000","last_clean":"2024-09-13T23:05:35.284985+0000","last_became_active":"2024-09-13T23:02:24.370112+0000","last_became_peered":"2024-09-13T23:02:24.370112+0000","last_unstale":"2024-09-13T23:05:35.284985+0000","last_undegraded":"2024-09-13T23:05:35.284985+0000","last_fullsized":"2024-09-13T23:05:35.284985+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'2","last_scrub_stamp":"2024-09-13T23:04:05.331012+0000","last_deep_scrub":"214'2","last_deep_scrub_stamp":"2024-09-13T22:47:07.022796+0000","last_clean_scrub_stamp":"2024-09-13T23:04:05.331012+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"309'6","reported_seq":2673,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:43.193649+0000","last_change":"2024-09-13T23:05:15.805499+0000","last_active":"2024-09-13T23:05:43.193649+0000","last_peered":"2024-09-13T23:05:43.193649+0000","last_clean":"2024-09-13T23:05:43.193649+0000","last_became_active":"2024-09-13T22:37:42.762638+0000","last_became_peered":"2024-09-13T22:37:42.762638+0000","last_unstale":"2024-09-13T23:05:43.193649+0000","last_undegraded":"2024-09-13T23:05:43.193649+0000","last_fullsized":"2024-09-13T23:05:43.193649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'6","last_scrub_stamp":"2024-09-13T23:05:15.805301+0000","last_deep_scrub":"309'6","last_deep_scrub_stamp":"2024-09-13T22:52:17.113713+0000","last_clean_scrub_stamp":"2024-09-13T23:05:15.805301+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:39.761119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013656600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1655,"num_read_kb":1650,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278321+0000","last_change":"2024-09-13T23:05:11.595837+0000","last_active":"2024-09-13T23:05:34.278321+0000","last_peered":"2024-09-13T23:05:34.278321+0000","last_clean":"2024-09-13T23:05:34.278321+0000","last_became_active":"2024-09-13T23:02:24.393819+0000","last_became_peered":"2024-09-13T23:02:24.393819+0000","last_unstale":"2024-09-13T23:05:34.278321+0000","last_undegraded":"2024-09-13T23:05:34.278321+0000","last_fullsized":"2024-09-13T23:05:34.278321+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:11.595653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.595653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:28.923365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278320+0000","last_change":"2024-09-13T23:05:22.705784+0000","last_active":"2024-09-13T23:05:34.278320+0000","last_peered":"2024-09-13T23:05:34.278320+0000","last_clean":"2024-09-13T23:05:34.278320+0000","last_became_active":"2024-09-13T23:02:24.399413+0000","last_became_peered":"2024-09-13T23:02:24.399413+0000","last_unstale":"2024-09-13T23:05:34.278320+0000","last_undegraded":"2024-09-13T23:05:34.278320+0000","last_fullsized":"2024-09-13T23:05:34.278320+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:22.705589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:46.659973+0000","last_clean_scrub_stamp":"2024-09-13T23:05:22.705589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:26.685351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"96'130","reported_seq":718,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.508133+0000","last_change":"2024-09-13T23:04:59.697982+0000","last_active":"2024-09-13T23:05:34.508133+0000","last_peered":"2024-09-13T23:05:34.508133+0000","last_clean":"2024-09-13T23:05:34.508133+0000","last_became_active":"2024-09-13T22:58:45.829511+0000","last_became_peered":"2024-09-13T22:58:45.829511+0000","last_unstale":"2024-09-13T23:05:34.508133+0000","last_undegraded":"2024-09-13T23:05:34.508133+0000","last_fullsized":"2024-09-13T23:05:34.508133+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:59.697738+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:04:59.697738+0000","last_clean_scrub_stamp":"2024-09-13T23:04:59.697738+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:07.670107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"397'64","reported_seq":1457,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112062+0000","last_change":"2024-09-13T23:04:25.311998+0000","last_active":"2024-09-13T23:05:37.112062+0000","last_peered":"2024-09-13T23:05:37.112062+0000","last_clean":"2024-09-13T23:05:37.112062+0000","last_became_active":"2024-09-13T22:37:43.301145+0000","last_became_peered":"2024-09-13T22:37:43.301145+0000","last_unstale":"2024-09-13T23:05:37.112062+0000","last_undegraded":"2024-09-13T23:05:37.112062+0000","last_fullsized":"2024-09-13T23:05:37.112062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'64","last_scrub_stamp":"2024-09-13T23:04:25.311656+0000","last_deep_scrub":"397'64","last_deep_scrub_stamp":"2024-09-13T23:04:25.311656+0000","last_clean_scrub_stamp":"2024-09-13T23:04:25.311656+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1035,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498736+0000","last_change":"2024-09-13T23:05:13.836769+0000","last_active":"2024-09-13T23:05:34.498736+0000","last_peered":"2024-09-13T23:05:34.498736+0000","last_clean":"2024-09-13T23:05:34.498736+0000","last_became_active":"2024-09-13T23:02:24.396411+0000","last_became_peered":"2024-09-13T23:02:24.396411+0000","last_unstale":"2024-09-13T23:05:34.498736+0000","last_undegraded":"2024-09-13T23:05:34.498736+0000","last_fullsized":"2024-09-13T23:05:34.498736+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:13.836583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:48.725054+0000","last_clean_scrub_stamp":"2024-09-13T23:05:13.836583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:41.008871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.507091+0000","last_change":"2024-09-13T23:04:53.644576+0000","last_active":"2024-09-13T23:05:34.507091+0000","last_peered":"2024-09-13T23:05:34.507091+0000","last_clean":"2024-09-13T23:05:34.507091+0000","last_became_active":"2024-09-13T22:58:42.740862+0000","last_became_peered":"2024-09-13T22:58:42.740862+0000","last_unstale":"2024-09-13T23:05:34.507091+0000","last_undegraded":"2024-09-13T23:05:34.507091+0000","last_fullsized":"2024-09-13T23:05:34.507091+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:53.644292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:53.644292+0000","last_clean_scrub_stamp":"2024-09-13T23:04:53.644292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:00.888838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1028,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.283554+0000","last_change":"2024-09-13T23:04:41.496868+0000","last_active":"2024-09-13T23:05:35.283554+0000","last_peered":"2024-09-13T23:05:35.283554+0000","last_clean":"2024-09-13T23:05:35.283554+0000","last_became_active":"2024-09-13T22:58:42.742606+0000","last_became_peered":"2024-09-13T22:58:42.742606+0000","last_unstale":"2024-09-13T23:05:35.283554+0000","last_undegraded":"2024-09-13T23:05:35.283554+0000","last_fullsized":"2024-09-13T23:05:35.283554+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:41.496672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:41.496672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:02.102150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"96'130","reported_seq":721,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.508369+0000","last_change":"2024-09-13T23:05:33.564698+0000","last_active":"2024-09-13T23:05:34.508369+0000","last_peered":"2024-09-13T23:05:34.508369+0000","last_clean":"2024-09-13T23:05:34.508369+0000","last_became_active":"2024-09-13T22:58:45.832840+0000","last_became_peered":"2024-09-13T22:58:45.832840+0000","last_unstale":"2024-09-13T23:05:34.508369+0000","last_undegraded":"2024-09-13T23:05:34.508369+0000","last_fullsized":"2024-09-13T23:05:34.508369+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:33.564501+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:05:33.564501+0000","last_clean_scrub_stamp":"2024-09-13T23:05:33.564501+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:46.370294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"397'48","reported_seq":1427,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112305+0000","last_change":"2024-09-13T23:05:13.288114+0000","last_active":"2024-09-13T23:05:37.112305+0000","last_peered":"2024-09-13T23:05:37.112305+0000","last_clean":"2024-09-13T23:05:37.112305+0000","last_became_active":"2024-09-13T22:37:43.302883+0000","last_became_peered":"2024-09-13T22:37:43.302883+0000","last_unstale":"2024-09-13T23:05:37.112305+0000","last_undegraded":"2024-09-13T23:05:37.112305+0000","last_fullsized":"2024-09-13T23:05:37.112305+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'48","last_scrub_stamp":"2024-09-13T23:05:13.287884+0000","last_deep_scrub":"103'24","last_deep_scrub_stamp":"2024-09-13T22:39:24.367843+0000","last_clean_scrub_stamp":"2024-09-13T23:05:13.287884+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:32.385851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"341'2","reported_seq":1045,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284727+0000","last_change":"2024-09-13T23:05:14.482221+0000","last_active":"2024-09-13T23:05:35.284727+0000","last_peered":"2024-09-13T23:05:35.284727+0000","last_clean":"2024-09-13T23:05:35.284727+0000","last_became_active":"2024-09-13T22:37:40.741328+0000","last_became_peered":"2024-09-13T22:37:40.741328+0000","last_unstale":"2024-09-13T23:05:35.284727+0000","last_undegraded":"2024-09-13T23:05:35.284727+0000","last_fullsized":"2024-09-13T23:05:35.284727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'2","last_scrub_stamp":"2024-09-13T23:05:14.482020+0000","last_deep_scrub":"341'2","last_deep_scrub_stamp":"2024-09-13T23:05:14.482020+0000","last_clean_scrub_stamp":"2024-09-13T23:05:14.482020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:22.426259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"348'3","reported_seq":1074,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284422+0000","last_change":"2024-09-13T23:04:43.436113+0000","last_active":"2024-09-13T23:05:35.284422+0000","last_peered":"2024-09-13T23:05:35.284422+0000","last_clean":"2024-09-13T23:05:35.284422+0000","last_became_active":"2024-09-13T23:02:24.394526+0000","last_became_peered":"2024-09-13T23:02:24.394526+0000","last_unstale":"2024-09-13T23:05:35.284422+0000","last_undegraded":"2024-09-13T23:05:35.284422+0000","last_fullsized":"2024-09-13T23:05:35.284422+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'3","last_scrub_stamp":"2024-09-13T23:04:43.435857+0000","last_deep_scrub":"348'3","last_deep_scrub_stamp":"2024-09-13T23:04:43.435857+0000","last_clean_scrub_stamp":"2024-09-13T23:04:43.435857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.531892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0027427570000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"414'14","reported_seq":1397,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.111964+0000","last_change":"2024-09-13T23:04:52.785033+0000","last_active":"2024-09-13T23:05:37.111964+0000","last_peered":"2024-09-13T23:05:37.111964+0000","last_clean":"2024-09-13T23:05:37.111964+0000","last_became_active":"2024-09-13T22:58:42.740603+0000","last_became_peered":"2024-09-13T22:58:42.740603+0000","last_unstale":"2024-09-13T23:05:37.111964+0000","last_undegraded":"2024-09-13T23:05:37.111964+0000","last_fullsized":"2024-09-13T23:05:37.111964+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-09-13T23:04:52.784826+0000","last_deep_scrub":"358'13","last_deep_scrub_stamp":"2024-09-13T22:58:05.208554+0000","last_clean_scrub_stamp":"2024-09-13T23:04:52.784826+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:00.899732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1024,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:38.709372+0000","last_change":"2024-09-13T23:05:38.709425+0000","last_active":"2024-09-13T23:05:38.709372+0000","last_peered":"2024-09-13T23:05:38.709372+0000","last_clean":"2024-09-13T23:05:38.709372+0000","last_became_active":"2024-09-13T22:37:46.923240+0000","last_became_peered":"2024-09-13T22:37:46.923240+0000","last_unstale":"2024-09-13T23:05:38.709372+0000","last_undegraded":"2024-09-13T23:05:38.709372+0000","last_fullsized":"2024-09-13T23:05:38.709372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:38.709223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:05:38.709223+0000","last_clean_scrub_stamp":"2024-09-13T23:05:38.709223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:07:03.708848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"96'130","reported_seq":1074,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.522982+0000","last_change":"2024-09-13T23:05:11.310119+0000","last_active":"2024-09-13T23:05:34.522982+0000","last_peered":"2024-09-13T23:05:34.522982+0000","last_clean":"2024-09-13T23:05:34.522982+0000","last_became_active":"2024-09-13T22:45:58.868901+0000","last_became_peered":"2024-09-13T22:45:58.868901+0000","last_unstale":"2024-09-13T23:05:34.522982+0000","last_undegraded":"2024-09-13T23:05:34.522982+0000","last_fullsized":"2024-09-13T23:05:34.522982+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:11.309805+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T23:03:46.549405+0000","last_clean_scrub_stamp":"2024-09-13T23:05:11.309805+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:34.948253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"468'47","reported_seq":134,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.540160+0000","last_change":"2024-09-13T23:05:37.540257+0000","last_active":"2024-09-13T23:05:37.540160+0000","last_peered":"2024-09-13T23:05:37.540160+0000","last_clean":"2024-09-13T23:05:37.540160+0000","last_became_active":"2024-09-13T23:02:24.392564+0000","last_became_peered":"2024-09-13T23:02:24.392564+0000","last_unstale":"2024-09-13T23:05:37.540160+0000","last_undegraded":"2024-09-13T23:05:37.540160+0000","last_fullsized":"2024-09-13T23:05:37.540160+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'47","last_scrub_stamp":"2024-09-13T23:05:37.540012+0000","last_deep_scrub":"468'47","last_deep_scrub_stamp":"2024-09-13T23:01:23.540241+0000","last_clean_scrub_stamp":"2024-09-13T23:05:37.540012+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:43.279521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1048,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524061+0000","last_change":"2024-09-13T23:05:18.312117+0000","last_active":"2024-09-13T23:05:34.524061+0000","last_peered":"2024-09-13T23:05:34.524061+0000","last_clean":"2024-09-13T23:05:34.524061+0000","last_became_active":"2024-09-13T22:58:42.741601+0000","last_became_peered":"2024-09-13T22:58:42.741601+0000","last_unstale":"2024-09-13T23:05:34.524061+0000","last_undegraded":"2024-09-13T23:05:34.524061+0000","last_fullsized":"2024-09-13T23:05:34.524061+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:18.311920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:05:18.311920+0000","last_clean_scrub_stamp":"2024-09-13T23:05:18.311920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:32.363297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1064,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278203+0000","last_change":"2024-09-13T23:05:05.868562+0000","last_active":"2024-09-13T23:05:34.278203+0000","last_peered":"2024-09-13T23:05:34.278203+0000","last_clean":"2024-09-13T23:05:34.278203+0000","last_became_active":"2024-09-13T23:02:24.404336+0000","last_became_peered":"2024-09-13T23:02:24.404336+0000","last_unstale":"2024-09-13T23:05:34.278203+0000","last_undegraded":"2024-09-13T23:05:34.278203+0000","last_fullsized":"2024-09-13T23:05:34.278203+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:05.868455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:53.889215+0000","last_clean_scrub_stamp":"2024-09-13T23:05:05.868455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:33.441126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1034,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.964814+0000","last_change":"2024-09-13T23:04:38.675390+0000","last_active":"2024-09-13T23:05:34.964814+0000","last_peered":"2024-09-13T23:05:34.964814+0000","last_clean":"2024-09-13T23:05:34.964814+0000","last_became_active":"2024-09-13T23:02:24.375086+0000","last_became_peered":"2024-09-13T23:02:24.375086+0000","last_unstale":"2024-09-13T23:05:34.964814+0000","last_undegraded":"2024-09-13T23:05:34.964814+0000","last_fullsized":"2024-09-13T23:05:34.964814+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:38.675138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:18.451343+0000","last_clean_scrub_stamp":"2024-09-13T23:04:38.675138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:41.137984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"96'130","reported_seq":1326,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.497471+0000","last_change":"2024-09-13T23:04:49.666637+0000","last_active":"2024-09-13T23:05:34.497471+0000","last_peered":"2024-09-13T23:05:34.497471+0000","last_clean":"2024-09-13T23:05:34.497471+0000","last_became_active":"2024-09-13T23:02:27.391190+0000","last_became_peered":"2024-09-13T23:02:27.391190+0000","last_unstale":"2024-09-13T23:05:34.497471+0000","last_undegraded":"2024-09-13T23:05:34.497471+0000","last_fullsized":"2024-09-13T23:05:34.497471+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":4,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:49.666443+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:04:49.666443+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:52.085906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"397'50","reported_seq":383,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.522899+0000","last_change":"2024-09-13T23:05:32.213315+0000","last_active":"2024-09-13T23:05:34.522899+0000","last_peered":"2024-09-13T23:05:34.522899+0000","last_clean":"2024-09-13T23:05:34.522899+0000","last_became_active":"2024-09-13T22:58:42.727647+0000","last_became_peered":"2024-09-13T22:58:42.727647+0000","last_unstale":"2024-09-13T23:05:34.522899+0000","last_undegraded":"2024-09-13T23:05:34.522899+0000","last_fullsized":"2024-09-13T23:05:34.522899+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-09-13T23:05:32.213065+0000","last_deep_scrub":"228'38","last_deep_scrub_stamp":"2024-09-13T22:56:56.760160+0000","last_clean_scrub_stamp":"2024-09-13T23:05:32.213065+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:52.221244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"470'2","reported_seq":1078,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.517402+0000","last_change":"2024-09-13T23:05:34.001657+0000","last_active":"2024-09-13T23:05:34.517402+0000","last_peered":"2024-09-13T23:05:34.517402+0000","last_clean":"2024-09-13T23:05:34.517402+0000","last_became_active":"2024-09-13T22:37:40.734203+0000","last_became_peered":"2024-09-13T22:37:40.734203+0000","last_unstale":"2024-09-13T23:05:34.517402+0000","last_undegraded":"2024-09-13T23:05:34.517402+0000","last_fullsized":"2024-09-13T23:05:34.517402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'2","last_scrub_stamp":"2024-09-13T23:05:34.001454+0000","last_deep_scrub":"470'2","last_deep_scrub_stamp":"2024-09-13T23:01:27.631212+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.001454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:39.769192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1052,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518994+0000","last_change":"2024-09-13T23:04:28.945472+0000","last_active":"2024-09-13T23:05:34.518994+0000","last_peered":"2024-09-13T23:05:34.518994+0000","last_clean":"2024-09-13T23:05:34.518994+0000","last_became_active":"2024-09-13T22:37:44.874358+0000","last_became_peered":"2024-09-13T22:37:44.874358+0000","last_unstale":"2024-09-13T23:05:34.518994+0000","last_undegraded":"2024-09-13T23:05:34.518994+0000","last_fullsized":"2024-09-13T23:05:34.518994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:28.945284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:26.734297+0000","last_clean_scrub_stamp":"2024-09-13T23:04:28.945284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.567460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1011,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498084+0000","last_change":"2024-09-13T23:04:48.697783+0000","last_active":"2024-09-13T23:05:34.498084+0000","last_peered":"2024-09-13T23:05:34.498084+0000","last_clean":"2024-09-13T23:05:34.498084+0000","last_became_active":"2024-09-13T22:58:42.764039+0000","last_became_peered":"2024-09-13T22:58:42.764039+0000","last_unstale":"2024-09-13T23:05:34.498084+0000","last_undegraded":"2024-09-13T23:05:34.498084+0000","last_fullsized":"2024-09-13T23:05:34.498084+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:48.697595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:48.697595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.917387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"96'130","reported_seq":1135,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.283990+0000","last_change":"2024-09-13T23:05:07.578268+0000","last_active":"2024-09-13T23:05:35.283990+0000","last_peered":"2024-09-13T23:05:35.283990+0000","last_clean":"2024-09-13T23:05:35.283990+0000","last_became_active":"2024-09-13T23:02:27.391398+0000","last_became_peered":"2024-09-13T23:02:27.391398+0000","last_unstale":"2024-09-13T23:05:35.283990+0000","last_undegraded":"2024-09-13T23:05:35.283990+0000","last_fullsized":"2024-09-13T23:05:35.283990+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":3,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:05:07.578066+0000","last_deep_scrub":"96'130","last_deep_scrub_stamp":"2024-09-13T22:52:20.386855+0000","last_clean_scrub_stamp":"2024-09-13T23:05:07.578066+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:29.562152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"485'49","reported_seq":1190,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277072+0000","last_change":"2024-09-13T23:04:38.946230+0000","last_active":"2024-09-13T23:05:34.277072+0000","last_peered":"2024-09-13T23:05:34.277072+0000","last_clean":"2024-09-13T23:05:34.277072+0000","last_became_active":"2024-09-13T23:02:24.403148+0000","last_became_peered":"2024-09-13T23:02:24.403148+0000","last_unstale":"2024-09-13T23:05:34.277072+0000","last_undegraded":"2024-09-13T23:05:34.277072+0000","last_fullsized":"2024-09-13T23:05:34.277072+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'49","last_scrub_stamp":"2024-09-13T23:04:38.946127+0000","last_deep_scrub":"485'49","last_deep_scrub_stamp":"2024-09-13T23:02:11.668680+0000","last_clean_scrub_stamp":"2024-09-13T23:04:38.946127+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:03.311497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":786,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.496607+0000","last_change":"2024-09-13T23:04:40.601476+0000","last_active":"2024-09-13T23:05:34.496607+0000","last_peered":"2024-09-13T23:05:34.496607+0000","last_clean":"2024-09-13T23:05:34.496607+0000","last_became_active":"2024-09-13T22:44:52.527135+0000","last_became_peered":"2024-09-13T22:44:52.527135+0000","last_unstale":"2024-09-13T23:05:34.496607+0000","last_undegraded":"2024-09-13T23:05:34.496607+0000","last_fullsized":"2024-09-13T23:05:34.496607+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:40.601280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:44:01.750360+0000","last_clean_scrub_stamp":"2024-09-13T23:04:40.601280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.214708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1027,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:41.201370+0000","last_change":"2024-09-13T23:05:41.201469+0000","last_active":"2024-09-13T23:05:41.201370+0000","last_peered":"2024-09-13T23:05:41.201370+0000","last_clean":"2024-09-13T23:05:41.201370+0000","last_became_active":"2024-09-13T22:37:44.883599+0000","last_became_peered":"2024-09-13T22:37:44.883599+0000","last_unstale":"2024-09-13T23:05:41.201370+0000","last_undegraded":"2024-09-13T23:05:41.201370+0000","last_fullsized":"2024-09-13T23:05:41.201370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:41.201197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:05:41.201197+0000","last_clean_scrub_stamp":"2024-09-13T23:05:41.201197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:51.177409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1024,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284078+0000","last_change":"2024-09-13T23:05:34.398965+0000","last_active":"2024-09-13T23:05:35.284078+0000","last_peered":"2024-09-13T23:05:35.284078+0000","last_clean":"2024-09-13T23:05:35.284078+0000","last_became_active":"2024-09-13T22:37:46.913920+0000","last_became_peered":"2024-09-13T22:37:46.913920+0000","last_unstale":"2024-09-13T23:05:35.284078+0000","last_undegraded":"2024-09-13T23:05:35.284078+0000","last_fullsized":"2024-09-13T23:05:35.284078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:34.398662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:59.725449+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.398662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:46.970537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"397'30","reported_seq":1088,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963655+0000","last_change":"2024-09-13T23:05:05.810722+0000","last_active":"2024-09-13T23:05:34.963655+0000","last_peered":"2024-09-13T23:05:34.963655+0000","last_clean":"2024-09-13T23:05:34.963655+0000","last_became_active":"2024-09-13T22:58:42.746819+0000","last_became_peered":"2024-09-13T22:58:42.746819+0000","last_unstale":"2024-09-13T23:05:34.963655+0000","last_undegraded":"2024-09-13T23:05:34.963655+0000","last_fullsized":"2024-09-13T23:05:34.963655+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:05:05.810496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:05.810496+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:29.324903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1036,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523970+0000","last_change":"2024-09-13T23:04:56.399458+0000","last_active":"2024-09-13T23:05:34.523970+0000","last_peered":"2024-09-13T23:05:34.523970+0000","last_clean":"2024-09-13T23:05:34.523970+0000","last_became_active":"2024-09-13T22:37:40.753968+0000","last_became_peered":"2024-09-13T22:37:40.753968+0000","last_unstale":"2024-09-13T23:05:34.523970+0000","last_undegraded":"2024-09-13T23:05:34.523970+0000","last_fullsized":"2024-09-13T23:05:34.523970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:56.399161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:56.399161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:04.611430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"96'130","reported_seq":1301,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498919+0000","last_change":"2024-09-13T23:04:36.552833+0000","last_active":"2024-09-13T23:05:34.498919+0000","last_peered":"2024-09-13T23:05:34.498919+0000","last_clean":"2024-09-13T23:05:34.498919+0000","last_became_active":"2024-09-13T22:58:45.829938+0000","last_became_peered":"2024-09-13T22:58:45.829938+0000","last_unstale":"2024-09-13T23:05:34.498919+0000","last_undegraded":"2024-09-13T23:05:34.498919+0000","last_fullsized":"2024-09-13T23:05:34.498919+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'130","last_scrub_stamp":"2024-09-13T23:04:36.552637+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2024-09-13T22:35:38.758953+0000","last_clean_scrub_stamp":"2024-09-13T23:04:36.552637+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:38.051986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1003,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.497801+0000","last_change":"2024-09-13T23:04:25.599319+0000","last_active":"2024-09-13T23:05:34.497801+0000","last_peered":"2024-09-13T23:05:34.497801+0000","last_clean":"2024-09-13T23:05:34.497801+0000","last_became_active":"2024-09-13T22:37:44.883888+0000","last_became_peered":"2024-09-13T22:37:44.883888+0000","last_unstale":"2024-09-13T23:05:34.497801+0000","last_undegraded":"2024-09-13T23:05:34.497801+0000","last_fullsized":"2024-09-13T23:05:34.497801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:25.599066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:38:56.676172+0000","last_clean_scrub_stamp":"2024-09-13T23:04:25.599066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:46.802603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1047,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524768+0000","last_change":"2024-09-13T23:04:36.520054+0000","last_active":"2024-09-13T23:05:34.524768+0000","last_peered":"2024-09-13T23:05:34.524768+0000","last_clean":"2024-09-13T23:05:34.524768+0000","last_became_active":"2024-09-13T23:02:24.384320+0000","last_became_peered":"2024-09-13T23:02:24.384320+0000","last_unstale":"2024-09-13T23:05:34.524768+0000","last_undegraded":"2024-09-13T23:05:34.524768+0000","last_fullsized":"2024-09-13T23:05:34.524768+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:36.519848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:36.519848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:05.163742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"397'45","reported_seq":1124,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498562+0000","last_change":"2024-09-13T23:05:26.907389+0000","last_active":"2024-09-13T23:05:34.498562+0000","last_peered":"2024-09-13T23:05:34.498562+0000","last_clean":"2024-09-13T23:05:34.498562+0000","last_became_active":"2024-09-13T22:58:42.764099+0000","last_became_peered":"2024-09-13T22:58:42.764099+0000","last_unstale":"2024-09-13T23:05:34.498562+0000","last_undegraded":"2024-09-13T23:05:34.498562+0000","last_fullsized":"2024-09-13T23:05:34.498562+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-09-13T23:05:26.907173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:26.907173+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:40.532856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1068,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:42.918202+0000","last_change":"2024-09-13T23:05:42.918265+0000","last_active":"2024-09-13T23:05:42.918202+0000","last_peered":"2024-09-13T23:05:42.918202+0000","last_clean":"2024-09-13T23:05:42.918202+0000","last_became_active":"2024-09-13T22:37:40.735873+0000","last_became_peered":"2024-09-13T22:37:40.735873+0000","last_unstale":"2024-09-13T23:05:42.918202+0000","last_undegraded":"2024-09-13T23:05:42.918202+0000","last_fullsized":"2024-09-13T23:05:42.918202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:42.918028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:05:42.918028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:47.612621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1052,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.024761+0000","last_change":"2024-09-13T23:05:35.024812+0000","last_active":"2024-09-13T23:05:35.024761+0000","last_peered":"2024-09-13T23:05:35.024761+0000","last_clean":"2024-09-13T23:05:35.024761+0000","last_became_active":"2024-09-13T22:37:44.874231+0000","last_became_peered":"2024-09-13T22:37:44.874231+0000","last_unstale":"2024-09-13T23:05:35.024761+0000","last_undegraded":"2024-09-13T23:05:35.024761+0000","last_fullsized":"2024-09-13T23:05:35.024761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:35.024630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:35.024630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:51.354571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":998,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:39.959576+0000","last_change":"2024-09-13T23:05:39.959673+0000","last_active":"2024-09-13T23:05:39.959576+0000","last_peered":"2024-09-13T23:05:39.959576+0000","last_clean":"2024-09-13T23:05:39.959576+0000","last_became_active":"2024-09-13T22:37:46.923865+0000","last_became_peered":"2024-09-13T22:37:46.923865+0000","last_unstale":"2024-09-13T23:05:39.959576+0000","last_undegraded":"2024-09-13T23:05:39.959576+0000","last_fullsized":"2024-09-13T23:05:39.959576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:39.959412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:43:59.658172+0000","last_clean_scrub_stamp":"2024-09-13T23:05:39.959412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:48.331051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"515'35","reported_seq":1109,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.524320+0000","last_change":"2024-09-13T23:04:14.434888+0000","last_active":"2024-09-13T23:05:34.524320+0000","last_peered":"2024-09-13T23:05:34.524320+0000","last_clean":"2024-09-13T23:05:34.524320+0000","last_became_active":"2024-09-13T22:58:42.731142+0000","last_became_peered":"2024-09-13T22:58:42.731142+0000","last_unstale":"2024-09-13T23:05:34.524320+0000","last_undegraded":"2024-09-13T23:05:34.524320+0000","last_fullsized":"2024-09-13T23:05:34.524320+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'35","last_scrub_stamp":"2024-09-13T23:04:14.434660+0000","last_deep_scrub":"515'35","last_deep_scrub_stamp":"2024-09-13T23:04:14.434660+0000","last_clean_scrub_stamp":"2024-09-13T23:04:14.434660+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1059,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.285143+0000","last_change":"2024-09-13T23:05:00.419977+0000","last_active":"2024-09-13T23:05:35.285143+0000","last_peered":"2024-09-13T23:05:35.285143+0000","last_clean":"2024-09-13T23:05:35.285143+0000","last_became_active":"2024-09-13T23:02:24.393975+0000","last_became_peered":"2024-09-13T23:02:24.393975+0000","last_unstale":"2024-09-13T23:05:35.285143+0000","last_undegraded":"2024-09-13T23:05:35.285143+0000","last_fullsized":"2024-09-13T23:05:35.285143+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:00.419783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:33.671179+0000","last_clean_scrub_stamp":"2024-09-13T23:05:00.419783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:09.887151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":509,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518306+0000","last_change":"2024-09-13T23:05:30.990576+0000","last_active":"2024-09-13T23:05:34.518306+0000","last_peered":"2024-09-13T23:05:34.518306+0000","last_clean":"2024-09-13T23:05:34.518306+0000","last_became_active":"2024-09-13T23:03:27.008041+0000","last_became_peered":"2024-09-13T23:03:27.008041+0000","last_unstale":"2024-09-13T23:05:34.518306+0000","last_undegraded":"2024-09-13T23:05:34.518306+0000","last_fullsized":"2024-09-13T23:05:34.518306+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:30.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:30.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.913782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1023,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277041+0000","last_change":"2024-09-13T23:04:46.921759+0000","last_active":"2024-09-13T23:05:34.277041+0000","last_peered":"2024-09-13T23:05:34.277041+0000","last_clean":"2024-09-13T23:05:34.277041+0000","last_became_active":"2024-09-13T22:37:46.915949+0000","last_became_peered":"2024-09-13T22:37:46.915949+0000","last_unstale":"2024-09-13T23:05:34.277041+0000","last_undegraded":"2024-09-13T23:05:34.277041+0000","last_fullsized":"2024-09-13T23:05:34.277041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:46.921560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:46.921560+0000","last_clean_scrub_stamp":"2024-09-13T23:04:46.921560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.585062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"397'46","reported_seq":1162,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.499080+0000","last_change":"2024-09-13T23:04:47.655901+0000","last_active":"2024-09-13T23:05:34.499080+0000","last_peered":"2024-09-13T23:05:34.499080+0000","last_clean":"2024-09-13T23:05:34.499080+0000","last_became_active":"2024-09-13T23:02:24.394074+0000","last_became_peered":"2024-09-13T23:02:24.394074+0000","last_unstale":"2024-09-13T23:05:34.499080+0000","last_undegraded":"2024-09-13T23:05:34.499080+0000","last_fullsized":"2024-09-13T23:05:34.499080+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:04:47.655681+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-09-13T23:01:36.239714+0000","last_clean_scrub_stamp":"2024-09-13T23:04:47.655681+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:54.281721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":495,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.519307+0000","last_change":"2024-09-13T23:04:41.954847+0000","last_active":"2024-09-13T23:05:34.519307+0000","last_peered":"2024-09-13T23:05:34.519307+0000","last_clean":"2024-09-13T23:05:34.519307+0000","last_became_active":"2024-09-13T23:03:27.005723+0000","last_became_peered":"2024-09-13T23:03:27.005723+0000","last_unstale":"2024-09-13T23:05:34.519307+0000","last_undegraded":"2024-09-13T23:05:34.519307+0000","last_fullsized":"2024-09-13T23:05:34.519307+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:41.954640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:41.954640+0000","last_clean_scrub_stamp":"2024-09-13T23:04:41.954640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.483498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"492'17","reported_seq":1420,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112212+0000","last_change":"2024-09-13T23:05:16.790332+0000","last_active":"2024-09-13T23:05:37.112212+0000","last_peered":"2024-09-13T23:05:37.112212+0000","last_clean":"2024-09-13T23:05:37.112212+0000","last_became_active":"2024-09-13T23:02:24.394764+0000","last_became_peered":"2024-09-13T23:02:24.394764+0000","last_unstale":"2024-09-13T23:05:37.112212+0000","last_undegraded":"2024-09-13T23:05:37.112212+0000","last_fullsized":"2024-09-13T23:05:37.112212+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'17","last_scrub_stamp":"2024-09-13T23:05:16.790128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:16.790128+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:42.432413+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1027,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523375+0000","last_change":"2024-09-13T23:04:35.498511+0000","last_active":"2024-09-13T23:05:34.523375+0000","last_peered":"2024-09-13T23:05:34.523375+0000","last_clean":"2024-09-13T23:05:34.523375+0000","last_became_active":"2024-09-13T22:58:42.732057+0000","last_became_peered":"2024-09-13T22:58:42.732057+0000","last_unstale":"2024-09-13T23:05:34.523375+0000","last_undegraded":"2024-09-13T23:05:34.523375+0000","last_fullsized":"2024-09-13T23:05:34.523375+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:35.498321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:27.309253+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.498321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:04.131799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"525'32","reported_seq":1134,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498664+0000","last_change":"2024-09-13T23:05:08.878192+0000","last_active":"2024-09-13T23:05:34.498664+0000","last_peered":"2024-09-13T23:05:34.498664+0000","last_clean":"2024-09-13T23:05:34.498664+0000","last_became_active":"2024-09-13T23:02:24.402569+0000","last_became_peered":"2024-09-13T23:02:24.402569+0000","last_unstale":"2024-09-13T23:05:34.498664+0000","last_undegraded":"2024-09-13T23:05:34.498664+0000","last_fullsized":"2024-09-13T23:05:34.498664+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'32","last_scrub_stamp":"2024-09-13T23:05:08.877954+0000","last_deep_scrub":"525'32","last_deep_scrub_stamp":"2024-09-13T23:05:08.877954+0000","last_clean_scrub_stamp":"2024-09-13T23:05:08.877954+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:30.064240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"112'4","reported_seq":114,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.280050+0000","last_change":"2024-09-13T23:04:42.736278+0000","last_active":"2024-09-13T23:05:34.280050+0000","last_peered":"2024-09-13T23:05:34.280050+0000","last_clean":"2024-09-13T23:05:34.280050+0000","last_became_active":"2024-09-13T23:02:24.393934+0000","last_became_peered":"2024-09-13T23:02:24.393934+0000","last_unstale":"2024-09-13T23:05:34.280050+0000","last_undegraded":"2024-09-13T23:05:34.280050+0000","last_fullsized":"2024-09-13T23:05:34.280050+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-09-13T23:04:42.736058+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-09-13T23:04:42.736058+0000","last_clean_scrub_stamp":"2024-09-13T23:04:42.736058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:44.223373+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":495,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:39.963665+0000","last_change":"2024-09-13T23:05:39.963665+0000","last_active":"2024-09-13T23:05:39.963665+0000","last_peered":"2024-09-13T23:05:39.963665+0000","last_clean":"2024-09-13T23:05:39.963665+0000","last_became_active":"2024-09-13T23:03:26.934191+0000","last_became_peered":"2024-09-13T23:03:26.934191+0000","last_unstale":"2024-09-13T23:05:39.963665+0000","last_undegraded":"2024-09-13T23:05:39.963665+0000","last_fullsized":"2024-09-13T23:05:39.963665+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:39.963525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:14.939085+0000","last_clean_scrub_stamp":"2024-09-13T23:05:39.963525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:54.492480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1006,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963995+0000","last_change":"2024-09-13T23:05:03.828589+0000","last_active":"2024-09-13T23:05:34.963995+0000","last_peered":"2024-09-13T23:05:34.963995+0000","last_clean":"2024-09-13T23:05:34.963995+0000","last_became_active":"2024-09-13T22:37:46.915044+0000","last_became_peered":"2024-09-13T22:37:46.915044+0000","last_unstale":"2024-09-13T23:05:34.963995+0000","last_undegraded":"2024-09-13T23:05:34.963995+0000","last_fullsized":"2024-09-13T23:05:34.963995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:03.828445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:12.611718+0000","last_clean_scrub_stamp":"2024-09-13T23:05:03.828445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.566595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"397'32","reported_seq":1162,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.496817+0000","last_change":"2024-09-13T23:04:55.695652+0000","last_active":"2024-09-13T23:05:34.496817+0000","last_peered":"2024-09-13T23:05:34.496817+0000","last_clean":"2024-09-13T23:05:34.496817+0000","last_became_active":"2024-09-13T23:02:24.395338+0000","last_became_peered":"2024-09-13T23:02:24.395338+0000","last_unstale":"2024-09-13T23:05:34.496817+0000","last_undegraded":"2024-09-13T23:05:34.496817+0000","last_fullsized":"2024-09-13T23:05:34.496817+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-09-13T23:04:55.695431+0000","last_deep_scrub":"397'32","last_deep_scrub_stamp":"2024-09-13T23:03:38.720852+0000","last_clean_scrub_stamp":"2024-09-13T23:04:55.695431+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:19.318431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1060,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.518550+0000","last_change":"2024-09-13T23:05:24.958255+0000","last_active":"2024-09-13T23:05:34.518550+0000","last_peered":"2024-09-13T23:05:34.518550+0000","last_clean":"2024-09-13T23:05:34.518550+0000","last_became_active":"2024-09-13T22:37:40.739841+0000","last_became_peered":"2024-09-13T22:37:40.739841+0000","last_unstale":"2024-09-13T23:05:34.518550+0000","last_undegraded":"2024-09-13T23:05:34.518550+0000","last_fullsized":"2024-09-13T23:05:34.518550+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:24.958078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:53.872730+0000","last_clean_scrub_stamp":"2024-09-13T23:05:24.958078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:26.778270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1001,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.965375+0000","last_change":"2024-09-13T23:04:30.658861+0000","last_active":"2024-09-13T23:05:34.965375+0000","last_peered":"2024-09-13T23:05:34.965375+0000","last_clean":"2024-09-13T23:05:34.965375+0000","last_became_active":"2024-09-13T22:37:44.871247+0000","last_became_peered":"2024-09-13T22:37:44.871247+0000","last_unstale":"2024-09-13T23:05:34.965375+0000","last_undegraded":"2024-09-13T23:05:34.965375+0000","last_fullsized":"2024-09-13T23:05:34.965375+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:30.658752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:05.183873+0000","last_clean_scrub_stamp":"2024-09-13T23:04:30.658752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:56.535714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1079,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:39.012894+0000","last_change":"2024-09-13T23:05:39.013001+0000","last_active":"2024-09-13T23:05:39.012894+0000","last_peered":"2024-09-13T23:05:39.012894+0000","last_clean":"2024-09-13T23:05:39.012894+0000","last_became_active":"2024-09-13T23:02:24.371637+0000","last_became_peered":"2024-09-13T23:02:24.371637+0000","last_unstale":"2024-09-13T23:05:39.012894+0000","last_undegraded":"2024-09-13T23:05:39.012894+0000","last_fullsized":"2024-09-13T23:05:39.012894+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:39.012728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:24.158640+0000","last_clean_scrub_stamp":"2024-09-13T23:05:39.012728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:53.681283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"397'14","reported_seq":275,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.506904+0000","last_change":"2024-09-13T23:04:03.566365+0000","last_active":"2024-09-13T23:05:34.506904+0000","last_peered":"2024-09-13T23:05:34.506904+0000","last_clean":"2024-09-13T23:05:34.506904+0000","last_became_active":"2024-09-13T22:58:42.736178+0000","last_became_peered":"2024-09-13T22:58:42.736178+0000","last_unstale":"2024-09-13T23:05:34.506904+0000","last_undegraded":"2024-09-13T23:05:34.506904+0000","last_fullsized":"2024-09-13T23:05:34.506904+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14","last_scrub_stamp":"2024-09-13T23:04:03.566160+0000","last_deep_scrub":"235'10","last_deep_scrub_stamp":"2024-09-13T22:48:31.992124+0000","last_clean_scrub_stamp":"2024-09-13T23:04:03.566160+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":283,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:41.716874+0000","last_change":"2024-09-13T23:05:41.716874+0000","last_active":"2024-09-13T23:05:41.716874+0000","last_peered":"2024-09-13T23:05:41.716874+0000","last_clean":"2024-09-13T23:05:41.716874+0000","last_became_active":"2024-09-13T23:02:24.403343+0000","last_became_peered":"2024-09-13T23:02:24.403343+0000","last_unstale":"2024-09-13T23:05:41.716874+0000","last_undegraded":"2024-09-13T23:05:41.716874+0000","last_fullsized":"2024-09-13T23:05:41.716874+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:41.716682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:53:26.915110+0000","last_clean_scrub_stamp":"2024-09-13T23:05:41.716682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:54.263381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1018,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.525395+0000","last_change":"2024-09-13T23:04:02.542656+0000","last_active":"2024-09-13T23:05:34.525395+0000","last_peered":"2024-09-13T23:05:34.525395+0000","last_clean":"2024-09-13T23:05:34.525395+0000","last_became_active":"2024-09-13T22:37:44.888504+0000","last_became_peered":"2024-09-13T22:37:44.888504+0000","last_unstale":"2024-09-13T23:05:34.525395+0000","last_undegraded":"2024-09-13T23:05:34.525395+0000","last_fullsized":"2024-09-13T23:05:34.525395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:02.542465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:04:02.542465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1033,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498322+0000","last_change":"2024-09-13T23:05:06.810218+0000","last_active":"2024-09-13T23:05:34.498322+0000","last_peered":"2024-09-13T23:05:34.498322+0000","last_clean":"2024-09-13T23:05:34.498322+0000","last_became_active":"2024-09-13T23:02:24.394992+0000","last_became_peered":"2024-09-13T23:02:24.394992+0000","last_unstale":"2024-09-13T23:05:34.498322+0000","last_undegraded":"2024-09-13T23:05:34.498322+0000","last_fullsized":"2024-09-13T23:05:34.498322+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:06.809938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:06.809938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:14.691502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"397'27","reported_seq":1157,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498036+0000","last_change":"2024-09-13T23:04:45.667263+0000","last_active":"2024-09-13T23:05:34.498036+0000","last_peered":"2024-09-13T23:05:34.498036+0000","last_clean":"2024-09-13T23:05:34.498036+0000","last_became_active":"2024-09-13T23:02:24.395825+0000","last_became_peered":"2024-09-13T23:02:24.395825+0000","last_unstale":"2024-09-13T23:05:34.498036+0000","last_undegraded":"2024-09-13T23:05:34.498036+0000","last_fullsized":"2024-09-13T23:05:34.498036+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-09-13T23:04:45.667060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:45.667060+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:14.137058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1038,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:42.202851+0000","last_change":"2024-09-13T23:05:42.202851+0000","last_active":"2024-09-13T23:05:42.202851+0000","last_peered":"2024-09-13T23:05:42.202851+0000","last_clean":"2024-09-13T23:05:42.202851+0000","last_became_active":"2024-09-13T22:37:40.754133+0000","last_became_peered":"2024-09-13T22:37:40.754133+0000","last_unstale":"2024-09-13T23:05:42.202851+0000","last_undegraded":"2024-09-13T23:05:42.202851+0000","last_fullsized":"2024-09-13T23:05:42.202851+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:42.202665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:50.627046+0000","last_clean_scrub_stamp":"2024-09-13T23:05:42.202665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:55.591953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":1377,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112248+0000","last_change":"2024-09-13T23:04:57.410753+0000","last_active":"2024-09-13T23:05:37.112248+0000","last_peered":"2024-09-13T23:05:37.112248+0000","last_clean":"2024-09-13T23:05:37.112248+0000","last_became_active":"2024-09-13T22:37:44.877389+0000","last_became_peered":"2024-09-13T22:37:44.877389+0000","last_unstale":"2024-09-13T23:05:37.112248+0000","last_undegraded":"2024-09-13T23:05:37.112248+0000","last_fullsized":"2024-09-13T23:05:37.112248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-13T23:04:57.410540+0000","last_deep_scrub":"325'12","last_deep_scrub_stamp":"2024-09-13T23:04:57.410540+0000","last_clean_scrub_stamp":"2024-09-13T23:04:57.410540+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:16.645440+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1014,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278097+0000","last_change":"2024-09-13T23:04:18.922767+0000","last_active":"2024-09-13T23:05:34.278097+0000","last_peered":"2024-09-13T23:05:34.278097+0000","last_clean":"2024-09-13T23:05:34.278097+0000","last_became_active":"2024-09-13T22:37:46.922930+0000","last_became_peered":"2024-09-13T22:37:46.922930+0000","last_unstale":"2024-09-13T23:05:34.278097+0000","last_undegraded":"2024-09-13T23:05:34.278097+0000","last_fullsized":"2024-09-13T23:05:34.278097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:18.922585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:18.922585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:43.268564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"397'21","reported_seq":1065,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277976+0000","last_change":"2024-09-13T23:05:29.672609+0000","last_active":"2024-09-13T23:05:34.277976+0000","last_peered":"2024-09-13T23:05:34.277976+0000","last_clean":"2024-09-13T23:05:34.277976+0000","last_became_active":"2024-09-13T22:37:42.763347+0000","last_became_peered":"2024-09-13T22:37:42.763347+0000","last_unstale":"2024-09-13T23:05:34.277976+0000","last_undegraded":"2024-09-13T23:05:34.277976+0000","last_fullsized":"2024-09-13T23:05:34.277976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:05:29.672403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:05:29.672403+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:50.208059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1059,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:45.820380+0000","last_change":"2024-09-13T23:05:45.820380+0000","last_active":"2024-09-13T23:05:45.820380+0000","last_peered":"2024-09-13T23:05:45.820380+0000","last_clean":"2024-09-13T23:05:45.820380+0000","last_became_active":"2024-09-13T23:02:24.401203+0000","last_became_peered":"2024-09-13T23:02:24.401203+0000","last_unstale":"2024-09-13T23:05:45.820380+0000","last_undegraded":"2024-09-13T23:05:45.820380+0000","last_fullsized":"2024-09-13T23:05:45.820380+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:45.820194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:14.878570+0000","last_clean_scrub_stamp":"2024-09-13T23:05:45.820194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:07:11.168541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'11","reported_seq":1356,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112191+0000","last_change":"2024-09-13T23:05:29.975146+0000","last_active":"2024-09-13T23:05:37.112191+0000","last_peered":"2024-09-13T23:05:37.112191+0000","last_clean":"2024-09-13T23:05:37.112191+0000","last_became_active":"2024-09-13T22:37:44.869557+0000","last_became_peered":"2024-09-13T22:37:44.869557+0000","last_unstale":"2024-09-13T23:05:37.112191+0000","last_undegraded":"2024-09-13T23:05:37.112191+0000","last_fullsized":"2024-09-13T23:05:37.112191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'11","last_scrub_stamp":"2024-09-13T23:05:29.974953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:29.974953+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:43.737369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":116,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.527677+0000","last_change":"2024-09-13T23:05:34.527731+0000","last_active":"2024-09-13T23:05:34.527677+0000","last_peered":"2024-09-13T23:05:34.527677+0000","last_clean":"2024-09-13T23:05:34.527677+0000","last_became_active":"2024-09-13T23:02:24.374176+0000","last_became_peered":"2024-09-13T23:02:24.374176+0000","last_unstale":"2024-09-13T23:05:34.527677+0000","last_undegraded":"2024-09-13T23:05:34.527677+0000","last_fullsized":"2024-09-13T23:05:34.527677+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:34.527540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:52:21.230166+0000","last_clean_scrub_stamp":"2024-09-13T23:05:34.527540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:51.848295+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"397'21","reported_seq":1103,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.498556+0000","last_change":"2024-09-13T23:04:37.584638+0000","last_active":"2024-09-13T23:05:34.498556+0000","last_peered":"2024-09-13T23:05:34.498556+0000","last_clean":"2024-09-13T23:05:34.498556+0000","last_became_active":"2024-09-13T23:02:24.397236+0000","last_became_peered":"2024-09-13T23:02:24.397236+0000","last_unstale":"2024-09-13T23:05:34.498556+0000","last_undegraded":"2024-09-13T23:05:34.498556+0000","last_fullsized":"2024-09-13T23:05:34.498556+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:04:37.584429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:37.584429+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.354922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":269,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.507696+0000","last_change":"2024-09-13T23:04:45.663056+0000","last_active":"2024-09-13T23:05:34.507696+0000","last_peered":"2024-09-13T23:05:34.507696+0000","last_clean":"2024-09-13T23:05:34.507696+0000","last_became_active":"2024-09-13T22:58:42.749062+0000","last_became_peered":"2024-09-13T22:58:42.749062+0000","last_unstale":"2024-09-13T23:05:34.507696+0000","last_undegraded":"2024-09-13T23:05:34.507696+0000","last_fullsized":"2024-09-13T23:05:34.507696+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:45.662801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:45.662801+0000","last_clean_scrub_stamp":"2024-09-13T23:04:45.662801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.902824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1009,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:39.010343+0000","last_change":"2024-09-13T23:05:39.010395+0000","last_active":"2024-09-13T23:05:39.010343+0000","last_peered":"2024-09-13T23:05:39.010343+0000","last_clean":"2024-09-13T23:05:39.010343+0000","last_became_active":"2024-09-13T22:37:44.864099+0000","last_became_peered":"2024-09-13T22:37:44.864099+0000","last_unstale":"2024-09-13T23:05:39.010343+0000","last_undegraded":"2024-09-13T23:05:39.010343+0000","last_fullsized":"2024-09-13T23:05:39.010343+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:39.010203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:24.673687+0000","last_clean_scrub_stamp":"2024-09-13T23:05:39.010203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:46.068110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1014,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.277448+0000","last_change":"2024-09-13T23:05:07.896069+0000","last_active":"2024-09-13T23:05:34.277448+0000","last_peered":"2024-09-13T23:05:34.277448+0000","last_clean":"2024-09-13T23:05:34.277448+0000","last_became_active":"2024-09-13T22:37:46.913471+0000","last_became_peered":"2024-09-13T22:37:46.913471+0000","last_unstale":"2024-09-13T23:05:34.277448+0000","last_undegraded":"2024-09-13T23:05:34.277448+0000","last_fullsized":"2024-09-13T23:05:34.277448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:07.895792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:05:07.895792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.567379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"397'30","reported_seq":128,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278671+0000","last_change":"2024-09-13T23:04:49.672838+0000","last_active":"2024-09-13T23:05:34.278671+0000","last_peered":"2024-09-13T23:05:34.278671+0000","last_clean":"2024-09-13T23:05:34.278671+0000","last_became_active":"2024-09-13T23:02:24.401214+0000","last_became_peered":"2024-09-13T23:02:24.401214+0000","last_unstale":"2024-09-13T23:05:34.278671+0000","last_undegraded":"2024-09-13T23:05:34.278671+0000","last_fullsized":"2024-09-13T23:05:34.278671+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-09-13T23:04:49.672613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:49.672613+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:05.325793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1034,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.285041+0000","last_change":"2024-09-13T23:05:25.566110+0000","last_active":"2024-09-13T23:05:35.285041+0000","last_peered":"2024-09-13T23:05:35.285041+0000","last_clean":"2024-09-13T23:05:35.285041+0000","last_became_active":"2024-09-13T22:37:40.739588+0000","last_became_peered":"2024-09-13T22:37:40.739588+0000","last_unstale":"2024-09-13T23:05:35.285041+0000","last_undegraded":"2024-09-13T23:05:35.285041+0000","last_fullsized":"2024-09-13T23:05:35.285041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:25.565914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:55:24.439970+0000","last_clean_scrub_stamp":"2024-09-13T23:05:25.565914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:50.066578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1026,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284081+0000","last_change":"2024-09-13T23:05:29.542039+0000","last_active":"2024-09-13T23:05:35.284081+0000","last_peered":"2024-09-13T23:05:35.284081+0000","last_clean":"2024-09-13T23:05:35.284081+0000","last_became_active":"2024-09-13T22:37:44.876819+0000","last_became_peered":"2024-09-13T22:37:44.876819+0000","last_unstale":"2024-09-13T23:05:35.284081+0000","last_undegraded":"2024-09-13T23:05:35.284081+0000","last_fullsized":"2024-09-13T23:05:35.284081+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:29.541845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:51.898279+0000","last_clean_scrub_stamp":"2024-09-13T23:05:29.541845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:37.539412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":997,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963173+0000","last_change":"2024-09-13T23:04:58.809306+0000","last_active":"2024-09-13T23:05:34.963173+0000","last_peered":"2024-09-13T23:05:34.963173+0000","last_clean":"2024-09-13T23:05:34.963173+0000","last_became_active":"2024-09-13T22:37:46.922300+0000","last_became_peered":"2024-09-13T22:37:46.922300+0000","last_unstale":"2024-09-13T23:05:34.963173+0000","last_undegraded":"2024-09-13T23:05:34.963173+0000","last_fullsized":"2024-09-13T23:05:34.963173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:58.809033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:19.327269+0000","last_clean_scrub_stamp":"2024-09-13T23:04:58.809033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:18.494703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"486'21","reported_seq":1060,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278156+0000","last_change":"2024-09-13T23:04:35.929849+0000","last_active":"2024-09-13T23:05:34.278156+0000","last_peered":"2024-09-13T23:05:34.278156+0000","last_clean":"2024-09-13T23:05:34.278156+0000","last_became_active":"2024-09-13T22:37:42.760456+0000","last_became_peered":"2024-09-13T22:37:42.760456+0000","last_unstale":"2024-09-13T23:05:34.278156+0000","last_undegraded":"2024-09-13T23:05:34.278156+0000","last_fullsized":"2024-09-13T23:05:34.278156+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'21","last_scrub_stamp":"2024-09-13T23:04:35.929519+0000","last_deep_scrub":"486'21","last_deep_scrub_stamp":"2024-09-13T23:02:16.665442+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.929519+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:03.393944+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1038,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.525652+0000","last_change":"2024-09-13T23:04:39.442052+0000","last_active":"2024-09-13T23:05:34.525652+0000","last_peered":"2024-09-13T23:05:34.525652+0000","last_clean":"2024-09-13T23:05:34.525652+0000","last_became_active":"2024-09-13T22:58:42.726569+0000","last_became_peered":"2024-09-13T22:58:42.726569+0000","last_unstale":"2024-09-13T23:05:34.525652+0000","last_undegraded":"2024-09-13T23:05:34.525652+0000","last_fullsized":"2024-09-13T23:05:34.525652+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:39.441867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:45.794843+0000","last_clean_scrub_stamp":"2024-09-13T23:04:39.441867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:01.565561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":275,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.507085+0000","last_change":"2024-09-13T23:04:58.701028+0000","last_active":"2024-09-13T23:05:34.507085+0000","last_peered":"2024-09-13T23:05:34.507085+0000","last_clean":"2024-09-13T23:05:34.507085+0000","last_became_active":"2024-09-13T23:02:24.402278+0000","last_became_peered":"2024-09-13T23:02:24.402278+0000","last_unstale":"2024-09-13T23:05:34.507085+0000","last_undegraded":"2024-09-13T23:05:34.507085+0000","last_fullsized":"2024-09-13T23:05:34.507085+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:58.700833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:03:31.274829+0000","last_clean_scrub_stamp":"2024-09-13T23:04:58.700833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.749174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1021,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.496919+0000","last_change":"2024-09-13T23:04:35.566223+0000","last_active":"2024-09-13T23:05:34.496919+0000","last_peered":"2024-09-13T23:05:34.496919+0000","last_clean":"2024-09-13T23:05:34.496919+0000","last_became_active":"2024-09-13T23:02:24.397699+0000","last_became_peered":"2024-09-13T23:02:24.397699+0000","last_unstale":"2024-09-13T23:05:34.496919+0000","last_undegraded":"2024-09-13T23:05:34.496919+0000","last_fullsized":"2024-09-13T23:05:34.496919+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:35.565966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:45.861252+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.565966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.870688+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"397'21","reported_seq":1056,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.525489+0000","last_change":"2024-09-13T23:03:57.577789+0000","last_active":"2024-09-13T23:05:34.525489+0000","last_peered":"2024-09-13T23:05:34.525489+0000","last_clean":"2024-09-13T23:05:34.525489+0000","last_became_active":"2024-09-13T22:37:42.763656+0000","last_became_peered":"2024-09-13T22:37:42.763656+0000","last_unstale":"2024-09-13T23:05:34.525489+0000","last_undegraded":"2024-09-13T23:05:34.525489+0000","last_fullsized":"2024-09-13T23:05:34.525489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:03:57.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:03:57.577560+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":268,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.508157+0000","last_change":"2024-09-13T23:04:43.657652+0000","last_active":"2024-09-13T23:05:34.508157+0000","last_peered":"2024-09-13T23:05:34.508157+0000","last_clean":"2024-09-13T23:05:34.508157+0000","last_became_active":"2024-09-13T22:58:42.742229+0000","last_became_peered":"2024-09-13T22:58:42.742229+0000","last_unstale":"2024-09-13T23:05:34.508157+0000","last_undegraded":"2024-09-13T23:05:34.508157+0000","last_fullsized":"2024-09-13T23:05:34.508157+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:43.657462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:21.874097+0000","last_clean_scrub_stamp":"2024-09-13T23:04:43.657462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:43.708479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1044,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.496772+0000","last_change":"2024-09-13T23:05:04.766151+0000","last_active":"2024-09-13T23:05:34.496772+0000","last_peered":"2024-09-13T23:05:34.496772+0000","last_clean":"2024-09-13T23:05:34.496772+0000","last_became_active":"2024-09-13T23:02:24.368599+0000","last_became_peered":"2024-09-13T23:02:24.368599+0000","last_unstale":"2024-09-13T23:05:34.496772+0000","last_undegraded":"2024-09-13T23:05:34.496772+0000","last_fullsized":"2024-09-13T23:05:34.496772+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:04.765960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:43.825402+0000","last_clean_scrub_stamp":"2024-09-13T23:05:04.765960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:09.752576+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1017,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.964234+0000","last_change":"2024-09-13T23:05:19.911310+0000","last_active":"2024-09-13T23:05:34.964234+0000","last_peered":"2024-09-13T23:05:34.964234+0000","last_clean":"2024-09-13T23:05:34.964234+0000","last_became_active":"2024-09-13T22:58:42.749118+0000","last_became_peered":"2024-09-13T22:58:42.749118+0000","last_unstale":"2024-09-13T23:05:34.964234+0000","last_undegraded":"2024-09-13T23:05:34.964234+0000","last_fullsized":"2024-09-13T23:05:34.964234+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:19.911118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:41:47.765137+0000","last_clean_scrub_stamp":"2024-09-13T23:05:19.911118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:35.119533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"380'2","reported_seq":1008,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:42.868601+0000","last_change":"2024-09-13T23:05:42.868666+0000","last_active":"2024-09-13T23:05:42.868601+0000","last_peered":"2024-09-13T23:05:42.868601+0000","last_clean":"2024-09-13T23:05:42.868601+0000","last_became_active":"2024-09-13T22:37:46.915029+0000","last_became_peered":"2024-09-13T22:37:46.915029+0000","last_unstale":"2024-09-13T23:05:42.868601+0000","last_undegraded":"2024-09-13T23:05:42.868601+0000","last_fullsized":"2024-09-13T23:05:42.868601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-13T23:05:42.868454+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-13T23:05:42.868454+0000","last_clean_scrub_stamp":"2024-09-13T23:05:42.868454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:07:00.608095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"397'46","reported_seq":400,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.517668+0000","last_change":"2024-09-13T23:04:51.853177+0000","last_active":"2024-09-13T23:05:34.517668+0000","last_peered":"2024-09-13T23:05:34.517668+0000","last_clean":"2024-09-13T23:05:34.517668+0000","last_became_active":"2024-09-13T23:02:24.364155+0000","last_became_peered":"2024-09-13T23:02:24.364155+0000","last_unstale":"2024-09-13T23:05:34.517668+0000","last_undegraded":"2024-09-13T23:05:34.517668+0000","last_fullsized":"2024-09-13T23:05:34.517668+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-09-13T23:04:51.852841+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-09-13T23:04:51.852841+0000","last_clean_scrub_stamp":"2024-09-13T23:04:51.852841+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:20.534556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1062,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.523225+0000","last_change":"2024-09-13T23:03:58.532100+0000","last_active":"2024-09-13T23:05:34.523225+0000","last_peered":"2024-09-13T23:05:34.523225+0000","last_clean":"2024-09-13T23:05:34.523225+0000","last_became_active":"2024-09-13T23:02:24.379222+0000","last_became_peered":"2024-09-13T23:02:24.379222+0000","last_unstale":"2024-09-13T23:05:34.523225+0000","last_undegraded":"2024-09-13T23:05:34.523225+0000","last_fullsized":"2024-09-13T23:05:34.523225+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:03:58.531905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:51.789525+0000","last_clean_scrub_stamp":"2024-09-13T23:03:58.531905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"492'18","reported_seq":376,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.112271+0000","last_change":"2024-09-13T23:05:05.718076+0000","last_active":"2024-09-13T23:05:37.112271+0000","last_peered":"2024-09-13T23:05:37.112271+0000","last_clean":"2024-09-13T23:05:37.112271+0000","last_became_active":"2024-09-13T23:02:24.402311+0000","last_became_peered":"2024-09-13T23:02:24.402311+0000","last_unstale":"2024-09-13T23:05:37.112271+0000","last_undegraded":"2024-09-13T23:05:37.112271+0000","last_fullsized":"2024-09-13T23:05:37.112271+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'18","last_scrub_stamp":"2024-09-13T23:05:05.717946+0000","last_deep_scrub":"266'14","last_deep_scrub_stamp":"2024-09-13T22:53:23.962463+0000","last_clean_scrub_stamp":"2024-09-13T23:05:05.717946+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:33.439629+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1046,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284196+0000","last_change":"2024-09-13T23:04:52.456602+0000","last_active":"2024-09-13T23:05:35.284196+0000","last_peered":"2024-09-13T23:05:35.284196+0000","last_clean":"2024-09-13T23:05:35.284196+0000","last_became_active":"2024-09-13T23:02:24.396581+0000","last_became_peered":"2024-09-13T23:02:24.396581+0000","last_unstale":"2024-09-13T23:05:35.284196+0000","last_undegraded":"2024-09-13T23:05:35.284196+0000","last_fullsized":"2024-09-13T23:05:35.284196+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:52.456408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:52.456408+0000","last_clean_scrub_stamp":"2024-09-13T23:04:52.456408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:20.159285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"397'21","reported_seq":1055,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963376+0000","last_change":"2024-09-13T23:04:22.669506+0000","last_active":"2024-09-13T23:05:34.963376+0000","last_peered":"2024-09-13T23:05:34.963376+0000","last_clean":"2024-09-13T23:05:34.963376+0000","last_became_active":"2024-09-13T22:57:37.356187+0000","last_became_peered":"2024-09-13T22:57:37.356187+0000","last_unstale":"2024-09-13T23:05:34.963376+0000","last_undegraded":"2024-09-13T23:05:34.963376+0000","last_fullsized":"2024-09-13T23:05:34.963376+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-09-13T23:04:22.669174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:41.731441+0000","last_clean_scrub_stamp":"2024-09-13T23:04:22.669174+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.716790+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"518'2","reported_seq":1023,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.963655+0000","last_change":"2024-09-13T23:04:26.663817+0000","last_active":"2024-09-13T23:05:34.963655+0000","last_peered":"2024-09-13T23:05:34.963655+0000","last_clean":"2024-09-13T23:05:34.963655+0000","last_became_active":"2024-09-13T22:58:42.748914+0000","last_became_peered":"2024-09-13T22:58:42.748914+0000","last_unstale":"2024-09-13T23:05:34.963655+0000","last_undegraded":"2024-09-13T23:05:34.963655+0000","last_fullsized":"2024-09-13T23:05:34.963655+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'2","last_scrub_stamp":"2024-09-13T23:04:26.663512+0000","last_deep_scrub":"518'2","last_deep_scrub_stamp":"2024-09-13T23:04:26.663512+0000","last_clean_scrub_stamp":"2024-09-13T23:04:26.663512+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:50.222772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1028,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278348+0000","last_change":"2024-09-13T23:05:08.860000+0000","last_active":"2024-09-13T23:05:34.278348+0000","last_peered":"2024-09-13T23:05:34.278348+0000","last_clean":"2024-09-13T23:05:34.278348+0000","last_became_active":"2024-09-13T22:37:44.878565+0000","last_became_peered":"2024-09-13T22:37:44.878565+0000","last_unstale":"2024-09-13T23:05:34.278348+0000","last_undegraded":"2024-09-13T23:05:34.278348+0000","last_fullsized":"2024-09-13T23:05:34.278348+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:05:08.859747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:48:30.083568+0000","last_clean_scrub_stamp":"2024-09-13T23:05:08.859747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:31.405458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.279543+0000","last_change":"2024-09-13T23:04:29.697193+0000","last_active":"2024-09-13T23:05:34.279543+0000","last_peered":"2024-09-13T23:05:34.279543+0000","last_clean":"2024-09-13T23:05:34.279543+0000","last_became_active":"2024-09-13T23:02:24.389985+0000","last_became_peered":"2024-09-13T23:02:24.389985+0000","last_unstale":"2024-09-13T23:05:34.279543+0000","last_undegraded":"2024-09-13T23:05:34.279543+0000","last_fullsized":"2024-09-13T23:05:34.279543+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:29.696999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:04:29.696999+0000","last_clean_scrub_stamp":"2024-09-13T23:04:29.696999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:59.339215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":282,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.507593+0000","last_change":"2024-09-13T23:04:47.663076+0000","last_active":"2024-09-13T23:05:34.507593+0000","last_peered":"2024-09-13T23:05:34.507593+0000","last_clean":"2024-09-13T23:05:34.507593+0000","last_became_active":"2024-09-13T23:02:24.375223+0000","last_became_peered":"2024-09-13T23:02:24.375223+0000","last_unstale":"2024-09-13T23:05:34.507593+0000","last_undegraded":"2024-09-13T23:05:34.507593+0000","last_fullsized":"2024-09-13T23:05:34.507593+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:47.662786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:47.662786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:55.870847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"397'11","reported_seq":1096,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278192+0000","last_change":"2024-09-13T23:04:31.962978+0000","last_active":"2024-09-13T23:05:34.278192+0000","last_peered":"2024-09-13T23:05:34.278192+0000","last_clean":"2024-09-13T23:05:34.278192+0000","last_became_active":"2024-09-13T23:02:24.401750+0000","last_became_peered":"2024-09-13T23:02:24.401750+0000","last_unstale":"2024-09-13T23:05:34.278192+0000","last_undegraded":"2024-09-13T23:05:34.278192+0000","last_fullsized":"2024-09-13T23:05:34.278192+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'11","last_scrub_stamp":"2024-09-13T23:04:31.962741+0000","last_deep_scrub":"397'11","last_deep_scrub_stamp":"2024-09-13T23:03:21.879061+0000","last_clean_scrub_stamp":"2024-09-13T23:04:31.962741+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:47.575075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1042,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:35.284897+0000","last_change":"2024-09-13T23:04:12.340625+0000","last_active":"2024-09-13T23:05:35.284897+0000","last_peered":"2024-09-13T23:05:35.284897+0000","last_clean":"2024-09-13T23:05:35.284897+0000","last_became_active":"2024-09-13T23:02:24.396469+0000","last_became_peered":"2024-09-13T23:02:24.396469+0000","last_unstale":"2024-09-13T23:05:35.284897+0000","last_undegraded":"2024-09-13T23:05:35.284897+0000","last_fullsized":"2024-09-13T23:05:35.284897+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:12.340435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T23:02:45.676834+0000","last_clean_scrub_stamp":"2024-09-13T23:04:12.340435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"498'2","reported_seq":1029,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:37.975096+0000","last_change":"2024-09-13T23:05:37.975158+0000","last_active":"2024-09-13T23:05:37.975096+0000","last_peered":"2024-09-13T23:05:37.975096+0000","last_clean":"2024-09-13T23:05:37.975096+0000","last_became_active":"2024-09-13T22:58:42.748249+0000","last_became_peered":"2024-09-13T22:58:42.748249+0000","last_unstale":"2024-09-13T23:05:37.975096+0000","last_undegraded":"2024-09-13T23:05:37.975096+0000","last_fullsized":"2024-09-13T23:05:37.975096+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'2","last_scrub_stamp":"2024-09-13T23:05:37.974952+0000","last_deep_scrub":"498'2","last_deep_scrub_stamp":"2024-09-13T23:02:57.250150+0000","last_clean_scrub_stamp":"2024-09-13T23:05:37.974952+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:45.031037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1021,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.964691+0000","last_change":"2024-09-13T23:04:35.688824+0000","last_active":"2024-09-13T23:05:34.964691+0000","last_peered":"2024-09-13T23:05:34.964691+0000","last_clean":"2024-09-13T23:05:34.964691+0000","last_became_active":"2024-09-13T22:58:42.769318+0000","last_became_peered":"2024-09-13T22:58:42.769318+0000","last_unstale":"2024-09-13T23:05:34.964691+0000","last_undegraded":"2024-09-13T23:05:34.964691+0000","last_fullsized":"2024-09-13T23:05:34.964691+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:35.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:37:39.717452+0000","last_clean_scrub_stamp":"2024-09-13T23:04:35.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:42.626403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"397'25","reported_seq":126,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278746+0000","last_change":"2024-09-13T23:05:17.676676+0000","last_active":"2024-09-13T23:05:34.278746+0000","last_peered":"2024-09-13T23:05:34.278746+0000","last_clean":"2024-09-13T23:05:34.278746+0000","last_became_active":"2024-09-13T23:02:24.374040+0000","last_became_peered":"2024-09-13T23:02:24.374040+0000","last_unstale":"2024-09-13T23:05:34.278746+0000","last_undegraded":"2024-09-13T23:05:34.278746+0000","last_fullsized":"2024-09-13T23:05:34.278746+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-09-13T23:05:17.676461+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-09-13T22:46:08.028593+0000","last_clean_scrub_stamp":"2024-09-13T23:05:17.676461+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:06:45.465858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":108,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-13T23:05:34.278365+0000","last_change":"2024-09-13T23:04:28.722797+0000","last_active":"2024-09-13T23:05:34.278365+0000","last_peered":"2024-09-13T23:05:34.278365+0000","last_clean":"2024-09-13T23:05:34.278365+0000","last_became_active":"2024-09-13T23:02:24.400638+0000","last_became_peered":"2024-09-13T23:02:24.400638+0000","last_unstale":"2024-09-13T23:05:34.278365+0000","last_undegraded":"2024-09-13T23:05:34.278365+0000","last_fullsized":"2024-09-13T23:05:34.278365+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T23:04:28.722603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T22:40:12.625120+0000","last_clean_scrub_stamp":"2024-09-13T23:04:28.722603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:05:48.172556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1655,"num_read_kb":1650,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1552,"num_read_kb":1309,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":522,"num_bytes_recovered":10450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":67,"seq":287762809178,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726388,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93018764,"statfs":{"total":95995035648,"available":95251214336,"internally_reserved":0,"allocated":1085440,"data_stored":681320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136229,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762128,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92983024,"statfs":{"total":95995035648,"available":95214616576,"internally_reserved":0,"allocated":720896,"data_stored":351098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":45,"seq":193273528688,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668316,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93076836,"statfs":{"total":95995035648,"available":95310680064,"internally_reserved":0,"allocated":1323008,"data_stored":941978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888442,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473396,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271756,"statfs":{"total":95995035648,"available":95510278144,"internally_reserved":0,"allocated":757760,"data_stored":352030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084676,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893208,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":92851944,"statfs":{"total":95995035648,"available":95080390656,"internally_reserved":0,"allocated":1318912,"data_stored":942289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":913308103},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379024,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618024,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93127128,"statfs":{"total":95995035648,"available":95362179072,"internally_reserved":0,"allocated":1007616,"data_stored":615142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575258,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812116,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92933036,"statfs":{"total":95995035648,"available":95163428864,"internally_reserved":0,"allocated":724992,"data_stored":352079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738789,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826004,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92919148,"statfs":{"total":95995035648,"available":95149207552,"internally_reserved":0,"allocated":724992,"data_stored":352729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T23:05:48.497 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T23:05:48.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ecab7be58963a5bd03910cd73eef66405d7fdd50 shell --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T23:05:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.845772+0000 osd.6 (osd.6) 410 : cluster [DBG] 3.a scrub starts 2024-09-13T23:05:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.850886+0000 osd.6 (osd.6) 411 : cluster [DBG] 3.a scrub ok 2024-09-13T23:05:48.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.992604+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:46.995277+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.133754+0000 mgr.y (mgr.14613) 1041 : cluster [DBG] pgmap v1255: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.582703+0000 osd.4 (osd.4) 376 : cluster [DBG] 2.f scrub starts 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.586243+0000 osd.4 (osd.4) 377 : cluster [DBG] 2.f scrub ok 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.596526+0000 osd.1 (osd.1) 540 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.598671+0000 osd.1 (osd.1) 541 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:05:48.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:48 smithi192 bash[24608]: audit 2024-09-13T23:05:47.666046+0000 mgr.y (mgr.14613) 1042 : audit [DBG] from='client.16317 v1:172.21.15.178:0/3224360496' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:05:48.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.845772+0000 osd.6 (osd.6) 410 : cluster [DBG] 3.a scrub starts 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.850886+0000 osd.6 (osd.6) 411 : cluster [DBG] 3.a scrub ok 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.992604+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:46.995277+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.133754+0000 mgr.y (mgr.14613) 1041 : cluster [DBG] pgmap v1255: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.582703+0000 osd.4 (osd.4) 376 : cluster [DBG] 2.f scrub starts 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.586243+0000 osd.4 (osd.4) 377 : cluster [DBG] 2.f scrub ok 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.596526+0000 osd.1 (osd.1) 540 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.598671+0000 osd.1 (osd.1) 541 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:05:48.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[22840]: audit 2024-09-13T23:05:47.666046+0000 mgr.y (mgr.14613) 1042 : audit [DBG] from='client.16317 v1:172.21.15.178:0/3224360496' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.845772+0000 osd.6 (osd.6) 410 : cluster [DBG] 3.a scrub starts 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.850886+0000 osd.6 (osd.6) 411 : cluster [DBG] 3.a scrub ok 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.992604+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:46.995277+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.133754+0000 mgr.y (mgr.14613) 1041 : cluster [DBG] pgmap v1255: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.582703+0000 osd.4 (osd.4) 376 : cluster [DBG] 2.f scrub starts 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.586243+0000 osd.4 (osd.4) 377 : cluster [DBG] 2.f scrub ok 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.596526+0000 osd.1 (osd.1) 540 : cluster [DBG] 5.1d scrub starts 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.598671+0000 osd.1 (osd.1) 541 : cluster [DBG] 5.1d scrub ok 2024-09-13T23:05:48.932 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:48 smithi178 bash[29889]: audit 2024-09-13T23:05:47.666046+0000 mgr.y (mgr.14613) 1042 : audit [DBG] from='client.16317 v1:172.21.15.178:0/3224360496' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T23:05:49.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.765291+0000 osd.0 (osd.0) 537 : cluster [DBG] 6.16 deep-scrub starts 2024-09-13T23:05:49.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[29889]: cluster 2024-09-13T23:05:47.767442+0000 osd.0 (osd.0) 538 : cluster [DBG] 6.16 deep-scrub ok 2024-09-13T23:05:49.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[29889]: cluster 2024-09-13T23:05:48.568242+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:05:49.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[29889]: cluster 2024-09-13T23:05:48.570846+0000 osd.4 (osd.4) 379 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:05:49.811 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.195396+0000 osd.5 (osd.5) 615 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.197679+0000 osd.5 (osd.5) 616 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.765291+0000 osd.0 (osd.0) 537 : cluster [DBG] 6.16 deep-scrub starts 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[22840]: cluster 2024-09-13T23:05:47.767442+0000 osd.0 (osd.0) 538 : cluster [DBG] 6.16 deep-scrub ok 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[22840]: cluster 2024-09-13T23:05:48.568242+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[22840]: cluster 2024-09-13T23:05:48.570846+0000 osd.4 (osd.4) 379 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.195396+0000 osd.5 (osd.5) 615 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:05:49.812 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.197679+0000 osd.5 (osd.5) 616 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:05:49.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:49 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.765291+0000 osd.0 (osd.0) 537 : cluster [DBG] 6.16 deep-scrub starts 2024-09-13T23:05:49.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:49 smithi192 bash[24608]: cluster 2024-09-13T23:05:47.767442+0000 osd.0 (osd.0) 538 : cluster [DBG] 6.16 deep-scrub ok 2024-09-13T23:05:49.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:49 smithi192 bash[24608]: cluster 2024-09-13T23:05:48.568242+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.1c scrub starts 2024-09-13T23:05:49.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:49 smithi192 bash[24608]: cluster 2024-09-13T23:05:48.570846+0000 osd.4 (osd.4) 379 : cluster [DBG] 5.1c scrub ok 2024-09-13T23:05:49.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:49 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.195396+0000 osd.5 (osd.5) 615 : cluster [DBG] 5.16 scrub starts 2024-09-13T23:05:49.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:49 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.197679+0000 osd.5 (osd.5) 616 : cluster [DBG] 5.16 scrub ok 2024-09-13T23:05:50.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:49 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:05:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:05:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:50 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.134871+0000 mgr.y (mgr.14613) 1043 : cluster [DBG] pgmap v1256: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:50 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.598632+0000 osd.1 (osd.1) 542 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:05:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:50 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.603791+0000 osd.1 (osd.1) 543 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:05:50.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:50 smithi192 bash[24608]: audit 2024-09-13T23:05:50.139394+0000 mon.b (mon.1) 326 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:50.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:50 smithi192 bash[24608]: cluster 2024-09-13T23:05:50.168647+0000 osd.5 (osd.5) 617 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:05:50.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:50 smithi192 bash[24608]: cluster 2024-09-13T23:05:50.172461+0000 osd.5 (osd.5) 618 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:05:50.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.134871+0000 mgr.y (mgr.14613) 1043 : cluster [DBG] pgmap v1256: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:50.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.598632+0000 osd.1 (osd.1) 542 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:05:50.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.603791+0000 osd.1 (osd.1) 543 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[22840]: audit 2024-09-13T23:05:50.139394+0000 mon.b (mon.1) 326 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[22840]: cluster 2024-09-13T23:05:50.168647+0000 osd.5 (osd.5) 617 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[22840]: cluster 2024-09-13T23:05:50.172461+0000 osd.5 (osd.5) 618 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.134871+0000 mgr.y (mgr.14613) 1043 : cluster [DBG] pgmap v1256: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.598632+0000 osd.1 (osd.1) 542 : cluster [DBG] 3.4 scrub starts 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.603791+0000 osd.1 (osd.1) 543 : cluster [DBG] 3.4 scrub ok 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[29889]: audit 2024-09-13T23:05:50.139394+0000 mon.b (mon.1) 326 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[29889]: cluster 2024-09-13T23:05:50.168647+0000 osd.5 (osd.5) 617 : cluster [DBG] 3.16 scrub starts 2024-09-13T23:05:50.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:50 smithi178 bash[29889]: cluster 2024-09-13T23:05:50.172461+0000 osd.5 (osd.5) 618 : cluster [DBG] 3.16 scrub ok 2024-09-13T23:05:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:51 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.969503+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:05:51.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:51 smithi192 bash[24608]: cluster 2024-09-13T23:05:49.972219+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:05:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:51 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.969503+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:05:51.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:51 smithi178 bash[29889]: cluster 2024-09-13T23:05:49.972219+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:05:51.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:51 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.969503+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.6 scrub starts 2024-09-13T23:05:51.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:51 smithi178 bash[22840]: cluster 2024-09-13T23:05:49.972219+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.6 scrub ok 2024-09-13T23:05:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:52 smithi192 bash[24608]: cluster 2024-09-13T23:05:50.983199+0000 osd.3 (osd.3) 555 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:05:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:52 smithi192 bash[24608]: cluster 2024-09-13T23:05:50.987111+0000 osd.3 (osd.3) 556 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:05:52.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:52 smithi192 bash[24608]: cluster 2024-09-13T23:05:51.136271+0000 mgr.y (mgr.14613) 1044 : cluster [DBG] pgmap v1257: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:52.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:52 smithi178 bash[22840]: cluster 2024-09-13T23:05:50.983199+0000 osd.3 (osd.3) 555 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:05:52.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:52 smithi178 bash[22840]: cluster 2024-09-13T23:05:50.987111+0000 osd.3 (osd.3) 556 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:05:52.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:52 smithi178 bash[22840]: cluster 2024-09-13T23:05:51.136271+0000 mgr.y (mgr.14613) 1044 : cluster [DBG] pgmap v1257: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:52 smithi178 bash[29889]: cluster 2024-09-13T23:05:50.983199+0000 osd.3 (osd.3) 555 : cluster [DBG] 3.18 scrub starts 2024-09-13T23:05:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:52 smithi178 bash[29889]: cluster 2024-09-13T23:05:50.987111+0000 osd.3 (osd.3) 556 : cluster [DBG] 3.18 scrub ok 2024-09-13T23:05:52.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:52 smithi178 bash[29889]: cluster 2024-09-13T23:05:51.136271+0000 mgr.y (mgr.14613) 1044 : cluster [DBG] pgmap v1257: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:53.308 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T23:05:53.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:53 smithi178 bash[29889]: cluster 2024-09-13T23:05:52.027897+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.e scrub starts 2024-09-13T23:05:53.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:53 smithi178 bash[29889]: cluster 2024-09-13T23:05:52.030275+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.e scrub ok 2024-09-13T23:05:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:53 smithi178 bash[22840]: cluster 2024-09-13T23:05:52.027897+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.e scrub starts 2024-09-13T23:05:53.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:53 smithi178 bash[22840]: cluster 2024-09-13T23:05:52.030275+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.e scrub ok 2024-09-13T23:05:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:53 smithi192 bash[24608]: cluster 2024-09-13T23:05:52.027897+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.e scrub starts 2024-09-13T23:05:53.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:53 smithi192 bash[24608]: cluster 2024-09-13T23:05:52.030275+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.e scrub ok 2024-09-13T23:05:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:54 smithi192 bash[24608]: cluster 2024-09-13T23:05:52.675080+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:05:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:54 smithi192 bash[24608]: cluster 2024-09-13T23:05:52.678468+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:05:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:54 smithi192 bash[24608]: cluster 2024-09-13T23:05:53.137197+0000 mgr.y (mgr.14613) 1045 : cluster [DBG] pgmap v1258: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:54 smithi192 bash[24608]: cluster 2024-09-13T23:05:53.929978+0000 osd.7 (osd.7) 610 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:05:54.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:54 smithi192 bash[24608]: cluster 2024-09-13T23:05:53.933703+0000 osd.7 (osd.7) 611 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:05:54.910 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[22840]: cluster 2024-09-13T23:05:52.675080+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[22840]: cluster 2024-09-13T23:05:52.678468+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[22840]: cluster 2024-09-13T23:05:53.137197+0000 mgr.y (mgr.14613) 1045 : cluster [DBG] pgmap v1258: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[22840]: cluster 2024-09-13T23:05:53.929978+0000 osd.7 (osd.7) 610 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[22840]: cluster 2024-09-13T23:05:53.933703+0000 osd.7 (osd.7) 611 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[29889]: cluster 2024-09-13T23:05:52.675080+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[29889]: cluster 2024-09-13T23:05:52.678468+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[29889]: cluster 2024-09-13T23:05:53.137197+0000 mgr.y (mgr.14613) 1045 : cluster [DBG] pgmap v1258: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[29889]: cluster 2024-09-13T23:05:53.929978+0000 osd.7 (osd.7) 610 : cluster [DBG] 3.15 scrub starts 2024-09-13T23:05:54.911 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:54 smithi178 bash[29889]: cluster 2024-09-13T23:05:53.933703+0000 osd.7 (osd.7) 611 : cluster [DBG] 3.15 scrub ok 2024-09-13T23:05:55.507 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[23095]: [13/Sep/2024:23:05:55] ENGINE Bus STOPPING 2024-09-13T23:05:55.507 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[23095]: [13/Sep/2024:23:05:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T23:05:55.507 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[23095]: [13/Sep/2024:23:05:55] ENGINE Bus STOPPED 2024-09-13T23:05:55.508 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[23095]: [13/Sep/2024:23:05:55] ENGINE Bus STARTING 2024-09-13T23:05:55.508 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[23095]: [13/Sep/2024:23:05:55] ENGINE Serving on http://:::9283 2024-09-13T23:05:55.508 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[23095]: [13/Sep/2024:23:05:55] ENGINE Bus STARTED 2024-09-13T23:05:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: cluster 2024-09-13T23:05:54.619022+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.a scrub starts 2024-09-13T23:05:55.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: cluster 2024-09-13T23:05:54.621632+0000 osd.1 (osd.1) 545 : cluster [DBG] 2.a scrub ok 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.237408+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.178:0/958462056' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.238039+0000 mon.a (mon.0) 1956 : audit [INF] from='client.26222 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.243883+0000 mon.a (mon.0) 1957 : audit [INF] from='client.26222 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.261773+0000 mon.b (mon.1) 327 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.263998+0000 mon.b (mon.1) 328 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.265070+0000 mon.b (mon.1) 329 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: audit 2024-09-13T23:05:55.270972+0000 mon.a (mon.0) 1958 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.296366+0000 osd.5 (osd.5) 619 : cluster [DBG] 1.a scrub starts 2024-09-13T23:05:55.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:55 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.298520+0000 osd.5 (osd.5) 620 : cluster [DBG] 1.a scrub ok 2024-09-13T23:05:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: cluster 2024-09-13T23:05:54.619022+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.a scrub starts 2024-09-13T23:05:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: cluster 2024-09-13T23:05:54.621632+0000 osd.1 (osd.1) 545 : cluster [DBG] 2.a scrub ok 2024-09-13T23:05:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.237408+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.178:0/958462056' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-13T23:05:55.930 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.238039+0000 mon.a (mon.0) 1956 : audit [INF] from='client.26222 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.243883+0000 mon.a (mon.0) 1957 : audit [INF] from='client.26222 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.261773+0000 mon.b (mon.1) 327 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.263998+0000 mon.b (mon.1) 328 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.265070+0000 mon.b (mon.1) 329 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: audit 2024-09-13T23:05:55.270972+0000 mon.a (mon.0) 1958 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.296366+0000 osd.5 (osd.5) 619 : cluster [DBG] 1.a scrub starts 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.298520+0000 osd.5 (osd.5) 620 : cluster [DBG] 1.a scrub ok 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: cluster 2024-09-13T23:05:54.619022+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.a scrub starts 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: cluster 2024-09-13T23:05:54.621632+0000 osd.1 (osd.1) 545 : cluster [DBG] 2.a scrub ok 2024-09-13T23:05:55.931 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.237408+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.178:0/958462056' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.238039+0000 mon.a (mon.0) 1956 : audit [INF] from='client.26222 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.243883+0000 mon.a (mon.0) 1957 : audit [INF] from='client.26222 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.261773+0000 mon.b (mon.1) 327 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.263998+0000 mon.b (mon.1) 328 : audit [DBG] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.265070+0000 mon.b (mon.1) 329 : audit [INF] from='mgr.14613 v1:172.21.15.178:0/1543572777' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: audit 2024-09-13T23:05:55.270972+0000 mon.a (mon.0) 1958 : audit [INF] from='mgr.14613 ' entity='mgr.y' 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.296366+0000 osd.5 (osd.5) 619 : cluster [DBG] 1.a scrub starts 2024-09-13T23:05:55.932 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:55 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.298520+0000 osd.5 (osd.5) 620 : cluster [DBG] 1.a scrub ok 2024-09-13T23:05:56.327 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T23:05:56.338 INFO:tasks.cephadm:Teardown begin 2024-09-13T23:05:56.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T23:05:56.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T23:05:56.462 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T23:05:56.462 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T23:05:56.478 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T23:05:56.492 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T23:05:56.492 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-13T23:05:56.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a 2024-09-13T23:05:56.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[48621]: debug there is no tcmu-runner data available 2024-09-13T23:05:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.039494+0000 osd.2 (osd.2) 391 : cluster [DBG] 6.f deep-scrub starts 2024-09-13T23:05:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.041802+0000 osd.2 (osd.2) 392 : cluster [DBG] 6.f deep-scrub ok 2024-09-13T23:05:56.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.138719+0000 mgr.y (mgr.14613) 1046 : cluster [DBG] pgmap v1259: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.652431+0000 osd.1 (osd.1) 546 : cluster [DBG] 5.c scrub starts 2024-09-13T23:05:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.654638+0000 osd.1 (osd.1) 547 : cluster [DBG] 5.c scrub ok 2024-09-13T23:05:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.915419+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.12 deep-scrub starts 2024-09-13T23:05:56.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:56 smithi192 bash[24608]: cluster 2024-09-13T23:05:55.917705+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.12 deep-scrub ok 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.039494+0000 osd.2 (osd.2) 391 : cluster [DBG] 6.f deep-scrub starts 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.041802+0000 osd.2 (osd.2) 392 : cluster [DBG] 6.f deep-scrub ok 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.138719+0000 mgr.y (mgr.14613) 1046 : cluster [DBG] pgmap v1259: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.652431+0000 osd.1 (osd.1) 546 : cluster [DBG] 5.c scrub starts 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.654638+0000 osd.1 (osd.1) 547 : cluster [DBG] 5.c scrub ok 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.915419+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.12 deep-scrub starts 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[29889]: cluster 2024-09-13T23:05:55.917705+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.12 deep-scrub ok 2024-09-13T23:05:56.680 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.039494+0000 osd.2 (osd.2) 391 : cluster [DBG] 6.f deep-scrub starts 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.041802+0000 osd.2 (osd.2) 392 : cluster [DBG] 6.f deep-scrub ok 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.138719+0000 mgr.y (mgr.14613) 1046 : cluster [DBG] pgmap v1259: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.652431+0000 osd.1 (osd.1) 546 : cluster [DBG] 5.c scrub starts 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.654638+0000 osd.1 (osd.1) 547 : cluster [DBG] 5.c scrub ok 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.915419+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.12 deep-scrub starts 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: cluster 2024-09-13T23:05:55.917705+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.12 deep-scrub ok 2024-09-13T23:05:56.681 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 systemd[1]: Stopping Ceph mon.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:05:56.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: debug 2024-09-13T23:05:56.713+0000 7f8b93868640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-13T23:05:56.930 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:56 smithi178 bash[22840]: debug 2024-09-13T23:05:56.713+0000 7f8b93868640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-13T23:05:57.977 INFO:journalctl@ceph.mon.a.smithi178.stdout:Sep 13 23:05:57 smithi178 bash[148005]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-mon-a 2024-09-13T23:05:57.989 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.a.service' 2024-09-13T23:05:58.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:05:58.054 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-13T23:05:58.054 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-13T23:05:58.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.c 2024-09-13T23:05:58.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:58 smithi178 systemd[1]: Stopping Ceph mon.c for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:05:58.430 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:58 smithi178 bash[29889]: debug 2024-09-13T23:05:58.193+0000 7f58fe193640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-13T23:05:58.431 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:58 smithi178 bash[29889]: debug 2024-09-13T23:05:58.193+0000 7f58fe193640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-13T23:05:59.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.c.service' 2024-09-13T23:05:59.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:59 smithi178 bash[148101]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-mon-c 2024-09-13T23:05:59.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:59 smithi178 bash[148161]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-mon-c 2024-09-13T23:05:59.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:59 smithi178 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-13T23:05:59.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Sep 13 23:05:59 smithi178 systemd[1]: Stopped Ceph mon.c for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T23:05:59.442 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:05:59.443 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-13T23:05:59.443 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-13T23:05:59.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.b 2024-09-13T23:05:59.855 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:59 smithi192 systemd[1]: Stopping Ceph mon.b for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:05:59.855 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:59 smithi192 bash[24608]: debug 2024-09-13T23:05:59.576+0000 7fe9f5fc3640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-13T23:05:59.855 INFO:journalctl@ceph.mon.b.smithi192.stdout:Sep 13 23:05:59 smithi192 bash[24608]: debug 2024-09-13T23:05:59.576+0000 7fe9f5fc3640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-13T23:06:00.180 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:05:59 smithi178 bash[23095]: ::ffff:172.21.15.192 - - [13/Sep/2024:23:05:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-13T23:06:00.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mon.b.service' 2024-09-13T23:06:00.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:00.718 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-13T23:06:00.718 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-13T23:06:00.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y 2024-09-13T23:06:01.136 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:06:00 smithi178 systemd[1]: Stopping Ceph mgr.y for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:01.136 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:06:00 smithi178 bash[23095]: debug 2024-09-13T23:06:00.877+0000 7f08fbb47640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:01.136 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:06:00 smithi178 bash[23095]: debug 2024-09-13T23:06:00.877+0000 7f08fbb47640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-13T23:06:01.933 INFO:journalctl@ceph.mgr.y.smithi178.stdout:Sep 13 23:06:01 smithi178 bash[148200]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-mgr-y 2024-09-13T23:06:01.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-13T23:06:01.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:01.959 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-13T23:06:01.959 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-13T23:06:01.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.x 2024-09-13T23:06:02.347 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Sep 13 23:06:01 smithi192 systemd[1]: Stopping Ceph mgr.x for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:03.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-13T23:06:03.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:03.027 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-13T23:06:03.027 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T23:06:03.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.0 2024-09-13T23:06:03.430 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 23:06:03 smithi178 systemd[1]: Stopping Ceph osd.0 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:03.430 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 23:06:03 smithi178 bash[32795]: debug 2024-09-13T23:06:03.173+0000 7f1c7570b640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:03.430 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 23:06:03 smithi178 bash[32795]: debug 2024-09-13T23:06:03.173+0000 7f1c7570b640 -1 osd.0 530 *** Got signal Terminated *** 2024-09-13T23:06:03.431 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 23:06:03 smithi178 bash[32795]: debug 2024-09-13T23:06:03.173+0000 7f1c7570b640 -1 osd.0 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:09.886 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 23:06:09 smithi178 bash[148304]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-0 2024-09-13T23:06:09.887 INFO:journalctl@ceph.osd.0.smithi178.stdout:Sep 13 23:06:09 smithi178 bash[148366]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-0 2024-09-13T23:06:12.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T23:06:12.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:12.236 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T23:06:12.236 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T23:06:12.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.1 2024-09-13T23:06:12.680 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 23:06:12 smithi178 systemd[1]: Stopping Ceph osd.1 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:12.681 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 23:06:12 smithi178 bash[38055]: debug 2024-09-13T23:06:12.425+0000 7f0114d2c640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:12.681 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 23:06:12 smithi178 bash[38055]: debug 2024-09-13T23:06:12.425+0000 7f0114d2c640 -1 osd.1 530 *** Got signal Terminated *** 2024-09-13T23:06:12.681 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 23:06:12 smithi178 bash[38055]: debug 2024-09-13T23:06:12.425+0000 7f0114d2c640 -1 osd.1 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:18.930 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 23:06:18 smithi178 bash[148500]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-1 2024-09-13T23:06:18.930 INFO:journalctl@ceph.osd.1.smithi178.stdout:Sep 13 23:06:18 smithi178 bash[148565]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-1 2024-09-13T23:06:20.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T23:06:20.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:20.988 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T23:06:20.988 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T23:06:20.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.2 2024-09-13T23:06:21.430 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 23:06:21 smithi178 systemd[1]: Stopping Ceph osd.2 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:21.430 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 23:06:21 smithi178 bash[43622]: debug 2024-09-13T23:06:21.169+0000 7f4f0ee14640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:21.431 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 23:06:21 smithi178 bash[43622]: debug 2024-09-13T23:06:21.169+0000 7f4f0ee14640 -1 osd.2 530 *** Got signal Terminated *** 2024-09-13T23:06:21.431 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 23:06:21 smithi178 bash[43622]: debug 2024-09-13T23:06:21.169+0000 7f4f0ee14640 -1 osd.2 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:27.796 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 23:06:27 smithi178 bash[148695]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-2 2024-09-13T23:06:27.797 INFO:journalctl@ceph.osd.2.smithi178.stdout:Sep 13 23:06:27 smithi178 bash[148762]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-2 2024-09-13T23:06:29.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T23:06:29.751 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:29.752 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T23:06:29.752 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T23:06:29.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.3 2024-09-13T23:06:30.180 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:29 smithi178 systemd[1]: Stopping Ceph osd.3 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:30.181 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:29 smithi178 bash[49028]: debug 2024-09-13T23:06:29.861+0000 7f14c4e32640 -1 osd.3 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.153385+0000 front 2024-09-13T23:06:03.153409+0000 (oldest deadline 2024-09-13T23:06:29.053630+0000) 2024-09-13T23:06:30.181 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:29 smithi178 bash[49028]: debug 2024-09-13T23:06:29.925+0000 7f14cc824640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:30.181 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:29 smithi178 bash[49028]: debug 2024-09-13T23:06:29.925+0000 7f14cc824640 -1 osd.3 530 *** Got signal Terminated *** 2024-09-13T23:06:30.181 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:29 smithi178 bash[49028]: debug 2024-09-13T23:06:29.925+0000 7f14cc824640 -1 osd.3 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:30.908 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:30 smithi192 bash[33643]: debug 2024-09-13T23:06:30.407+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:31.180 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:30 smithi178 bash[49028]: debug 2024-09-13T23:06:30.825+0000 7f14c4e32640 -1 osd.3 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.153385+0000 front 2024-09-13T23:06:03.153409+0000 (oldest deadline 2024-09-13T23:06:29.053630+0000) 2024-09-13T23:06:31.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:31 smithi192 bash[33643]: debug 2024-09-13T23:06:31.359+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:32.159 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:31 smithi192 bash[38879]: debug 2024-09-13T23:06:31.811+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:32.180 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:31 smithi178 bash[49028]: debug 2024-09-13T23:06:31.797+0000 7f14c4e32640 -1 osd.3 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.153385+0000 front 2024-09-13T23:06:03.153409+0000 (oldest deadline 2024-09-13T23:06:29.053630+0000) 2024-09-13T23:06:32.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:32 smithi192 bash[33643]: debug 2024-09-13T23:06:32.315+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:33.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:32 smithi192 bash[38879]: debug 2024-09-13T23:06:32.807+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:33.180 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:32 smithi178 bash[49028]: debug 2024-09-13T23:06:32.785+0000 7f14c4e32640 -1 osd.3 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.153385+0000 front 2024-09-13T23:06:03.153409+0000 (oldest deadline 2024-09-13T23:06:29.053630+0000) 2024-09-13T23:06:33.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:33 smithi192 bash[33643]: debug 2024-09-13T23:06:33.327+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:33.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:33 smithi192 bash[27998]: debug 2024-09-13T23:06:33.495+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:34.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:33 smithi192 bash[38879]: debug 2024-09-13T23:06:33.807+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:34.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:33 smithi192 bash[44341]: debug 2024-09-13T23:06:33.875+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:34.180 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:33 smithi178 bash[49028]: debug 2024-09-13T23:06:33.800+0000 7f14c4e32640 -1 osd.3 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.153385+0000 front 2024-09-13T23:06:03.153409+0000 (oldest deadline 2024-09-13T23:06:29.053630+0000) 2024-09-13T23:06:34.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:34 smithi192 bash[27998]: debug 2024-09-13T23:06:34.495+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:34.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:34 smithi192 bash[33643]: debug 2024-09-13T23:06:34.339+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:35.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:34 smithi192 bash[38879]: debug 2024-09-13T23:06:34.759+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:35.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:34 smithi192 bash[44341]: debug 2024-09-13T23:06:34.911+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:35.180 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:34 smithi178 bash[49028]: debug 2024-09-13T23:06:34.832+0000 7f14c4e32640 -1 osd.3 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.153385+0000 front 2024-09-13T23:06:03.153409+0000 (oldest deadline 2024-09-13T23:06:29.053630+0000) 2024-09-13T23:06:35.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:35 smithi192 bash[27998]: debug 2024-09-13T23:06:35.451+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:35.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:35 smithi192 bash[33643]: debug 2024-09-13T23:06:35.383+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:36.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:35 smithi192 bash[38879]: debug 2024-09-13T23:06:35.759+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:36.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:35 smithi192 bash[44341]: debug 2024-09-13T23:06:35.895+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:36.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:36 smithi192 bash[27998]: debug 2024-09-13T23:06:36.447+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:36.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:36 smithi192 bash[33643]: debug 2024-09-13T23:06:36.363+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:36.680 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:36 smithi178 bash[148890]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-3 2024-09-13T23:06:36.680 INFO:journalctl@ceph.osd.3.smithi178.stdout:Sep 13 23:06:36 smithi178 bash[148956]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-3 2024-09-13T23:06:37.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:36 smithi192 bash[44341]: debug 2024-09-13T23:06:36.935+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:37.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:36 smithi192 bash[38879]: debug 2024-09-13T23:06:36.795+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:37.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[33643]: debug 2024-09-13T23:06:37.391+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:37.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[27998]: debug 2024-09-13T23:06:37.431+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:38.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[38879]: debug 2024-09-13T23:06:37.819+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:38.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[44341]: debug 2024-09-13T23:06:37.955+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[51599]: ts=2024-09-13T23:06:37.807Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T23:06:38.158 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[51599]: ts=2024-09-13T23:06:37.807Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T23:06:38.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[51599]: ts=2024-09-13T23:06:37.807Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T23:06:38.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[51599]: ts=2024-09-13T23:06:37.809Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T23:06:38.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:06:37 smithi192 bash[51599]: ts=2024-09-13T23:06:37.809Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.178:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.178:8765: connect: connection refused" 2024-09-13T23:06:38.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T23:06:38.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:38.244 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T23:06:38.244 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T23:06:38.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.4 2024-09-13T23:06:38.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[33643]: debug 2024-09-13T23:06:38.375+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:38.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[33643]: debug 2024-09-13T23:06:38.375+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:38.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:38 smithi192 systemd[1]: Stopping Ceph osd.4 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:38.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[27998]: debug 2024-09-13T23:06:38.463+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:38.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[27998]: debug 2024-09-13T23:06:38.475+0000 7f13888e2640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:38.659 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[27998]: debug 2024-09-13T23:06:38.475+0000 7f13888e2640 -1 osd.4 530 *** Got signal Terminated *** 2024-09-13T23:06:38.659 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[27998]: debug 2024-09-13T23:06:38.475+0000 7f13888e2640 -1 osd.4 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:39.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[38879]: debug 2024-09-13T23:06:38.811+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:39.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[44341]: debug 2024-09-13T23:06:38.927+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:39.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:38 smithi192 bash[44341]: debug 2024-09-13T23:06:38.927+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:39.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:39 smithi192 bash[27998]: debug 2024-09-13T23:06:39.515+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:39.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:39 smithi192 bash[33643]: debug 2024-09-13T23:06:39.387+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:39.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:39 smithi192 bash[33643]: debug 2024-09-13T23:06:39.387+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:40.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:39 smithi192 bash[38879]: debug 2024-09-13T23:06:39.783+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:40.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:39 smithi192 bash[44341]: debug 2024-09-13T23:06:39.903+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:40.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:39 smithi192 bash[44341]: debug 2024-09-13T23:06:39.903+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:40.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:40 smithi192 bash[27998]: debug 2024-09-13T23:06:40.491+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:40.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:40 smithi192 bash[33643]: debug 2024-09-13T23:06:40.343+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:40.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:40 smithi192 bash[33643]: debug 2024-09-13T23:06:40.343+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:41.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:40 smithi192 bash[38879]: debug 2024-09-13T23:06:40.831+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:41.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:40 smithi192 bash[44341]: debug 2024-09-13T23:06:40.947+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:41.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:40 smithi192 bash[44341]: debug 2024-09-13T23:06:40.947+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:41.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[27998]: debug 2024-09-13T23:06:41.523+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:41.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[33643]: debug 2024-09-13T23:06:41.323+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:41.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[33643]: debug 2024-09-13T23:06:41.323+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:42.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[38879]: debug 2024-09-13T23:06:41.827+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:42.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[38879]: debug 2024-09-13T23:06:41.827+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:42.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[44341]: debug 2024-09-13T23:06:41.959+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:42.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:41 smithi192 bash[44341]: debug 2024-09-13T23:06:41.959+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:42.658 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[27998]: debug 2024-09-13T23:06:42.539+0000 7f1380ef0640 -1 osd.4 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:07.798497+0000 front 2024-09-13T23:06:07.798250+0000 (oldest deadline 2024-09-13T23:06:33.097859+0000) 2024-09-13T23:06:42.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[33643]: debug 2024-09-13T23:06:42.291+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:42.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[33643]: debug 2024-09-13T23:06:42.291+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:43.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[38879]: debug 2024-09-13T23:06:42.835+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:43.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[38879]: debug 2024-09-13T23:06:42.835+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:43.159 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[44341]: debug 2024-09-13T23:06:42.967+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:43.159 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:42 smithi192 bash[44341]: debug 2024-09-13T23:06:42.967+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:43.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:43 smithi192 bash[33643]: debug 2024-09-13T23:06:43.315+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:43.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:43 smithi192 bash[33643]: debug 2024-09-13T23:06:43.315+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:44.069 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[44341]: debug 2024-09-13T23:06:44.003+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:44.069 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[44341]: debug 2024-09-13T23:06:44.003+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:44.069 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:43 smithi192 bash[38879]: debug 2024-09-13T23:06:43.791+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:44.069 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:43 smithi192 bash[38879]: debug 2024-09-13T23:06:43.791+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:44.408 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[33643]: debug 2024-09-13T23:06:44.267+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:44.408 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[33643]: debug 2024-09-13T23:06:44.267+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:44.836 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[101451]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-4 2024-09-13T23:06:44.836 INFO:journalctl@ceph.osd.4.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[101514]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-4 2024-09-13T23:06:44.836 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[38879]: debug 2024-09-13T23:06:44.747+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:44.836 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:44 smithi192 bash[38879]: debug 2024-09-13T23:06:44.747+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:45.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:45 smithi192 bash[44341]: debug 2024-09-13T23:06:45.039+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:45.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:45 smithi192 bash[44341]: debug 2024-09-13T23:06:45.039+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:45.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:45 smithi192 bash[33643]: debug 2024-09-13T23:06:45.275+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:45.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:45 smithi192 bash[33643]: debug 2024-09-13T23:06:45.275+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:46.017 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:45 smithi192 bash[38879]: debug 2024-09-13T23:06:45.755+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:46.017 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:45 smithi192 bash[38879]: debug 2024-09-13T23:06:45.755+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:46.306 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[44341]: debug 2024-09-13T23:06:46.015+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:46.306 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[44341]: debug 2024-09-13T23:06:46.015+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:46.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[33643]: debug 2024-09-13T23:06:46.303+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:46.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[33643]: debug 2024-09-13T23:06:46.303+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:46.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T23:06:46.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:46.989 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T23:06:46.989 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T23:06:46.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.5 2024-09-13T23:06:46.995 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[38879]: debug 2024-09-13T23:06:46.735+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:46.995 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[38879]: debug 2024-09-13T23:06:46.735+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:47.329 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[44341]: debug 2024-09-13T23:06:46.991+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:47.329 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:46 smithi192 bash[44341]: debug 2024-09-13T23:06:46.991+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:47.329 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:47 smithi192 systemd[1]: Stopping Ceph osd.5 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:47.329 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[33643]: debug 2024-09-13T23:06:47.139+0000 7fdffa36e640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:47.329 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[33643]: debug 2024-09-13T23:06:47.139+0000 7fdffa36e640 -1 osd.5 530 *** Got signal Terminated *** 2024-09-13T23:06:47.329 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[33643]: debug 2024-09-13T23:06:47.139+0000 7fdffa36e640 -1 osd.5 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:47.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[33643]: debug 2024-09-13T23:06:47.327+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:47.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[33643]: debug 2024-09-13T23:06:47.327+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:48.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[44341]: debug 2024-09-13T23:06:47.955+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:48.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[44341]: debug 2024-09-13T23:06:47.955+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:48.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[38879]: debug 2024-09-13T23:06:47.731+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:48.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:47 smithi192 bash[38879]: debug 2024-09-13T23:06:47.731+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:48.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[33643]: debug 2024-09-13T23:06:48.335+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:48.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[33643]: debug 2024-09-13T23:06:48.335+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:48.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[33643]: debug 2024-09-13T23:06:48.335+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:23.239697+0000 front 2024-09-13T23:06:23.239894+0000 (oldest deadline 2024-09-13T23:06:47.339567+0000) 2024-09-13T23:06:48.973 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[38879]: debug 2024-09-13T23:06:48.703+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:48.973 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[38879]: debug 2024-09-13T23:06:48.703+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:49.384 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[44341]: debug 2024-09-13T23:06:48.971+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:49.385 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:48 smithi192 bash[44341]: debug 2024-09-13T23:06:48.971+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:49.653 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[33643]: debug 2024-09-13T23:06:49.383+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:49.654 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[33643]: debug 2024-09-13T23:06:49.383+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:49.654 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[33643]: debug 2024-09-13T23:06:49.383+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:23.239697+0000 front 2024-09-13T23:06:23.239894+0000 (oldest deadline 2024-09-13T23:06:47.339567+0000) 2024-09-13T23:06:49.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[38879]: debug 2024-09-13T23:06:49.651+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:49.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[38879]: debug 2024-09-13T23:06:49.651+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:50.335 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[44341]: debug 2024-09-13T23:06:49.975+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:50.335 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[44341]: debug 2024-09-13T23:06:49.975+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:50.336 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:49 smithi192 bash[44341]: debug 2024-09-13T23:06:49.975+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:50.644 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[33643]: debug 2024-09-13T23:06:50.335+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:50.644 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[33643]: debug 2024-09-13T23:06:50.335+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:50.644 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[33643]: debug 2024-09-13T23:06:50.335+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:23.239697+0000 front 2024-09-13T23:06:23.239894+0000 (oldest deadline 2024-09-13T23:06:47.339567+0000) 2024-09-13T23:06:50.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[38879]: debug 2024-09-13T23:06:50.639+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:50.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[38879]: debug 2024-09-13T23:06:50.639+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:50.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[38879]: debug 2024-09-13T23:06:50.639+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:51.308 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[44341]: debug 2024-09-13T23:06:50.987+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:51.309 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[44341]: debug 2024-09-13T23:06:50.987+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:51.309 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:50 smithi192 bash[44341]: debug 2024-09-13T23:06:50.987+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:51.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:51 smithi192 bash[33643]: debug 2024-09-13T23:06:51.303+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:03.535168+0000 front 2024-09-13T23:06:03.535523+0000 (oldest deadline 2024-09-13T23:06:29.434919+0000) 2024-09-13T23:06:51.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:51 smithi192 bash[33643]: debug 2024-09-13T23:06:51.303+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:15.237013+0000 front 2024-09-13T23:06:15.237386+0000 (oldest deadline 2024-09-13T23:06:38.037603+0000) 2024-09-13T23:06:51.658 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:51 smithi192 bash[33643]: debug 2024-09-13T23:06:51.303+0000 7fdff297c640 -1 osd.5 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:23.239697+0000 front 2024-09-13T23:06:23.239894+0000 (oldest deadline 2024-09-13T23:06:47.339567+0000) 2024-09-13T23:06:52.029 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:51 smithi192 bash[38879]: debug 2024-09-13T23:06:51.687+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:52.029 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:51 smithi192 bash[38879]: debug 2024-09-13T23:06:51.687+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:52.029 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:51 smithi192 bash[38879]: debug 2024-09-13T23:06:51.687+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:52.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:52 smithi192 bash[44341]: debug 2024-09-13T23:06:52.027+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:52.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:52 smithi192 bash[44341]: debug 2024-09-13T23:06:52.027+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:52.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:52 smithi192 bash[44341]: debug 2024-09-13T23:06:52.027+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:53.039 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:52 smithi192 bash[38879]: debug 2024-09-13T23:06:52.723+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:53.039 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:52 smithi192 bash[38879]: debug 2024-09-13T23:06:52.723+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:53.039 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:52 smithi192 bash[38879]: debug 2024-09-13T23:06:52.723+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:53.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[44341]: debug 2024-09-13T23:06:53.039+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:53.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[44341]: debug 2024-09-13T23:06:53.039+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:53.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[44341]: debug 2024-09-13T23:06:53.039+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:53.908 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[101646]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-5 2024-09-13T23:06:53.908 INFO:journalctl@ceph.osd.5.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[101713]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-5 2024-09-13T23:06:53.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[38879]: debug 2024-09-13T23:06:53.739+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:53.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[38879]: debug 2024-09-13T23:06:53.739+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:53.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:53 smithi192 bash[38879]: debug 2024-09-13T23:06:53.739+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:54.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:54 smithi192 bash[44341]: debug 2024-09-13T23:06:54.091+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:54.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:54 smithi192 bash[44341]: debug 2024-09-13T23:06:54.091+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:54.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:54 smithi192 bash[44341]: debug 2024-09-13T23:06:54.091+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:55.004 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:54 smithi192 bash[38879]: debug 2024-09-13T23:06:54.719+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:55.004 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:54 smithi192 bash[38879]: debug 2024-09-13T23:06:54.719+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:55.004 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:54 smithi192 bash[38879]: debug 2024-09-13T23:06:54.719+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:55.278 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:55 smithi192 bash[44341]: debug 2024-09-13T23:06:55.091+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:55.278 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:55 smithi192 bash[44341]: debug 2024-09-13T23:06:55.091+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:55.278 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:55 smithi192 bash[44341]: debug 2024-09-13T23:06:55.091+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:55.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T23:06:55.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:06:55.933 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T23:06:55.933 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T23:06:55.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.6 2024-09-13T23:06:55.984 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:55 smithi192 bash[38879]: debug 2024-09-13T23:06:55.719+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:55.984 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:55 smithi192 bash[38879]: debug 2024-09-13T23:06:55.719+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:55.985 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:55 smithi192 bash[38879]: debug 2024-09-13T23:06:55.719+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:56.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:55 smithi192 systemd[1]: Stopping Ceph osd.6 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:06:56.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[38879]: debug 2024-09-13T23:06:56.111+0000 7f9672d9e640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:06:56.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[38879]: debug 2024-09-13T23:06:56.111+0000 7f9672d9e640 -1 osd.6 530 *** Got signal Terminated *** 2024-09-13T23:06:56.158 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[38879]: debug 2024-09-13T23:06:56.111+0000 7f9672d9e640 -1 osd.6 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:06:56.158 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[44341]: debug 2024-09-13T23:06:56.099+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:56.159 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[44341]: debug 2024-09-13T23:06:56.099+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:56.159 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[44341]: debug 2024-09-13T23:06:56.099+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:57.095 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[38879]: debug 2024-09-13T23:06:56.755+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:57.096 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[38879]: debug 2024-09-13T23:06:56.755+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:57.096 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:56 smithi192 bash[38879]: debug 2024-09-13T23:06:56.755+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:57.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:57 smithi192 bash[44341]: debug 2024-09-13T23:06:57.095+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:57.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:57 smithi192 bash[44341]: debug 2024-09-13T23:06:57.095+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:57.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:57 smithi192 bash[44341]: debug 2024-09-13T23:06:57.095+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:58.130 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:57 smithi192 bash[38879]: debug 2024-09-13T23:06:57.715+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:58.130 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:57 smithi192 bash[38879]: debug 2024-09-13T23:06:57.715+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:58.130 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:57 smithi192 bash[38879]: debug 2024-09-13T23:06:57.715+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:58.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:58 smithi192 bash[44341]: debug 2024-09-13T23:06:58.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:58.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:58 smithi192 bash[44341]: debug 2024-09-13T23:06:58.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:58.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:58 smithi192 bash[44341]: debug 2024-09-13T23:06:58.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:06:59.106 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:58 smithi192 bash[38879]: debug 2024-09-13T23:06:58.763+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:06:59.106 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:58 smithi192 bash[38879]: debug 2024-09-13T23:06:58.763+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:06:59.106 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:58 smithi192 bash[38879]: debug 2024-09-13T23:06:58.763+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:06:59.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:59 smithi192 bash[44341]: debug 2024-09-13T23:06:59.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:06:59.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:59 smithi192 bash[44341]: debug 2024-09-13T23:06:59.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:06:59.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:06:59 smithi192 bash[44341]: debug 2024-09-13T23:06:59.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:00.111 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:59 smithi192 bash[38879]: debug 2024-09-13T23:06:59.739+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:07:00.111 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:59 smithi192 bash[38879]: debug 2024-09-13T23:06:59.739+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:07:00.111 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:06:59 smithi192 bash[38879]: debug 2024-09-13T23:06:59.739+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:07:00.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[44341]: debug 2024-09-13T23:07:00.107+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:00.409 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[44341]: debug 2024-09-13T23:07:00.107+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:00.409 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[44341]: debug 2024-09-13T23:07:00.107+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:01.082 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[38879]: debug 2024-09-13T23:07:00.747+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:06.610943+0000 front 2024-09-13T23:06:06.611222+0000 (oldest deadline 2024-09-13T23:06:31.310872+0000) 2024-09-13T23:07:01.082 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[38879]: debug 2024-09-13T23:07:00.747+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:16.612581+0000 front 2024-09-13T23:06:16.612429+0000 (oldest deadline 2024-09-13T23:06:41.312130+0000) 2024-09-13T23:07:01.082 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[38879]: debug 2024-09-13T23:07:00.747+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:25.115405+0000 front 2024-09-13T23:06:25.115316+0000 (oldest deadline 2024-09-13T23:06:50.415134+0000) 2024-09-13T23:07:01.083 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:07:00 smithi192 bash[38879]: debug 2024-09-13T23:07:00.747+0000 7f966b3ac640 -1 osd.6 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:33.916983+0000 front 2024-09-13T23:06:33.916958+0000 (oldest deadline 2024-09-13T23:06:59.816611+0000) 2024-09-13T23:07:01.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:01 smithi192 bash[44341]: debug 2024-09-13T23:07:01.079+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:01.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:01 smithi192 bash[44341]: debug 2024-09-13T23:07:01.079+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:01.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:01 smithi192 bash[44341]: debug 2024-09-13T23:07:01.079+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:01.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:01 smithi192 bash[44341]: debug 2024-09-13T23:07:01.079+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:02.407 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:07:02 smithi192 bash[101843]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-6 2024-09-13T23:07:02.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:02 smithi192 bash[44341]: debug 2024-09-13T23:07:02.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:02.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:02 smithi192 bash[44341]: debug 2024-09-13T23:07:02.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:02.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:02 smithi192 bash[44341]: debug 2024-09-13T23:07:02.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:02.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:02 smithi192 bash[44341]: debug 2024-09-13T23:07:02.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:02.908 INFO:journalctl@ceph.osd.6.smithi192.stdout:Sep 13 23:07:02 smithi192 bash[101907]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-6 2024-09-13T23:07:03.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:03 smithi192 bash[44341]: debug 2024-09-13T23:07:03.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:03.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:03 smithi192 bash[44341]: debug 2024-09-13T23:07:03.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:03.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:03 smithi192 bash[44341]: debug 2024-09-13T23:07:03.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:03.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:03 smithi192 bash[44341]: debug 2024-09-13T23:07:03.103+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:04.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.143+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:04.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.143+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:04.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.143+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:04.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.143+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:04.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T23:07:04.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:07:04.675 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T23:07:04.675 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T23:07:04.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.7 2024-09-13T23:07:04.908 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 systemd[1]: Stopping Ceph osd.7 for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:04.908 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.831+0000 7f20395e3640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:07:04.908 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.831+0000 7f20395e3640 -1 osd.7 530 *** Got signal Terminated *** 2024-09-13T23:07:04.908 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:04 smithi192 bash[44341]: debug 2024-09-13T23:07:04.831+0000 7f20395e3640 -1 osd.7 530 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T23:07:05.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:05 smithi192 bash[44341]: debug 2024-09-13T23:07:05.135+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:05.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:05 smithi192 bash[44341]: debug 2024-09-13T23:07:05.135+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:05.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:05 smithi192 bash[44341]: debug 2024-09-13T23:07:05.135+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:05.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:05 smithi192 bash[44341]: debug 2024-09-13T23:07:05.135+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:06.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:06 smithi192 bash[44341]: debug 2024-09-13T23:07:06.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:06.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:06 smithi192 bash[44341]: debug 2024-09-13T23:07:06.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:06.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:06 smithi192 bash[44341]: debug 2024-09-13T23:07:06.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:06.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:06 smithi192 bash[44341]: debug 2024-09-13T23:07:06.127+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:07.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:07 smithi192 bash[44341]: debug 2024-09-13T23:07:07.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:07.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:07 smithi192 bash[44341]: debug 2024-09-13T23:07:07.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:07.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:07 smithi192 bash[44341]: debug 2024-09-13T23:07:07.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:07.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:07 smithi192 bash[44341]: debug 2024-09-13T23:07:07.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:07.409 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:07 smithi192 bash[44341]: debug 2024-09-13T23:07:07.119+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.192:6802 osd.4 since back 2024-09-13T23:06:40.407049+0000 front 2024-09-13T23:06:40.406998+0000 (oldest deadline 2024-09-13T23:07:06.306784+0000) 2024-09-13T23:07:08.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:08 smithi192 bash[44341]: debug 2024-09-13T23:07:08.071+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:08.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:08 smithi192 bash[44341]: debug 2024-09-13T23:07:08.071+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:08.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:08 smithi192 bash[44341]: debug 2024-09-13T23:07:08.071+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:08.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:08 smithi192 bash[44341]: debug 2024-09-13T23:07:08.071+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:08.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:08 smithi192 bash[44341]: debug 2024-09-13T23:07:08.071+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.192:6802 osd.4 since back 2024-09-13T23:06:40.407049+0000 front 2024-09-13T23:06:40.406998+0000 (oldest deadline 2024-09-13T23:07:06.306784+0000) 2024-09-13T23:07:09.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:09 smithi192 bash[44341]: debug 2024-09-13T23:07:09.087+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6803 osd.0 since back 2024-09-13T23:06:04.101045+0000 front 2024-09-13T23:06:04.101441+0000 (oldest deadline 2024-09-13T23:06:32.901634+0000) 2024-09-13T23:07:09.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:09 smithi192 bash[44341]: debug 2024-09-13T23:07:09.087+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6807 osd.1 since back 2024-09-13T23:06:17.002918+0000 front 2024-09-13T23:06:17.003385+0000 (oldest deadline 2024-09-13T23:06:38.702972+0000) 2024-09-13T23:07:09.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:09 smithi192 bash[44341]: debug 2024-09-13T23:07:09.087+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6811 osd.2 since back 2024-09-13T23:06:24.004708+0000 front 2024-09-13T23:06:24.004637+0000 (oldest deadline 2024-09-13T23:06:49.204903+0000) 2024-09-13T23:07:09.408 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:09 smithi192 bash[44341]: debug 2024-09-13T23:07:09.087+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.178:6815 osd.3 since back 2024-09-13T23:06:34.506395+0000 front 2024-09-13T23:06:34.506479+0000 (oldest deadline 2024-09-13T23:07:00.406178+0000) 2024-09-13T23:07:09.409 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:09 smithi192 bash[44341]: debug 2024-09-13T23:07:09.087+0000 7f2031bf1640 -1 osd.7 530 heartbeat_check: no reply from 172.21.15.192:6802 osd.4 since back 2024-09-13T23:06:40.407049+0000 front 2024-09-13T23:06:40.406998+0000 (oldest deadline 2024-09-13T23:07:06.306784+0000) 2024-09-13T23:07:11.118 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:10 smithi192 bash[102037]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-7 2024-09-13T23:07:11.118 INFO:journalctl@ceph.osd.7.smithi192.stdout:Sep 13 23:07:11 smithi192 bash[102103]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-osd-7 2024-09-13T23:07:12.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T23:07:12.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:07:12.926 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T23:07:12.927 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-13T23:07:12.927 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-13T23:07:13.430 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 23:07:12 smithi178 systemd[1]: Stopping Ceph rgw.foo.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:13.430 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 23:07:13 smithi178 bash[52826]: debug 2024-09-13T23:07:13.076+0000 7f35957d9640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-13T23:07:13.430 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 23:07:13 smithi178 bash[52826]: debug 2024-09-13T23:07:13.076+0000 7f3599045880 -1 shutting down 2024-09-13T23:07:23.680 INFO:journalctl@ceph.rgw.foo.a.smithi178.stdout:Sep 13 23:07:23 smithi178 bash[149085]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-13T23:07:23.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-13T23:07:23.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:07:23.843 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-13T23:07:23.843 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-13T23:07:23.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-03ae43e2-721f-11ef-bceb-c7b262605968@prometheus.a 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 systemd[1]: Stopping Ceph prometheus.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.995Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.995Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.995Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.995Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.996Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T23:07:24.159 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.995Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T23:07:24.160 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.996Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T23:07:24.160 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.996Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T23:07:24.160 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:23 smithi192 bash[51599]: ts=2024-09-13T23:07:23.996Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T23:07:24.160 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:24 smithi192 bash[51599]: ts=2024-09-13T23:07:24.000Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T23:07:24.160 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:24 smithi192 bash[51599]: ts=2024-09-13T23:07:24.000Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T23:07:24.160 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Sep 13 23:07:24 smithi192 bash[51599]: ts=2024-09-13T23:07:24.000Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T23:07:24.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-03ae43e2-721f-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-13T23:07:24.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T23:07:24.713 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-13T23:07:24.713 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T23:07:24.897 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T23:07:29.752 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:29 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:29.752 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:29 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:30.075 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:29 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:30.075 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:29 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:30.346 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:30.347 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:30.679 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:30.680 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.021 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.021 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.022 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.022 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:30 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.316 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:31 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.317 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:31 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.639 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:31 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.639 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:31 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:31.640 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:31 smithi178 systemd[1]: Stopping Ceph alertmanager.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:31.901 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:31 smithi178 bash[60120]: ts=2024-09-13T23:07:31.640Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T23:07:32.650 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:32 smithi178 bash[149489]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T23:07:32.650 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:32 smithi178 bash[149551]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T23:07:32.650 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:32 smithi178 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T23:07:32.650 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:32 smithi178 systemd[1]: Stopped Ceph alertmanager.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T23:07:32.929 INFO:journalctl@ceph.alertmanager.a.smithi178.stdout:Sep 13 23:07:32 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:32.930 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:32 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:32.930 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:32 smithi178 systemd[1]: Stopping Ceph node-exporter.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:33.894 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:33 smithi178 bash[149620]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T23:07:33.894 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:33 smithi178 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T23:07:33.894 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:33 smithi178 bash[149678]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T23:07:33.894 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:33 smithi178 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T23:07:33.894 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:33 smithi178 systemd[1]: Stopped Ceph node-exporter.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T23:07:33.895 INFO:journalctl@ceph.node-exporter.a.smithi178.stdout:Sep 13 23:07:33 smithi178 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:34.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T23:07:34.591 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T23:07:39.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:39.479 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:39.479 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:39.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:39.775 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:39.775 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.031 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:39 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.394 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.394 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.644 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.645 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.645 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.907 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:40.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:40 smithi192 systemd[1]: Stopping Ceph iscsi.iscsi.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:40.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:40 smithi192 bash[48621]: debug Shutdown received 2024-09-13T23:07:51.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:51 smithi192 bash[102584]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-13T23:07:51.877 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:51 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:51.878 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:51 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:51.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:51 smithi192 bash[102700]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-13T23:07:51.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:51 smithi192 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-13T23:07:51.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:51 smithi192 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-13T23:07:51.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:51 smithi192 systemd[1]: Stopped Ceph iscsi.iscsi.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T23:07:51.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Sep 13 23:07:51 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:53.148 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:53.148 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:53 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:53.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:52 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:53.149 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:53 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:53.407 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:53 smithi192 systemd[1]: Stopping Ceph node-exporter.b for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:54.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:53 smithi192 bash[102901]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T23:07:54.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:54 smithi192 bash[102962]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T23:07:54.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:54 smithi192 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T23:07:54.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:54 smithi192 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T23:07:54.145 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:54 smithi192 systemd[1]: Stopped Ceph node-exporter.b for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T23:07:54.396 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Sep 13 23:07:54 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:54.396 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:54 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:54.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:54 smithi192 systemd[1]: Stopping Ceph grafana.a for 03ae43e2-721f-11ef-bceb-c7b262605968... 2024-09-13T23:07:54.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:54 smithi192 bash[50858]: logger=server t=2024-09-13T23:07:54.38372824Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-13T23:07:54.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:54 smithi192 bash[50858]: logger=tracing t=2024-09-13T23:07:54.383833106Z level=info msg="Closing tracing" 2024-09-13T23:07:54.397 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:54 smithi192 bash[50858]: logger=ticker t=2024-09-13T23:07:54.383982646Z level=info msg=stopped last_tick=2024-09-13T23:07:50Z 2024-09-13T23:07:55.158 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:54 smithi192 bash[103030]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-grafana-a 2024-09-13T23:07:55.158 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:55 smithi192 bash[103090]: Error response from daemon: No such container: ceph-03ae43e2-721f-11ef-bceb-c7b262605968-grafana-a 2024-09-13T23:07:55.158 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:55 smithi192 systemd[1]: ceph-03ae43e2-721f-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-13T23:07:55.158 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:55 smithi192 systemd[1]: Stopped Ceph grafana.a for 03ae43e2-721f-11ef-bceb-c7b262605968. 2024-09-13T23:07:55.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Sep 13 23:07:55 smithi192 systemd[1]: /etc/systemd/system/ceph-03ae43e2-721f-11ef-bceb-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-09-13T23:07:55.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T23:07:55.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T23:07:55.769 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T23:07:55.770 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289/remote/smithi178/crash 2024-09-13T23:07:55.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/crash -- . 2024-09-13T23:07:55.824 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T23:07:55.824 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2024-09-13T23:07:55.826 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289/remote/smithi192/crash 2024-09-13T23:07:55.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/crash -- . 2024-09-13T23:07:55.842 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T23:07:55.842 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-09-13T23:07:55.844 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T23:07:55.844 DEBUG:teuthology.orchestra.run.smithi178:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-13T23:07:55.949 INFO:tasks.cephadm:Compressing logs... 2024-09-13T23:07:55.950 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T23:07:55.952 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T23:07:55.961 INFO:teuthology.orchestra.run.smithi178.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-09-13T23:07:55.965 INFO:teuthology.orchestra.run.smithi192.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-09-13T23:07:55.970 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T23:07:55.973 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-13T23:07:55.973 INFO:teuthology.orchestra.run.smithi178.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-13T23:07:55.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-13T23:07:55.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-13T23:07:55.975 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T23:07:55.975 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.2.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-13T23:07:55.976 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-13T23:07:55.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.cephadm.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log 2024-09-13T23:07:55.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T23:07:55.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-13T23:07:55.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T23:07:55.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-13T23:07:55.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.audit.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log: 59.3% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-13T23:07:55.984 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T23:07:55.984 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-13T23:07:55.985 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-13T23:07:55.985 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T23:07:55.986 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.7.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-13T23:07:55.986 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-13T23:07:55.986 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-13T23:07:55.987 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/tcmu-runner.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log 2024-09-13T23:07:55.987 INFO:teuthology.orchestra.run.smithi192.stderr: 63.3% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-13T23:07:55.987 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T23:07:55.987 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mgr.x.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T23:07:55.988 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-13T23:07:55.988 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-volume.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-13T23:07:55.990 INFO:teuthology.orchestra.run.smithi178.stderr: 91.8% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T23:07:55.996 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.4.log: 92.3% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-13T23:07:56.035 INFO:teuthology.orchestra.run.smithi192.stderr: 79.3% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T23:07:56.089 INFO:teuthology.orchestra.run.smithi178.stderr: 88.9% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T23:07:56.116 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T23:07:56.125 INFO:teuthology.orchestra.run.smithi178.stderr: 94.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T23:07:56.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T23:07:56.166 INFO:teuthology.orchestra.run.smithi178.stderr: 94.1% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T23:07:56.172 INFO:teuthology.orchestra.run.smithi192.stderr: 85.4% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T23:07:56.179 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T23:07:56.392 INFO:teuthology.orchestra.run.smithi178.stderr: 91.2% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T23:07:57.384 INFO:teuthology.orchestra.run.smithi178.stderr: 89.7% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-13T23:07:59.053 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-13T23:08:00.900 INFO:teuthology.orchestra.run.smithi178.stderr: 91.4% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-13T23:08:09.212 INFO:teuthology.orchestra.run.smithi178.stderr: 90.2% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-13T23:10:37.022 INFO:teuthology.orchestra.run.smithi192.stderr: 94.0% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-13T23:13:01.038 INFO:teuthology.orchestra.run.smithi192.stderr: 93.5% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-13T23:13:05.246 INFO:teuthology.orchestra.run.smithi192.stderr: 93.7% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-13T23:13:22.177 INFO:teuthology.orchestra.run.smithi178.stderr: 93.7% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-13T23:13:23.848 INFO:teuthology.orchestra.run.smithi192.stderr: 92.7% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-13T23:13:23.848 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-13T23:13:23.849 INFO:teuthology.orchestra.run.smithi192.stderr:real 5m27.893s 2024-09-13T23:13:23.849 INFO:teuthology.orchestra.run.smithi192.stderr:user 7m4.369s 2024-09-13T23:13:23.849 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m18.468s 2024-09-13T23:21:02.464 INFO:teuthology.orchestra.run.smithi178.stderr: 93.0% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-13T23:21:17.787 INFO:teuthology.orchestra.run.smithi178.stderr: 92.5% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-13T23:21:19.987 INFO:teuthology.orchestra.run.smithi178.stderr: 92.8% -- replaced with /var/log/ceph/03ae43e2-721f-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-13T23:21:19.989 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-09-13T23:21:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:real 13m24.035s 2024-09-13T23:21:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:user 15m6.580s 2024-09-13T23:21:19.989 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m41.132s 2024-09-13T23:21:19.990 INFO:tasks.cephadm:Archiving logs... 2024-09-13T23:21:19.990 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289/remote/smithi178/log 2024-09-13T23:21:19.991 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T23:22:08.976 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289/remote/smithi192/log 2024-09-13T23:22:08.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T23:22:42.106 INFO:tasks.cephadm:Removing cluster... 2024-09-13T23:22:42.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 --force 2024-09-13T23:22:42.329 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T23:22:44.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 03ae43e2-721f-11ef-bceb-c7b262605968 --force 2024-09-13T23:22:44.558 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 03ae43e2-721f-11ef-bceb-c7b262605968 2024-09-13T23:22:46.357 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T23:22:46.357 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T23:22:46.371 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T23:22:46.375 INFO:tasks.cephadm:Teardown complete 2024-09-13T23:22:46.376 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T23:22:46.385 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T23:22:46.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T23:22:46.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T23:22:46.511 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T23:22:46.520 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T23:22:46.520 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T23:22:46.522 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T23:22:46.586 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T23:22:46.586 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2024-09-13T23:22:46.586 INFO:teuthology.orchestra.run.smithi178.stdout:+hv01.front.sepi 67.205.162.81 3 u 56 128 377 0.086 -2.191 0.177 2024-09-13T23:22:46.586 INFO:teuthology.orchestra.run.smithi178.stdout:*hv02.front.sepi 96.245.170.99 3 u 20 128 377 0.113 -2.199 0.159 2024-09-13T23:22:46.586 INFO:teuthology.orchestra.run.smithi178.stdout:+hv03.front.sepi 74.6.168.72 3 u 90 128 377 0.102 -2.283 0.135 2024-09-13T23:22:46.586 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-13T23:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T23:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2024-09-13T23:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout:*hv01.front.sepi 67.205.162.81 3 u 37 64 377 0.099 -2.208 0.382 2024-09-13T23:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout:+hv02.front.sepi 96.245.170.99 3 u 34 64 377 0.114 -1.646 0.262 2024-09-13T23:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout:+hv03.front.sepi 74.6.168.72 3 u 26 64 377 0.081 -1.796 0.233 2024-09-13T23:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-13T23:22:46.588 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T23:22:46.597 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T23:22:46.597 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T23:22:46.606 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T23:22:46.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T23:22:46.624 INFO:teuthology.task.internal:Duration was 4028.514410 seconds 2024-09-13T23:22:46.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T23:22:46.633 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T23:22:46.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T23:22:46.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T23:22:46.752 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T23:22:46.752 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-09-13T23:22:46.753 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T23:22:46.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-09-13T23:22:46.779 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T23:22:46.789 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T23:22:46.789 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T23:22:46.821 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T23:22:46.983 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T23:22:46.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T23:22:46.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T23:22:47.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T23:22:47.417 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T23:22:47.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T23:22:47.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T23:22:47.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T23:22:47.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T23:22:47.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T23:22:47.507 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-09-13T23:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-09-13T23:22:47.574 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T23:22:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:22:47.578 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T23:22:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T23:22:47.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T23:22:47.630 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T23:22:47.631 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289/remote/smithi178 2024-09-13T23:22:47.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T23:22:47.665 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-13_21:08:03-orch-squid-distro-default-smithi/7904289/remote/smithi192 2024-09-13T23:22:47.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T23:22:47.696 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T23:22:47.696 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T23:22:47.706 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T23:22:47.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T23:22:47.747 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T23:22:47.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T23:22:47.756 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T23:22:47.757 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T23:22:47.759 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T23:22:47.763 INFO:teuthology.orchestra.run.smithi178.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 23:22 /home/ubuntu/cephtest 2024-09-13T23:22:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: 658633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 23:22 /home/ubuntu/cephtest 2024-09-13T23:22:47.786 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T23:22:47.795 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T23:22:47.815 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 4028.5144095420837 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-13T23:22:47.816 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T23:22:47.933 INFO:teuthology.run:pass